| *** dave-mccowan has quit IRC | 00:19 | |
| *** gaoyanami has joined #openstack-ansible | 01:07 | |
| *** gaoyanami has quit IRC | 01:12 | |
| *** cjloader has joined #openstack-ansible | 01:15 | |
| *** cjloader has quit IRC | 01:19 | |
| *** exodusftw has joined #openstack-ansible | 01:19 | |
| *** gkadam has quit IRC | 01:43 | |
| *** gkadam has joined #openstack-ansible | 01:45 | |
| *** thyism has joined #openstack-ansible | 01:46 | |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-galera_server master: Tune-up the galera role for efficiency https://review.openstack.org/466827 | 01:51 |
|---|---|---|
| thyism | so, anyone else noticed indeterminant behaviour in openstack-ansible? running a stable/ocata release and there seems to be some kind of race condition....sometime on reboot, the haproxy fails to load | 01:52 |
| thyism | changed the dhcp timeout like someone recommended in a bug report, but not sure that made any difference | 01:53 |
| thyism | weirdest thing is... when it happens, i cant even do an apt-get update on the host because it tries to resolve the ubuntu cloud repo to the address space of the containers | 01:54 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible master: Allow the haproxy configuration to run with limited groups https://review.openstack.org/521331 | 01:55 |
| *** SmearedBeard has quit IRC | 01:56 | |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible master: Remove the AIO scenario & add new scenarios to maintain coverage https://review.openstack.org/516002 | 01:58 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible master: Allow the haproxy configuration to run with limited groups https://review.openstack.org/521331 | 01:58 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible master: Add dynamic table for our tested Scenarios https://review.openstack.org/520294 | 01:58 |
| cloudnull | opps ^ | 01:59 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible master: Add dynamic table for our tested Scenarios https://review.openstack.org/520294 | 02:01 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible master: Allow the haproxy configuration to run with limited groups https://review.openstack.org/521331 | 02:01 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible master: Remove the AIO scenario & add new scenarios to maintain coverage https://review.openstack.org/516002 | 02:01 |
| cloudnull | odyssey4me: https://review.openstack.org/#/c/521331 | 02:01 |
| *** ajmaidak has quit IRC | 02:04 | |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible master: Remove the AIO scenario & add new scenarios to maintain coverage https://review.openstack.org/516002 | 02:06 |
| *** ajmaidak has joined #openstack-ansible | 02:07 | |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible master: Remove the AIO scenario & add new scenarios to maintain coverage https://review.openstack.org/516002 | 02:10 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible master: Remove the AIO scenario & add new scenarios to maintain coverage https://review.openstack.org/516002 | 02:12 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-galera_server master: Add gitkeep file for reno https://review.openstack.org/521335 | 02:21 |
| *** chhavi has joined #openstack-ansible | 02:57 | |
| *** john51 has quit IRC | 03:03 | |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-galera_server master: Add gitkeep file for reno https://review.openstack.org/521335 | 03:07 |
| *** john51 has joined #openstack-ansible | 03:07 | |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-galera_server master: Add gitkeep file for reno https://review.openstack.org/521335 | 03:12 |
| *** szaher has quit IRC | 03:23 | |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-galera_server master: Tune-up the galera role for efficiency https://review.openstack.org/466827 | 03:25 |
| *** szaher has joined #openstack-ansible | 03:34 | |
| *** germs1 has joined #openstack-ansible | 03:43 | |
| *** germs has quit IRC | 03:44 | |
| *** germs1 has quit IRC | 03:47 | |
| *** chhavi has quit IRC | 03:50 | |
| *** omiday has joined #openstack-ansible | 04:01 | |
| *** jamesdenton has quit IRC | 04:04 | |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-galera_server master: Tune-up the galera role for efficiency https://review.openstack.org/466827 | 04:38 |
| *** omiday has quit IRC | 04:38 | |
| *** bhujay has joined #openstack-ansible | 04:49 | |
| cloudnull | looks like the suse repos are still out of sync :( | 04:56 |
| cloudnull | hwoarang: ^ | 04:56 |
| cloudnull | also evrardjp odyssey4me logan- IDK whats up with some of our release notes processing. is that maybe an infra change? | 04:57 |
| *** bhujay has quit IRC | 04:58 | |
| *** bhujay has joined #openstack-ansible | 04:59 | |
| *** bhujay has quit IRC | 05:06 | |
| *** bhujay has joined #openstack-ansible | 05:06 | |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible master: Remove the AIO scenario & add new scenarios to maintain coverage https://review.openstack.org/516002 | 05:15 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible master: Run gate playbooks in parallel https://review.openstack.org/497742 | 05:15 |
| *** gkadam has quit IRC | 05:27 | |
| openstackgerrit | Merged openstack/openstack-ansible master: zuul: project.yaml: Move openSUSE Ceph job to 'check' queue https://review.openstack.org/520589 | 06:13 |
| *** SmearedBeard has joined #openstack-ansible | 06:15 | |
| *** gouthamr has joined #openstack-ansible | 06:16 | |
| *** gouthamr has quit IRC | 06:28 | |
| *** bhujay has quit IRC | 06:39 | |
| *** sxc731 has joined #openstack-ansible | 08:10 | |
| *** bhujay_ has joined #openstack-ansible | 08:12 | |
| odyssey4me | cloudnull yes, some adjustments to the releasenotes jobs by infra broken them - they're on the way to being fixed | 09:29 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-lxc_hosts master: Fix ansible-lint errors https://review.openstack.org/521357 | 09:47 |
| odyssey4me | gun1x you set the number of forks ansible uses via an env var | 09:50 |
| odyssey4me | gun1x https://docs.openstack.org/openstack-ansible/latest/admin/advanced-config.html#ansible-forks | 09:51 |
| odyssey4me | savvas errr most times that's an issue with config related to networking in some way (keepalived, cidr mismatch, bad indentation, etc) | 09:52 |
| odyssey4me | thyism yes, because the repo container is configured as an apt cache by default once it's deployed | 09:53 |
| odyssey4me | ah, thanks cloudnull | 09:53 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible master: Allow the haproxy configuration to run with limited groups https://review.openstack.org/521331 | 09:55 |
| *** sxc731 has quit IRC | 09:57 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-lxc_hosts master: Fix ansible-lint errors https://review.openstack.org/521357 | 10:01 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-lxc_hosts master: Fix ansible-lint errors https://review.openstack.org/521357 | 10:02 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_glance master: Implement testing using tempest https://review.openstack.org/521268 | 10:20 |
| *** bhujay_ has quit IRC | 10:22 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-lxc_hosts master: Make the cache prep timeout configurable https://review.openstack.org/521313 | 10:22 |
| gun1x | odyssey4me: so if i have 5 hosts, i can leave this to default :)) | 10:23 |
| odyssey4me | gun1x you can set up to 10 forks without making life complicated, and it may benefit you even with 5 hosts - I don't know | 10:24 |
| odyssey4me | it's mostly beneficial when you have many, many compute hosts and are executing the nova playbook | 10:25 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible master: Allow the haproxy configuration to run with limited groups https://review.openstack.org/521331 | 10:28 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible master: Remove the AIO scenario & add new scenarios to maintain coverage https://review.openstack.org/516002 | 10:28 |
| gun1x | is there anybody else using DLD (detailed level design) for tracking details about every infrastructure? | 10:31 |
| gun1x | or do you use something else ? (like internal wiki whatever) | 10:33 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible master: Add dynamic table for our tested Scenarios https://review.openstack.org/520294 | 10:39 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible master: Allow the haproxy configuration to run with limited groups https://review.openstack.org/521331 | 10:39 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible master: Remove the AIO scenario & add new scenarios to maintain coverage https://review.openstack.org/516002 | 10:39 |
| odyssey4me | cloudnull apologies - I messed up that patch due to rebasing - I think it's fixed again | 10:42 |
| *** SmearedBeard has quit IRC | 10:51 | |
| *** bhujay has joined #openstack-ansible | 10:51 | |
| *** pbandark has joined #openstack-ansible | 11:14 | |
| *** bhujay has quit IRC | 11:32 | |
| bndzor | hmm, i rebooted my cluster and galera does not want to come back up | 11:44 |
| bndzor | ah, galera_new_cluster.. | 11:56 |
| *** dave-mcc_ has joined #openstack-ansible | 12:32 | |
| bndzor | is there a procedure that should be followed if everything was shutdown ? | 12:39 |
| bndzor | after shutting down all and booting it up, galera did not come up, so i started it myself. Now, when trying to login to the dash board, im getting Unable to establish connection to keystone endpoint. | 12:47 |
| bndzor | and on infra1-keystone-container i can see *1793 recv() failed (104: Connection reset by peer) while sending to client, client: 172.29.236.11, server: , request: "HEAD / HTTP/1.0", upstream: "uwsgi://127.0.0.1:35358" | 12:48 |
| bndzor | aaand i found the issue, so my lb was not coming up correctly. My bad. | 12:53 |
| gun1x | bndzor: there was some documentation on OSA and galera | 13:18 |
| gun1x | AFAIK they have some playlist or something, or a parameter for galera | 13:19 |
| gun1x | did you get it done, or should i help you search? | 13:19 |
| *** SmearedBeard has joined #openstack-ansible | 13:36 | |
| *** SmearedBeard has quit IRC | 13:38 | |
| *** SmearedBeard has joined #openstack-ansible | 13:39 | |
| *** SmearedBeard has quit IRC | 13:42 | |
| bndzor | ye it works | 13:45 |
| bndzor | now im having a issue where i cant start instances | 13:45 |
| bndzor | http://paste.openstack.org/show/626744/ | 13:47 |
| bndzor | issue communicating with rmq possibly ? | 13:47 |
| *** SmearedBeard has joined #openstack-ansible | 13:48 | |
| bndzor | hmm i dont get it | 13:58 |
| bndzor | nova service-list and neutron agent-list shows all up | 13:59 |
| bndzor | nova hypervisor-stats shows all values as 0 | 13:59 |
| *** woodard has quit IRC | 14:12 | |
| *** woodard has joined #openstack-ansible | 14:12 | |
| bndzor | so im getting 2017-11-19 14:12:50.684 6445 WARNING nova.compute.manager [req-bca53908-823b-40f1-a13c-4e3f8884403f - - - - -] No compute node record found for host compute1. If this is the first time this service is starting on this host, then you can ignore this warning.: ComputeHostNotFound_Remote: Compute host compute1 could not be found. | 14:13 |
| bndzor | and no, its not the first time obviously | 14:24 |
| odyssey4me | bndzor hmm, that seems to be related to the placement api probably - not sure, it's been a while since I worked with any of that | 14:24 |
| odyssey4me | but it definitely sounds like one of the services isn't quite right | 14:24 |
| bndzor | il reinstall everything once again and see if this occurs | 14:25 |
| odyssey4me | you might find using https://review.openstack.org/#/c/495492/ handy, although that doesn't go to all service just yet - just the infra bits | 14:26 |
| *** SmearedBeard has quit IRC | 14:36 | |
| *** SmearedBeard has joined #openstack-ansible | 14:48 | |
| savvas | odyssey4me: like I mentioned via PM my issue related to my haproxy.cfg whitelist not having the proper entries. The playbook ran like a charm after that | 15:00 |
| odyssey4me | savvas great! | 15:13 |
| savvas | just started testing the infra, ran into 2 issues so far. | 15:13 |
| savvas | Even though disabling HTTPS, the horizon containers still rewrite to ssl so it tries to redirect | 15:13 |
| savvas | is that another variable I am missing? | 15:13 |
| savvas | also running openstack image create results in a broken pipe error | 15:14 |
| odyssey4me | hmm, is your keystone service catalog or endpoint list showing any https endpoints? | 15:14 |
| savvas | no they are all http | 15:21 |
| odyssey4me | odd, you'll have to verify the haproxy & horizon config to see whether something in there is doing the redirect | 15:22 |
| *** SmearedBeard has quit IRC | 15:23 | |
| odyssey4me | and the broken pipe error you'll have to work through client -> haproxy -> service -> image store to see whether the logs are showing you why | 15:23 |
| odyssey4me | https://docs.openstack.org/openstack-ansible/latest/admin/troubleshooting.html#diagnose-image-service-issues | 15:23 |
| *** sxc731 has joined #openstack-ansible | 15:24 | |
| savvas | yes odyssey4me the haproxy works only on http as it should | 15:30 |
| savvas | but the horizon apache config suggests a rewrite condition to https | 15:30 |
| odyssey4me | so a bug in the logic here perhaps? | 15:32 |
| odyssey4me | https://github.com/openstack/openstack-ansible-os_horizon/blob/stable/pike/templates/openstack_dashboard.conf.j2 | 15:32 |
| savvas | RewriteEngine On | 15:32 |
| savvas | RewriteCond %{HTTPS} !=on | 15:32 |
| savvas | RewriteRule ^/?(.*) https://%{HTTP_HOST}/$1 [R,L] | 15:32 |
| savvas | ye this right here | 15:32 |
| savvas | I don't know if there's a user variable to be defined which makes the playbook do this different | 15:33 |
| savvas | otherwise this is something I'd need to manually correct for now | 15:33 |
| odyssey4me | do you have openstack_external_ssl set to 'no' or 'false' ? | 15:33 |
| savvas | false | 15:33 |
| odyssey4me | hmm, I think there's a bug in https://github.com/openstack/openstack-ansible-os_horizon/blob/stable/pike/templates/openstack_dashboard.conf.j2#L3 - it needs to check the protocol set too | 15:35 |
| odyssey4me | you should be able to set horizon_external_ssl to 'yes' in your user_vars and it'll do the right thing - just re-run the horizon playbook after setting it | 15:35 |
| odyssey4me | if you can confirm that, then write it up in a bug report that'd be great | 15:36 |
| savvas | alright give me a few mins | 15:39 |
| savvas | playbook's running | 15:39 |
| savvas | odyssey4me: in regards to glance | 15:41 |
| savvas | think it has something to do with ceph | 15:42 |
| savvas | Nov 19 14:48:37 compute03-glance-container-6129b3a6 glance-api: 2017-11-19 14:48:36.636 3902 ERROR glance.common.wsgi AttributeError: 'NoneType' object has no attribute 'Rados' | 15:42 |
| odyssey4me | ok, did you deploy ceph with OSA or seperately? | 15:42 |
| odyssey4me | but yes, it looks like you're missing something that activates using ceph | 15:42 |
| savvas | with OSA | 15:43 |
| savvas | horizon playbook finished now but didn't fix the problem | 15:43 |
| savvas | I checked haproxy.cfg and it didn't make any changes there, should I run the haproxy setup again as well? | 15:44 |
| savvas | because I am guessing it should enable https there for horizon? | 15:44 |
| odyssey4me | ok, I'm a little tied up right now and losing focus ... so see if you can figure out what's going on there | 15:44 |
| savvas | ye no worries I'll figure it out, I'll let you know if it is anything worth mentioning | 15:45 |
| *** pbandark has quit IRC | 16:08 | |
| *** SmearedBeard has joined #openstack-ansible | 16:17 | |
| *** jwitko_ has joined #openstack-ansible | 16:19 | |
| *** pbandark has joined #openstack-ansible | 16:21 | |
| *** pbandark has quit IRC | 16:30 | |
| *** pbandark has joined #openstack-ansible | 16:44 | |
| *** pbandark has quit IRC | 16:53 | |
| *** viktor_ has joined #openstack-ansible | 17:00 | |
| *** pbandark has joined #openstack-ansible | 17:06 | |
| *** cjloader has joined #openstack-ansible | 17:11 | |
| *** woodard has quit IRC | 17:13 | |
| *** woodard has joined #openstack-ansible | 17:13 | |
| *** woodard has quit IRC | 17:18 | |
| *** pbandark has quit IRC | 17:23 | |
| *** SmearedBeard has quit IRC | 17:38 | |
| *** pbandark has joined #openstack-ansible | 17:38 | |
| savvas | logan-: the containers for for example glance and cinder refer to /etc/ceph/ceph.conf for using rbd, but those files are not present on the containers. How do they communicate with Ceph? | 17:43 |
| savvas | in my case neither work and cinder for example complains that rbd can't initialize "Update driver status failed: (config name RBD) is uninitialized." and neither will glance "AttributeError: 'NoneType' object has no attribute 'Rados'" | 17:44 |
| savvas | I thought at first it may have something to do with missing python packages on the containers so I tried installing python-ceph but that doesn't solve it. | 17:44 |
| *** viktor_ has quit IRC | 17:46 | |
| *** cjloader has quit IRC | 17:55 | |
| *** savvas_ has joined #openstack-ansible | 18:15 | |
| bndzor | everytime i run a fresh install | 18:15 |
| bndzor | it fails on the rabbit mq | 18:15 |
| bndzor | when i rerun the setup infrastructure, its ok | 18:16 |
| odyssey4me | bndzor are you sure it fails? because there's a try/rescue block where the try will always fail for a new install, but the rescue kicks in and the playbook doesn't fail | 18:16 |
| *** savvas has quit IRC | 18:17 | |
| bndzor | fatal: [controller3_rabbit_mq_container-6ce07033]: FAILED! => {"changed": false, "cmd": "rabbitmqctl cluster_status | grep -w '<<\"openstack\">>'", "delta": "0:00:00.975624", "end": "2017-11-19 17:53:01.767258", "failed": true, "rc": 1, "start": "2017-11-19 17:53:00.791634", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} | 18:17 |
| odyssey4me | odd | 18:17 |
| bndzor | and the one before | 18:17 |
| bndzor | fatal: [controller3_rabbit_mq_container-6ce07033]: FAILED! => {"changed": false, "content": "Rk9JRkNDUFpHQ0ZDUVNSWVVUUlc=", "encoding": "base64", "failed": true, "failed_when_result": true, "source": "/var/lib/rabbitmq/.erlang.cookie"} | 18:17 |
| bndzor | now runnig it a second time, and im sure its going to work.. that happend last time i did a fresh install also (yday) | 18:18 |
| *** woodard has joined #openstack-ansible | 18:18 | |
| odyssey4me | odd, sorry I can't help atm - busy knee deep in other stuff right now | 18:19 |
| bndzor | its alright, just wanted to point it out :) | 18:20 |
| *** woodard has quit IRC | 18:23 | |
| *** pbandark has quit IRC | 18:45 | |
| *** viktor_ has joined #openstack-ansible | 18:51 | |
| *** sxc731 has quit IRC | 18:53 | |
| openstackgerrit | Jimmy McCrory proposed openstack/openstack-ansible master: Move inventory files to folder in root of repo https://review.openstack.org/516032 | 19:09 |
| openstackgerrit | Jimmy McCrory proposed openstack/openstack-ansible master: [TEST] Update Ansible to 2.4.2.0-0.4.beta4 https://review.openstack.org/501814 | 19:12 |
| *** woodard has joined #openstack-ansible | 19:15 | |
| *** woodard has quit IRC | 19:19 | |
| *** savvas_ has quit IRC | 19:44 | |
| *** savvas has joined #openstack-ansible | 19:45 | |
| savvas | bndzor: I had the same issue with rabbitmq, exactly the same, have you by any chance defined container/vxlan/storage networks outside of 172.16, 10.0 ? | 19:45 |
| savvas | gotta go but if you're still stuck there, check that and if needed whitelist your own blocks in user_variables for haproxy, then rerun haproxy.yml and proceed with setu-infra | 19:53 |
| *** hybridpollo has joined #openstack-ansible | 20:06 | |
| *** hybridpollo has quit IRC | 20:13 | |
| *** hybridpollo has joined #openstack-ansible | 20:13 | |
| *** woodard has joined #openstack-ansible | 20:15 | |
| *** woodard has quit IRC | 20:20 | |
| *** hybridpollo has quit IRC | 20:23 | |
| *** hybridpollo has joined #openstack-ansible | 20:24 | |
| *** viktor_ has quit IRC | 20:25 | |
| bndzor | savvas: no, all are in 172.16 | 20:38 |
| *** dave-mcc_ has quit IRC | 20:47 | |
| *** dave-mccowan has joined #openstack-ansible | 20:49 | |
| bndzor | reinstalled | 21:16 |
| bndzor | compute nodes still not work | 21:16 |
| *** dave-mccowan has quit IRC | 21:16 | |
| bndzor | nova-compute.log says: Error updating resources for node appserver2.openstack.local.: RuntimeError: rbd python libraries not found | 21:16 |
| bndzor | and pip install -vvv python-cephlibs returns | 21:17 |
| bndzor | Could not find a version that satisfies the requirement python-cephlibs (from versions: ) | 21:17 |
| bndzor | any ideas ? | 21:18 |
| *** woodard has joined #openstack-ansible | 21:20 | |
| *** woodard has quit IRC | 21:25 | |
| bndzor | logan-: any ideas ? | 21:28 |
| *** dave-mccowan has joined #openstack-ansible | 21:39 | |
| *** dgonzalez has joined #openstack-ansible | 22:12 | |
| *** hamzy has joined #openstack-ansible | 22:15 | |
| *** woodard has joined #openstack-ansible | 22:16 | |
| *** woodard has quit IRC | 22:20 | |
| *** threestrands has joined #openstack-ansible | 22:27 | |
| *** threestrands has quit IRC | 22:27 | |
| *** threestrands has joined #openstack-ansible | 22:27 | |
| *** ajmaidak has quit IRC | 22:36 | |
| *** ajmaidak has joined #openstack-ansible | 22:40 | |
| logan- | bndzor: could you show me os-nova-install playbook output? | 22:43 |
| bndzor | logan-: is it ok if i rerun it ? | 22:53 |
| *** pmannidi has joined #openstack-ansible | 23:04 | |
| *** gouthamr has joined #openstack-ansible | 23:06 | |
| *** mrtenio-afk has quit IRC | 23:10 | |
| *** mrtenio-afk has joined #openstack-ansible | 23:13 | |
| *** woodard has joined #openstack-ansible | 23:17 | |
| *** woodard has quit IRC | 23:22 | |
| logan- | bndzor: yes | 23:34 |
| bndzor | alright, will run it | 23:53 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!