| *** macza has quit IRC | 00:20 | |
| *** DanyC has quit IRC | 00:23 | |
| *** sdake has quit IRC | 00:23 | |
| *** sdake has joined #openstack-ansible | 00:24 | |
| *** ansmith has joined #openstack-ansible | 00:34 | |
| *** cmart has quit IRC | 00:35 | |
| *** gyee has quit IRC | 00:51 | |
| *** ansmith has quit IRC | 01:03 | |
| *** ansmith has joined #openstack-ansible | 01:04 | |
| *** markvoelker has joined #openstack-ansible | 01:10 | |
| *** macza has joined #openstack-ansible | 01:10 | |
| *** markvoelker has quit IRC | 01:15 | |
| *** macza has quit IRC | 01:15 | |
| *** ThiagoCMC has joined #openstack-ansible | 01:25 | |
| ThiagoCMC | Just a quick note... Looks like that new Rock OSA deployments with Cinder with Ceph is broken due to: https://bugs.launchpad.net/cinder/+bug/1806156 | 01:26 |
|---|---|---|
| openstack | Launchpad bug 1806156 in Cinder "shared_targets_online_data_migration fails when cinder-volume service not running" [Undecided,Confirmed] | 01:26 |
| ThiagoCMC | My cinder-volume is running and it failes anyway. | 01:26 |
| *** sdake has quit IRC | 01:28 | |
| ThiagoCMC | I'm just commenting out the Ansible block "Perform online database migrations", hope it's okay to not run it! | 01:28 |
| ThiagoCMC | =P | 01:28 |
| *** ansmith has quit IRC | 01:32 | |
| *** tosky has quit IRC | 01:34 | |
| *** TxGirlGeek has quit IRC | 01:40 | |
| *** nurdie has joined #openstack-ansible | 02:16 | |
| *** nurdie has quit IRC | 02:21 | |
| *** cmart has joined #openstack-ansible | 02:21 | |
| *** sdake has joined #openstack-ansible | 02:31 | |
| *** sdake has quit IRC | 02:37 | |
| *** sdake has joined #openstack-ansible | 02:38 | |
| *** sdake has quit IRC | 02:58 | |
| *** sdake has joined #openstack-ansible | 02:59 | |
| *** bgmccollum has quit IRC | 03:00 | |
| *** bgmccollum has joined #openstack-ansible | 03:02 | |
| *** gkadam has joined #openstack-ansible | 03:11 | |
| *** sdake has quit IRC | 03:26 | |
| *** sdake has joined #openstack-ansible | 03:30 | |
| *** sdake has quit IRC | 03:42 | |
| *** udesale has joined #openstack-ansible | 04:02 | |
| *** macza has joined #openstack-ansible | 04:05 | |
| *** macza_ has joined #openstack-ansible | 04:07 | |
| *** macza has quit IRC | 04:09 | |
| *** macza_ has quit IRC | 04:11 | |
| *** jpward1981 has quit IRC | 04:11 | |
| *** macza has joined #openstack-ansible | 04:21 | |
| *** chkumar|out is now known as chandankumar | 04:22 | |
| *** macza has quit IRC | 04:25 | |
| *** cmart has quit IRC | 04:44 | |
| *** sdake has joined #openstack-ansible | 04:49 | |
| *** spsurya has joined #openstack-ansible | 05:05 | |
| *** shyamb has joined #openstack-ansible | 05:11 | |
| *** udesale has quit IRC | 05:29 | |
| *** nurdie has joined #openstack-ansible | 05:32 | |
| *** sdake has quit IRC | 05:41 | |
| chandankumar | jrosser: odyssey4me \o/ | 05:42 |
| chandankumar | jrosser: odyssey4me https://review.openstack.org/633655 will fix nova lxd tempest listing issue | 05:42 |
| openstackgerrit | Chandan Kumar proposed openstack/openstack-ansible-os_nova master: Use venv_packages_to_symlink to symlink to import libvirt-python https://review.openstack.org/633474 | 05:43 |
| *** radeks_ has joined #openstack-ansible | 05:44 | |
| *** shyamb has quit IRC | 05:44 | |
| *** shyamb has joined #openstack-ansible | 05:48 | |
| *** sdake has joined #openstack-ansible | 05:49 | |
| *** udesale has joined #openstack-ansible | 05:51 | |
| *** hwoarang has quit IRC | 05:52 | |
| *** hwoarang has joined #openstack-ansible | 05:55 | |
| *** udesale has quit IRC | 05:59 | |
| *** udesale has joined #openstack-ansible | 06:00 | |
| openstackgerrit | Chandan Kumar proposed openstack/openstack-ansible-os_tempest master: Update cirros from 3.5 to 3.6 https://review.openstack.org/633208 | 06:14 |
| *** markvoelker has joined #openstack-ansible | 06:20 | |
| *** markvoelker has quit IRC | 06:24 | |
| jrosser | chandankumar: for the lxd patch, aren’t there other references to self.client...? That doesn’t look right? | 06:42 |
| chandankumar | jrosser: self.client is used at one place only | 06:43 |
| chandankumar | jrosser: let me pass the file url | 06:45 |
| chandankumar | jrosser: sorry correct, I think I need to fix more stuff | 06:47 |
| *** shyamb has quit IRC | 06:52 | |
| *** shyamb has joined #openstack-ansible | 06:55 | |
| *** arbrandes has joined #openstack-ansible | 07:01 | |
| *** radeks_ has quit IRC | 07:01 | |
| *** arbrandes1 has quit IRC | 07:04 | |
| *** faizy98 has joined #openstack-ansible | 07:05 | |
| jrosser | chandankumar: I spoke with nova-lxd yesterday (check yesterday morning #osa eavesdrop) and it looks like that tempest test duplicates low level tests which should only be done in nova-lxd gate, not a tempest plugin | 07:07 |
| jrosser | It is not right for tempest to need to contact the compute host lxd daemon | 07:09 |
| jrosser | IMHO the right thing to do here is to disable the nova-lxd tempest plugin, if that is possible | 07:10 |
| *** udesale has quit IRC | 07:10 | |
| *** jawad_axd has joined #openstack-ansible | 07:15 | |
| *** udesale has joined #openstack-ansible | 07:18 | |
| chandankumar | jrosser: only way to do that to remove nova-lxd tempest plugin from there since it is not used then | 07:18 |
| chandankumar | jrosser: or let me find another way to disable tempest plugin | 07:21 |
| jrosser | Can we make a list of excluded plugins? That would be neat, then it can be done on a case by case basis | 07:21 |
| *** udesale has quit IRC | 07:22 | |
| *** udesale has joined #openstack-ansible | 07:26 | |
| *** shyamb has quit IRC | 07:38 | |
| chandankumar | jrosser: I think last year I have added a gate on tempest side to filter out broken plugin http://logs.openstack.org/98/631998/1/check/tempest-tox-plugin-sanity-check/d7bfeb3/ | 07:45 |
| chandankumar | jrosser: But I didnot get a chance to cleanup its code | 07:46 |
| chandankumar | jrosser: will i remove the tempest plugin entry point from nova-lxd? | 07:46 |
| chandankumar | jrosser: it will be not discovered by tempest then | 07:47 |
| chandankumar | jrosser: just this part https://github.com/openstack/nova-lxd/blob/master/setup.cfg#L26 | 07:47 |
| jrosser | Perhaps best to talk to tinwood about a long term fix | 07:48 |
| jrosser | However in the short term we need to unstick the osa nova tests | 07:48 |
| *** radeks_ has joined #openstack-ansible | 07:49 | |
| jrosser | If the os_tempest role has a way of filtering out known troublesome plugins we can work around upstream issues | 07:50 |
| chandankumar | jrosser: I will take a look on blacklist plugin | 07:57 |
| chandankumar | jrosser: this one is good to go https://review.openstack.org/#/c/633513/ | 07:57 |
| *** shardy has joined #openstack-ansible | 08:02 | |
| *** sdake has quit IRC | 08:11 | |
| *** shardy has quit IRC | 08:11 | |
| *** shardy has joined #openstack-ansible | 08:12 | |
| *** kopecmartin|off is now known as kopecmartin | 08:19 | |
| evrardjp | mnaser: could you have a look at https://review.openstack.org/#/c/631326/ ? | 08:20 |
| *** markvoelker has joined #openstack-ansible | 08:20 | |
| chandankumar | jrosser: these errors are known http://logs.openstack.org/08/633208/9/check/openstack-ansible-functional-centos-7/95a334e/logs/openstack/openstack1/neutron/neutron-l3-agent.log.txt.gz | 08:26 |
| chandankumar | and this one http://logs.openstack.org/08/633208/9/check/openstack-ansible-functional-centos-7/95a334e/logs/openstack/openstack1/nova/nova-api-wsgi.log.txt.gz#_2019-01-29_07_47_46_941 | 08:27 |
| chandankumar | ? | 08:27 |
| *** electrofelix has joined #openstack-ansible | 08:35 | |
| *** django has quit IRC | 08:36 | |
| jrosser | chandankumar: the first one looks like the neutron vhost setup on rabbitmq is not done until later | 08:36 |
| jrosser | chandankumar: the second one looks more fundamental | 08:37 |
| *** django has joined #openstack-ansible | 08:40 | |
| *** tosky has joined #openstack-ansible | 08:42 | |
| chandankumar | jrosser: is there a way to fix those issue? | 08:42 |
| *** nurdie has quit IRC | 08:45 | |
| *** nurdie has joined #openstack-ansible | 08:45 | |
| *** shyamb has joined #openstack-ansible | 08:47 | |
| *** nurdie has quit IRC | 08:50 | |
| *** pcaruana has joined #openstack-ansible | 08:51 | |
| *** rgogunskiy has joined #openstack-ansible | 08:54 | |
| *** priteau has joined #openstack-ansible | 08:54 | |
| *** markvoelker has quit IRC | 08:54 | |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible-os_nova master: Disable nova-lxd tempest plugin during nova-lxd test https://review.openstack.org/633677 | 09:03 |
| jrosser | chandankumar: lets try that ^ | 09:03 |
| *** Darcidride_ has joined #openstack-ansible | 09:04 | |
| *** Darcidride_ has quit IRC | 09:04 | |
| chandankumar | jrosser: yes, it will not clone nova-lxd repo, thanks :-) | 09:13 |
| chandankumar | jrosser: one more things, Is there a way to clone those depends on also in test-evirnonment https://review.openstack.org/#/c/633208/ for example this one? | 09:14 |
| *** faizy98 has quit IRC | 09:18 | |
| *** DanyC has joined #openstack-ansible | 09:18 | |
| *** DanyC has quit IRC | 09:19 | |
| *** DanyC has joined #openstack-ansible | 09:19 | |
| *** DanyC has quit IRC | 09:22 | |
| *** DanyC has joined #openstack-ansible | 09:22 | |
| *** DanyC has quit IRC | 09:24 | |
| *** DanyC has joined #openstack-ansible | 09:24 | |
| *** DanyC has quit IRC | 09:28 | |
| *** DanyC has joined #openstack-ansible | 09:43 | |
| *** markvoelker has joined #openstack-ansible | 09:51 | |
| jrosser | chandankumar: sorry i'm not sure about that - i'd need to spin up a vagrant box and start digging..... i'm not very familiar with the tox test stuff | 09:56 |
| jrosser | and afaik the depends-on is resolved by zuul, not the test code in the repo? | 09:57 |
| chandankumar | jrosser: ok | 09:58 |
| *** PTO has joined #openstack-ansible | 10:02 | |
| *** CeeMac has quit IRC | 10:08 | |
| *** shyamb has quit IRC | 10:10 | |
| *** shyamb has joined #openstack-ansible | 10:11 | |
| *** exbob has joined #openstack-ansible | 10:16 | |
| *** markvoelker has quit IRC | 10:24 | |
| *** shyamb has quit IRC | 10:38 | |
| *** shyamb has joined #openstack-ansible | 10:40 | |
| *** shyamb has quit IRC | 10:45 | |
| jrosser | chandankumar: so this https://review.openstack.org/#/c/633677/ unsticks the lxd bit | 10:59 |
| jrosser | we are just left with whatever is upsetting centos | 10:59 |
| chandankumar | jrosser: we have kept one node on hold | 11:06 |
| chandankumar | for debugging temepst issude | 11:06 |
| jrosser | excellent - hopefully we can find whats up there | 11:06 |
| chandankumar | jrosser: but sshing into the node, it appears that all the tempest related venv files are not there | 11:06 |
| *** d3n14l has joined #openstack-ansible | 11:06 | |
| *** mkuf_ has quit IRC | 11:07 | |
| chandankumar | jrosser: does it wipes out all the stuff | 11:07 |
| chandankumar | ? | 11:07 |
| jrosser | it should all stay afaik | 11:07 |
| *** mkuf has joined #openstack-ansible | 11:07 | |
| chandankumar | in /root there is only openrc | 11:08 |
| *** udesale has quit IRC | 11:09 | |
| d3n14l | Hey guys - trying to deploy octavia (osa rocky) - I have set up the network and it looks good until my dhcp-Agent ports for the lbaas mgmt net are put into VLAN 4095 in OVS. Any hint is appreciated… | 11:09 |
| *** d3n14l has quit IRC | 11:12 | |
| *** markvoelker has joined #openstack-ansible | 11:21 | |
| *** ansmith has joined #openstack-ansible | 11:33 | |
| *** slaweq has joined #openstack-ansible | 11:34 | |
| slaweq | hi | 11:34 |
| slaweq | chandankumar | 11:34 |
| chandankumar | slaweq: jrosser Hello | 11:34 |
| jrosser | hi | 11:34 |
| slaweq | jrosser: hi | 11:34 |
| chandankumar | jrosser: slaweq is in temepst container | 11:35 |
| jrosser | so ssh fails from the tempest container to the vm | 11:35 |
| *** shyamb has joined #openstack-ansible | 11:35 | |
| slaweq | jrosser: I didn't check it yet but it looks so | 11:35 |
| slaweq | for now I tried manually to create network/subnet/router and spawn vm | 11:35 |
| jrosser | here is the error http://logs.openstack.org/77/633677/1/check/openstack-ansible-functional-centos-7/8d55690/logs/openstack/infra1/stestr_results.html | 11:35 |
| slaweq | all worked fine for me | 11:36 |
| jrosser | what is the ip of the vm? | 11:36 |
| slaweq | jrosser: my manually created vm is 4b6e7448-2ab4-49d5-bdf8-6cfcd1b3a90c | 11:37 |
| slaweq | jrosser: it pings from host but not from tempest-1 container | 11:38 |
| jrosser | i cant ping the ip of the router 73a5c739-1758-44ad-9ea7-891a043b9945 10.1.3.197 from inside the tempest container | 11:38 |
| jrosser | also cannot ping router 6f50d731-c43e-4e55-80fd-d4f429e3c444 10.1.3.101 | 11:39 |
| slaweq | yep, all is reachable from host but not from container | 11:40 |
| chandankumar | jrosser: slaweq this review was added https://github.com/openstack/openstack-ansible-tests/commit/fe6c8344d1cdf23add574a461af280d4033b8428 to use systemd networkd role | 11:42 |
| slaweq | jrosser: when I added IP address from "public" network inside container on eth12 it works fine | 11:43 |
| slaweq | ip a a 10.1.3.254/24 dev eth12 | 11:43 |
| jrosser | ok i can now ping the router | 11:44 |
| slaweq | [root@tempest1 ~]# ping 10.1.3.197 | 11:44 |
| jrosser | yes we did the same thing at the same tim e:) | 11:44 |
| slaweq | PING 10.1.3.197 (10.1.3.197) 56(84) bytes of data. | 11:44 |
| slaweq | 64 bytes from 10.1.3.197: icmp_seq=1 ttl=64 time=0.924 ms | 11:44 |
| slaweq | :) | 11:44 |
| slaweq | so IMHO this is missing in container config | 11:44 |
| jrosser | eth12 is in the same bridge on the host as br-vlan | 11:44 |
| jrosser | ^ ykwim | 11:44 |
| slaweq | yes, it's on same bridge | 11:45 |
| jrosser | i wonder why this works on the other distros | 11:45 |
| slaweq | but in container You don't have route to it | 11:45 |
| jrosser | i tried adding a route via eth0 but that didnt work | 11:45 |
| slaweq | so You have: | 11:45 |
| slaweq | [root@tempest1 ~]# ip route get 10.1.3.197 | 11:45 |
| slaweq | 10.1.3.197 dev eth0 src 10.100.100.51 | 11:45 |
| slaweq | cache | 11:45 |
| chandankumar | slaweq: https://github.com/openstack/openstack-ansible-os_tempest/search?q=eth12&unscoped_q=eth12 | 11:45 |
| jrosser | perhaps the default route works on the other platforms | 11:45 |
| slaweq | and Your ping is going through eth0 | 11:45 |
| slaweq | when we added IP from same subnet on eth12 packets to IP from this network were send via eth12 | 11:46 |
| slaweq | not via default route and it works then | 11:46 |
| slaweq | jrosser: but I can't help with OSA containers config - I have zero experience with that :/ | 11:46 |
| jrosser | i think we need to add vlan_address in here https://github.com/openstack/openstack-ansible-os_tempest/blob/8bc47db7c00d1dbdcba2464ff0c22a364f44faf8/tests/host_vars/tempest1.yml | 11:47 |
| jrosser | and then wire it into the setup of the tempest container | 11:48 |
| jrosser | let me take a look at that - i have some meetings but will try to do it after lunch | 11:48 |
| slaweq | jrosser: thx | 11:48 |
| chandankumar | slaweq: jrosser thanks :-) | 11:48 |
| jrosser | i have no idea why only on centos this breaks though :/ | 11:48 |
| slaweq | jrosser: I'm disconnecting from this nodes now | 11:48 |
| slaweq | if You would need help from neutron side, ping me or someone on neutron channel later | 11:49 |
| slaweq | I will be afk in few minutes | 11:49 |
| chandankumar | jrosser: let me know when you are done, so that we can inform the openstack-infra team for reuse :-) | 11:49 |
| jrosser | i'm logged out now, you can release the node unless we want it for anything else | 11:49 |
| *** rgogunskiy has quit IRC | 11:49 | |
| chandankumar | jrosser: sure | 11:50 |
| *** ansmith has quit IRC | 11:52 | |
| *** markvoelker has quit IRC | 11:53 | |
| *** d3n14l has joined #openstack-ansible | 11:53 | |
| *** shyamb has quit IRC | 11:53 | |
| *** shyamb has joined #openstack-ansible | 11:55 | |
| *** kukacz has quit IRC | 12:04 | |
| *** kukacz has joined #openstack-ansible | 12:04 | |
| *** fdegir is now known as fdegir_ | 12:10 | |
| odyssey4me | ThiagoCMC two things - if the repo masters group has no hosts, then something is broken in your inventory/host group config.... without that, there are no repo servers... and without those, you'll be running an OSA deployment i a very slow and non-repeatable way | 12:19 |
| odyssey4me | ThiagoCMC also, online migrations for cinder are important - if not running them makes the playbook continue, then you have something broken somewhere | 12:19 |
| chandankumar | jrosser: so basically we need to centos7 job then we can able to land lxd changes | 12:20 |
| *** mkuf has quit IRC | 12:20 | |
| odyssey4me | chandankumar if https://review.openstack.org/633655 fixes the issue, then perhaps that means that we need to add pylxd as a package into the OSA venv, or we need to ensure the plugin includes it as a dependency? | 12:21 |
| odyssey4me | jrosser chandankumar given that our tests aren't using the nova-lxd tempest plugin, perhaps it's best for us to just remove it from the tempest role - or perhaps to prevent re-addition by mistake, we should comment it out of the lists with a note that it is intentionally left out | 12:23 |
| chandankumar | odyssey4me: +1 on removal | 12:23 |
| chandankumar | odyssey4me: jrosser is working on a patch to fix tempest container vxlan networking issue | 12:24 |
| odyssey4me | mnaser evrardjp I noticed today that Stein is past M2, and OSA has not released M1 or M2, and as I recall, you have to release at least 2 milestones in order to be accepted for the final release. Is this something that we know about and are sorting out, or are we dropping the ball? | 12:24 |
| evrardjp | I don't think that's still the case | 12:25 |
| evrardjp | I think with the move to cycle-with-rc that requirement dropped | 12:26 |
| evrardjp | but let me double check | 12:26 |
| jrosser | odyssey4me: yes we have two options, remove the nova-lxd plugin entirely or just disable it like this https://review.openstack.org/#/c/633677/ | 12:26 |
| nowster | Sigh. Just been chasing a failure. If one reboots a compute node, nova starts before the libvirt framework, and promptly disables itself. | 12:26 |
| *** mkuf has joined #openstack-ansible | 12:26 | |
| evrardjp | odyssey4me: I have a confirmation | 12:27 |
| evrardjp | it's not needed anymore | 12:27 |
| evrardjp | odyssey4me: I will check if the docs in release is up to date | 12:29 |
| *** fdegir has joined #openstack-ansible | 12:30 | |
| odyssey4me | evrardjp oh ok, thanks - I didn't realise there was a governance change... I guess this takes the pressure off a bit | 12:30 |
| evrardjp | odyssey4me: you can see the text was adapted in here: https://releases.openstack.org/reference/release_models.html#cycle-trailing . The cycle-with-milestones where this applied is considered legacy | 12:31 |
| evrardjp | (we moved to cycle-with-rc) | 12:31 |
| odyssey4me | nowster hmm, which release is that - we had a patch go in some time ago to fic that | 12:31 |
| evrardjp | odyssey4me: I think the requirement for trailing was changed further longer ago, but anyway, long story short, we don't need to release that often, as it doesn't really make sense for us (we should rather point to master instead) | 12:32 |
| openstackgerrit | Chandan Kumar proposed openstack/openstack-ansible-os_tempest master: Remove nova_lxd tempest plugin https://review.openstack.org/633711 | 12:32 |
| *** fdegir_ has left #openstack-ansible | 12:32 | |
| chandankumar | odyssey4me: ^^ | 12:32 |
| chandankumar | odyssey4me: will I remove the nova_lxd service available flag also? | 12:32 |
| chandankumar | evrardjp: cloudnull https://review.openstack.org/633513 is good to go | 12:33 |
| evrardjp | chandankumar: will have a look | 12:33 |
| PTO | Is the openstack ansible pike release dead? | 12:33 |
| odyssey4me | chandankumar I think it might be better to just set the enablement to 'false' like jrosser did in https://review.openstack.org/#/c/633677/, with the same note to ensure we remember why... but leaving that does mean we can still enable it later if we want to | 12:34 |
| odyssey4me | PTO nope | 12:34 |
| chandankumar | odyssey4me: sure | 12:34 |
| odyssey4me | PTO it passed the deploy test just yesterday: http://zuul.openstack.org/build/ddae77d295a944fea1593ab7a6759fe4 | 12:34 |
| evrardjp | PTO: ocata is "kinda" as we decided to not release anymore | 12:34 |
| odyssey4me | well, this morning | 12:35 |
| evrardjp | Pike and others are fine | 12:35 |
| PTO | I just tried to bootstrap the pike release again and some git repos are missing (have been deleted on github.com) | 12:35 |
| evrardjp | I haven't tagged Pike last week-end, which I had to do on monday which got pretty busy. But it's still on my todolist for today. | 12:35 |
| evrardjp | PTO which release of pike? | 12:35 |
| odyssey4me | PTO yeah, that's a known issue which is fixed at stable/pike, but not in a release yet | 12:36 |
| evrardjp | oh the ceph bit? | 12:36 |
| odyssey4me | yep | 12:36 |
| evrardjp | ok | 12:36 |
| evrardjp | yeah I tried to find time on sunday, but I got sidetracked | 12:36 |
| PTO | the ceph-defaults | 12:36 |
| evrardjp | I am only part time in this (0% of my full time:p ) | 12:36 |
| odyssey4me | PTO the ceph folks deleted all the ceph-* role repositories. | 12:37 |
| PTO | I'm gonna upgrade very soon. Can i jump from pike to rocky in one step? | 12:37 |
| PTO | Or should i goto queens first? | 12:37 |
| odyssey4me | PTO I know we have done some tests - maybe antonym can comment when he comes online. However, the official response would be that you must go through each release. | 12:38 |
| jamesdenton | mornin' folks | 12:38 |
| odyssey4me | You *might* be able to skip some things, but you'd have to do test runs of that in a lab env to qualify parts you can skip. | 12:38 |
| PTO | So better be safe and jump to queens first - agree? | 12:39 |
| odyssey4me | Yes, absolutely. | 12:39 |
| PTO | So just follow the guide, checkout and run-update.sh | 12:39 |
| odyssey4me | PTO either that, or run the steps the script does yourself by hand - it's up to you and your config. | 12:40 |
| odyssey4me | And your uptime expectations. | 12:40 |
| *** pcaruana has quit IRC | 12:40 | |
| PTO | I think im gonna try the script (easy mode) and if it fails then go through the steps | 12:40 |
| PTO | Should I apply any minor updates in pike before the major upgrade to queens? | 12:41 |
| jamesdenton | Pay special attention to the notes of neutron agent going baremetal (from container). odyssey4me worked out a lot of the automation to make that smooth, but any feedback is appreciated | 12:42 |
| odyssey4me | PTO yeah, pike->queens includes a consolidation of multiple containers per service to one per service, and a move of the neutron agents on to bare metal. | 12:43 |
| PTO | Interesting... I have not yet deployed all minor patches. Should these be deployed before attempting the major upgrade? | 12:44 |
| odyssey4me | PTO it's usually better to update to the latest release tag in the series, then do the upgrade, because that's closest to what we test... but you can do a validation test in a lab to see whether your current tag will just work... or of course you can look through the changes in the same series to see whether they look like they should be done before upgrading... it's really down to your specific use case | 12:46 |
| PTO | I was planning to update the minor releases, but im not able to bootstrap the pike package. Any ETA on when you will release a fix? | 12:47 |
| ioni | PTO, odyssey4me what's important and not mentioned in the upgrade documentation is to move the agents from container to bare metal before deleting the container | 12:47 |
| PTO | @ioni good point! I will write that down :-) | 12:48 |
| ioni | i had issues with dhcp ports not being moved automatically to bare metal | 12:48 |
| ioni | also make sure to update network configuration | 12:48 |
| ioni | br-vxlan didn't had any ip on the controller | 12:48 |
| PTO | I assume the script will redeploy the agents on bare metal during the process - correct? | 12:48 |
| odyssey4me | ioni oh? then that's a bug and we should fix that - if you can register the bug and the steps you had to do manually, then we can automate it in | 12:48 |
| ioni | because it ip was inside the container | 12:48 |
| ioni | now you have to create the right configuration for br-vxlan and br-vlan(the eth12 pair) | 12:49 |
| evrardjp | yeah that sounds like a bug | 12:49 |
| odyssey4me | PTO you can just use stable/pike rather than a tag for now... or wait for the tag release | 12:49 |
| *** pcaruana has joined #openstack-ansible | 12:50 | |
| ioni | odyssey4me, well, i'm not sure if is a bug or not, on my 5 regions that i've done the upgrade, only one didn't moved automatically the port | 12:50 |
| *** kaiokmo has joined #openstack-ansible | 12:50 | |
| ioni | i had to disable dhcp and reenable it | 12:50 |
| odyssey4me | ioni oh, that's odd | 12:50 |
| PTO | odyssey4me: git checkout stable/pike | 12:50 |
| *** markvoelker has joined #openstack-ansible | 12:50 | |
| odyssey4me | PTO yep | 12:50 |
| ioni | because the agent id was missing in order to move it using neutron commands like this: https://www.openstackfaq.com/openstack-migrate-routers-and-dhcp/ | 12:51 |
| PTO | cool | 12:51 |
| PTO | I have bootstraped the queens package. Should I manually remove /etc/ansible/roles/* before? | 12:52 |
| ioni | there is a playbook for that | 12:52 |
| ioni | https://docs.openstack.org/openstack-ansible/queens/admin/upgrades/major-upgrades.html | 12:52 |
| odyssey4me | PTO yeah, if you're using stable/queens then there is... I don't think the adjustment made it into the last queens release | 12:53 |
| odyssey4me | but yes, it's advisable to wipe out /etc/ansible/roles/ceph* before doing the pike/queens upgrades | 12:53 |
| ioni | PTO, i was able to boostrap pike 16.0.25 | 12:54 |
| ioni | i have a forked version of openstack-ansible that i manage into my private git | 12:54 |
| ioni | and from time to time a sync | 12:54 |
| *** d3n14l has quit IRC | 12:55 | |
| odyssey4me | I'll propose https://github.com/openstack/openstack-ansible/commit/d528daf069559c3686f05a26f9b4d68c84a34b77#diff-0e0b5a4ebeeb2dd9a60106998e218e0b to pike to make sure people know to do that | 12:55 |
| PTO | I'm currently running pike (16.0.9). Got the queens branch and did a bootstrap. | 12:55 |
| *** shyamb has quit IRC | 12:55 | |
| evrardjp | Ok it seems there is some kind of urgency to release so I will stop what I am doing to do it | 12:56 |
| evrardjp | it thought it could wait a few more hours. | 12:56 |
| PTO | I can go with the stable/pike branch - no problems | 12:57 |
| PTO | Just want to know how to "roll back" the queens bootstrap | 12:57 |
| *** mkuf_ has joined #openstack-ansible | 12:59 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible stable/pike: Add release note about ceph role changes https://review.openstack.org/633718 | 12:59 |
| odyssey4me | PTO to roll back the ansible bootstrap, you'll need to wipe /opt/ansible-runtime and wipe /etc/ansible/roles - then checkout stable/pike and do the bootstrap again | 13:01 |
| odyssey4me | evrardjp no urgency for a pike release just yet, because https://review.openstack.org/633718 should still go in first :) | 13:01 |
| evrardjp | ok | 13:02 |
| *** mkuf has quit IRC | 13:02 | |
| evrardjp | well | 13:02 |
| openstackgerrit | Jean-Philippe Evrard proposed openstack/openstack-ansible stable/rocky: Bump version to 18.1.4 https://review.openstack.org/633720 | 13:02 |
| evrardjp | your patch can merge fast | 13:03 |
| evrardjp | let me -w another patch then | 13:03 |
| *** priteau has quit IRC | 13:04 | |
| *** priteau has joined #openstack-ansible | 13:04 | |
| evrardjp | odyssey4me: I guess your patch should be merged first, so we should probably get another vote from cores? | 13:04 |
| evrardjp | maybe jrosser? | 13:04 |
| nowster | Sigh². It appears that the VXLAN is being mapped to br-vxlan on the infra host, but br-mgmt's IP on the compute node. | 13:05 |
| evrardjp | anyway now that I am interrupted, I will just do the other releases now | 13:05 |
| evrardjp | :p | 13:05 |
| nowster | wrong values in the linuxbridge conf. | 13:07 |
| ioni | nowster, are you sure that on compute node has an ip to br-vxlan ? | 13:09 |
| ioni | i had this issue on infra when i forgot to apply networking modification from pike to queens | 13:09 |
| openstackgerrit | Jean-Philippe Evrard proposed openstack/openstack-ansible stable/queens: Bump version to 17.1.8 https://review.openstack.org/633725 | 13:09 |
| nowster | ioni: I'm checking now. | 13:10 |
| *** priteau has quit IRC | 13:12 | |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible-tests master: Set a defined IP address range for tempest test public addresses https://review.openstack.org/633728 | 13:14 |
| *** strattao has joined #openstack-ansible | 13:18 | |
| *** markvoelker has quit IRC | 13:20 | |
| *** gkadam has quit IRC | 13:20 | |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible-os_tempest master: Add an ip address to eth12 in OSA test containers https://review.openstack.org/633732 | 13:28 |
| jrosser | chandankumar: ^ that should fix it | 13:28 |
| odyssey4me | jrosser oh nice, and thank you so much for figuring it out - assuming the test passes ;) | 13:29 |
| jrosser | well i hope so - this had been on my mind as the proxy scenario is failing, and i'd come to a very similar conclusion about why that was | 13:30 |
| nowster | ioni: the interface is there and has the right address | 13:32 |
| jrosser | nowster: i think that this https://github.com/openstack/openstack-ansible/blob/master/playbooks/common-tasks/dynamic-address-fact.yml is used in the neutron setup to decide which IP to pick | 13:33 |
| nowster | ta. I fixed up the linuxbridge config and it seems to have done the right thing | 13:34 |
| openstackgerrit | Merged openstack/openstack-ansible stable/pike: Add release note about ceph role changes https://review.openstack.org/633718 | 13:34 |
| nowster | jrosser: looks like it picked the wrong one, as it had the ip address of br-mgmt in there | 13:36 |
| *** ansmith has joined #openstack-ansible | 13:36 | |
| chandankumar | jrosser: thanks, In a meeting, will take a look soon | 13:38 |
| *** zul has quit IRC | 13:38 | |
| PTO | odyssey4me: thx for clarifying | 13:38 |
| jrosser | nowster: or the data being fed in is wrong, it will take a bridge name and return the IP, so if the openstack_user_config has br-mgmt somewhere instead of br-vlxan then the same thing will happen | 13:40 |
| *** nurdie has joined #openstack-ansible | 13:42 | |
| *** mkuf has joined #openstack-ansible | 13:43 | |
| *** mkuf_ has quit IRC | 13:45 | |
| odyssey4me | evrardjp you can release the -w on https://review.openstack.org/633348 now | 13:45 |
| evrardjp | releasing the kraken! | 13:47 |
| evrardjp | and pike | 13:47 |
| evrardjp | I am sad release k was kilo and not kraken | 13:48 |
| openstackgerrit | Merged openstack/openstack-ansible-tests master: Ensure selinux bindings are linked into the venv https://review.openstack.org/633513 | 13:48 |
| PTO | I have read somewhere that its possible to use swift with ceph as storage backend. Is this somethink you have looked at? | 13:51 |
| odyssey4me | PTO I don't know if that's ever been a thing, but ceph has ceph rgw, which provides a swift API to a ceph back-end | 13:52 |
| *** cmart has joined #openstack-ansible | 13:53 | |
| ioni | not related top OSA, but i know that you guys also operate public or private clouds. | 13:54 |
| ioni | how do you "disable" obsolete images ? | 13:54 |
| ioni | i use --deactivate but this has problems with nova when instances want to resize, i got Not authorized for image and instances is then in error state | 13:54 |
| *** CeeMac has joined #openstack-ansible | 13:55 | |
| PTO | Just wanted to check if you had a code snippet in your stash for testing :-) | 13:55 |
| CeeMac | afternoon channel | 13:55 |
| jrosser | PTO i run ceph rgw with both swift and S3 API | 13:55 |
| jrosser | you should find an example setup in the osa ceph test scenario iirc | 13:56 |
| jrosser | certainly for swift api | 13:56 |
| ThiagoCMC | odyssey4me, thanks for clarifying that! Not sure why repo masters aren't there, here is my conf: https://github.com/tmartinx/openstack_deploy/blob/master/openstack_user_config.yml - can you see something wrong? | 14:02 |
| odyssey4me | ThiagoCMC odd, repo_infra-hosts should put it there. | 14:02 |
| ThiagoCMC | The cinder thing finally worked! But Cinder still can't create volumes on Ceph. When I try to create a vol, the cinder-vlumes becomes "Down" but, daemon still running, very weird. | 14:02 |
| ThiagoCMC | odyssey4me, I'll do it now! Thanks! | 14:03 |
| ThiagoCMC | I already have "repo-infra_hosts:" | 14:03 |
| ThiagoCMC | :-/ | 14:03 |
| ThiagoCMC | is it with underscore or dash? | 14:04 |
| odyssey4me | ThiagoCMC ok, OSA is built in such a way that if it's got the right config it'll just work - so if something turns out broken, then usually it's missing config or missing underlying network/storage config... so I'd suggest ensuring that you get it running through setup-infrastructure without error before trying to fix setup-openstack | 14:04 |
| odyssey4me | ThiagoCMC you have 'repo-infra_hosts' which is correct, so it should be there - if you run through the repo-server playbook, does it work? | 14:05 |
| ThiagoCMC | yeah, I just did this yesterday for the first time, setup-everything.yml worked without a single error. | 14:05 |
| ThiagoCMC | But still, Glance can't upload images to Ceph (while openstack images list works), Cinder can't create volumes and Heat (openstack stack list) returns error 500. | 14:06 |
| ThiagoCMC | Hard time... lol | 14:06 |
| ThiagoCMC | The syntax check tells me about the repo masters with a warning. | 14:06 |
| ThiagoCMC | but I can deploy Rocky anyway | 14:07 |
| odyssey4me | ThiagoCMC yeah, the syntax check will return that - that's not a concern | 14:07 |
| ThiagoCMC | Hmm... ok lol | 14:07 |
| odyssey4me | ok, if glance can list but not upload images, then that points at some sort of inability to write to the back-end | 14:07 |
| odyssey4me | check the cinder-volume service log... | 14:08 |
| ThiagoCMC | Nothing hits the cinder-volume logs. | 14:08 |
| ThiagoCMC | I was watching with journalctl, nothing. | 14:08 |
| odyssey4me | ThiagoCMC ok, check the systemd journal on the cinder-volume host? | 14:08 |
| ThiagoCMC | yep, that's what I did | 14:09 |
| odyssey4me | ok, try watching that and restarting the cinder-volume service? | 14:09 |
| ioni | questions regarding keystone in rocky | 14:09 |
| ioni | is not used anymore 35357 ? | 14:09 |
| odyssey4me | ioni nope | 14:09 |
| ThiagoCMC | journalctl -f -u cinder-volume - after systemctl restart cinder-volume | 14:09 |
| ioni | so we have to delete the old nginx configuration for that port | 14:09 |
| ioni | is not going to work if updating from a version that had one | 14:09 |
| ioni | i had this problem now | 14:09 |
| ThiagoCMC | I can see it is restarted and up, then, when I try to create a vol, it fails and status down but daemon still running. | 14:09 |
| odyssey4me | ioni I think we baked that all in already | 14:10 |
| ioni | cool | 14:10 |
| ioni | wainting for new tag then | 14:10 |
| odyssey4me | ioni https://github.com/openstack/openstack-ansible-os_keystone/commit/ff63ec8a3ef0057eced6467980f4f5c4833e0db6 | 14:10 |
| *** zul has joined #openstack-ansible | 14:10 | |
| ioni | cool | 14:10 |
| ThiagoCMC | I have to drive to office now, chat soon! I'm in desperate need for help! ^_^ | 14:10 |
| ioni | odyssey4me, so what happens for old endpoints that point to 35357? | 14:11 |
| odyssey4me | ThiagoCMC hmm, so the cinder agent list shows it as down? | 14:11 |
| odyssey4me | ioni hmm, I don't think we have something to delete it...? | 14:11 |
| ioni | odyssey4me, right now i have a mix setup with rocky and queens | 14:12 |
| ioni | rocky is where the keystone is | 14:12 |
| chandankumar | jrosser: http://logs.openstack.org/32/633732/1/check/openstack-ansible-functional-centos-7/cff6c8f/job-output.txt.gz#_2019-01-29_14_01_52_150170 | 14:12 |
| nowster | jrosser: can't see where that might be wrong. I've just been through openstack_user_config.yml | 14:12 |
| *** shyamb has joined #openstack-ansible | 14:12 | |
| chandankumar | jrosser: failing at create container mac script | 14:13 |
| odyssey4me | ioni yeah, it looks like we don't have a task to remove the old admin endpoint - could you register a bug or submit a patch for that? | 14:14 |
| ioni | odyssey4me, well, i think it got updated to 5000 for admin | 14:14 |
| odyssey4me | ioni oh yes, that is right | 14:15 |
| ioni | odyssey4me, i see it on rocky region having 5000 | 14:15 |
| ioni | odyssey4me, i need to see if queens works with admin being 5000 | 14:15 |
| odyssey4me | oh, that's correct - there is still an admin endpoint, just on the same port as all other endpoints | 14:15 |
| odyssey4me | it's no longer a seperate wsgi app | 14:15 |
| ioni | cool | 14:15 |
| ioni | less memory :D | 14:15 |
| odyssey4me | yep | 14:15 |
| ioni | sorry for the noise, i wasn't up to date from git | 14:16 |
| ioni | i'm testing the branch with latest commit being Merge "Update all SHAs for 18.1.3" into stable/rocky | 14:17 |
| jrosser | chandankumar: yes just having a look | 14:18 |
| *** samc-bbc has joined #openstack-ansible | 14:19 | |
| *** shyamb has quit IRC | 14:20 | |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible-os_tempest master: Add an ip address to eth12 in OSA test containers https://review.openstack.org/633732 | 14:21 |
| jrosser | chandankumar: i missed one of the containers, that should be better | 14:22 |
| nowster | Good news after that is that I have IPv6 pings having fixed the vxlan binding. | 14:23 |
| *** udesale has joined #openstack-ansible | 14:24 | |
| jrosser | nowster: what did you need to fix in the end - is there a bug we need to look at? | 14:29 |
| *** gshippey has joined #openstack-ansible | 14:33 | |
| nowster | jrosser: it was "local_ip = 172.29.xxx.12" in /etc/neutron/plugins/ml2/linuxbridge_agent.ini on the compute node. | 14:33 |
| nowster | xxx = 236 from ansible, I changed it to 240, and things meshed correctly | 14:34 |
| nowster | 236 is mgmt, 240 is vxlan (as per example config) | 14:34 |
| openstackgerrit | Jean-Philippe Evrard proposed openstack/openstack-ansible stable/pike: Bump version to 16.0.26 https://review.openstack.org/633751 | 14:36 |
| *** SimAloo has joined #openstack-ansible | 14:37 | |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible-os_tempest stable/rocky: Update all plugin urls to use https rather than git https://review.openstack.org/633752 | 14:37 |
| *** SimAloo has quit IRC | 14:37 | |
| *** sum12 has quit IRC | 14:38 | |
| *** sum12 has joined #openstack-ansible | 14:38 | |
| *** dave-mccowan has joined #openstack-ansible | 14:38 | |
| *** SimAloo has joined #openstack-ansible | 14:40 | |
| *** dave-mccowan has quit IRC | 14:45 | |
| *** pcaruana has quit IRC | 14:45 | |
| *** sdake has joined #openstack-ansible | 14:50 | |
| *** sdake has quit IRC | 14:51 | |
| *** pcaruana has joined #openstack-ansible | 14:53 | |
| evrardjp | odyssey4me and cores: does it sounds reasonable to say bootstrap-ansible is always run when doing a minor update in a branch | 14:54 |
| odyssey4me | evrardjp absolutely, yes | 14:54 |
| *** sdake has joined #openstack-ansible | 14:54 | |
| evrardjp | I thought too. | 14:54 |
| jrosser | evrardjp: otherwise you dont get the roles checked out to the right points | 14:54 |
| odyssey4me | evrardjp shall I put a patch together to take the option of using ansible-galaxy out? | 14:55 |
| evrardjp | well | 14:55 |
| jrosser | or a potential minor ansible version upgrade | 14:55 |
| evrardjp | jrosser: I thought ppl could use the play to fetch latest, and not update ansible for example | 14:55 |
| evrardjp | but yeah, I think it's fair to say so | 14:55 |
| evrardjp | odyssey4me: good idea | 14:55 |
| odyssey4me | ok, will do that now - then we can add a tracking branch into a-r-r too | 14:55 |
| evrardjp | yeah | 14:56 |
| evrardjp | explicit is better than implicit :) | 14:56 |
| evrardjp | but next to that I got an idea for the openstack-ansible wrapper that removes the need to update the update file. Simple :) | 14:56 |
| evrardjp | the version file* | 14:57 |
| evrardjp | I found a bug in bootstrap ansible at the same time | 14:57 |
| evrardjp | so I will file a few things | 14:57 |
| evrardjp | with that in mind we should do a first alpha release of master branch | 14:59 |
| evrardjp | then we can go full auto | 14:59 |
| evrardjp | I am thrilled :) | 14:59 |
| * nowster tries to work out what sets neutron_local_ip on each type of host | 15:00 | |
| odyssey4me | :) | 15:00 |
| nowster | but meeting first | 15:00 |
| *** nurdie has quit IRC | 15:00 | |
| *** nurdie has joined #openstack-ansible | 15:01 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible master: Remove ANSIBLE_ROLE_FETCH_MODE https://review.openstack.org/633756 | 15:04 |
| odyssey4me | evrardjp ^ | 15:05 |
| *** nurdie has quit IRC | 15:06 | |
| evrardjp | woot | 15:10 |
| evrardjp | I am in a meeting but reviewing | 15:10 |
| chandankumar | jrosser: thanks ! | 15:11 |
| jamesdenton | nowster neutron_local_ip should be populated from tunnel_address for a given host/container | 15:11 |
| antonym | PTO: odyssey4me: i was able to jump from newton to queens (and separately rocky) last week in the lab by migrating all DBs for each release, and then only running the final release upgrade. i pulled some cleanup stuff from queens along with the neutron bare metal migration scripts and everything still seemed to work afterwards, there were a few oddies that come up but nothing really major and easy | 15:13 |
| antonym | to fix up. i'm continuing testing on it this week to automate | 15:13 |
| odyssey4me | antonym oh nice :) I'm guessing that's mostly just doing a subset of actions from each series? | 15:14 |
| antonym | yeah, i tossed together some playbooks that stitch together all of the actions migrations for each release and runs them from a venv, just loop through that for each release and then run the final target run-upgrade.sh... then just have to go back and pick up all the cleanup items from older upgrades | 15:15 |
| openstackgerrit | Jean-Philippe Evrard proposed openstack/openstack-ansible master: Define OSA clone dir in the openstack-ansible.sh script https://review.openstack.org/633759 | 15:17 |
| nowster | this is odd: http://paste.openstack.org/show/744170/ | 15:17 |
| antonym | it's running all the ansible_fact_cleanup, config changes, secrets adjustments, etc too for each release so we're not missing anything | 15:17 |
| nowster | I'd be expecting an "After=libvirtd.service" in there. | 15:18 |
| *** udesale has quit IRC | 15:18 | |
| jrosser | nowster: can you give this a spin? https://review.openstack.org/#/c/633104/ | 15:19 |
| *** jwitko has joined #openstack-ansible | 15:22 | |
| nowster | jrosser: that seems sensible to me | 15:23 |
| * nowster = meetinging | 15:24 | |
| *** jawad_axd has quit IRC | 15:27 | |
| *** jawad_axd has joined #openstack-ansible | 15:28 | |
| *** jawad_axd has quit IRC | 15:29 | |
| *** sdake has quit IRC | 15:32 | |
| evrardjp | antonym: nice | 15:34 |
| evrardjp | odyssey4me: what do you think of https://review.openstack.org/#/c/633759/1 ? | 15:34 |
| *** sdake has joined #openstack-ansible | 15:35 | |
| *** nurdie has joined #openstack-ansible | 15:38 | |
| openstackgerrit | Jean-Philippe Evrard proposed openstack/openstack-ansible master: Mark OSA version in the wrapper script https://review.openstack.org/633762 | 15:43 |
| openstackgerrit | Jean-Philippe Evrard proposed openstack/openstack-ansible master: Use an env lookup to determine the OSA version https://review.openstack.org/633763 | 15:43 |
| *** mattheca has joined #openstack-ansible | 15:44 | |
| odyssey4me | evrardjp will look in a bit after my meeting | 15:46 |
| openstackgerrit | Jean-Philippe Evrard proposed openstack/openstack-ansible master: Use an env lookup to determine the OSA version https://review.openstack.org/633763 | 15:47 |
| *** ztr has joined #openstack-ansible | 15:50 | |
| *** openstackgerrit has quit IRC | 15:51 | |
| *** openstackgerrit has joined #openstack-ansible | 15:51 | |
| openstackgerrit | Francois Deppierraz proposed openstack/openstack-ansible-lxc_hosts stable/rocky: Increase LXC container default shutdown delay https://review.openstack.org/633767 | 15:51 |
| *** mbuil has joined #openstack-ansible | 15:53 | |
| mbuil | guys, could you check https://review.openstack.org/#/c/622216/ please? thx! | 15:53 |
| openstackgerrit | Francois Deppierraz proposed openstack/openstack-ansible-lxc_hosts stable/rocky: Increase LXC container default shutdown delay https://review.openstack.org/633767 | 15:55 |
| CeeMac | anyone got a fix for dhcp-agent unable to bind port, with vif_type=binding_failed noted in dhcp-agent logs? | 15:58 |
| CeeMac | enabled dubug, can't see anything more useful in there sadly :( | 15:59 |
| mnaser | cloudnull, DimGR, d34dh0r53, hughsaunders, b3rnard0, palendae, odyssey4me, serverascode, rromans, erikmwilson, mancdaz, _shaps_, BjoernT, claco, echiu, dstanek, jwagner, ayoung, prometheanfire, evrardjp, arbrandes, scarlisle, luckyinva, ntt, javeriak, spotz, vdo, jmccrory, alextricity25, jasondotstar, admin0, michaelgugino, ametts, bgmccollum, darrenc, JRobinson__, colinmcnamara, thorst, adreznec, eil397, | 16:00 |
| mnaser | qwang,nishpatwa_, cathrichardson, drifterza, hwoarang, cshen, ullbeking, mnaser, nicolasbock, jrosser, cjloader, antonym, dcdamien, jamesdenton | 16:00 |
| mnaser | meeting time! | 16:00 |
| mnaser | #startmeeting openstack_ansible_meeting | 16:00 |
| openstack | Meeting started Tue Jan 29 16:00:29 2019 UTC and is due to finish in 60 minutes. The chair is mnaser. Information about MeetBot at http://wiki.debian.org/MeetBot. | 16:00 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 16:00 |
| *** openstack changes topic to " (Meeting topic: openstack_ansible_meeting)" | 16:00 | |
| openstack | The meeting name has been set to 'openstack_ansible_meeting' | 16:00 |
| mnaser | #topic rollcall | 16:00 |
| mnaser | o/ | 16:00 |
| *** openstack changes topic to "rollcall (Meeting topic: openstack_ansible_meeting)" | 16:00 | |
| hwoarang | o/ | 16:00 |
| evrardjp | o/ | 16:00 |
| prometheanfire | o/ | 16:01 |
| mnaser | (sorry for the past 2, i meant to send an email asking if someone could run it, only 2 weeks off :) | 16:01 |
| mnaser | not much attendance | 16:02 |
| mnaser | #topic last week highlights | 16:02 |
| *** openstack changes topic to "last week highlights (Meeting topic: openstack_ansible_meeting)" | 16:02 | |
| mnaser | section seems empty, is anyone around to share anything in specific? | 16:02 |
| evrardjp | not really | 16:02 |
| evrardjp | maybe jrosser or odyssey4me | 16:03 |
| prometheanfire | sure :D | 16:03 |
| odyssey4me | apologies - I'm stuck in another meeting | 16:03 |
| mnaser | i see some gentoo patches finally ;) | 16:03 |
| prometheanfire | https://review.openstack.org/#/q/topic:add-gentoo-support+status:open gentoo stuff is working, though we need systemd-241 to finally be released | 16:03 |
| evrardjp | don't say it's a highlight of last week: p | 16:03 |
| prometheanfire | of course it is, for me :P | 16:03 |
| evrardjp | :) | 16:04 |
| mnaser | hehe, gentoo is an interesting deployment target | 16:04 |
| evrardjp | prometheanfire: if this isn;t a done deal yet, should we speak about that during open discussion? | 16:04 |
| prometheanfire | once the dib change merges and dib release is made (I don't think os-infra builds from master) and then the gentoo image is rebiult osa-tests should pass | 16:04 |
| evrardjp | mnaser: so is tumbleweed ? :D | 16:04 |
| hwoarang | gentoo is always a highlight | 16:04 |
| prometheanfire | evrardjp: sure | 16:04 |
| jrosser | we need to get the tempest and nova stuff unstuck, but there seems to be progress on that today | 16:04 |
| mnaser | evrardjp: you sold out! :P | 16:05 |
| evrardjp | mnaser: :) | 16:05 |
| mnaser | i was hoping to try and dive through, i know centos-7 is been bad | 16:05 |
| mnaser | and as part of that was just trying to rip out the container bits | 16:05 |
| evrardjp | I thought of comparing how much it would take me to build an arch linux OSA thing. Probably faster than doing it on gentoo :p but I will stop the flamebait there | 16:05 |
| mnaser | but anyways, our triage list has grown big but i feel like everyone gets bored and disappears in triage :) | 16:06 |
| evrardjp | that's kinda true | 16:06 |
| mnaser | so i'm proposing a short open discussion portion where we can talk about this stuff now, then we can do bug triage with whoever survives hah | 16:06 |
| evrardjp | it's sad | 16:06 |
| evrardjp | yeah that sounds fair | 16:06 |
| evrardjp | what about organising a bug killing day? | 16:06 |
| mnaser | evrardjp: sounds like a good idea, i'll try to gather up and see what everyone's availabilties seem like over the ML | 16:07 |
| evrardjp | I haven't done one in the last cycles, but I used to run one. | 16:07 |
| evrardjp | thanks | 16:07 |
| mnaser | jrosser: i see you had policy-in-code stuff in open discusison, was that from last week or meant for todays? | 16:07 |
| jrosser | it was for last week but we were time out | 16:07 |
| *** jpward1981 has joined #openstack-ansible | 16:07 | |
| mnaser | i assume it's removing all the hard coded stuff we ship in our roles | 16:08 |
| jrosser | just really for someone who knows the deal there to update on what still needs to be done | 16:08 |
| chandankumar | odyssey4me: mnaser https://review.openstack.org/#/c/633732/ needs merges unblocks centos gates | 16:08 |
| mnaser | there is a list of all projects that have moved to policy in code | 16:08 |
| evrardjp | oh yeah I have another topic for open discussion: releasing. Bumping is now automatic, and I have a few patches in to have automatic versioning with setuptools, which should be good enough to not change code anymore. Releases would still require manual intervention to say what/when to tag, until releases CLI is working for us at 100% (stein and above) | 16:09 |
| chandankumar | jrosser: it worked | 16:09 |
| chandankumar | jrosser: we are good to go now :-) | 16:09 |
| chandankumar | thanks to jrosser and slaweq for the gates fixes :-) | 16:09 |
| odyssey4me | yeah, I was thinking that perhaps we need to organise a hack day around each milestone and get agreement from our employers to do it | 16:10 |
| jrosser | mnaser: there seem to have been a few bugs crop up which felt related to policy stuff | 16:10 |
| mnaser | odyssey4me: just sent an email to the ML about that, so that'd be cool :) | 16:10 |
| odyssey4me | it's been quite tough to get focused attention, and loads of bugs are just sitting there with no attention | 16:10 |
| mnaser | ++ | 16:10 |
| mnaser | chandankumar: good work on catching that, thank you. | 16:11 |
| chandankumar | mnaser: it's a team work, we jrosser odyssey4me and slaweq did it :-) | 16:11 |
| mnaser | evrardjp: i like that, simplifying our life is always a good thing. we're all quite busy | 16:11 |
| guilhermesp | now with more time to take a look at the osa bugs we found during some deployments, this week Im focusing on a bunch of PR to submit. One of the focus is related to upgrade jobs https://review.openstack.org/#/c/627782/ | 16:13 |
| evrardjp | should that series of patches merge, stein will be able to be released fully automatically. The patches can still be backported for simpler releasing in older branches, but not 100% perfect solution there. | 16:13 |
| guilhermesp | I'm going to take a look at the failures but me and mnaser agreed that the workspace fix is still not complete https://review.openstack.org/#/c/633549/ | 16:13 |
| mnaser | right, upgrades has been rough for us, and i'm pretty sure there's a bug with the way we deploy rabbitmq too where a cluster failure results in the cluster not routing anything anymore unless you delete all queues | 16:14 |
| mnaser | i've seen this repeatedly over multiple rocky envs, so there's still some clean up and work to do | 16:14 |
| odyssey4me | ouch | 16:14 |
| mnaser | deleting a vhost isnt enough, you have to delete every single queue, and it just magically starts working again | 16:14 |
| mnaser | it's affected us and a few customers. i'm confident it's a confirmed issue by now as it's always been fixed this way. i haven't had time to dig deeper, but yeah. | 16:14 |
| mnaser | anyhow, subjects so far: releases, upgrades and hackday. | 16:15 |
| mnaser | releases => we will try to use the new tooling that evrardjp worked on and then *IF* someone has time, we could backport i guess | 16:15 |
| mnaser | upgrades => guilhermesp is working on it and will continue to iterate, we're so so so close because it's failing in tempest after a full upgrade, so that's great news overall | 16:16 |
| mnaser | hackday => i sent an email to ML, so if you can respond to it, that'd be awesome :) | 16:16 |
| odyssey4me | yeah, let's see how it goes with stein - then work it back if it all goes well | 16:16 |
| odyssey4me | for upgrades, I'm happy to help - although I need to focus back on figuring out the final bits for the python builds | 16:17 |
| chandankumar | mnaser: on centos Jobs, we find errors in neutron logs, is there any plan to get rid of that | 16:17 |
| chandankumar | in the morning, jrosser and I were discussing about that | 16:17 |
| mnaser | odyssey4me: i think your time is well invested in the python build to wrap it up, in the meantime i'll work with guilhermesp to get upgrades done, it should be minor things afaik | 16:17 |
| mnaser | chandankumar: do you mind explaining more about that? | 16:17 |
| chandankumar | mnaser: grabbing the logs | 16:18 |
| chandankumar | mnaser: http://logs.openstack.org/32/633732/2/check/openstack-ansible-functional-centos-7/a8cb2f1/logs/openstack/openstack1/neutron/neutron-dhcp-agent.log.txt.gz#_2019-01-29_15_09_31_526 | 16:20 |
| mnaser | chandankumar: thats probably because the service goes up before we setup the mq's | 16:20 |
| odyssey4me | yeah, it'd be nice to sort that out | 16:21 |
| odyssey4me | it should be a straightforward fix - just re-ordering some tasks | 16:21 |
| chandankumar | mnaser: http://logs.openstack.org/32/633732/2/check/openstack-ansible-functional-centos-7/a8cb2f1/logs/openstack/openstack1/neutron/neutron-l3-agent.log.txt.gz#_2019-01-29_15_09_30_544 | 16:21 |
| chandankumar | we fixed libvirt import error issues | 16:21 |
| chandankumar | mnaser: on tripleo side, we have a role named collect-logs to dump all errors in a single file | 16:22 |
| chandankumar | mnaser: I will check with wes tomorrow how we can use it here | 16:22 |
| mnaser | oh that's super awesome. yes, let's share tooling. chandankumar | 16:23 |
| mnaser | i have a subject -- evrardjp brought this up before but i think we should move to office hours instead of an actual meeting | 16:24 |
| chandankumar | mnaser: odyssey4me something like this http://logs.openstack.org/85/633185/8/check/tripleo-ci-centos-7-standalone/9c2e95c/logs/undercloud/var/log/extra/errors.txt.gz | 16:24 |
| mnaser | if you use a role to collect the logs, we can probably reuse it in the gate together | 16:24 |
| chandankumar | https://github.com/openstack/tripleo-quickstart-extras/tree/master/roles/collect-logs | 16:25 |
| chandankumar | there was a plan to move it to a seperate project but stalled due to other priorities | 16:25 |
| chandankumar | I will check with team tomorrow and let you know | 16:25 |
| mnaser | ok cool, it might be pretty beneficial in our gates too | 16:26 |
| mnaser | i mean like, in all of openstack | 16:26 |
| *** fdegir has quit IRC | 16:26 | |
| mnaser | so, thoughts about office hours instead of meetings? | 16:28 |
| jrosser | i would be concerned that the bug triage gets even more out of hand - how would we handle that? | 16:28 |
| *** jawad_axd has joined #openstack-ansible | 16:28 | |
| jrosser | imho it's quite a good way of socialising whats broken and how folks are using our stuff | 16:28 |
| odyssey4me | what's the difference between the two? | 16:29 |
| odyssey4me | (office hours vs meetings) | 16:29 |
| mnaser | jrosser: office hours is just a time where we try to all be available to discuss things (rather than async reaching each other), without a specific agenda, just a time where we're all there | 16:29 |
| chandankumar | office hours ~= meeting without predefined agenda | 16:30 |
| mnaser | the bug triage, i'm hoping that we can do some sort of bug smash thing every here and there. | 16:30 |
| mnaser | the difficult part is that it ends up being 1 or 2 people doing most of the triage | 16:30 |
| odyssey4me | well, we kinda have office hours daily during the crossover time between UK and US | 16:30 |
| ThiagoCMC | I finally have OSA/Rock up and running with Ceph! At least Glance and Cinder are working! Wheee! | 16:31 |
| openstackgerrit | Merged openstack/openstack-ansible stable/pike: Bump SHAs for stable/pike https://review.openstack.org/633348 | 16:31 |
| mnaser | ThiagoCMC: w00t | 16:31 |
| ThiagoCMC | Trying to boot a VM now | 16:31 |
| ThiagoCMC | I'm so happy! | 16:31 |
| ThiagoCMC | :-D | 16:31 |
| odyssey4me | I would rather try and do a bug triage/fix team rotation than let it slip to happening once every so often. | 16:31 |
| jrosser | are we struggling for people to attend the meeting due to $dayjob pressure? | 16:31 |
| * redrobot sneaks in through the back | 16:31 | |
| mnaser | jrosser: i'm not sure. i don't have much of an explanation. but i think it's largely a time constraint | 16:32 |
| mnaser | i think its late in EU timezone, and conflicts with a lot of other meeting timeslots | 16:32 |
| mnaser | i often see people mention they're inbetween meetings (and that's fine, i understand people need to get their jobs done), but yeah | 16:32 |
| odyssey4me | I unfortunately have two meetings at the same time today - this one and my internal team meeting. | 16:32 |
| mnaser | right, i'm all for keeping doing bug triage, but it ends up being a subset of folks that do it. we can either look into a rotation, or maybe we can come up with another time where we have more resources/people to help do it | 16:34 |
| *** chandankumar is now known as chkumar|out | 16:35 | |
| *** jawad_axd has quit IRC | 16:35 | |
| jrosser | perhaps we should look at some bugs? | 16:36 |
| mnaser | anyhow, we can defer this to next week and see how this weeks bug triage goes :) | 16:36 |
| mnaser | #topic bug triage | 16:36 |
| *** openstack changes topic to "bug triage (Meeting topic: openstack_ansible_meeting)" | 16:36 | |
| mnaser | #link https://bugs.launchpad.net/openstack-ansible/+bug/1813660 | 16:36 |
| openstack | Launchpad bug 1813660 in openstack-ansible "Upgrade from Pike to Queens skips setup-hosts when running neutron on bare metal" [Undecided,New] - Assigned to Bjoern Teipel (bjoern-teipel) | 16:36 |
| mnaser | looks like that's already assigned | 16:37 |
| *** tstrul has joined #openstack-ansible | 16:37 | |
| jrosser | there may even be a patch for that | 16:37 |
| mnaser | yeah, i'm trying to search under that name :p | 16:37 |
| mnaser | https://review.openstack.org/#/q/owner:%22Bjoern+Teipel+%253Cbjoern.teipel%2540rackspace.com%253E%22 i don't think so | 16:37 |
| guilhermesp | worth to ask updates for that guy? | 16:38 |
| *** tstrul has quit IRC | 16:38 | |
| prometheanfire | guilhermesp: he's a coworker, should I bug him about something specific? | 16:38 |
| prometheanfire | #1813660 ? | 16:38 |
| mnaser | yep | 16:39 |
| mnaser | i mean | 16:39 |
| mnaser | reported 19 hours ago | 16:39 |
| jrosser | odyssey4me: didnt you have a patch for this? | 16:39 |
| prometheanfire | ya, kinda recent | 16:39 |
| mnaser | ok so i think we can mark this down as confirmed medium | 16:40 |
| mnaser | and we'll have a patch soon :) | 16:40 |
| prometheanfire | ya, pinged him | 16:40 |
| odyssey4me | jrosser sort-f, I made it work better - then for master I fixed it properly | 16:41 |
| mnaser | oh, so fixed? | 16:41 |
| odyssey4me | hang a sec | 16:41 |
| odyssey4me | the issue there is pike->queens, right? | 16:41 |
| guilhermesp | yep odyssey4me | 16:41 |
| odyssey4me | ok, I think that bug is relating to the thing I fixed - yes, lemme provide a review | 16:42 |
| *** TxGirlGeek has joined #openstack-ansible | 16:42 | |
| odyssey4me | hmm: https://review.openstack.org/625898 | 16:42 |
| odyssey4me | that was rocky - there was a reason I didn't port that back to queens... but I can't remember what that reason is | 16:43 |
| odyssey4me | in master I did a bunch more: https://review.openstack.org/624773 | 16:44 |
| mnaser | so we can safely triage this and figure out fix later? :) | 16:44 |
| odyssey4me | yeah, it's valid and already set to medium | 16:45 |
| odyssey4me | I'll comment what's already in place for queens & master. Bjoern can then decide what to do about Pike. | 16:46 |
| *** sdake has quit IRC | 16:46 | |
| evrardjp | odyssey4me: for once you don't remember? :p | 16:46 |
| mnaser | #link https://bugs.launchpad.net/openstack-ansible/+bug/1813300 | 16:46 |
| openstack | Launchpad bug 1813300 in openstack-ansible "NFS mount point for Glance is created with wrong permissions" [Undecided,New] | 16:46 |
| evrardjp | that rings me a bell ... haven't we changed that already in the past? | 16:47 |
| evrardjp | but there is a patch included! | 16:47 |
| odyssey4me | Yeah - I feel that this one keeps coming up, and a new patch goes in, and then another one later... and so on. | 16:47 |
| chkumar|out | mnaser: if we have time I want to discuss about using https://trunk.rdoproject.org/centos7-master/delorean-deps.repo in OSa for installing dependencies not maintained around openstack ecosystem | 16:50 |
| chkumar|out | mnaser: I was checking the openstack-ansible-tests code on nodepool test file but no clue how to use it | 16:51 |
| chkumar|out | mnaser: http://codesearch.openstack.org/?q=delorean-deps.repo&i=nope&files=&repos= | 16:51 |
| chkumar|out | mnaser: it is used in POI and tripleo | 16:51 |
| chkumar|out | mnaser: can we use it here also? | 16:52 |
| odyssey4me | chkumar|out I think we already do? | 16:52 |
| chkumar|out | odyssey4me: we only use delorean.repo only | 16:52 |
| odyssey4me | oh, I see | 16:53 |
| odyssey4me | would this repo be used in production at all? | 16:53 |
| chkumar|out | odyssey4me: https://github.com/openstack/openstack-ansible-tests/blob/401fc3d5cdef09f99470f20256c2ecd7e36925fa/common-tasks/test-set-nodepool-vars.yml#L49 | 16:53 |
| chkumar|out | odyssey4me: in downstream, we import packages from same | 16:53 |
| mnaser | confirmed/medium for the nfs bug, i asked Juri if it's possible to work with them to get them to push it to gerrit | 16:54 |
| chkumar|out | odyssey4me: it is maintained here https://github.com/redhat-openstack/rdoinfo/blob/master/deps.yml | 16:54 |
| mnaser | chkumar|out: i'd be in favour, using delorean deps was very helpful and made our gate usually quite stable in poi times (it also helped crossgate with rdo) | 16:55 |
| chkumar|out | mnaser: I need some pointers and I can make the changes in openstack-ansible-tests | 16:55 |
| mnaser | chkumar|out: we can discuss post meeting if you're not "out" :) | 16:55 |
| *** hamzy has quit IRC | 16:56 | |
| chkumar|out | mnaser: may be tomorrow, I can ping you in evening from my time zone | 16:56 |
| mnaser | chkumar|out: great! | 16:56 |
| mnaser | we're running close to time, maybe we can get one more triage in | 16:57 |
| mnaser | #link https://bugs.launchpad.net/openstack-ansible/+bug/1813187 | 16:57 |
| openstack | Launchpad bug 1813187 in openstack-ansible "CentOS tempest test_server_basic_ops failure" [Undecided,New] | 16:57 |
| mnaser | oh, that was resolved by the patch listed above | 16:57 |
| mnaser | done | 16:58 |
| mnaser | #link https://bugs.launchpad.net/openstack-ansible/+bug/1813149 | 16:58 |
| openstack | Launchpad bug 1813149 in openstack-ansible "Missing git respo: https://github.com/ceph/ansible-ceph-defaults" [Undecided,New] | 16:58 |
| prometheanfire | cjloader: ^? | 16:59 |
| odyssey4me | ja, that's all fixed | 16:59 |
| mnaser | did we release since | 16:59 |
| mnaser | looks like 16.0.24 is the tag the user used | 16:59 |
| cjloader | yes was fixed | 16:59 |
| odyssey4me | ocata: https://review.openstack.org/632182 & pike: https://review.openstack.org/632142 | 17:00 |
| *** hamzy has joined #openstack-ansible | 17:00 | |
| mnaser | nice work cjloader | 17:00 |
| odyssey4me | no release based on that yet, I think evrardjp did the release requests earlier today | 17:00 |
| mnaser | cool, ill update the bug | 17:00 |
| mnaser | ok, we're over time, but it looks like we don't need any bug triage cause everything just works ;) haha. | 17:01 |
| *** pcaruana has quit IRC | 17:01 | |
| mnaser | thanks everyone, and please please take time to respond to the hackday ML post on openstack-discuss | 17:02 |
| mnaser | <3 | 17:02 |
| mnaser | #endmeeting | 17:02 |
| *** openstack changes topic to "Launchpad: https://launchpad.net/openstack-ansible || Weekly Meetings: https://wiki.openstack.org/wiki/Meetings/openstack-ansible || Review Dashboard: http://bit.ly/2xA1eZC" | 17:02 | |
| openstack | Meeting ended Tue Jan 29 17:02:07 2019 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 17:02 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/openstack_ansible_meeting/2019/openstack_ansible_meeting.2019-01-29-16.00.html | 17:02 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/openstack_ansible_meeting/2019/openstack_ansible_meeting.2019-01-29-16.00.txt | 17:02 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/openstack_ansible_meeting/2019/openstack_ansible_meeting.2019-01-29-16.00.log.html | 17:02 |
| cjloader | mnaser: https://review.openstack.org/#/c/632182/ | 17:02 |
| cjloader | https://review.openstack.org/#/c/632142/ | 17:03 |
| *** gyee has joined #openstack-ansible | 17:03 | |
| prometheanfire | question while people are here, nothing sets up the main nginx.conf, so nothing tells nginx to point to a sites-enabled type of directory | 17:04 |
| prometheanfire | is there a better way of handling this? https://review.openstack.org/#/c/633423/1/tasks/keystone_nginx.yml@62 | 17:04 |
| prometheanfire | https://review.openstack.org/#/c/633423/1/files/nginx.conf | 17:04 |
| mnaser | prometheanfire: i've been thinking that we should just rip out nginx from there | 17:05 |
| odyssey4me | mnaser keystone requires a web server for federation support | 17:05 |
| mnaser | odyssey4me: we fallback to apache2 when we do federation | 17:06 |
| odyssey4me | so we can't just do uwsgi | 17:06 |
| prometheanfire | mnaser: I'd be fine with that, it seems default though (it was installed in my gentoo testing) | 17:06 |
| mnaser | but for some reason we do both nginx *and* uwsgi for non-federated deployments | 17:06 |
| odyssey4me | yeah, we implemented nginx because the keystone team recommended we do so | 17:06 |
| odyssey4me | the plan was to switch the apache config for federation over to nginx too, whenever someone had the time to figure out how | 17:06 |
| mnaser | ah i see | 17:07 |
| *** macza has joined #openstack-ansible | 17:07 | |
| prometheanfire | gentoo stuuf doesn't support federation yet, I'd have to package some things I think | 17:07 |
| odyssey4me | however, it seems that RDO still does apache/mod_wsgi - and a lot of openstack docs still help people config that way, so our config is confusing to many... I find myself wondering whether we shouldn't just confirm to what everyone else does as a default | 17:08 |
| odyssey4me | (even if it is a bit crappy) | 17:08 |
| prometheanfire | meh (no strong opinion) | 17:10 |
| jrosser | this is needed to make the tempest vm ssh stuff even more robust https://review.openstack.org/#/c/633728/ | 17:11 |
| *** hamzy has quit IRC | 17:11 | |
| * prometheanfire is testing if the same problem exists with apache | 17:12 | |
| prometheanfire | jrosser: how does limiting it help? | 17:13 |
| jrosser | becasue we now assign IPs to bridges on the containers in that subnet | 17:13 |
| jrosser | and neutron needs to know to not try to use one of those for an instance IP | 17:14 |
| prometheanfire | ah, and could get a conflict | 17:14 |
| jrosser | if thats not clear enough perhaps we should reference the commit which consumes some of those IP in the commit msg? | 17:15 |
| chkumar|out | jrosser: regarding today's debugging we need to add some validation also | 17:16 |
| chkumar|out | jrosser: if something fails we should have some data to verify those stuffs | 17:16 |
| jrosser | what did you have in mind? | 17:17 |
| *** sdake has joined #openstack-ansible | 17:17 | |
| chkumar|out | jrosser: I will propose a patch tomorrow to test ping stiff from container | 17:17 |
| jrosser | i guess once the tempest resources are created you would expect to be able to ping the router | 17:17 |
| jrosser | even before the tests are run | 17:18 |
| *** sdake has quit IRC | 17:18 | |
| prometheanfire | jrosser: hardcoding the fix makes it more likely to fail in the future (though it's still useful) | 17:18 |
| jrosser | well it's not a fix | 17:18 |
| *** kopecmartin is now known as kopecmartin|off | 17:24 | |
| *** ThiagoCMC has quit IRC | 17:28 | |
| openstackgerrit | Merged openstack/openstack-ansible-os_tempest master: Add an ip address to eth12 in OSA test containers https://review.openstack.org/633732 | 17:31 |
| openstackgerrit | Chandan Kumar proposed openstack/openstack-ansible-os_tempest master: Update cirros from 3.5 to 3.6 https://review.openstack.org/633208 | 17:34 |
| openstackgerrit | Chandan Kumar proposed openstack/openstack-ansible-os_tempest master: Added dependencies of os_tempest role https://review.openstack.org/632726 | 17:34 |
| openstackgerrit | Chandan Kumar proposed openstack/openstack-ansible-os_tempest master: Always generate stackviz irrespective of tests pass or fail https://review.openstack.org/631967 | 17:34 |
| openstackgerrit | Chandan Kumar proposed openstack/openstack-ansible-os_tempest master: Use tempest_cloud_name in tempestconf https://review.openstack.org/631708 | 17:34 |
| openstackgerrit | Chandan Kumar proposed openstack/openstack-ansible-os_tempest master: Added tempest.conf for heat_plugin https://review.openstack.org/632021 | 17:35 |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible-os_tempest master: Add telemetry distro plugin install for aodh https://review.openstack.org/632125 | 17:36 |
| jrosser | odyssey4me: which way would you prefer this to go? do it in nova role test vars or in tempest role? | 17:39 |
| jrosser | https://review.openstack.org/#/c/633677/ | 17:39 |
| odyssey4me | jrosser tempest role, I think - then it's universally applied | 17:42 |
| odyssey4me | jrosser rather than override a default - fix the default | 17:42 |
| jrosser | ok, i'll fix that up | 17:43 |
| odyssey4me | great, thanks | 17:43 |
| *** sdake has joined #openstack-ansible | 17:44 | |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible-os_tempest master: Disable nova_lxd tempest plugin https://review.openstack.org/633711 | 17:57 |
| openstackgerrit | Merged openstack/openstack-ansible-tests master: Set a defined IP address range for tempest test public addresses https://review.openstack.org/633728 | 17:58 |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible-os_tempest master: Disable nova-lxd tempest plugin https://review.openstack.org/633711 | 17:58 |
| *** sdake has quit IRC | 18:10 | |
| *** cmart has quit IRC | 18:19 | |
| *** electrofelix has quit IRC | 18:34 | |
| *** sdake has joined #openstack-ansible | 18:37 | |
| openstackgerrit | Jacob Wagner proposed openstack/openstack-ansible-ops master: Add ability to deploy Designate (DNSaaS) https://review.openstack.org/633801 | 18:50 |
| *** hamzy has joined #openstack-ansible | 18:52 | |
| *** hamzaachi has joined #openstack-ansible | 18:59 | |
| *** exbob has quit IRC | 19:01 | |
| *** cmart has joined #openstack-ansible | 19:08 | |
| *** strattao has quit IRC | 19:32 | |
| ioni | quick questions, i hope that's not a stupid one but i'm curious | 19:36 |
| ioni | what's in openstack-ansible-ops? | 19:36 |
| ioni | and MNAIO ? | 19:37 |
| *** ztr has quit IRC | 19:37 | |
| jamesdenton | MNAIO is a 'multi-node all-in-one' deploy. Basically, a set of scripts that deploys OSA in a set of VMs. infra, compute, storage, etc | 19:57 |
| *** ThiagoCMC has joined #openstack-ansible | 19:57 | |
| jamesdenton | openstack-ansible-ops is the kitchen sink repo | 19:58 |
| ThiagoCMC | Guys, I just finished a fresh OSA/Rocky deployment on top of Ubuntu 18.04 (all hosts deployed via MaaS) and almost everything is working! Except Heat. | 19:58 |
| ThiagoCMC | The command `openstack stack list` returns: ERROR: Internal Error | 19:58 |
| jamesdenton | sounds like everything is good then. lol | 19:58 |
| ThiagoCMC | With --debug: http://paste.openstack.org/show/744195/ | 19:59 |
| ioni | jamesdenton, thanks | 19:59 |
| ThiagoCMC | Maybe it is broken due to this: https://github.com/openstack/openstack-ansible-os_heat/commit/785fcfd33d29ddfee54f09cd6bf126990d64e4dd ? | 19:59 |
| jamesdenton | ThiagoCMC you may want to check the heat api logs to see the traceback | 20:00 |
| ThiagoCMC | ok | 20:00 |
| jamesdenton | does openrc point to the internal or public uri for heat? | 20:01 |
| *** cmart has quit IRC | 20:01 | |
| ThiagoCMC | "OS_AUTH_URL=http://172.29.239.250:5000/v3" internal (br-mgmt | 20:02 |
| *** cmart has joined #openstack-ansible | 20:02 | |
| ThiagoCMC | Sorry, you said "for heat"? It's all "internalURL" | 20:03 |
| jamesdenton | k | 20:05 |
| ThiagoCMC | No errors on heat-api logs, only this: http://paste.openstack.org/show/744197/ - everytime that I run `openstack stack list` | 20:07 |
| ThiagoCMC | Here is ful `openstack stack list --debug` output: http://paste.openstack.org/show/744198/ | 20:10 |
| jamesdenton | hmm, i'm sure. I'm about to head out, but may be worth filing a bug and hopefully someone can look at that if you don't figure it out beforehand | 20:12 |
| jamesdenton | *not sure, rather | 20:12 |
| jamesdenton | the only thing i see is its hitting http url and complaining of an sslerror, so that needs to be looked at | 20:13 |
| ThiagoCMC | I see, thanks anyway! | 20:14 |
| ThiagoCMC | I'll chat with admin0 to see if he can help =) | 20:14 |
| *** fdegir_ has joined #openstack-ansible | 20:14 | |
| jamesdenton | cool. see ya | 20:14 |
| ThiagoCMC | See U! | 20:15 |
| *** cmart has quit IRC | 20:31 | |
| *** strattao has joined #openstack-ansible | 20:40 | |
| *** strattao has quit IRC | 20:44 | |
| *** SimAloo has quit IRC | 20:45 | |
| *** sdake has quit IRC | 20:45 | |
| *** cmart has joined #openstack-ansible | 20:48 | |
| *** fdegir_ is now known as fdegir | 20:55 | |
| *** SimAloo has joined #openstack-ansible | 20:58 | |
| openstackgerrit | Merged openstack/openstack-ansible-os_tempest master: Adds tempest run command with --test-list option https://review.openstack.org/631351 | 20:59 |
| *** DanyC has quit IRC | 21:04 | |
| *** DanyC has joined #openstack-ansible | 21:04 | |
| *** DanyC has quit IRC | 21:08 | |
| ThiagoCMC | Guys, I'm seeing the following haproxy.log entry: "keystone_service-front-1/1: SSL handshake failure", any idea to where start looking? | 21:31 |
| ThiagoCMC | It's on my third controller (the one with the VIPs) | 21:31 |
| *** ansmith has quit IRC | 21:38 | |
| *** hamzy has quit IRC | 21:42 | |
| jrosser | ThiagoCMC: if it were me i'd start with very simple tools, like wget, from a host that is nothing to do with your openstack deploy but has connectivity to the external vip | 21:48 |
| jrosser | try wget https://<external endpoint>:5000 | 21:49 |
| ThiagoCMC | Trying it now... | 21:52 |
| ThiagoCMC | The `wget --no-check-certificate https://172.29.235.250:5000/` just worked... | 22:05 |
| ThiagoCMC | That's my br-public subnet IP | 22:05 |
| ThiagoCMC | The SSL handshake problem (haproxy.log line message) is always close/before to the heat_api thing. | 22:07 |
| ThiagoCMC | And my `openstack stack list` is returning Error 500 | 22:07 |
| ThiagoCMC | I believe that there is a bug on stable OSA/Rocky branch. Where heat_api is trying to talk clear text against a https endpoint. | 22:10 |
| ThiagoCMC | This SSL handshake problem and Heat Error 500 might be related, because of this: http://paste.openstack.org/show/744211/ | 22:11 |
| ThiagoCMC | Always the two lines together. | 22:11 |
| *** sdake has joined #openstack-ansible | 22:15 | |
| jrosser | ThiagoCMC: Jan 29 22:08:18 localhost haproxy[13236]: 10.0.3.41:54086...... | 22:25 |
| jrosser | i don't like that, it suggests that eth0 on a container has been used to contact the external keystone endpoint, which just feels wrong | 22:26 |
| jrosser | mnaser: are you sure about this patch? https://github.com/openstack/openstack-ansible-os_heat/commit/785fcfd33d29ddfee54f09cd6bf126990d64e4dd | 22:27 |
| mnaser | yep, why's that jrosser ? | 22:28 |
| jrosser | i'm concerned that there is an assumption that the mgmt network can talk to the external endpoint, which isn't necessarily the case | 22:28 |
| mnaser | so all those urls i actually provided are ones which are presented to the user | 22:28 |
| mnaser | but not used for auth | 22:28 |
| mnaser | for example, www_authenticate_uri is just exposed in the headers etc | 22:28 |
| *** nurdie has quit IRC | 22:28 | |
| jrosser | what about this http://paste.openstack.org/show/744211/ | 22:28 |
| jrosser | where the external vip is hit from a 10.x address which looks like a container eth0 | 22:29 |
| jrosser | i.e mgmt net traffic to external endpoint must take the default rout | 22:29 |
| jrosser | e | 22:29 |
| openstackgerrit | Merged openstack/openstack-ansible-os_tempest master: Enable port security https://review.openstack.org/617719 | 22:29 |
| mnaser | i think this introduces another issue | 22:29 |
| mnaser | heat-agent actually uses that url to give to vms | 22:30 |
| jrosser | somehow there the wires look crossed between internal mgmt net and external | 22:30 |
| mnaser | os-cloud-config or whatever | 22:30 |
| mnaser | no, not os-cloud-config, grr | 22:30 |
| jrosser | late here, i need to stop, but i think theres a few folks having trouble with heat | 22:31 |
| jrosser | and it just all has a bit of a smell of internal/external networks getting mixed up | 22:32 |
| *** TxGirlGeek has quit IRC | 22:32 | |
| *** gyee has quit IRC | 22:35 | |
| ThiagoCMC | AHA! | 22:36 |
| *** slaweq has quit IRC | 22:36 | |
| ThiagoCMC | jrosser, mnaser I just reverted https://github.com/openstack/openstack-ansible-os_heat/commit/785fcfd33d29ddfee54f09cd6bf126990d64e4dd and executed os-heat-install.yml, no more Erro 500! `openstack stack list` is finally working! | 22:38 |
| mnaser | yikes, that breaks magnum though | 22:38 |
| mnaser | ThiagoCMC: can you try reverting the values one by one and seeing which one breaks? | 22:38 |
| ThiagoCMC | yes | 22:39 |
| ThiagoCMC | It will take a few minutes to try again, waiting for ansible to finish... | 22:43 |
| ThiagoCMC | Also, no more SSL handshake error message! ;-) | 22:43 |
| ThiagoCMC | mnaser, I reverted only line 46, under [clients_keystone], auth_uri. | 22:58 |
| ThiagoCMC | The others still points to the public one. | 22:58 |
| ThiagoCMC | I think that the bug is with clients_keystone that tries to talk in clear text over a https connection. | 22:59 |
| *** SimAloo has quit IRC | 23:03 | |
| -openstackstatus- NOTICE: http://zuul.openstack.org is not working. https://zuul.openstack.org does work. Please use that while we investigate. | 23:12 | |
| *** radeks_ has quit IRC | 23:15 | |
| *** TxGirlGeek has joined #openstack-ansible | 23:28 | |
| *** TxGirlGeek has quit IRC | 23:29 | |
| *** TxGirlGe_ has joined #openstack-ansible | 23:29 | |
| *** cmart has quit IRC | 23:34 | |
| *** sdake has quit IRC | 23:35 | |
| *** sdake has joined #openstack-ansible | 23:37 | |
| *** cmart has joined #openstack-ansible | 23:43 | |
| *** errr_ has joined #openstack-ansible | 23:53 | |
| *** sdake has quit IRC | 23:55 | |
| *** sdake has joined #openstack-ansible | 23:55 | |
| *** errr has quit IRC | 23:56 | |
| *** hamzaachi_ has joined #openstack-ansible | 23:58 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!