17:04:28 <evrardjp> #startmeeting openstack_ansible_meeting 17:04:29 <openstack> Meeting started Tue Mar 6 17:04:28 2018 UTC and is due to finish in 60 minutes. The chair is evrardjp. Information about MeetBot at http://wiki.debian.org/MeetBot. 17:04:30 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 17:04:32 <openstack> The meeting name has been set to 'openstack_ansible_meeting' 17:05:05 <spotz> o/ 17:05:14 <evrardjp> #topic focus of the week 17:05:41 <evrardjp> This week's focus will be on releasing queens first 17:06:06 <evrardjp> then releasing all the other branches, as it should have been done during end of last month, which was ptg 17:06:35 <evrardjp> #topic previous week's conversations 17:06:50 <andymccr> o/ 17:06:55 <andymccr> sorry im late 17:06:58 <hwoarang> o/ 17:07:23 <evrardjp> we had many conversations at the PTG, and it's too long to write them down here, so I'll try to write up a blog or ML post. Anyone willing to help is welcome 17:07:25 <spotz> andymccr: You're back!!!! 17:07:28 <evrardjp> welcomed* 17:07:40 <andymccr> spotz: its like i never left :D 17:07:49 <evrardjp> ok next topic 17:07:50 <spotz> :) 17:07:54 <odyssey4me> o/ 17:07:56 <evrardjp> #topic bug triage 17:07:57 <andymccr> evrardjp: if you need help on the blog let me know - happy to read over it or w/e 17:08:03 <evrardjp> #link https://bugs.launchpad.net/openstack-ansible/+bug/1753543 17:08:04 <openstack> Launchpad bug 1753543 in openstack-ansible "LXC interface template does not set MTU for provider interface" [Undecided,New] 17:08:26 <odyssey4me> I think cloudnull just pushed up a slew of patches for that 17:08:34 <evrardjp> looks like it 17:09:03 <andymccr> yeah 17:09:08 <andymccr> i guess we can move to next 17:09:25 <andymccr> and review the patches as usual 17:09:52 <evrardjp> confirmed at least :) 17:10:15 <evrardjp> ok next then 17:10:17 <evrardjp> #link https://bugs.launchpad.net/openstack-ansible/+bug/1753125 17:10:18 <openstack> Launchpad bug 1753125 in openstack-ansible "os_cinder : Ensure cinder api is available timeout" [Undecided,New] 17:10:56 <odyssey4me> need more info 17:11:10 <evrardjp> Incomplete 17:11:12 <evrardjp> indeed 17:11:40 <evrardjp> next 17:11:51 <evrardjp> #link https://bugs.launchpad.net/openstack-ansible/+bug/1752932 17:11:52 <openstack> Launchpad bug 1752932 in openstack-ansible "hostvars breaks on Queens" [Undecided,New] 17:12:05 <evrardjp> we've seen this at the PTG. 17:12:30 <evrardjp> I'd like to debug this in more details. It sounds serious -- high or critical 17:12:44 <odyssey4me> yeah, nice that we have a full config to work with :) 17:12:49 <evrardjp> but the conditions are not clear to me -- python/jinja/ansible version dependant 17:12:57 <andymccr> yeah that seems pretty critical 17:13:10 <evrardjp> we weirdly don't see that in gates. 17:13:13 <andymccr> yeah 17:13:42 <evrardjp> let me leave it as unconfirmed now 17:13:54 <odyssey4me> might have to use vagrant, or the mnaio to reproduce 17:13:56 <evrardjp> if next week I haven't reported 17:14:03 <evrardjp> then we can continue 17:14:08 <evrardjp> I'll assign that to myself 17:14:21 <evrardjp> ok for everyone? 17:14:30 <hwoarang> οκ 17:14:34 <hwoarang> ok 17:14:37 <evrardjp> haha 17:14:43 <evrardjp> next 17:14:45 <evrardjp> #link https://bugs.launchpad.net/openstack-ansible/+bug/1752898 17:14:45 <openstack> Launchpad bug 1752898 in openstack-ansible "gateway option does not work" [Undecided,New] 17:16:16 <evrardjp> I see this listed in all branches up to queens, probably master too. 17:16:29 <evrardjp> so let me check if that's a docs bug or a code bug :) 17:17:20 <andymccr> hmm 17:17:31 <evrardjp> https://github.com/openstack/openstack-ansible-lxc_container_create/blob/aee117fc09981930ac1500e6375905cea2bfaab9/templates/container_network.network.j2#L31 17:17:35 <evrardjp> looks okay to me 17:18:09 <spotz> I blame mhayden.... 17:18:15 <odyssey4me> so it should work, but apparently doesn't 17:18:23 <evrardjp> but our test coverage doesn't test it 17:18:25 <evrardjp> https://github.com/openstack/openstack-ansible-lxc_container_create/blob/dc034ff5c32e373efeab127128327e361430b2c0/tests/group_vars/all_containers.yml#L27 17:18:35 <odyssey4me> it might be because the containers use dhcp on eth0, which gives it a route 17:19:04 <odyssey4me> so it's plausible that the option there was something that worked when we hard coded everything - newton onwards we rely more on auto config 17:19:16 <odyssey4me> so perhaps we should just remove those docs and suggest static routes 17:19:25 <odyssey4me> it would seem a better approach to me anyway 17:20:41 <odyssey4me> actually, this is for a secondary interface 17:20:44 <odyssey4me> ... 17:21:55 * odyssey4me needs to think about this more 17:23:35 <evrardjp> docs issue 17:23:57 <odyssey4me> adding that extra gateway is only useful if you have disabled lxcbr0, otherwise eth0 has a default gw and that will be there first 17:23:59 <odyssey4me> you can't have two 17:24:25 <evrardjp> I think the code is fine indeed, on all branches. 17:25:30 <evrardjp> Marked it invalid 17:25:47 <evrardjp> Tahvok: we might need your review on the docs, because you submitted this bug 17:27:08 <evrardjp> ok next 17:27:10 <evrardjp> #link https://bugs.launchpad.net/openstack-ansible/+bug/1752848 17:27:11 <openstack> Launchpad bug 1752848 in openstack-ansible "Cinder backup on ceph when ceph backend not on all cinder-volume" [Undecided,New] 17:30:28 <evrardjp> Definitely need more eyes 17:30:40 <hwoarang> i can't parse it right now 17:30:45 * hwoarang doesn't understand ceph that much yet 17:30:47 <evrardjp> haha 17:31:08 <evrardjp> I think what we can summarize from that is that we are checking what cinder backend is used for cinder_backend_rbd_inuse 17:31:24 <evrardjp> but we should probably have cinder_service_backup_driver 17:31:38 <evrardjp> I mean cinder_service_backup_rbd_inuse 17:32:01 <evrardjp> and rbd_in_use if either is true 17:33:27 <odyssey4me> currently we only automagically set the right vars when there is swift in the env.. we haven't done the same for ceph yet, you have to set them yourself... and if you have a mixed env, then using global overrides in user_*.yml will break things 17:34:08 <odyssey4me> so we should perhaps look at either simplifying the implementation and doing some clever things in the role, like we did for nova for the different settings used per hypervisor 17:34:22 <odyssey4me> or we should wire ceph up in the group vars like we've done with swift 17:35:48 <pabelanger> afternoon, any change I can get a few eyes on https://review.openstack.org/549887/ Remove fedora-26 nodes from testing (migrates to fedora-27) 17:36:01 <evrardjp> I asked the overrides to make sure we can triage this appropriately 17:36:22 <evrardjp> but I expect this to be indeed a lack of user friendliness on our side -- poor variable wiring 17:36:28 <odyssey4me> pabelanger if mhayden says it's fine, then it's fine to me - he likes breaking things :) 17:36:40 <evrardjp> pabelanger: I will have a look after the bug triage :) 17:37:06 <evrardjp> ok next bug 17:37:09 <pabelanger> odyssey4me: evrardjp: thanks 17:37:39 <spotz> odyssey4me: hhehe 17:37:49 <evrardjp> ok next 17:37:50 <evrardjp> #link https://bugs.launchpad.net/openstack-ansible/+bug/1752073 17:37:51 <openstack> Launchpad bug 1752073 in openstack-ansible "Neutron firewall_v2 driver missing" [Undecided,New] 17:38:35 <openstackgerrit> Merged openstack/openstack-ansible-lxc_container_create master: templates: networkd: Drop Link=ARP from networkd configuration https://review.openstack.org/549878 17:38:44 <evrardjp> that's pike 17:39:02 <hwoarang> interesting 17:39:27 <evrardjp> our docs seem wrong if it doesn't work: https://docs.openstack.org/openstack-ansible-os_neutron/pike/configure-network-services.html#deploying-fwaas-v2 17:39:32 <evrardjp> do we have a scenario for that? 17:39:36 <evrardjp> not sure 17:39:54 <odyssey4me> nope 17:40:00 <evrardjp> we should probably add a scenario to validate that 17:40:06 <evrardjp> and update the docs 17:40:12 <evrardjp> who has cycle to work on that? 17:40:18 <odyssey4me> yeah, hopefully it's not broken most of the time due to upstream bitrot 17:40:29 <evrardjp> firewalling, that must be interesting for some ppl I guess?? 17:41:26 <odyssey4me> this is accurate according to upstream docs: https://docs.openstack.org/neutron/queens/admin/fwaas-v2-scenario.html 17:41:37 <odyssey4me> but maybe we're missing something 17:41:41 <hwoarang> i think that we prob forget to pull in some package or something 17:42:25 <evrardjp> I think we should probably exercice it before confirming/infirming it. 17:42:58 <evrardjp> deny* 17:43:05 <odyssey4me> yeah, a new scenario would be good 17:43:28 <evrardjp> leaving it as is 17:43:30 <hwoarang> i can look into that maybe i will ask mbuil or someone else from opnfv 17:43:32 <odyssey4me> I'm afraid my networking knowledge is too sketchy. 17:43:39 <odyssey4me> ah yes, goo dplan 17:43:39 <evrardjp> hwoarang: oh that would be great 17:43:42 <hwoarang> i will assign it to myself 17:43:47 <evrardjp> great 17:43:47 <hwoarang> and i will delegate :D 17:43:54 <evrardjp> haha 17:44:32 <evrardjp> next 17:44:34 <evrardjp> #link https://bugs.launchpad.net/openstack-ansible/+bug/1751722 17:44:35 <openstack> Launchpad bug 1751722 in openstack-ansible "ceph_mon check fails" [Undecided,New] 17:45:47 <evrardjp> andymccr: :D 17:46:36 <evrardjp> I think we need to know more 17:46:41 <evrardjp> about the environment 17:46:59 <evrardjp> is the ceph mon not authorized with current deploy keys? 17:47:18 <evrardjp> I think it would be worth knowing what were the expectations and what's the failyre 17:47:23 <evrardjp> so marking it as incomplete 17:48:10 <logan-> ssh: connect to host 192.168.10.30 port 22: Connection timed out 17:48:16 <logan-> doesn't look like a key issue 17:49:05 <odyssey4me> well, jrosser and boxrick discovered that there's a port check, then it fails back to an ssh connection 17:49:44 <odyssey4me> I'm sorry - it's port check, then a "ping" check using the ansible module 17:49:56 <evrardjp> omg 17:50:20 <logan-> yeah it just needs to turn into a wait_for_connection and get rid of the whole rest of the stuff 17:50:30 <odyssey4me> this is fixed in ansible 2.4 I think, but a problem in 2.3 - and they work through a bastion so the initial check always fails 17:50:30 <logan-> :/ 17:51:11 <odyssey4me> logan- I think that bit is in wait_for_connection too for 2.3, but in 2.4 it's done right... but yeah, perhaps some of the ceph stuff is still using older methods 17:51:54 <odyssey4me> in their case they are maintaining a 2.3 fork with patches from newer ansible, which is less than ideal, but aapparently 2.3 is near EOL 17:52:22 <boxrick> We ended up backporting the fix to a fork of 2.3x we point at for now 17:52:33 <boxrick> Doh connection fall 17:52:37 <boxrick> Fail 17:53:42 <jrosser> thats all a bit confusing, if copying the priv key around allegedly fixes it, suggests its not really a connection issue? 17:53:55 <evrardjp> Yeah exactly. 17:54:03 <evrardjp> I vote for incomplete 17:54:22 <evrardjp> Did you see my comment I just posted? 17:54:26 <odyssey4me> https://github.com/openstack/openstack-ansible-ceph_client/blob/159d8164f6080db68d641f0ff5a54b0fd92449d8/tasks/ceph_get_mon_host.yml#L16-L28 17:54:35 <odyssey4me> it's actually doing ssh 17:54:43 <odyssey4me> that;d be why 17:55:27 <evrardjp> yeah. 17:55:54 <odyssey4me> wow, it used to be even more interesting: https://github.com/openstack/openstack-ansible-ceph_client/commit/f30ee47ed0e423a0555888b45b14999e7da517bd 17:56:08 <openstackgerrit> Kevin Carter (cloudnull) proposed openstack/openstack-ansible-lxc_container_create stable/newton: Set the MTU regardless if an address is present https://review.openstack.org/550147 17:56:09 <odyssey4me> ah, and logan- is to blame :p https://github.com/openstack/openstack-ansible-ceph_client/commit/a039741521ae90b5003c21284c93dc887baa76df 17:56:19 <openstackgerrit> Kevin Carter (cloudnull) proposed openstack/openstack-ansible-lxc_container_create stable/queens: Set the MTU regardless if an address is present https://review.openstack.org/550145 17:56:21 <openstackgerrit> Merged openstack/ansible-hardening master: Update to fedora-27 for testing https://review.openstack.org/549887 17:56:29 <openstackgerrit> Kevin Carter (cloudnull) proposed openstack/openstack-ansible-lxc_container_create stable/pike: Set the MTU regardless if an address is present https://review.openstack.org/550148 17:56:40 <openstackgerrit> Kevin Carter (cloudnull) proposed openstack/openstack-ansible-lxc_container_create stable/ocata: Set the MTU regardless if an address is present https://review.openstack.org/550146 17:56:40 <odyssey4me> it would seem that we have many more tools in the box now to do this better - perhaps logan- could pick up doing this better? 17:56:53 <logan-> yes please assign it to me 17:56:59 <evrardjp> that would be nice -- but I still think we should be aware of the expectations first. 17:57:01 <evrardjp> logan-: done! 17:57:24 <evrardjp> thanks logan- ! 17:58:02 <evrardjp> ok last one 17:58:04 <evrardjp> #link https://bugs.launchpad.net/openstack-ansible/+bug/1750841 17:58:05 <openstack> Launchpad bug 1750841 in openstack-ansible "playbook fails if /opt is missing" [Undecided,New] 17:59:35 <odyssey4me> possible due to https://github.com/openstack/openstack-ansible-pip_install/blob/26a5868fff0f52db977e6be52dcc2786cf1b8c43/tasks/install_online.yml#L20 and many other places where we assume it's there 17:59:54 <odyssey4me> it's a fair bug, I'd say confirmed 18:00:08 <odyssey4me> medium/low, because clearly most environments have that folder otherwise we'd have resolved that by now 18:01:08 <evrardjp> I'd like to see the task that fails 18:01:11 <evrardjp> because https://github.com/openstack/openstack-ansible/blob/stable/pike/playbooks/common-tasks/set-upper-constraints.yml#L41 18:01:14 <evrardjp> is the first I see 18:01:21 <evrardjp> and it should be fairly standard 18:01:31 <evrardjp> it's not like /opt/openstack-ansible 18:01:39 <evrardjp> it's the whole damn /opt 18:02:42 <evrardjp> ok I will ask for more info, mark it incomplete 18:02:45 <evrardjp> ok for everyone? 18:02:47 <odyssey4me> we could add a docs entry which says that /opt, /var, /bin, etc must exist :p 18:02:55 <evrardjp> hahahaha 18:02:58 <evrardjp> ok on that fun note 18:03:03 <evrardjp> let's wrap up 18:04:44 <evrardjp> thanks everyone! 18:04:46 <evrardjp> #endmeeting