17:04:28 <evrardjp> #startmeeting openstack_ansible_meeting
17:04:29 <openstack> Meeting started Tue Mar  6 17:04:28 2018 UTC and is due to finish in 60 minutes.  The chair is evrardjp. Information about MeetBot at http://wiki.debian.org/MeetBot.
17:04:30 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
17:04:32 <openstack> The meeting name has been set to 'openstack_ansible_meeting'
17:05:05 <spotz> o/
17:05:14 <evrardjp> #topic focus of the week
17:05:41 <evrardjp> This week's focus will be on releasing queens first
17:06:06 <evrardjp> then releasing all the other branches, as it should have been done during end of last month, which was ptg
17:06:35 <evrardjp> #topic previous week's conversations
17:06:50 <andymccr> o/
17:06:55 <andymccr> sorry im late
17:06:58 <hwoarang> o/
17:07:23 <evrardjp> we had many conversations at the PTG, and it's too long to write them down here, so I'll try to write up a blog or ML post. Anyone willing to help is welcome
17:07:25 <spotz> andymccr: You're back!!!!
17:07:28 <evrardjp> welcomed*
17:07:40 <andymccr> spotz: its like i never left :D
17:07:49 <evrardjp> ok next topic
17:07:50 <spotz> :)
17:07:54 <odyssey4me> o/
17:07:56 <evrardjp> #topic bug triage
17:07:57 <andymccr> evrardjp: if you need help on the blog let me know - happy to read over it or w/e
17:08:03 <evrardjp> #link https://bugs.launchpad.net/openstack-ansible/+bug/1753543
17:08:04 <openstack> Launchpad bug 1753543 in openstack-ansible "LXC interface template does not set MTU for provider interface" [Undecided,New]
17:08:26 <odyssey4me> I think cloudnull just pushed up a slew of patches for that
17:08:34 <evrardjp> looks like it
17:09:03 <andymccr> yeah
17:09:08 <andymccr> i guess we can move to next
17:09:25 <andymccr> and review the patches as usual
17:09:52 <evrardjp> confirmed at least :)
17:10:15 <evrardjp> ok next then
17:10:17 <evrardjp> #link https://bugs.launchpad.net/openstack-ansible/+bug/1753125
17:10:18 <openstack> Launchpad bug 1753125 in openstack-ansible "os_cinder : Ensure cinder api is available timeout" [Undecided,New]
17:10:56 <odyssey4me> need more info
17:11:10 <evrardjp> Incomplete
17:11:12 <evrardjp> indeed
17:11:40 <evrardjp> next
17:11:51 <evrardjp> #link https://bugs.launchpad.net/openstack-ansible/+bug/1752932
17:11:52 <openstack> Launchpad bug 1752932 in openstack-ansible "hostvars breaks on Queens" [Undecided,New]
17:12:05 <evrardjp> we've seen this at the PTG.
17:12:30 <evrardjp> I'd like to debug this in more details. It sounds serious -- high or critical
17:12:44 <odyssey4me> yeah, nice that we have a full config to work with :)
17:12:49 <evrardjp> but the conditions are not clear to me -- python/jinja/ansible version dependant
17:12:57 <andymccr> yeah that seems pretty critical
17:13:10 <evrardjp> we weirdly don't see that in gates.
17:13:13 <andymccr> yeah
17:13:42 <evrardjp> let me leave it as unconfirmed now
17:13:54 <odyssey4me> might have to use vagrant, or the mnaio to reproduce
17:13:56 <evrardjp> if next week I haven't reported
17:14:03 <evrardjp> then we can continue
17:14:08 <evrardjp> I'll assign that to myself
17:14:21 <evrardjp> ok for everyone?
17:14:30 <hwoarang> οκ
17:14:34 <hwoarang> ok
17:14:37 <evrardjp> haha
17:14:43 <evrardjp> next
17:14:45 <evrardjp> #link https://bugs.launchpad.net/openstack-ansible/+bug/1752898
17:14:45 <openstack> Launchpad bug 1752898 in openstack-ansible "gateway option does not work" [Undecided,New]
17:16:16 <evrardjp> I see this listed in all branches up to queens, probably master too.
17:16:29 <evrardjp> so let me check if that's a docs bug or a code bug :)
17:17:20 <andymccr> hmm
17:17:31 <evrardjp> https://github.com/openstack/openstack-ansible-lxc_container_create/blob/aee117fc09981930ac1500e6375905cea2bfaab9/templates/container_network.network.j2#L31
17:17:35 <evrardjp> looks okay to me
17:18:09 <spotz> I blame mhayden....
17:18:15 <odyssey4me> so it should work, but apparently doesn't
17:18:23 <evrardjp> but our test coverage doesn't test it
17:18:25 <evrardjp> https://github.com/openstack/openstack-ansible-lxc_container_create/blob/dc034ff5c32e373efeab127128327e361430b2c0/tests/group_vars/all_containers.yml#L27
17:18:35 <odyssey4me> it might be because the containers use dhcp on eth0, which gives it a route
17:19:04 <odyssey4me> so it's plausible that the option there was something that worked when we hard coded everything - newton onwards we rely more on auto config
17:19:16 <odyssey4me> so perhaps we should just remove those docs and suggest static routes
17:19:25 <odyssey4me> it would seem a better approach to me anyway
17:20:41 <odyssey4me> actually, this is for a secondary interface
17:20:44 <odyssey4me> ...
17:21:55 * odyssey4me needs to think about this more
17:23:35 <evrardjp> docs issue
17:23:57 <odyssey4me> adding that extra gateway is only useful if you have disabled lxcbr0, otherwise eth0 has a default gw and that will be there first
17:23:59 <odyssey4me> you can't have two
17:24:25 <evrardjp> I think the code is fine indeed, on all branches.
17:25:30 <evrardjp> Marked it invalid
17:25:47 <evrardjp> Tahvok: we might need your review on the docs, because you submitted this bug
17:27:08 <evrardjp> ok next
17:27:10 <evrardjp> #link https://bugs.launchpad.net/openstack-ansible/+bug/1752848
17:27:11 <openstack> Launchpad bug 1752848 in openstack-ansible "Cinder backup on ceph when ceph backend not on all cinder-volume" [Undecided,New]
17:30:28 <evrardjp> Definitely need more eyes
17:30:40 <hwoarang> i can't parse it right now
17:30:45 * hwoarang doesn't understand ceph that much yet
17:30:47 <evrardjp> haha
17:31:08 <evrardjp> I think what we can summarize from that is that we are checking what cinder backend is used for cinder_backend_rbd_inuse
17:31:24 <evrardjp> but we should probably have cinder_service_backup_driver
17:31:38 <evrardjp> I mean cinder_service_backup_rbd_inuse
17:32:01 <evrardjp> and rbd_in_use if either is true
17:33:27 <odyssey4me> currently we only automagically set the right vars when there is swift in the env.. we haven't done the same for ceph yet, you have to set them yourself... and if you have a mixed env, then using global overrides in user_*.yml will break things
17:34:08 <odyssey4me> so we should perhaps look at either simplifying the implementation and doing some clever things in the role, like we did for nova for the different settings used per hypervisor
17:34:22 <odyssey4me> or we should wire ceph up in the group vars like we've done with swift
17:35:48 <pabelanger> afternoon, any change I can get a few eyes on https://review.openstack.org/549887/ Remove fedora-26 nodes from testing (migrates to fedora-27)
17:36:01 <evrardjp> I asked the overrides to make sure we can triage this appropriately
17:36:22 <evrardjp> but I expect this to be indeed a lack of user friendliness on our side -- poor variable wiring
17:36:28 <odyssey4me> pabelanger if mhayden says it's fine, then it's fine to me - he likes breaking things :)
17:36:40 <evrardjp> pabelanger: I will have a look after the bug triage :)
17:37:06 <evrardjp> ok next bug
17:37:09 <pabelanger> odyssey4me: evrardjp: thanks
17:37:39 <spotz> odyssey4me: hhehe
17:37:49 <evrardjp> ok next
17:37:50 <evrardjp> #link https://bugs.launchpad.net/openstack-ansible/+bug/1752073
17:37:51 <openstack> Launchpad bug 1752073 in openstack-ansible "Neutron firewall_v2 driver missing" [Undecided,New]
17:38:35 <openstackgerrit> Merged openstack/openstack-ansible-lxc_container_create master: templates: networkd: Drop Link=ARP from networkd configuration  https://review.openstack.org/549878
17:38:44 <evrardjp> that's pike
17:39:02 <hwoarang> interesting
17:39:27 <evrardjp> our docs seem wrong if it doesn't work: https://docs.openstack.org/openstack-ansible-os_neutron/pike/configure-network-services.html#deploying-fwaas-v2
17:39:32 <evrardjp> do we have a scenario for that?
17:39:36 <evrardjp> not sure
17:39:54 <odyssey4me> nope
17:40:00 <evrardjp> we should probably add a scenario to validate that
17:40:06 <evrardjp> and update the docs
17:40:12 <evrardjp> who has cycle to work on that?
17:40:18 <odyssey4me> yeah, hopefully it's not broken most of the time due to upstream bitrot
17:40:29 <evrardjp> firewalling, that must be interesting for some ppl I guess??
17:41:26 <odyssey4me> this is accurate according to upstream docs: https://docs.openstack.org/neutron/queens/admin/fwaas-v2-scenario.html
17:41:37 <odyssey4me> but maybe we're missing something
17:41:41 <hwoarang> i think that we prob forget to pull in some package or something
17:42:25 <evrardjp> I think we should probably exercice it before confirming/infirming it.
17:42:58 <evrardjp> deny*
17:43:05 <odyssey4me> yeah, a new scenario would be good
17:43:28 <evrardjp> leaving it as is
17:43:30 <hwoarang> i can look into that maybe i will ask mbuil or someone else from opnfv
17:43:32 <odyssey4me> I'm afraid my networking knowledge is too sketchy.
17:43:39 <odyssey4me> ah yes, goo dplan
17:43:39 <evrardjp> hwoarang: oh that would be great
17:43:42 <hwoarang> i will assign it to myself
17:43:47 <evrardjp> great
17:43:47 <hwoarang> and i will delegate :D
17:43:54 <evrardjp> haha
17:44:32 <evrardjp> next
17:44:34 <evrardjp> #link https://bugs.launchpad.net/openstack-ansible/+bug/1751722
17:44:35 <openstack> Launchpad bug 1751722 in openstack-ansible "ceph_mon check fails" [Undecided,New]
17:45:47 <evrardjp> andymccr: :D
17:46:36 <evrardjp> I think we need to know more
17:46:41 <evrardjp> about the environment
17:46:59 <evrardjp> is the ceph mon not authorized with current deploy keys?
17:47:18 <evrardjp> I think it would be worth knowing what were the expectations and what's the failyre
17:47:23 <evrardjp> so marking it as incomplete
17:48:10 <logan-> ssh: connect to host 192.168.10.30 port 22: Connection timed out
17:48:16 <logan-> doesn't look like a key issue
17:49:05 <odyssey4me> well, jrosser and boxrick discovered that there's a port check, then it fails back to an ssh connection
17:49:44 <odyssey4me> I'm sorry - it's port check, then a "ping" check using the ansible module
17:49:56 <evrardjp> omg
17:50:20 <logan-> yeah it just needs to turn into a wait_for_connection and get rid of the whole rest of the stuff
17:50:30 <odyssey4me> this is fixed in ansible 2.4 I think, but a problem in 2.3 - and they work through a bastion so the initial check always fails
17:50:30 <logan-> :/
17:51:11 <odyssey4me> logan- I think that bit is in wait_for_connection too for 2.3, but in 2.4 it's done right... but yeah, perhaps some of the ceph stuff is still using older methods
17:51:54 <odyssey4me> in their case they are maintaining a 2.3 fork with patches from newer ansible, which is less than ideal, but aapparently 2.3 is near EOL
17:52:22 <boxrick> We ended up backporting the fix to a fork of 2.3x we point at for now
17:52:33 <boxrick> Doh connection fall
17:52:37 <boxrick> Fail
17:53:42 <jrosser> thats all a bit confusing, if copying the priv key around allegedly fixes it, suggests its not really a connection issue?
17:53:55 <evrardjp> Yeah exactly.
17:54:03 <evrardjp> I vote for incomplete
17:54:22 <evrardjp> Did you see my comment I just posted?
17:54:26 <odyssey4me> https://github.com/openstack/openstack-ansible-ceph_client/blob/159d8164f6080db68d641f0ff5a54b0fd92449d8/tasks/ceph_get_mon_host.yml#L16-L28
17:54:35 <odyssey4me> it's actually doing ssh
17:54:43 <odyssey4me> that;d be why
17:55:27 <evrardjp> yeah.
17:55:54 <odyssey4me> wow, it used to be even more interesting: https://github.com/openstack/openstack-ansible-ceph_client/commit/f30ee47ed0e423a0555888b45b14999e7da517bd
17:56:08 <openstackgerrit> Kevin Carter (cloudnull) proposed openstack/openstack-ansible-lxc_container_create stable/newton: Set the MTU regardless if an address is present  https://review.openstack.org/550147
17:56:09 <odyssey4me> ah, and logan- is to blame :p https://github.com/openstack/openstack-ansible-ceph_client/commit/a039741521ae90b5003c21284c93dc887baa76df
17:56:19 <openstackgerrit> Kevin Carter (cloudnull) proposed openstack/openstack-ansible-lxc_container_create stable/queens: Set the MTU regardless if an address is present  https://review.openstack.org/550145
17:56:21 <openstackgerrit> Merged openstack/ansible-hardening master: Update to fedora-27 for testing  https://review.openstack.org/549887
17:56:29 <openstackgerrit> Kevin Carter (cloudnull) proposed openstack/openstack-ansible-lxc_container_create stable/pike: Set the MTU regardless if an address is present  https://review.openstack.org/550148
17:56:40 <openstackgerrit> Kevin Carter (cloudnull) proposed openstack/openstack-ansible-lxc_container_create stable/ocata: Set the MTU regardless if an address is present  https://review.openstack.org/550146
17:56:40 <odyssey4me> it would seem that we have many more tools in the box now to do this better - perhaps logan- could pick up doing this better?
17:56:53 <logan-> yes please assign it to me
17:56:59 <evrardjp> that would be nice -- but I still think we should be aware of the expectations first.
17:57:01 <evrardjp> logan-: done!
17:57:24 <evrardjp> thanks logan- !
17:58:02 <evrardjp> ok last one
17:58:04 <evrardjp> #link https://bugs.launchpad.net/openstack-ansible/+bug/1750841
17:58:05 <openstack> Launchpad bug 1750841 in openstack-ansible "playbook fails if /opt is missing" [Undecided,New]
17:59:35 <odyssey4me> possible due to https://github.com/openstack/openstack-ansible-pip_install/blob/26a5868fff0f52db977e6be52dcc2786cf1b8c43/tasks/install_online.yml#L20 and many other places where we assume it's there
17:59:54 <odyssey4me> it's a fair bug, I'd say confirmed
18:00:08 <odyssey4me> medium/low, because clearly most environments have that folder otherwise we'd have resolved that by now
18:01:08 <evrardjp> I'd like to see the task that fails
18:01:11 <evrardjp> because https://github.com/openstack/openstack-ansible/blob/stable/pike/playbooks/common-tasks/set-upper-constraints.yml#L41
18:01:14 <evrardjp> is the first I see
18:01:21 <evrardjp> and it should be fairly standard
18:01:31 <evrardjp> it's not like /opt/openstack-ansible
18:01:39 <evrardjp> it's the whole damn /opt
18:02:42 <evrardjp> ok I will ask for more info, mark it incomplete
18:02:45 <evrardjp> ok for everyone?
18:02:47 <odyssey4me> we could add a docs entry which says that /opt, /var, /bin, etc must exist :p
18:02:55 <evrardjp> hahahaha
18:02:58 <evrardjp> ok on that fun note
18:03:03 <evrardjp> let's wrap up
18:04:44 <evrardjp> thanks everyone!
18:04:46 <evrardjp> #endmeeting