17:00:25 <evrardjp> #startmeeting openstack_ansible_meeting
17:00:25 <openstack> Meeting started Tue Mar 20 17:00:25 2018 UTC and is due to finish in 60 minutes.  The chair is evrardjp. Information about MeetBot at http://wiki.debian.org/MeetBot.
17:00:26 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
17:00:29 <openstack> The meeting name has been set to 'openstack_ansible_meeting'
17:00:40 <d34dh0r53> o/
17:00:52 <Tahvok> o/
17:01:05 <RandomTech> Hey i have an instance that would finish deleting, any suggestions on were to look for errors?
17:01:10 <evrardjp> #topic What happened since last meeting
17:01:14 <hwoarang> o/
17:01:18 <RandomTech> oops sorry didnt notice the meeting started
17:01:25 <odyssey4me> o/
17:01:29 <jmccrory> o/
17:01:33 <cjloader> o/
17:01:39 <evrardjp> Under this section we are just friendly reminding what went on last week
17:01:50 <evrardjp> cloudnull and evrardjp talked about the integration of external roles
17:01:56 <evrardjp> #link http://eavesdrop.openstack.org/irclogs/%23openstack-ansible/%23openstack-ansible.2018-03-14.log.html#t2018-03-14T15:05:20
17:02:02 <cloudnull> o/
17:02:07 <evrardjp> evrardjp worked with tonyb on newton EOLing
17:02:14 <evrardjp> #link http://lists.openstack.org/pipermail/openstack-dev/2018-March/128313.html
17:02:21 <evrardjp> evrardjp's regular bump was not done last week, due to already pending O/P bump patches
17:02:31 <evrardjp> https://review.openstack.org/#/c/550599/ https://review.openstack.org/#/c/550593/
17:02:40 <evrardjp> spotz introduced the idea of office hours, but no formal office hours will be held for now.
17:02:47 <evrardjp> #link http://eavesdrop.openstack.org/irclogs/%23openstack-ansible/%23openstack-ansible.2018-03-14.log.html#t2018-03-14T16:38:58
17:03:11 <evrardjp> I know that's a lot to process, but it's not over yet!
17:03:15 <evrardjp> evrardjp promoted the Vancouver Forum brainstorming etherpad and would love some brainstorm to happen
17:03:21 <evrardjp> #link https://etherpad.openstack.org/p/YVR-openstack-ansible-brainstorming
17:03:33 <evrardjp> evrardjp added gokhan as core member of the telemetry roles
17:03:34 <spotz> hola
17:03:39 <evrardjp> #link http://eavesdrop.openstack.org/irclogs/%23openstack-ansible/%23openstack-ansible.2018-03-19.log.html#t2018-03-19T13:37:53
17:03:48 <evrardjp> niraj_singh is working on the inclusion of masakari into OSA
17:03:54 <evrardjp> #link http://eavesdrop.openstack.org/irclogs/%23openstack-ansible/%23openstack-ansible.2018-03-19.log.html#t2018-03-19T13:46:12
17:04:01 <evrardjp> odyssey4me would like some input for the two approaches outlined in the etherpad for the python build simplification
17:04:09 <evrardjp> #link https://etherpad.openstack.org/p/osa-source-deploy-role-planning
17:04:15 <andymccr> o/
17:04:37 <evrardjp> Now that you have all those links opened in your browsers.... Did I miss something?
17:04:51 <evrardjp> If not, we can go ahead with the triage.
17:05:01 <evrardjp> We can discuss those topics after the triage
17:05:13 <logan-> o/
17:05:27 <odyssey4me> evrardjp thanks for pulling all that together - it's useful to get a quick summary to digest
17:05:41 <hwoarang> yeah awesome work
17:06:07 <hwoarang> evrardjp: yes mbuil is core in neutron too ;p
17:06:17 <evrardjp> odyssey4me: the format is not easy, but at least it would leave the ppl the time to digest things after the meeting. Any addition for next week should go into the meeting page https://wiki.openstack.org/wiki/Meetings/openstack-ansible
17:06:23 <hwoarang> mbuil: wake up meeting time!
17:06:41 <evrardjp> hwoarang: that's not this week, I think it was previous week, but I might have the dates wrong!
17:06:46 <hwoarang> ah could be
17:06:56 <evrardjp> but thanks for the reminder hwoarang
17:07:24 <evrardjp> could we go ahead with the triage? I know we have lots of bugs, and I know odyssey4me would like to discuss his etherpad link after the triage
17:07:57 <evrardjp> 5
17:07:59 <evrardjp> 4
17:08:01 <evrardjp> 3
17:08:03 <evrardjp> 2
17:08:05 <evrardjp> 1
17:08:08 <evrardjp> #topic bug triage
17:08:23 <evrardjp> #link https://bugs.launchpad.net/openstack-ansible/+bug/1756005
17:08:24 <openstack> Launchpad bug 1756005 in openstack-ansible "cinder_volumes_container bind mount for /var/log not configured" [Undecided,New]
17:09:55 <evrardjp> I don't know why we have cinder_backend_lvm_inuse | bool  there
17:10:01 <andymccr> yeah
17:10:07 <andymccr> id say high on that-  not confirmed it though
17:10:43 <andymccr> seems sensible based ont he report though
17:11:01 <odyssey4me> agreed
17:11:03 <evrardjp> oh we did like that everywhere: we say we run containers based on backend x or y, not based on is_metal or not
17:14:03 <evrardjp> I think we should just add another task that does the same kind of things, but without the unconfined profile I guess
17:14:31 <evrardjp> when cinder_volume is in group_names and is_metal is False
17:14:37 <evrardjp> sounds plausible?
17:14:51 <evrardjp> confirmed high?
17:15:13 <andymccr> yeah
17:15:17 <evrardjp> ok next
17:15:19 <evrardjp> #link https://bugs.launchpad.net/openstack-ansible/+bug/1755929
17:15:21 <openstack> Launchpad bug 1755929 in openstack-ansible "Octavia config drop fails on multiple hosts" [Undecided,New]
17:16:37 <evrardjp> that sound like bad orchestration
17:16:54 <andymccr> yeah
17:17:08 <odyssey4me> yeah, not very good validation either
17:17:21 <evrardjp> it looks legit
17:17:23 <odyssey4me> can't confirm, but seems like it needs someone to pick that up pronto
17:17:32 <evrardjp> yup
17:17:39 <evrardjp> xgerman_: could you check at this?
17:17:42 <odyssey4me> xgerman_ ^ that looks like something that needs urgent follow up
17:17:48 <odyssey4me> heh
17:17:52 <evrardjp> :)
17:18:09 <odyssey4me> I'd say High
17:18:14 <evrardjp> ok Let's mark it as confirmed and high.
17:18:26 <odyssey4me> It's pretty serious for production.
17:18:26 <andymccr> yeah
17:18:27 <xgerman_> k
17:18:57 <xgerman_> have a patch up for that but need ot review
17:19:19 <evrardjp> Could you mark the bug in your patch?
17:19:26 <evrardjp> It would be listed then
17:19:26 <xgerman_> yep, will do
17:19:29 <evrardjp> ok next
17:19:31 <evrardjp> #link https://bugs.launchpad.net/openstack-ansible/+bug/1755821
17:19:32 <openstack> Launchpad bug 1755821 in openstack-ansible "config_template fails to parse template if it contains a comment with leading spaces" [Undecided,New]
17:20:25 <evrardjp> I haven't tested this myself
17:20:29 <evrardjp> but that sounds very serious
17:20:30 <odyssey4me> me neither
17:20:36 <cloudnull> yea that doesnt sound good
17:20:44 <evrardjp> cloudnull: could you have a look?
17:21:04 <cloudnull> you can add me to that
17:21:12 <cloudnull> I wont be able to look at it today
17:21:28 <openstackgerrit> Andy McCrae proposed openstack/openstack-ansible-plugins stable/queens: Utilise sorted to ensure no random changes  https://review.openstack.org/554301
17:21:40 <evrardjp> it's alright, I am leaving it unconfirmed for now, so that if you forget it would reappear next week
17:21:48 <evrardjp> leaving the triage status at high though
17:22:00 <evrardjp> next
17:22:02 <evrardjp> #link https://bugs.launchpad.net/openstack-ansible/+bug/1755790
17:22:03 <openstack> Launchpad bug 1755790 in openstack-ansible "vxlan ttl defaults to 1" [Undecided,New]
17:22:26 <evrardjp> opinion?
17:22:52 <evrardjp> I think vxlan ttl default behavior is to have ttl to 1 though, to avoid traffic crossing through routers, on purpose
17:23:13 <andymccr> hmm
17:23:47 <evrardjp> but the pod example is exactly the opposite, you need to cross routers
17:23:48 <jrosser> this is
17:23:54 <jrosser> multicast
17:24:12 <jrosser> sorry on crap 3g
17:24:19 <evrardjp> oh yeah true
17:24:23 <evrardjp> but still
17:24:40 <evrardjp> jrosser: haha
17:24:54 <jrosser> ttl 1 is usually set for things like mdns which are explocitly link local
17:24:54 <evrardjp> I think this deserves confirming and adapting in the docs
17:26:03 <evrardjp> The deployer will evaluate what's best for his/her case.
17:26:18 <evrardjp> I am fine with adapting the documentation, fine for you jrosser ?
17:26:36 <jrosser> tes it needs to certatainly go in the pods example
17:26:41 <evrardjp> I am not sure it's a good idea to change the ttl by default
17:26:53 <andymccr> documentation sounds like the way
17:26:55 <evrardjp> jrosser: I'd do it ONLY for the pods example
17:27:16 <jrosser> the unicast part of vxlan will be generally routtable
17:27:59 <evrardjp> jrosser: but it doesn't need to be on non-pod
17:28:09 <jrosser> so as it stands the behaviour will be different for generally encapsualted packets and any BUM like dhcp or arp
17:28:34 <evrardjp> yes
17:28:52 <evrardjp> I think we agree and we can discuss in the bug and patches.
17:28:59 <evrardjp> I'll assign that to me
17:29:01 <jrosser> yes
17:29:18 <evrardjp> next
17:29:19 <evrardjp> #link https://bugs.launchpad.net/openstack-ansible/+bug/1755609
17:29:20 <openstack> Launchpad bug 1755609 in openstack-ansible "Galera xinetd only_from listens to load balancers' "ansible_host" rather than container management IP" [Undecided,New]
17:30:13 <evrardjp> tricky one this one
17:30:26 <evrardjp> due to our extreme configurability
17:31:13 <andymccr> hmm
17:32:18 <jmccrory> are there facts that could be used to list interface addresses on haproxy_hosts?
17:32:20 <cloudnull> I think it makes sense to whitelist the cidrs
17:32:22 <evrardjp> so in its example, the ip used for ansible_host doesn't matter
17:32:34 <evrardjp> jmccrory: yes
17:33:05 <evrardjp> jmccrory: I think that's a good solution in fact, whitelist all the haproxy interfaces ip
17:33:10 <evrardjp> same for galera ips
17:33:16 <evrardjp> good thinking there!
17:33:35 <evrardjp> cloudnull: I think it would be okay, but opening very wide, I prefer jmccrory solution :)
17:33:46 <evrardjp> if that's possible
17:33:56 <cloudnull> we shouldn't just limit it to haproxy
17:34:03 <odyssey4me> if jmccrory says it, it must be possible ;)
17:34:15 <spotz> heheheh
17:34:21 <evrardjp> yeah, it should be a setup delegate_facts delegate_to
17:34:32 <evrardjp> to get the fact of the interfaces facts
17:34:40 <evrardjp> cloudnull: why?
17:34:48 <cloudnull> at rax we have external LBs which are not haproxy
17:35:00 <cloudnull> so limiting to haproxy would break me
17:35:23 <evrardjp> cloudnull: you can still add the hosts
17:35:31 <evrardjp> of your load balancers
17:35:41 <cloudnull> but ansible wont have access to them
17:35:48 <cloudnull> so there are no facts to gather
17:35:59 <odyssey4me> assuming an override is available, surely any environment can override the defaults?
17:36:13 <evrardjp> that's what I meant
17:36:27 <cloudnull> so long as we can set the networks and are not forced to use facts from a given host, +1
17:37:00 <odyssey4me> perhaps we could have a smart default - use haproxy addresses if there are haproxy hosts, otherwise fall back to the CIDR's?
17:37:08 <cloudnull> ^ ++
17:37:30 <odyssey4me> Use the inventory Luke...
17:37:32 <evrardjp> sounds good. The fallback would require work, because networks_cidr aren't directly exposed IIRC
17:37:36 <evrardjp> only the ips
17:38:18 <evrardjp> oh yes we do
17:38:19 <cloudnull> maybe the fall back is like the hap whitelist where it just any rfc1918 address
17:38:24 <evrardjp> {name}_cidr
17:38:56 <evrardjp> cloudnull: great security :p
17:38:59 <odyssey4me> seems sensible, given that the whitelist is well established
17:39:13 <evrardjp> ok we agree on the idea anyway, let's triage this: confirmed and medium?
17:39:16 <evrardjp> or low
17:39:20 <odyssey4me> if you care about security, you'd have a whitelist override in place
17:39:22 <evrardjp> it's a very edge case
17:39:40 <odyssey4me> confirmed/medium methinks
17:39:44 <cloudnull> something like https://github.com/openstack/openstack-ansible/blob/master/inventory/group_vars/haproxy_all/haproxy.yml#L63
17:40:02 <cloudnull> https://github.com/openstack/openstack-ansible/blob/master/inventory/group_vars/haproxy_all/haproxy.yml#L25-L28
17:40:33 <evrardjp> next
17:40:35 <evrardjp> #link https://bugs.launchpad.net/openstack-ansible/+bug/1755593
17:40:36 <openstack> Launchpad bug 1755593 in openstack-ansible "Rally can't install encapsulated Tempest verifier" [Undecided,New]
17:40:47 <cloudnull> so use hap hosts by default, fall back to somethign that will work providing a minimal level of security but is configurable.
17:41:13 <evrardjp> cloudnull: we agree.
17:41:18 <cloudnull> cool
17:41:18 <evrardjp> Rally?
17:42:12 <odyssey4me> ja, apparently rally does its own preparation of some kind
17:42:20 <evrardjp> indeed
17:42:26 <evrardjp> installing ansible and all that jazz!
17:42:30 <odyssey4me> we had a discussion in channel about it
17:42:35 <evrardjp> oh
17:42:37 <evrardjp> ?
17:42:42 <evrardjp> about this bug?
17:42:48 <odyssey4me> yeah, me and the bug reporter
17:43:00 <evrardjp> what was the summary of that? It's not written in the bug
17:43:11 <odyssey4me> he's been looking into hacks and things but it doesn't look like he's reported back into the bug
17:43:18 <evrardjp> ok
17:43:39 <evrardjp> how do we triage this then?
17:43:51 <odyssey4me> well, it's definitely true - so confirmed
17:44:03 <odyssey4me> it seems to affect using rally, so I'd say medium
17:44:26 <odyssey4me> not high because this is the first we've heard of it - so either no-one uses rally, or this is a new issue in pike
17:44:40 <evrardjp> sold!
17:44:44 <evrardjp> next
17:44:51 <evrardjp> #link https://bugs.launchpad.net/openstack-ansible/+bug/1755241
17:44:52 <openstack> Launchpad bug 1755241 in openstack-ansible "The Documentation for Pike is conistantly telling users to pull the wrong version" [Undecided,New]
17:45:27 <evrardjp> confirmed and medium, it's telling the right version, but the version is confusing
17:45:45 <evrardjp> ok for everyone?
17:45:47 <andymccr> yeah
17:46:05 <odyssey4me> ja
17:46:22 <evrardjp> next
17:46:24 <evrardjp> #link https://bugs.launchpad.net/openstack-ansible/+bug/1754591
17:46:25 <openstack> Launchpad bug 1754591 in openstack-ansible "Neutron agent delete: remove the wrong argument" [Undecided,New]
17:47:03 <andymccr> i dont see what we can do abotu that
17:47:25 <evrardjp> yeah, so?
17:47:48 <andymccr> it'll be fixed in a future version of 16.0.x?
17:47:57 <evrardjp> oh it doesn't say it's part of the deploy of our playbooks
17:48:15 <evrardjp> andymccr: most likely it won't get fixed
17:48:26 <evrardjp> because we are still constrained
17:48:28 <andymccr> its backported to stable/pike?
17:48:30 <evrardjp> so we'll not get that bump
17:48:44 <evrardjp> I doubt it
17:48:49 <andymccr> https://review.openstack.org/#/c/518371/
17:49:01 <evrardjp> oh it was
17:49:09 <andymccr> i guess the point im makgin is its not an OSA issue
17:49:10 <odyssey4me> looks like it merged into the stable branch, so there needs to be a release, then a u0c bump
17:49:16 <andymccr> ^ yeah
17:49:23 <odyssey4me> then we get it for free
17:49:35 <evrardjp> it's merged into stable branch
17:49:42 <evrardjp> it's just not bumped into u-c
17:49:53 <odyssey4me> yes, but it still has to be released - then bumped in u-c
17:49:58 <andymccr> yeah
17:50:05 <evrardjp> the patch for inclusion in pike is 2017.
17:50:09 <odyssey4me> u-c doesn't bump libraries to SHA's ;)
17:50:26 <odyssey4me> well, someone needs to be chased to do the release then
17:50:37 <evrardjp> "This issue was fixed in the openstack/python-openstackclient 3.13.0 "
17:50:43 <evrardjp> (2017-12-13)
17:50:46 <odyssey4me> or we need to implement our own git source usage for it
17:50:59 <evrardjp> it's just that this hasn't been bumped into requirements
17:51:04 <evrardjp> so I am not sure what we can do
17:51:26 <odyssey4me> we can push the review to bump u-c
17:51:42 <odyssey4me> for stable branches that's not an automatic process
17:52:07 <evrardjp> I thought that was kinda taboo
17:52:09 <evrardjp> :p
17:52:21 <evrardjp> "do not bump osc on stable branches!"
17:52:36 <evrardjp> I guess we can ask prometheanfire
17:52:37 <odyssey4me> you can always push up a review - it may not be accepted
17:52:47 <evrardjp> prometheanfire: can you have a look?
17:52:51 <odyssey4me> I've done it before
17:52:54 <jmccrory> why do they have stable branches otherwise?
17:53:05 <andymccr> yeah you can have a discussion about it at least.
17:53:08 <evrardjp> I've done it once, and it was not very welcomed by tonyb
17:53:14 <odyssey4me> for this bug though, I'd mark it as a duplicate and link the original bug to openstack-ansible
17:53:23 <evrardjp> good point
17:53:51 <evrardjp> next
17:54:01 <evrardjp> #link https://bugs.launchpad.net/openstack-ansible/+bug/1754139
17:54:02 <openstack> Launchpad bug 1754139 in openstack-ansible "openvswitch does not start on new AIO deployment" [Undecided,New]
17:54:34 <evrardjp> ovs... mbuil could you by chance have a look at this?
17:55:07 <evrardjp> except if someone else is willing to have a look?
17:55:42 <evrardjp> leaving it as is in 5 seconds!
17:55:45 <evrardjp> 4
17:55:47 <evrardjp> 3
17:55:50 <evrardjp> 2
17:55:53 <evrardjp> 1
17:55:57 <evrardjp> (large seconds!)
17:56:03 <evrardjp> next
17:56:05 <evrardjp> #link https://bugs.launchpad.net/openstack-ansible/+bug/1753801
17:56:07 <openstack> Launchpad bug 1753801 in openstack-ansible "Galera connection contention on full base deployment on Xeon E3 " [Undecided,New]
17:57:28 <evrardjp> I guess we'll wait for logan- 's report and patches :)
17:57:35 <evrardjp> good job there
17:57:47 <evrardjp> let's move on, shall we?
17:57:52 <andymccr> yeah
17:57:54 <evrardjp> #link https://bugs.launchpad.net/openstack-ansible/+bug/1753790
17:57:55 <openstack> Launchpad bug 1753790 in openstack-ansible "spurious 'no space left on device' errors during venv build" [Undecided,New]
17:57:55 <logan-> im not sure if it is valid, gnocchi is totally broken on pike currently and restarts endlessly
17:58:03 <hwoarang> hmm
17:58:03 <prometheanfire> wat
17:58:08 <hwoarang> that may have been fixed
17:58:11 <evrardjp> logan-: oh so we'll wait for your feedback :)
17:58:26 <evrardjp> hwoarang: are we sure it's done now?
17:58:29 <prometheanfire> evrardjp: bumping thing in stable branches is fine, if it a stable branch release
17:58:33 <hwoarang> i haven't seen it lately
17:58:36 <hwoarang> have you?
17:58:38 <odyssey4me> no, I've still seen disk space issues come up
17:58:43 <hwoarang> ok then...
17:58:48 <evrardjp> odyssey4me: on which branch?
17:58:59 <odyssey4me> master
17:59:02 <evrardjp> I have seen things on Q
17:59:08 <evrardjp> oh
17:59:16 <evrardjp> yeah let's leave that open then
17:59:21 <evrardjp> as a reminder!
17:59:34 <odyssey4me> cloudnull and I discussed it... but couldn't figure it out
17:59:43 <odyssey4me> I didn't think I'd seen it on Queens though.
17:59:54 <evrardjp> mmm. Let's rollback to standard lxc image managemnt then :p
18:00:01 <odyssey4me> might need to dig into logstash to see how frequently that's happening
18:00:14 <evrardjp> I still need to learn that thing.
18:00:29 <evrardjp> next time you do it, let's discuss on the channel
18:00:31 <evrardjp> ok
18:00:36 <odyssey4me> sure
18:00:38 <evrardjp> It's time to wrap up
18:00:56 <evrardjp> #endmeeting