16:00:56 <mnaser> #startmeeting openstack_ansible_meeting
16:00:57 <openstack> Meeting started Tue Jul 16 16:00:56 2019 UTC and is due to finish in 60 minutes.  The chair is mnaser. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:00:58 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
16:01:01 <openstack> The meeting name has been set to 'openstack_ansible_meeting'
16:01:11 <mnaser> #topic rollcall
16:01:15 <mnaser> o/
16:01:18 <noonedeadpunk> o/
16:02:07 <mnaser> looks like we're a bit less than the usual
16:02:09 <jrosser> o/
16:02:17 <mnaser> i think generally things have been progressing nicely
16:02:36 <mnaser> afaik releae was just about to merge?
16:02:50 <mnaser> nvm
16:02:52 <mnaser> it's out :D
16:02:54 <mnaser> https://review.opendev.org/#/c/666728/
16:02:57 <mnaser> #topic open discussion
16:03:18 <raukadah> \o/
16:03:32 <openstackgerrit> Merged openstack/openstack-ansible-os_ceilometer master: Replace git.openstack.org URLs with opendev.org URLs  https://review.opendev.org/670073
16:03:37 <mnaser> https://opendev.org/openstack/openstack-ansible/src/tag/19.0.0
16:03:40 <mnaser> it has been tagged indeed
16:04:01 <guilhermesp> o/
16:04:03 <mnaser> so w00t
16:04:09 <mnaser> tweet it out, tell your friends, etc, etc
16:04:27 <jrosser> Do we need to do anything re. the reports of vms seeing all traffic? That sounds terribad.
16:04:40 <mnaser> uh, i haven't seen anything about that
16:04:45 <mnaser> context?
16:05:29 <jrosser> ^^^up there in the history I think but both jamesdenton and Logan see fdb weirdness on stein
16:05:38 <logan-> grabes noticed it on his environment and I have verified the behavior on there. we see bridge fdb entries on the hv nodes for the VM side of the tap interface, rather than the host's tap mac address
16:05:50 <mnaser> ovs or lxb?
16:05:50 <logan-> this is a stein env
16:05:52 <logan-> lxb
16:06:13 <mnaser> this does sound like a neutron bug though, or is it something we do?
16:06:16 <mnaser> cause afaik we dont build the bridges
16:06:19 <logan-> not something we do, I'm thinking neutron bug
16:06:21 <mnaser> we let neutron do it
16:06:23 <logan-> correct
16:06:41 <logan-> i don't think it is osa config related
16:07:00 <jrosser> I think I meant more about getting some neutron people to see what’s up, like we need to make a bug report I guess
16:07:12 <logan-> but if someone has a stein lxb env id love some confirmation on what we're seeing in grabes env
16:07:26 <mnaser> yeah we don't run lxb ever :X
16:08:03 <jrosser> I have a stein lxb setup but will likely be tomorrow before I can poke at it
16:08:17 <mnaser> ok sound sgood
16:08:26 <logan-> jrosser: thanks!
16:08:43 <mnaser> other stuff, noonedeadpunk has been working on patchsets to move all our logging to journald
16:09:04 <mnaser> the topic is here https://review.opendev.org/#/q/topic:use_journalctl+(status:open+OR+status:merged)
16:09:07 <logan-> neato
16:09:25 <noonedeadpunk> And found neutron-ha-tool.py - does anyone use it for stein (or rocky at least)?
16:09:26 <jrosser> That’s great, i was reading about all the journal extra fields
16:09:29 <guilhermesp> nice
16:09:32 <openstackgerrit> Merged openstack/openstack-ansible-nspawn_hosts master: Replace git.openstack.org URLs with opendev.org URLs  https://review.opendev.org/657241
16:09:44 <mnaser> yeah, the neutron-ha-tool.py existance seemed like a rackspace-ism
16:09:59 <guilhermesp> mnaser: this minimizes our disk space issues right?
16:10:02 <openstackgerrit> Merged openstack/openstack-ansible-os_heat master: Replace git.openstack.org URLs with opendev.org URLs  https://review.opendev.org/657253
16:10:15 <mnaser> guilhermesp: yeah, kinda, it means we can just ship systemd logs elsewhere and not have t oworry about services logging locally
16:10:22 * guilhermesp looking with passion to this topic
16:10:54 <noonedeadpunk> So I probably should raise this by an email before deprecation
16:11:08 <logan-> thanks for working on that noonedeadpunk
16:11:11 <noonedeadpunk> since neutron has ha since pike I guess
16:11:17 <logan-> i do think we'll need renos in these patches
16:11:35 <jrosser> Sort of related I have the elk7 stuff almost done
16:11:41 <noonedeadpunk> logan-: on all of them?
16:12:16 <logan-> noonedeadpunk: idk. it'll cluster up our integrated build renos pretty bad if we do a reno per role.
16:12:16 <noonedeadpunk> so can't we place reno just for openstack-ansible repo? And in case of removal some variables per repo?
16:12:24 <logan-> yeah wfm
16:12:31 <mnaser> ++ i like that idea
16:12:36 <mnaser> (a single reno that is)
16:12:40 <logan-> clutter not cluster*
16:12:51 <mnaser> clustered integrated builds?!
16:12:54 <mnaser> that sounds cool =P
16:13:39 <mnaser> but yeah jrosser elk7  work might be good in combination
16:13:42 <mnaser> ship systemd and call it a day
16:14:07 <mnaser> great, so overall things are good in that side of things
16:14:09 <logan-> yeah and having journalbeat upstreamed into elastic proper makes it easy to do id think
16:14:30 <mnaser> i had an idea earlier i think i can work with noonedeadpunk to work on which is the uwsgi role
16:14:34 <logan-> instead of having to install the 3rd party journalbeat
16:14:47 <jrosser> Elk7 has some major changes so I want to get something minimal up so we can all have a hack
16:14:56 <mnaser> (alongside the service_setup.yml task file similar to {mq,db}_setup.yml)
16:15:20 <logan-> mnaser: cool makes sense.. so it'd be a common task file not a role?
16:15:35 <mnaser> i think the idea is to make service_setup.yml (the keystone bits) a common task
16:15:46 <mnaser> but uwsgi a common role (it seems too big to be a task, esp when it involves handlers to restart things, etc)
16:15:55 <logan-> gotcha
16:16:13 <mnaser> (btw, noonedeadpunk, i think adding a
16:16:33 <mnaser> pushing up patch to drop neutron-ha-tool and a ML post saying "we will drop it speak up if you need it" is probably ok)
16:16:41 <logan-> i wonder if we should move these common task sync files out of -tests into something else, maybe integrated repo, and move the post pipeline job there. it seems kind of weird that we sync a bunch of production task files from -tests
16:17:10 <logan-> since they are not part of the testing framework at all
16:17:17 <mnaser> yeah thats' something i thought about
16:17:19 <jrosser> I was thinking the same
16:17:27 <mnaser> slowly but surely openstack-ansible-tests should/will disappear
16:17:29 <grabes> logan-: spatel is running stein on centos, and doesn’t have the fdb issues.
16:17:40 <mnaser> i think spatel runs ovs tho
16:18:00 <logan-> ^ i thought the same
16:19:04 <logan-> if jrosser has time tomorrow it shouldnt take long to confirm the behavior on one of his hvs too. would be great to have some output from 2 different envs showing it when i write up the neutron bug
16:19:08 <mnaser> but yeah overall i think we're doing okay in terms of 'production' deployments
16:19:08 <grabes> I thought he was as well, but speaking with him yesterday he confirmed it was lxb
16:19:16 <logan-> interesting ok
16:19:27 <mnaser> i think we should probably schedule a hack/bugfix day like we did last time
16:19:44 <noonedeadpunk> btw, I guess we have some work for upgrade script?
16:20:20 <noonedeadpunk> which we've scheduled after release?
16:20:21 <mnaser> i think before that we should try and get functional upgrade jobs
16:20:41 <mnaser> if it takes 1hr to do a debian-based deploy, and our gate max is 3 hours, i think we should totally be ok in doing an upgrade check
16:22:15 <guilhermesp> btw is there a script already for placemente for upgrade jobs?
16:22:22 <logan-> no
16:22:26 <logan-> not that im aware of
16:22:29 <mnaser> i think the ansible bits are done
16:22:33 <mnaser> but the upgrade scripts bit are not
16:22:43 <guilhermesp> not sure if jesse started
16:22:43 <openstackgerrit> Merged openstack/openstack-ansible-os_gnocchi master: Replace git.openstack.org URLs with opendev.org URLs  https://review.opendev.org/657227
16:22:44 <noonedeadpunk> and we didn't backport placement to stein iirc
16:22:58 <mnaser> https://review.opendev.org/#/c/664862/
16:22:58 <jrosser> and we need something for the galera wsrep method change too
16:23:01 <mnaser> oh it wasn't complte but its a start
16:23:12 <jrosser> thats been catching folks out and we've not really documented it
16:23:24 <mnaser> i think all of these will get uncovered once we have a functional upgrade job
16:23:26 <guilhermesp> yeah mnaser that was the patch
16:23:33 <mnaser> btw
16:23:35 <mnaser> the job already exists
16:23:40 <mnaser> its just a matter of adding it to check & gate
16:23:45 <mnaser> and then just doing the fixes to make it pass
16:23:54 <grabes> one thing I do not have enabled is the l2population.  However, I wasn’t sure that would do as intended on VLAN based networks
16:24:02 <grabes> in neutron
16:24:09 <logan-> yeah greenfield is done in master, but upgrade from S->T will require some db manage stuff in upgrade playbooks (i don't know what all needs to be done, is there a nova doc showing this?) and I don't think any of it is done yet
16:24:12 <logan-> ah cool
16:24:25 * logan- hadn't seen that patch yet
16:24:26 <mnaser> it would be a good exercise to know if these issues have to do with using l2pop + dvr + lxb
16:25:04 <grabes> Let me add it and restart a VM and see what happens
16:25:34 <mnaser> i think guilhermesp actually got working upgrade jobs in our gate at some point
16:25:37 <mnaser> but we just never merged it
16:25:38 <jrosser> neutron related can we land this? https://review.opendev.org/#/c/668526/
16:25:49 <jrosser> it needs to be backported to S
16:25:50 <logan-> yeah I suggested ml2 pop as a hacky workaround in grabes env, but honestly with neutron populating local node fdb macs incorrectly i don't know if it would even help to do ml2 pop.
16:25:54 <logan-> grabes: we should be able to turn ml2 pop on in one hv and see what the fdb looks like
16:25:55 <guilhermesp> the only thing I did in the upgrade jobs in the past was fixing tempest workspace
16:26:08 <grabes> logan- doing that now, give me 5
16:26:09 <mnaser> yeah im sure it will uncover other weird things
16:27:49 <jrosser> mnaser: i think we are very close on the bind-to-mgmt stuff too
16:28:12 <jrosser> i need to update the db_setup task, then the bit things to land withh be rabbitmq and galera
16:29:38 <logan-> jrosser: i have not had a chance to review those patches but one question as someone using the galera role outside of osa, does the role default behavior change? im guessing not, and the bind to management stuff is piped through from the osa integrated repo vars?
16:30:44 <jrosser> logan-: yes the first thing to do has been going round adding defaults binding to 0.0.0.0 where they were missing, like this https://review.opendev.org/#/c/670403/
16:30:49 <openstackgerrit> Mohammed Naser proposed openstack/openstack-ansible master: Add metal upgrade jobs  https://review.opendev.org/671105
16:30:52 <logan-> gotcha
16:31:07 <jrosser> then that can all get rewired onto the mgmt ip in the integrated repo group vars
16:31:13 <mnaser> fyi there's a patch that will likely fail but it's a good start point if someone wants to do the ugprade work
16:31:26 <logan-> thanks jrosser
16:31:51 <grabes> logan-: looks the same, I didn’t see it add the fa mac
16:31:54 <jrosser> logan-: heres whats done so far https://review.opendev.org/#/q/topic:bind-to-mgmt+(status:open+OR+status:merged)
16:32:57 <mnaser> can someone review any of the patches here if they're okay with the approach used https://review.opendev.org/#/q/topic:use_journalctl+(status:open+OR+status:merged) ? i don't want to start reviewing/merging those two as it'll be everyone from same org push and commit
16:33:07 <mnaser> esp when its large cross-role stuff
16:33:19 <openstackgerrit> Dmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible-os_nova master: Use systemd-journald instead of log files  https://review.opendev.org/670882
16:33:43 <openstackgerrit> Merged openstack/openstack-ansible-ceph_client master: Replace git.openstack.org URLs with opendev.org URLs  https://review.opendev.org/657232
16:35:15 <logan-> jrosser: should we bind :: instead of 0.0.0.0? does it make a difference?
16:35:29 <mnaser> the :: vs 0.0.0.0 is a tricky one
16:35:36 <mnaser> apparently certain distros treat :: as in listen to ipv6 only
16:35:42 <mnaser> and some treat it as listen to ipv6 and ipv4
16:35:44 <jrosser> ^ reviews welcome :)
16:35:57 <noonedeadpunk> Also I'd love to ask to review https://review.opendev.org/#/c/670471/ since there was a change in service_setup.yml which I'll probably need to implement for already merged roles
16:35:57 <jrosser> i pretty much ignored ipv6
16:36:29 <jrosser> because i figured that in itself was a whole patch set for a ton of stuff that we'd do as a seperate piece of work
16:36:43 <logan-> yeah
16:36:44 <logan-> wfm
16:37:34 <mnaser> noonedeadpunk: i like that, what was the change that was implemetned that was different from tehoo thers?
16:38:01 <noonedeadpunk> default for validate_certs and condition which involves _service_in_ldap
16:38:03 <mnaser> *the others***
16:38:08 <mnaser> aaah
16:38:29 <mnaser> oh i think i see something
16:38:38 <mnaser> actually nvm
16:38:50 * jrosser heads out - catchup later
16:39:05 <mnaser> noonedeadpunk: i wonder if we also need to have multiple roles instead of one role
16:39:15 <mnaser> i think heat is one of othose that needs multiple items
16:39:29 <mnaser> later thanks for dropping by jrosser
16:40:19 <noonedeadpunk> what do you mean under multiple roles?
16:40:40 <noonedeadpunk> ah, keystone user role?
16:40:43 <mnaser> yeah
16:42:21 <noonedeadpunk> you're right....
16:43:11 <noonedeadpunk> and several users as well...
16:43:52 <noonedeadpunk> ok, there's smth to do then regarding this topic
16:44:07 <mnaser> yeah so i think addressing that would be useful (which would need a "revision" of the api)
16:45:15 <noonedeadpunk> so you mean to try to work out with single user?
16:45:52 <mnaser> nono i mean like the service_setup.yml will have to revised so it says _service_users and _service_roles being a list
16:46:04 <noonedeadpunk> ah, yep, sure
16:46:19 <noonedeadpunk> that what I was going to do at the first place:)
16:46:21 <mnaser> and then the existing role sthat merged will probably have to be revised so we dont break them when we make the change is openstack-ansible-tests
16:46:23 <mnaser> perfect :D
16:49:30 <openstackgerrit> Merged openstack/openstack-ansible-os_ironic master: Cap sphinx for py2 to match global requirements  https://review.opendev.org/663582
16:53:26 <mnaser> i think that spretty much covers it all
16:53:32 <mnaser> anything else? :)
16:56:14 <mnaser> i guess not :)
16:56:16 <mnaser> #endmeeting