16:00:56 #startmeeting openstack_ansible_meeting 16:00:57 Meeting started Tue Jul 16 16:00:56 2019 UTC and is due to finish in 60 minutes. The chair is mnaser. Information about MeetBot at http://wiki.debian.org/MeetBot. 16:00:58 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 16:01:01 The meeting name has been set to 'openstack_ansible_meeting' 16:01:11 #topic rollcall 16:01:15 o/ 16:01:18 o/ 16:02:07 looks like we're a bit less than the usual 16:02:09 o/ 16:02:17 i think generally things have been progressing nicely 16:02:36 afaik releae was just about to merge? 16:02:50 nvm 16:02:52 it's out :D 16:02:54 https://review.opendev.org/#/c/666728/ 16:02:57 #topic open discussion 16:03:18 \o/ 16:03:32 Merged openstack/openstack-ansible-os_ceilometer master: Replace git.openstack.org URLs with opendev.org URLs https://review.opendev.org/670073 16:03:37 https://opendev.org/openstack/openstack-ansible/src/tag/19.0.0 16:03:40 it has been tagged indeed 16:04:01 o/ 16:04:03 so w00t 16:04:09 tweet it out, tell your friends, etc, etc 16:04:27 Do we need to do anything re. the reports of vms seeing all traffic? That sounds terribad. 16:04:40 uh, i haven't seen anything about that 16:04:45 context? 16:05:29 ^^^up there in the history I think but both jamesdenton and Logan see fdb weirdness on stein 16:05:38 grabes noticed it on his environment and I have verified the behavior on there. we see bridge fdb entries on the hv nodes for the VM side of the tap interface, rather than the host's tap mac address 16:05:50 ovs or lxb? 16:05:50 this is a stein env 16:05:52 lxb 16:06:13 this does sound like a neutron bug though, or is it something we do? 16:06:16 cause afaik we dont build the bridges 16:06:19 not something we do, I'm thinking neutron bug 16:06:21 we let neutron do it 16:06:23 correct 16:06:41 i don't think it is osa config related 16:07:00 I think I meant more about getting some neutron people to see what’s up, like we need to make a bug report I guess 16:07:12 but if someone has a stein lxb env id love some confirmation on what we're seeing in grabes env 16:07:26 yeah we don't run lxb ever :X 16:08:03 I have a stein lxb setup but will likely be tomorrow before I can poke at it 16:08:17 ok sound sgood 16:08:26 jrosser: thanks! 16:08:43 other stuff, noonedeadpunk has been working on patchsets to move all our logging to journald 16:09:04 the topic is here https://review.opendev.org/#/q/topic:use_journalctl+(status:open+OR+status:merged) 16:09:07 neato 16:09:25 And found neutron-ha-tool.py - does anyone use it for stein (or rocky at least)? 16:09:26 That’s great, i was reading about all the journal extra fields 16:09:29 nice 16:09:32 Merged openstack/openstack-ansible-nspawn_hosts master: Replace git.openstack.org URLs with opendev.org URLs https://review.opendev.org/657241 16:09:44 yeah, the neutron-ha-tool.py existance seemed like a rackspace-ism 16:09:59 mnaser: this minimizes our disk space issues right? 16:10:02 Merged openstack/openstack-ansible-os_heat master: Replace git.openstack.org URLs with opendev.org URLs https://review.opendev.org/657253 16:10:15 guilhermesp: yeah, kinda, it means we can just ship systemd logs elsewhere and not have t oworry about services logging locally 16:10:22 * guilhermesp looking with passion to this topic 16:10:54 So I probably should raise this by an email before deprecation 16:11:08 thanks for working on that noonedeadpunk 16:11:11 since neutron has ha since pike I guess 16:11:17 i do think we'll need renos in these patches 16:11:35 Sort of related I have the elk7 stuff almost done 16:11:41 logan-: on all of them? 16:12:16 noonedeadpunk: idk. it'll cluster up our integrated build renos pretty bad if we do a reno per role. 16:12:16 so can't we place reno just for openstack-ansible repo? And in case of removal some variables per repo? 16:12:24 yeah wfm 16:12:31 ++ i like that idea 16:12:36 (a single reno that is) 16:12:40 clutter not cluster* 16:12:51 clustered integrated builds?! 16:12:54 that sounds cool =P 16:13:39 but yeah jrosser elk7 work might be good in combination 16:13:42 ship systemd and call it a day 16:14:07 great, so overall things are good in that side of things 16:14:09 yeah and having journalbeat upstreamed into elastic proper makes it easy to do id think 16:14:30 i had an idea earlier i think i can work with noonedeadpunk to work on which is the uwsgi role 16:14:34 instead of having to install the 3rd party journalbeat 16:14:47 Elk7 has some major changes so I want to get something minimal up so we can all have a hack 16:14:56 (alongside the service_setup.yml task file similar to {mq,db}_setup.yml) 16:15:20 mnaser: cool makes sense.. so it'd be a common task file not a role? 16:15:35 i think the idea is to make service_setup.yml (the keystone bits) a common task 16:15:46 but uwsgi a common role (it seems too big to be a task, esp when it involves handlers to restart things, etc) 16:15:55 gotcha 16:16:13 (btw, noonedeadpunk, i think adding a 16:16:33 pushing up patch to drop neutron-ha-tool and a ML post saying "we will drop it speak up if you need it" is probably ok) 16:16:41 i wonder if we should move these common task sync files out of -tests into something else, maybe integrated repo, and move the post pipeline job there. it seems kind of weird that we sync a bunch of production task files from -tests 16:17:10 since they are not part of the testing framework at all 16:17:17 yeah thats' something i thought about 16:17:19 I was thinking the same 16:17:27 slowly but surely openstack-ansible-tests should/will disappear 16:17:29 logan-: spatel is running stein on centos, and doesn’t have the fdb issues. 16:17:40 i think spatel runs ovs tho 16:18:00 ^ i thought the same 16:19:04 if jrosser has time tomorrow it shouldnt take long to confirm the behavior on one of his hvs too. would be great to have some output from 2 different envs showing it when i write up the neutron bug 16:19:08 but yeah overall i think we're doing okay in terms of 'production' deployments 16:19:08 I thought he was as well, but speaking with him yesterday he confirmed it was lxb 16:19:16 interesting ok 16:19:27 i think we should probably schedule a hack/bugfix day like we did last time 16:19:44 btw, I guess we have some work for upgrade script? 16:20:20 which we've scheduled after release? 16:20:21 i think before that we should try and get functional upgrade jobs 16:20:41 if it takes 1hr to do a debian-based deploy, and our gate max is 3 hours, i think we should totally be ok in doing an upgrade check 16:22:15 btw is there a script already for placemente for upgrade jobs? 16:22:22 no 16:22:26 not that im aware of 16:22:29 i think the ansible bits are done 16:22:33 but the upgrade scripts bit are not 16:22:43 not sure if jesse started 16:22:43 Merged openstack/openstack-ansible-os_gnocchi master: Replace git.openstack.org URLs with opendev.org URLs https://review.opendev.org/657227 16:22:44 and we didn't backport placement to stein iirc 16:22:58 https://review.opendev.org/#/c/664862/ 16:22:58 and we need something for the galera wsrep method change too 16:23:01 oh it wasn't complte but its a start 16:23:12 thats been catching folks out and we've not really documented it 16:23:24 i think all of these will get uncovered once we have a functional upgrade job 16:23:26 yeah mnaser that was the patch 16:23:33 btw 16:23:35 the job already exists 16:23:40 its just a matter of adding it to check & gate 16:23:45 and then just doing the fixes to make it pass 16:23:54 one thing I do not have enabled is the l2population. However, I wasn’t sure that would do as intended on VLAN based networks 16:24:02 in neutron 16:24:09 yeah greenfield is done in master, but upgrade from S->T will require some db manage stuff in upgrade playbooks (i don't know what all needs to be done, is there a nova doc showing this?) and I don't think any of it is done yet 16:24:12 ah cool 16:24:25 * logan- hadn't seen that patch yet 16:24:26 it would be a good exercise to know if these issues have to do with using l2pop + dvr + lxb 16:25:04 Let me add it and restart a VM and see what happens 16:25:34 i think guilhermesp actually got working upgrade jobs in our gate at some point 16:25:37 but we just never merged it 16:25:38 neutron related can we land this? https://review.opendev.org/#/c/668526/ 16:25:49 it needs to be backported to S 16:25:50 yeah I suggested ml2 pop as a hacky workaround in grabes env, but honestly with neutron populating local node fdb macs incorrectly i don't know if it would even help to do ml2 pop. 16:25:54 grabes: we should be able to turn ml2 pop on in one hv and see what the fdb looks like 16:25:55 the only thing I did in the upgrade jobs in the past was fixing tempest workspace 16:26:08 logan- doing that now, give me 5 16:26:09 yeah im sure it will uncover other weird things 16:27:49 mnaser: i think we are very close on the bind-to-mgmt stuff too 16:28:12 i need to update the db_setup task, then the bit things to land withh be rabbitmq and galera 16:29:38 jrosser: i have not had a chance to review those patches but one question as someone using the galera role outside of osa, does the role default behavior change? im guessing not, and the bind to management stuff is piped through from the osa integrated repo vars? 16:30:44 logan-: yes the first thing to do has been going round adding defaults binding to 0.0.0.0 where they were missing, like this https://review.opendev.org/#/c/670403/ 16:30:49 Mohammed Naser proposed openstack/openstack-ansible master: Add metal upgrade jobs https://review.opendev.org/671105 16:30:52 gotcha 16:31:07 then that can all get rewired onto the mgmt ip in the integrated repo group vars 16:31:13 fyi there's a patch that will likely fail but it's a good start point if someone wants to do the ugprade work 16:31:26 thanks jrosser 16:31:51 logan-: looks the same, I didn’t see it add the fa mac 16:31:54 logan-: heres whats done so far https://review.opendev.org/#/q/topic:bind-to-mgmt+(status:open+OR+status:merged) 16:32:57 can someone review any of the patches here if they're okay with the approach used https://review.opendev.org/#/q/topic:use_journalctl+(status:open+OR+status:merged) ? i don't want to start reviewing/merging those two as it'll be everyone from same org push and commit 16:33:07 esp when its large cross-role stuff 16:33:19 Dmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible-os_nova master: Use systemd-journald instead of log files https://review.opendev.org/670882 16:33:43 Merged openstack/openstack-ansible-ceph_client master: Replace git.openstack.org URLs with opendev.org URLs https://review.opendev.org/657232 16:35:15 jrosser: should we bind :: instead of 0.0.0.0? does it make a difference? 16:35:29 the :: vs 0.0.0.0 is a tricky one 16:35:36 apparently certain distros treat :: as in listen to ipv6 only 16:35:42 and some treat it as listen to ipv6 and ipv4 16:35:44 ^ reviews welcome :) 16:35:57 Also I'd love to ask to review https://review.opendev.org/#/c/670471/ since there was a change in service_setup.yml which I'll probably need to implement for already merged roles 16:35:57 i pretty much ignored ipv6 16:36:29 because i figured that in itself was a whole patch set for a ton of stuff that we'd do as a seperate piece of work 16:36:43 yeah 16:36:44 wfm 16:37:34 noonedeadpunk: i like that, what was the change that was implemetned that was different from tehoo thers? 16:38:01 default for validate_certs and condition which involves _service_in_ldap 16:38:03 *the others*** 16:38:08 aaah 16:38:29 oh i think i see something 16:38:38 actually nvm 16:38:50 * jrosser heads out - catchup later 16:39:05 noonedeadpunk: i wonder if we also need to have multiple roles instead of one role 16:39:15 i think heat is one of othose that needs multiple items 16:39:29 later thanks for dropping by jrosser 16:40:19 what do you mean under multiple roles? 16:40:40 ah, keystone user role? 16:40:43 yeah 16:42:21 you're right.... 16:43:11 and several users as well... 16:43:52 ok, there's smth to do then regarding this topic 16:44:07 yeah so i think addressing that would be useful (which would need a "revision" of the api) 16:45:15 so you mean to try to work out with single user? 16:45:52 nono i mean like the service_setup.yml will have to revised so it says _service_users and _service_roles being a list 16:46:04 ah, yep, sure 16:46:19 that what I was going to do at the first place:) 16:46:21 and then the existing role sthat merged will probably have to be revised so we dont break them when we make the change is openstack-ansible-tests 16:46:23 perfect :D 16:49:30 Merged openstack/openstack-ansible-os_ironic master: Cap sphinx for py2 to match global requirements https://review.opendev.org/663582 16:53:26 i think that spretty much covers it all 16:53:32 anything else? :) 16:56:14 i guess not :) 16:56:16 #endmeeting