15:01:34 #startmeeting kolla 15:01:35 Meeting started Wed Aug 7 15:01:34 2019 UTC and is due to finish in 60 minutes. The chair is mgoddard. Information about MeetBot at http://wiki.debian.org/MeetBot. 15:01:36 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 15:01:39 The meeting name has been set to 'kolla' 15:01:45 Marcin Juszkiewicz proposed openstack/kolla-ansible master: Configure docker without guessing how it starts https://review.opendev.org/672700 15:01:58 #topic rollcall 15:02:01 \o 15:02:03 o/ 15:02:04 o/ 15:02:04 o/ 15:02:25 o/ 15:02:33 0/ 15:02:37 o/ 15:02:41 o/ 15:02:46 sorry, it is my first meeting. What is a rollcall?! 15:02:56 It's where everyone waves 15:03:04 o/ 15:03:07 exactly! 15:03:08 rafaelweingartne: thanks for joining. Raise your hand: o/ 15:03:17 o/ 15:03:21 like in a classroom 15:03:28 This is my first meeting as well :) 15:03:32 it just records who is present 15:03:44 nice turnout today 15:03:48 #topic agenda 15:03:58 mgoddard: everyone waiting for you probably 15:04:20 * Roll-call 15:04:22 * Announcements 15:04:24 ** 99cloud working on kolla-cli - it lives on 15:04:26 ** PTL (mgoddard) is back 15:04:28 ** Added review priority for kayobe in gerrit 15:04:30 ** Kayobe now covered in kolla meetings 15:04:32 * Review action items from last meeting 15:04:34 * Kolla whiteboard https://etherpad.openstack.org/p/KollaWhiteBoard 15:04:36 * Kayobe Stein release status 15:04:38 * Train release planning 15:04:40 * Ocata EOL 15:04:42 * Shanghai summit 15:04:44 * Ceph ansible migration 15:04:46 #topic announcements 15:04:48 #info 99cloud working on kolla-cli - it lives on 15:04:57 hurray 15:05:11 Jeffrey4l_ is the only one I'm aware of who uses IRC 15:05:20 thanks for picking it up 15:05:46 I added a deliverable file to the releases repo for the Train cycle, so hopefully it will get its first release then 15:05:57 #info PTL (mgoddard) is back 15:06:18 welcome back 15:06:22 everyone missed you 15:06:32 <3 15:06:34 Family all doing well, trying to get used to working on less sleep 15:06:40 aw thanks :) 15:06:48 #info Added review priority for kayobe in gerrit 15:07:02 mgoddard: sleep when baby sleeps 15:07:40 This puts kayobe in sync with other kolla projects - you can now set the review priority field in gerrit to highlight 'important' reviews 15:07:52 +1 for important, +2 for critical 15:07:58 mgoddard: and no longer x/kayobe but kolla/kayobe? 15:08:05 openstack/ 15:08:15 hrw: not yet, still waiting on gerrit restart :( 15:08:16 or. just no x/ 15:08:22 :-( 15:08:26 all patches in review and ready to go 15:08:36 maybe I should nudge them 15:08:51 #action mgoddard to ask infra about restarting gerrit 15:08:59 #info Kayobe now covered in kolla meetings 15:08:59 mgoddard: delivery of kayobe for train? 15:09:22 yoctozepto: yes - patch is approved, but wanted to wait for the rename 15:10:10 We didn't do a proper introduction as I was away, but please welcome the kayobe team to our meeting 15:10:39 currently we have jovial[m] in attendance from the core team, and Wasaac is also a contributor 15:10:42 hi - I've moved over from kayobe land 15:10:57 welcome! 15:10:58 welcome jovial[m] 15:11:04 dougsz has a clash with monasca meeting and priteau is with a customer but should join next time 15:11:08 mgoddard; I meant first official release :-) 15:11:09 (both cores) 15:11:16 yoctozepto: yeah 15:11:35 great! 15:11:45 is there a merger of cores? 15:11:53 all have contributed to kolla too, so I'm sure you recognise each other 15:13:11 yeah, sure - wondered if we end up in one team or still two separate? 15:13:14 I don't want to do an automatic merge of teams - let's keep it based on performance. Anyone who contributes & provides useful reviews on a regular basis can be considered core for that team 15:13:50 we have several groups in gerrit - kolla-core, kolla-ansible-core, kolla-cli-core, kayobe-core 15:13:54 mgoddard: fully agree 15:14:18 mgoddard: +1 15:14:34 great 15:14:39 lots of announcements today :) 15:14:51 yoctozepto: you got all kolla at once as it was normal way but not everyone went that way 15:15:09 if anyone has more questions about the new setup, please raise in open discussion 15:15:09 * yoctozepto feeling special ;D 15:15:34 #topic Review action items from last meeting 15:16:03 no action items 15:16:14 #topic Kolla whiteboard https://etherpad.openstack.org/p/KollaWhiteBoard 15:16:41 I merged the kayobe whiteboard into ours 15:17:29 I think the main CI issues recently are around Ceph, and we have fixes in place or in review 15:17:33 right yoctozepto? 15:17:38 right mgoddard 15:17:44 rocky waiting for reviews 15:17:48 never again 15:18:03 I guess we should add kolla-cli CI to the list 15:18:24 yoctozepto: don't worry, you fixed a deprecated feature :) 15:18:43 can kayobe guys clean line 135+ in whiteboard? 15:18:58 now as you have review priority 15:20:41 hrw: sure 15:20:41 done 15:20:50 thx 15:21:05 need to do another round of stable backporting 15:21:16 last checked about 6 weeks ago 15:21:41 #action mgoddard or someone else to check stable backports 15:22:18 #topic Kayobe Stein release status 15:22:52 Now that kolla stein has been released we have no excuses in kayobe, and should also release 15:23:10 ;P 15:23:41 dougsz, jovial[m], priteau: I marked a few patches as RP+1, I'd like to try getting these into master before we branch 15:23:57 https://tiny.cc/kayobe-review-dash 15:24:06 will take a look, thanks 15:24:18 I'll take a look as soon as possible 15:24:30 thanks 15:25:05 #topic Train release planning 15:25:41 yoctozepto and I were discussing earlier that Train is fast approaching 15:26:43 feature freeze is september 13th, we normally freeze a few weeks after - so end of september 15:26:55 #link https://releases.openstack.org/train/schedule.html 15:27:39 it seems like a good time to reflect on our priority features 15:27:51 L133 in https://etherpad.openstack.org/p/KollaWhiteBoard 15:28:07 Anyone know the status of CentOS 8? 15:28:31 https://wiki.centos.org/About/Building_8 15:28:52 looks like they are quite far through, but I don't know how long they normally tak 15:29:23 we are dependent on it for python3 on centos 15:29:42 I expect we'll have a last minute push 15:30:20 most likely, and then RDO will send another pack of nails for our coffin :) 15:30:42 ;) 15:31:01 also interesting was RHEL 7.7 with inplace kernel upgrades 15:31:02 not much we can do there, unless we try using fedora 15:31:32 jovial[m]: i.e. no reboot? 15:31:39 mgoddard: there is some f28 change by someone from RedHat, but I don't know the status 15:31:58 mgoddard: yeah - I believe that to the case 15:32:11 mnasiadka: yeah, for kolla. Would be nice to have a kolla-ansible run with py3 15:32:49 Ok, I guess we just wait on this one 15:32:54 mgoddard: ah, that one - shouldn't be hard to do a preliminary test 15:33:08 mgoddard: but who knows what won't work on CentOS 8 :) 15:33:29 Define support matrix has not been started AFAIK 15:33:41 anyone planning on picking it up? 15:33:55 support matrix is important but I'm too young core to handle this 15:34:00 ;p 15:34:06 I vaguely remember someone saying they were interested, but they didn't get added as an owner 15:34:17 not me 15:34:30 yoctozepto: since you're the youngest core - you can create a framework, and then we can populate :) 15:34:47 mgoddard, kplant no ? 15:34:58 I think a lot of it will need collaboration, but as mnasiadka says someone needs to drive 15:35:19 part of it is agreeing on vocabulary 15:35:50 goldyfruit_: I added kplant as an owner, we can remove if he disagrees :) 15:36:12 Nova cells is underway by dougsz 15:36:45 it wasn't me :-) 15:37:26 ok, removing kplant :) 15:37:35 hum, my bad thenĀ“ 15:37:51 Support newer ansible versions: mnasiadka working on it 15:38:34 yup 15:38:34 anything to say on it mnasiadka? 15:38:55 well, there are two changes - one for bumping up Ansible in kolla-toolbox 15:39:21 but in order to properly constrain pip packages versions in subsequent release branches - we need to use upper-constraints.txt 15:39:27 quick reject that was 15:39:29 which is not available in binary 15:39:46 so I need to rework some of the kolla/common/config.py stuff - should be ready this week 15:40:49 then another piece of work is migrating from kolla_* Ansible modules to upstream os_* - where it makes sense 15:40:54 but it's like an extra thing to do 15:40:57 is this to include get upper-constraints for binary? 15:41:19 how would you do it? curl the official URL? 15:41:35 e.g. https://releases.openstack.org/constraints/upper/master 15:41:43 mgoddard: yeah, that was the idea - but then we need to have a variable that points to current openstack release :-) 15:41:49 right 15:41:58 ok, thanks 15:41:59 (mgoddard, it was ohwhyosa) 15:42:09 aha, nice one goldyfruit_ 15:43:18 Debian/Ubuntu Python 3 support is mostly done, but there is a bug in kolla-ansible that shows we need some changes there too 15:43:40 hrw: https://review.opendev.org/#/c/674241/ 15:43:56 python path stuff 15:44:02 will look 15:44:08 thanks 15:44:26 TLS everywhere has a number of patches in flight, kklimonda working on it 15:44:31 what's the status kklimonda? 15:44:53 he was not around 15:45:20 patches need updating IIRC 15:46:12 libvirt TLS needs another review 15:46:18 and merge conflicts fixing 15:47:10 [kolla] Health checks 15:47:16 mnasiadka: started looking at it? 15:47:48 mgoddard: yes, had some preliminary code somewhere, will look into that after Ansible bump up 15:48:04 k 15:48:24 [kolla ansible] Improving test coverage 15:48:36 lots of good stuff done here already 15:48:43 a few more in progress 15:49:02 dougsz: would be nice to get an intial monasca test merged that we can iterate on 15:49:28 ah yes, sorry, I need to debug why it is failing 15:49:34 mgoddard: I was thinking the same about prometheus and all those exporters that we have... 15:49:49 I mean if somebody would like to do it :) 15:50:09 yeah, would be nice to have that covered 15:50:28 Simplifying handler conditionals - done 15:50:38 [kolla] Podman/buildah support - no owner yet 15:50:59 [kolla ansible] IPv6 - yoctozepto have you started it? 15:51:10 I am afraid that podman thing will slip into Ucycle 15:51:35 mgoddard: yeah, sketching 15:51:37 hrw: I expect so. It's low on priorities so that's ok 15:51:40 BaptisteGer is working on ipv6 deployment in Linaro. no idea how much time during week he has for it 15:51:42 yoctozepto: cool 15:51:54 hrw: using kolla ansible? 15:51:57 mgoddard: yes 15:52:10 cool yoctozepto & BaptisteGer should talk :) 15:52:15 yeah, he sent me his diff 15:52:20 nice 15:52:38 hrw: can I add BaptisteGer as another owner of that feature or is it just downstream? 15:52:41 obviously it drops ipv4 support the way he did it :-) 15:52:52 lol 15:53:01 ipv6 only man 15:53:03 got to start somewhere 15:53:04 done right 15:53:18 mgoddard: add as interested rather than owner 15:53:35 I will bother him to test my proposal when ready 15:53:36 hrw: done 15:53:45 yoctozepto: he may base on some old patch which did that way 15:53:57 and it is easier to get ipv6 machines than ipv4 ones 15:54:21 the diff is good for scoping, no problem 15:54:38 [kolla ansible] Upgrade checkers - no progress since PoC 15:55:05 [kolla] Define source image config via JSON - I think mnasiadka picked this up 15:55:30 mnasiadka: if it has a bug/blueprint, please link it on the whiteboard 15:55:41 [kolla ansible] SELinux - needs an owner 15:56:01 mgoddard: yes, as I wandered into the abyss called common/config.py - but I will most probably first fix the Ansible bump up to make it fast, and then continue with the JSON stuff later on - if that's ok 15:56:14 sure 15:56:28 ok let's move on in the last few minutes 15:56:30 I have working selinux experience but not willing to be blamed for failing the delivery :-) 15:56:32 if SELinux doesn't have an owner now - it will be hard to get anywhere before end of Sep 15:56:40 hopefully that was useful though 15:56:53 add as potential owner or sth 15:56:56 well sdake claimed it used to 'just work' :) 15:57:05 could be 15:57:12 mgoddard: it did, but neutron-ovs-agent stopped working long time ago 15:57:15 tripleo does it with kolla images 15:57:16 docker is free of selinux checks most likely 15:57:34 mgoddard: yes, but they don't use libvirt and ovs in container :) 15:57:44 #topic: Ocata EOL 15:57:54 Who wants to EOL ocata? 15:58:04 We have waited the required 6 months with failing CI 15:58:24 I think the question should be who wants to keep it alive :-) 15:58:51 I guess so 15:58:58 anyone want to keep it alive? 15:59:22 I want to EOL pike 15:59:25 don't look at me 15:59:49 can't EOL pike until CI has been failing for 6 months :) 15:59:53 yeah, I wanted to point out that pike is EM (or should be) 15:59:54 officially 15:59:59 ocata is dead. 16:00:11 ok I think we agree on EOL ocata 16:00:20 for me we still need to keep queens alive 16:00:41 queens is dead, long live rocky? 16:00:42 until EM 16:00:58 #topic Shanghai summit 16:01:04 Anyone going? 16:01:17 I'll try 16:01:18 * hrw skips 16:02:10 goldyfruit_: would you be available to help with kolla forum sessions e.g. answering questions etc.? 16:02:31 spsurya has agreed to handle the project update and project onboarding as I won't be going 16:02:43 with help from Jeffrey4l_ and colleagues 16:02:55 mgoddard, If I'm going yes 16:03:01 I want to get more involved on kolla project in general, will start reading the https://docs.openstack.org/kolla/latest/contributor/index.html before asking questions on how everything is getting organized :) 16:03:23 nafiux: wait for open discussion part 16:03:39 Sure! 16:04:19 we're over time on the meeting, does anyone need to go or can we finish off? 16:04:53 mgoddard: rafaelweingartne 16:04:59 is it the open discussions moment? 16:05:20 I was waiting for it 16:05:31 Let's discuss Ceph ansible migration next time (or after meeting in channel) 16:05:32 yeah, the meeting was a bit longer than usual 16:05:37 #topic open discussion 16:05:46 rafaelweingartne: you're up 16:06:30 ok, so I have been working on this PR (https://review.opendev.org/#/c/670626/) ``Standardize the configuration of "oslo_messaging" section`` 16:06:53 and we have had some debate regarding the use of those "oslo_messaging" sections 16:07:31 our team here, when we first used Kolla-ansible, we expected it to deploy a minimum viable openstack setup, which in our perspective should include event messaging 16:08:14 I guess 'minimum viable' is open to interpretation :) 16:08:16 just when we were debugging an bug that is caused by Ceilometer and barbican keystone listener, when they are using the same topic that we found out what was happening 16:08:28 mgoddard: yes 16:08:31 rafaelweingartne: those notifications goes to ceilometer, right? 16:08:41 yes and no 16:08:46 there are some misconfigurations 16:09:03 I mean 'notifications require ceilometer' 16:09:15 for instance, barbican cannot connect to the same topic. Otherwise, it will get events that were not destinated to it 16:09:16 hrw: no 16:09:38 hrw: ceilometer is just the main consumer 16:09:43 ok 16:09:48 hrw: yes, that is our interpretation, but there is nothing hardcoded, all of it can be customized, and some projects do that 16:10:37 rafaelweingartne: ok 16:10:38 gugug proposed openstack/kolla-cli master: Move kolla prechecks action into kolla_actiom model https://review.opendev.org/675140 16:10:40 so, after discussing with Radek, and later with Mark, we proposed to apply the same method that was being used in Nova role in all of the other projects that had the oslo messaging section declared, but not configured 16:11:28 just to give you guys a quick highlight of that idea, in Nova we can decide if oslo_messaging is going to be configured if there are at least of topic enabled 16:11:48 and then, the topics are enabled/disabled based on components that are enabled/disabled (e.g. ceilometer) 16:12:53 What I think we need to discuss here is: "do we have an idea what are the projects that need notifications so the deployment can be considered an MVP?" 16:13:34 by asking that, I am assuming that the method/standard applied in Nova role is widely accepted by the community, and therefore, we just need to discuss where to apply it 16:13:43 well, it seems ceilometer can be configured to listen for any project's notifications 16:13:50 yes 16:13:59 so we can coordinate that as well 16:14:12 and then configure any service that: 16:14:15 1) exists? 16:14:24 2) we know produces notifications? 16:14:30 ^ choose one 16:14:44 I would say 2 16:14:48 well, I was not aware of all of the projects that produced notifications 16:15:04 therefore, I just went to the roles, and looked for the ones that had the "oslo_messaging" section declared 16:15:22 and worked to configure them using the approach I described 16:15:39 but sure, we can extend this to all service that push some notification 16:16:45 I would more say "restrict" this to services that push notifications 16:16:59 +1 to mgoddard way 16:17:03 all of them then? 16:17:07 review the modified services 16:17:11 it should be easier 16:17:31 and drop the notifications for those that do not produce them for sure 16:17:34 for example, I don't believe mistral produces notifications, so we should not add variables for it 16:17:41 because right now, we seem to be enabling notifications in just a few (more than a few actually) of them 16:17:42 (I am open to being proven wrong) 16:18:14 and regarding the drop of the oslo messaging sections 16:18:18 rafaelweingartne: the thing is none of current cores was responsible for notifications integration in k-a 16:18:29 hmm 16:18:42 got it 16:18:46 so we cannot say why it was done this way 16:18:57 might be copy-paste syndrome 16:18:59 yes, this caused a bit of confusion for us 16:19:11 it actually looked quite a lot of copy/paste indeed 16:19:46 and that is why we decided to fix it by apply the same standard that we have in some well-known project roles such as the nova one 16:20:30 dropping in randomly, but as an example, Octavia should have an oslo_messaging configuration section, but we don't produce notifications (we just READ from a queue -- which I think is the same for a lot of projects?) 16:20:31 I think we agree that is a good approach, we just need to review where we apply it 16:20:43 apologies if I missed some context 16:20:55 yes, but this is a problem 16:21:06 for instance for the projects that had the section declared, but did not use it 16:21:11 rm_work: that is very good context, and one reason why I was concerned about removing these sections 16:21:13 should I do the removal in the same PR? 16:21:25 or should I wait and do that in a different one? 16:21:37 rafaelweingartne: based on rm_work's comment, please do not remove any oslo_messaging_notification sections 16:21:37 I would rather do in a second PR just with the removals 16:21:44 I thought the section configured producers, not consumers 16:21:53 ahh hold on 16:22:08 I just read this: 16:22:08 [09:15:04] rafaelweingartne: therefore, I just went to the roles, and looked for the ones that had the "oslo_messaging" section declared 16:22:10 me too 16:22:25 if it's [oslo_messaging_notifications], then that's maybe fine, we don't declare that 16:22:31 rm_work: ok, thanks 16:22:34 but i am not sure if it's relevant for readers-only 16:22:35 thanks rm_work 16:22:54 lemme check then, but I'm 99% sure it is for producers only 16:22:57 I'm still a little wary of removing them 16:23:17 if we can verify it's only used for producers I would be less wary 16:23:18 yoctozepto: I checked, and it was for producers only 16:23:29 but I am quite new to openstack, so I might be mistaken 16:23:43 either way, separate PR please 16:23:51 https://docs.openstack.org/oslo.messaging/stein/configuration/opts.html#oslo-messaging-notifications 16:24:07 *sending* 16:24:09 ok we're approaching 30 minutes over time in the meeting. The discussion can continue in here if people are free 16:24:14 Thanks everyone 16:24:14 cool, just got concerned due to the abbreviation :D 16:24:18 #endmeeting