15:07:41 <yoctozepto> #startmeeting kolla 15:07:41 <opendevmeet> Meeting started Wed Jun 30 15:07:41 2021 UTC and is due to finish in 60 minutes. The chair is yoctozepto. Information about MeetBot at http://wiki.debian.org/MeetBot. 15:07:41 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 15:07:41 <opendevmeet> The meeting name has been set to 'kolla' 15:07:47 <yoctozepto> #chair mgoddard 15:07:47 <opendevmeet> Current chairs: mgoddard yoctozepto 15:07:53 <yoctozepto> #topic Roll-call 15:07:55 <yoctozepto> o/ 15:07:57 <mgoddard> erm 15:08:03 <mgoddard> oop 15:08:04 <mgoddard> s 15:08:13 <yoctozepto> no problem 15:08:20 <mgoddard> I'm not sure where today went :) 15:08:26 <yoctozepto> let's harvest participants 15:08:32 <yoctozepto> crazy day here as well :-) 15:08:43 <mgoddard> mgoddard mnasiadka hrw egonzalez yoctozepto rafaelweingartne cosmicsound osmanlicilegi bbezak parallax Fl1nt 15:08:52 <mgoddard> koalas assemble 15:08:52 <yoctozepto> meeting now, folks :-) 15:08:55 <yoctozepto> <3 15:08:59 <mnasiadka> seriously? 15:09:02 <mnasiadka> :) 15:09:06 <yoctozepto> but you have to whisper "assemble" 15:09:41 <mgoddard> #topic agenda 15:09:43 <mgoddard> * Roll-call 15:09:45 <mgoddard> * Agenda 15:09:47 <mgoddard> * Announcements 15:09:49 <mgoddard> * Review action items from the last meeting 15:09:51 <mgoddard> * CI status 15:09:53 <mgoddard> * Release tasks 15:09:55 <mgoddard> * Xena cycle planning 15:09:57 <mgoddard> * Open discussion 15:09:59 <mgoddard> #topic announcements 15:10:41 <mgoddard> #info Kolla Wallaby releases now available 15:11:34 <yoctozepto> YEAH 15:11:42 <mgoddard> #link http://lists.openstack.org/pipermail/openstack-discuss/2021-June/023289.html 15:11:49 <mgoddard> #topic Review action items from the last meeting 15:12:03 <mgoddard> there were none 15:12:07 <mgoddard> #topic CI status 15:12:57 <yoctozepto> need to merge patches I think? 15:13:07 <mgoddard> hmm 15:13:25 <yoctozepto> on the stable branches 15:14:19 <yoctozepto> for kolla 15:14:20 <mgoddard> #link https://review.opendev.org/q/topic:%22ci-emergency-fix-for-zuul-4-6%22 15:15:19 <mgoddard> added my +2 15:16:08 <mgoddard> upgrades to Ussuri fail sporadically on Ubuntu due to Neutron migrations failing 15:16:12 <mgoddard> monasca scenario fails all the time 15:16:38 <yoctozepto> yes, sir 15:16:52 <mgoddard> something to do with the ES index 15:16:58 <mgoddard> unsure of the cause 15:17:15 <mgoddard> I think dougsz is aware 15:17:32 <yoctozepto> ok 15:17:52 <mgoddard> #topic Release tasks 15:17:55 <yoctozepto> ubuntu ussuri upgrades are a mystery though 15:18:55 <mgoddard> I think we still have some R-17 release patches open 15:19:19 <mnasiadka> Yes, and Heat is a mystery 15:19:41 <mgoddard> https://review.opendev.org/c/openstack/kolla-ansible/+/796422 15:19:43 <yoctozepto> refresh our memory please 15:20:00 <mnasiadka> and only on debuntu 15:20:37 <mnasiadka> https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_d4c/796415/3/check/kolla-ansible-debian-source/d4c358b/primary/logs/kolla/heat/heat-api-error.txt 15:20:46 <mgoddard> Can we at least merge this one? https://review.opendev.org/c/openstack/kolla-ansible/+/796422 15:20:51 <mnasiadka> https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_d4c/796415/3/check/kolla-ansible-debian-source/d4c358b/primary/logs/kolla/heat/apache-error.txt 15:21:34 <mnasiadka> mgoddard: I guess we can 15:24:29 <yoctozepto> why enable chrony again? 15:24:37 <mgoddard> has anyone tried speaking to the heat team? 15:25:20 <mgoddard> yoctozepto: only for upgrades, to test the cleanup command 15:25:53 <mnasiadka> mgoddard: well, once somebody will debug it properly, and find some meaningful error message - it would make sense, but I haven't made it to this point. 15:28:38 <yoctozepto> mgoddard: are not we testing that in wallaby? 15:28:50 <mgoddard> yoctozepto: yes, testing the wallaby code 15:28:53 <yoctozepto> ah, you want to avoid regressions 15:28:59 <mgoddard> in master we test master code 15:29:04 <mgoddard> indeed 15:29:12 <yoctozepto> there is one issue 15:29:27 <yoctozepto> enabled will kill the host one 15:29:33 <mgoddard> "we disabled CI testing in master because we test ussuri" :) 15:29:51 <yoctozepto> ? 15:30:00 <mgoddard> I'm being silly 15:30:23 <yoctozepto> I see 15:30:33 <mgoddard> if that is an issue in master then it is an issue in wallaby also 15:30:59 <yoctozepto> yeah, we don't restore it for users 15:31:11 <yoctozepto> ok 15:31:21 <yoctozepto> as long as it works in ci 15:32:09 <mgoddard> hold on, don't we disable the host one first, then start it after cleaning the container? 15:32:14 <mgoddard> to avoid having them both running 15:33:23 <mgoddard> on the heat issue, should we disable testing of heat in CI for now? 15:34:13 <mnasiadka> that's also an option 15:34:26 <mnasiadka> I'll do a recheck to check if the issue still persists, if it does will disable heat testing in CI 15:34:27 <mgoddard> it's only doing openstack orchestration service list 15:35:37 <yoctozepto> does it break on the api call? 15:35:41 <yoctozepto> or earlier? 15:36:04 <mgoddard> looks like api call https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_d4c/796415/3/check/kolla-ansible-debian-source/d4c358b/primary/logs/kolla/heat/heat-api-error.txt 15:36:19 <mnasiadka> yeah, it's breaking on that 15:37:34 <mgoddard> Any other release tasks? 15:37:38 <yoctozepto> can we call something else? 15:37:43 <yoctozepto> on that heat api 15:38:27 <mnasiadka> it's crashing, so what do you want to call? 15:38:47 <yoctozepto> I'm just curious if it's failing listing these 15:39:09 <mnasiadka> like the whole api wsgi service is not being run, it's crashing on mod_wsgi running the binary 15:39:26 <yoctozepto> yeah, but mgoddard said only after that api call 15:39:31 <yoctozepto> and it looks like it 15:40:08 <yoctozepto> - - - [22/Jun/2021:06:36:44 +0000] "GET / HTTP/1.1" 500 531 1321112 "-" "curl-healthcheck" 15:40:08 <mgoddard> seems to affect ubuntu only 15:40:09 <yoctozepto> hmm 15:40:17 <yoctozepto> it seems to work on version discovery from curl 15:40:22 <yoctozepto> or does it? 15:40:24 <mgoddard> 500 15:40:33 <mnasiadka> well, docker healthcheck does something 15:40:35 * yoctozepto blind 15:40:45 <yoctozepto> let's see the same for centos 15:41:13 <mnasiadka> and gets 500 15:42:54 <mgoddard> I think we should move on 15:43:42 <mgoddard> I propose we disable the heat service list on ubuntu, and add it to the workarounds list to be fixed before release 15:44:17 <mnasiadka> will do 15:45:36 <yoctozepto> - - - [22/Jun/2021:06:35:03 +0000] "GET / HTTP/1.1" 300 117 2108380 "-" "curl-healthcheck" 15:45:39 <yoctozepto> working gets 300 15:45:43 <yoctozepto> so it's utterly broken 15:45:47 <yoctozepto> hmm 15:45:55 <yoctozepto> must be mod_wsgi version 15:45:57 <mgoddard> re other release tasks, we should probably do some stable releases soon 15:46:51 <mgoddard> #action mgoddard check open backports & propose stable releases 15:48:24 <mgoddard> #topic Xena cycle planning 15:49:20 <mgoddard> Do we need to take some time to look at open patches, and try to agree some things to focus on 15:49:23 <mgoddard> ? 15:49:47 <headphoneJames> Would like to get the Lets Encrypt integration in Xena :) 15:49:53 <mgoddard> +1 15:49:56 <headphoneJames> https://review.opendev.org/c/openstack/kolla-ansible/+/741340 15:50:02 <headphoneJames> part 1 is there 15:50:18 <mgoddard> we could borrow an idea from ironic 15:50:54 <yoctozepto> how so? 15:50:56 <mgoddard> core reviewers can sign up to review particular features 15:51:01 <yoctozepto> ah 15:51:45 <mgoddard> which means that the developer of the feature has an idea if their patch is likely to get merged, and who to ask/bug for help/reviews 15:52:00 <mgoddard> it might also help focus attention of cores 15:52:20 <mgoddard> and maybe get more cores involved in reviewing 15:52:35 <mgoddard> since the review firehose can be a bit overwhelming 15:52:45 <yoctozepto> true that 15:53:07 <mgoddard> and it is easy to focus on easy wins 15:53:24 <mgoddard> just to get them out of the queue :) 15:53:27 <mgoddard> any thoughts? 15:53:43 <yoctozepto> (sorry for mentioning but I need to before I forget, there is a thread on ml which could use mgoddard replying - some kayobe slides are gone) 15:53:56 <yoctozepto> mgoddard: I agree we could adopt this strategy 15:53:57 <mgoddard> k, thanks 15:54:09 <yoctozepto> yw 15:54:22 <yoctozepto> (the one from Amit) 15:54:54 <mgoddard> I see it 15:56:11 <mgoddard> right now we don't have a great view of features being developed 15:57:38 <mgoddard> well, something to think about 15:57:44 <mgoddard> bright ideas on a postcard please 15:57:47 <mnasiadka> Well, it would be nice to have a list of features that need reviewing on some board/etherpad 15:58:02 <mgoddard> we previously had priorities on https://etherpad.opendev.org/p/KollaWhiteBoard 15:58:11 <mgoddard> but we stopped doing priorities :) 15:58:24 <mgoddard> I guess they had two flaws 15:58:36 <mgoddard> 1. no guarantee that anyone picks up priorities 15:59:05 <mgoddard> 2. people often contribute unanticipated features that do not get added to the list 15:59:42 <yoctozepto> yeah, need to adapt that approach 15:59:45 <yoctozepto> to reality 15:59:54 <mgoddard> so maybe we need more of a separation between 'community priorities as a guide for project direction' 15:59:54 <yoctozepto> ok, we have to wrap up I guess 16:00:00 <mgoddard> vs work in progress 16:00:01 <yoctozepto> yup 16:00:34 <mgoddard> sounds like a good challenge for the next PTL :) 16:00:44 <mgoddard> on that note, thanks all 16:00:52 <mgoddard> #endmeeting