15:07:41 <yoctozepto> #startmeeting kolla
15:07:41 <opendevmeet> Meeting started Wed Jun 30 15:07:41 2021 UTC and is due to finish in 60 minutes.  The chair is yoctozepto. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:07:41 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:07:41 <opendevmeet> The meeting name has been set to 'kolla'
15:07:47 <yoctozepto> #chair mgoddard
15:07:47 <opendevmeet> Current chairs: mgoddard yoctozepto
15:07:53 <yoctozepto> #topic Roll-call
15:07:55 <yoctozepto> o/
15:07:57 <mgoddard> erm
15:08:03 <mgoddard> oop
15:08:04 <mgoddard> s
15:08:13 <yoctozepto> no problem
15:08:20 <mgoddard> I'm not sure where today went :)
15:08:26 <yoctozepto> let's harvest participants
15:08:32 <yoctozepto> crazy day here as well :-)
15:08:43 <mgoddard> mgoddard mnasiadka hrw egonzalez yoctozepto rafaelweingartne cosmicsound osmanlicilegi bbezak parallax Fl1nt
15:08:52 <mgoddard> koalas assemble
15:08:52 <yoctozepto> meeting now, folks :-)
15:08:55 <yoctozepto> <3
15:08:59 <mnasiadka> seriously?
15:09:02 <mnasiadka> :)
15:09:06 <yoctozepto> but you have to whisper "assemble"
15:09:41 <mgoddard> #topic agenda
15:09:43 <mgoddard> * Roll-call
15:09:45 <mgoddard> * Agenda
15:09:47 <mgoddard> * Announcements
15:09:49 <mgoddard> * Review action items from the last meeting
15:09:51 <mgoddard> * CI status
15:09:53 <mgoddard> * Release tasks
15:09:55 <mgoddard> * Xena cycle planning
15:09:57 <mgoddard> * Open discussion
15:09:59 <mgoddard> #topic announcements
15:10:41 <mgoddard> #info Kolla Wallaby releases now available
15:11:34 <yoctozepto> YEAH
15:11:42 <mgoddard> #link http://lists.openstack.org/pipermail/openstack-discuss/2021-June/023289.html
15:11:49 <mgoddard> #topic  Review action items from the last meeting
15:12:03 <mgoddard> there were none
15:12:07 <mgoddard> #topic CI status
15:12:57 <yoctozepto> need to merge patches I think?
15:13:07 <mgoddard> hmm
15:13:25 <yoctozepto> on the stable branches
15:14:19 <yoctozepto> for kolla
15:14:20 <mgoddard> #link https://review.opendev.org/q/topic:%22ci-emergency-fix-for-zuul-4-6%22
15:15:19 <mgoddard> added my +2
15:16:08 <mgoddard> upgrades to Ussuri fail sporadically on Ubuntu due to Neutron migrations failing
15:16:12 <mgoddard> monasca scenario fails all the time
15:16:38 <yoctozepto> yes, sir
15:16:52 <mgoddard> something to do with the ES index
15:16:58 <mgoddard> unsure of the cause
15:17:15 <mgoddard> I think dougsz is aware
15:17:32 <yoctozepto> ok
15:17:52 <mgoddard> #topic Release tasks
15:17:55 <yoctozepto> ubuntu ussuri upgrades are a mystery though
15:18:55 <mgoddard> I think we still have some R-17 release patches open
15:19:19 <mnasiadka> Yes, and Heat is a mystery
15:19:41 <mgoddard> https://review.opendev.org/c/openstack/kolla-ansible/+/796422
15:19:43 <yoctozepto> refresh our memory please
15:20:00 <mnasiadka> and only on debuntu
15:20:37 <mnasiadka> https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_d4c/796415/3/check/kolla-ansible-debian-source/d4c358b/primary/logs/kolla/heat/heat-api-error.txt
15:20:46 <mgoddard> Can we at least merge this one? https://review.opendev.org/c/openstack/kolla-ansible/+/796422
15:20:51 <mnasiadka> https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_d4c/796415/3/check/kolla-ansible-debian-source/d4c358b/primary/logs/kolla/heat/apache-error.txt
15:21:34 <mnasiadka> mgoddard: I guess we can
15:24:29 <yoctozepto> why enable chrony again?
15:24:37 <mgoddard> has anyone tried speaking to the heat team?
15:25:20 <mgoddard> yoctozepto: only for upgrades, to test the cleanup command
15:25:53 <mnasiadka> mgoddard: well, once somebody will debug it properly, and find some meaningful error message - it would make sense, but I haven't made it to this point.
15:28:38 <yoctozepto> mgoddard: are not we testing that in wallaby?
15:28:50 <mgoddard> yoctozepto: yes, testing the wallaby code
15:28:53 <yoctozepto> ah, you want to avoid regressions
15:28:59 <mgoddard> in master we test master code
15:29:04 <mgoddard> indeed
15:29:12 <yoctozepto> there is one issue
15:29:27 <yoctozepto> enabled will kill the host one
15:29:33 <mgoddard> "we disabled CI testing in master because we test ussuri" :)
15:29:51 <yoctozepto> ?
15:30:00 <mgoddard> I'm being silly
15:30:23 <yoctozepto> I see
15:30:33 <mgoddard> if that is an issue in master then it is an issue in wallaby also
15:30:59 <yoctozepto> yeah, we don't restore it for users
15:31:11 <yoctozepto> ok
15:31:21 <yoctozepto> as long as it works in ci
15:32:09 <mgoddard> hold on, don't we disable the host one first, then start it after cleaning the container?
15:32:14 <mgoddard> to avoid having them both running
15:33:23 <mgoddard> on the heat issue, should we disable testing of heat in CI for now?
15:34:13 <mnasiadka> that's also an option
15:34:26 <mnasiadka> I'll do a recheck to check if the issue still persists, if it does will disable heat testing in CI
15:34:27 <mgoddard> it's only doing openstack orchestration service list
15:35:37 <yoctozepto> does it break on the api call?
15:35:41 <yoctozepto> or earlier?
15:36:04 <mgoddard> looks like api call https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_d4c/796415/3/check/kolla-ansible-debian-source/d4c358b/primary/logs/kolla/heat/heat-api-error.txt
15:36:19 <mnasiadka> yeah, it's breaking on that
15:37:34 <mgoddard> Any other release tasks?
15:37:38 <yoctozepto> can we call something else?
15:37:43 <yoctozepto> on that heat api
15:38:27 <mnasiadka> it's crashing, so what do you want to call?
15:38:47 <yoctozepto> I'm just curious if it's failing listing these
15:39:09 <mnasiadka> like the whole api wsgi service is not being run, it's crashing on mod_wsgi running the binary
15:39:26 <yoctozepto> yeah, but mgoddard said only after that api call
15:39:31 <yoctozepto> and it looks like it
15:40:08 <yoctozepto> - - - [22/Jun/2021:06:36:44 +0000] "GET / HTTP/1.1" 500 531 1321112 "-" "curl-healthcheck"
15:40:08 <mgoddard> seems to affect ubuntu only
15:40:09 <yoctozepto> hmm
15:40:17 <yoctozepto> it seems to work on version discovery from curl
15:40:22 <yoctozepto> or does it?
15:40:24 <mgoddard> 500
15:40:33 <mnasiadka> well, docker healthcheck does something
15:40:35 * yoctozepto blind
15:40:45 <yoctozepto> let's see the same for centos
15:41:13 <mnasiadka> and gets 500
15:42:54 <mgoddard> I think we should move on
15:43:42 <mgoddard> I propose we disable the heat service list on ubuntu, and add it to the workarounds list to be fixed before release
15:44:17 <mnasiadka> will do
15:45:36 <yoctozepto> - - - [22/Jun/2021:06:35:03 +0000] "GET / HTTP/1.1" 300 117 2108380 "-" "curl-healthcheck"
15:45:39 <yoctozepto> working gets 300
15:45:43 <yoctozepto> so it's utterly broken
15:45:47 <yoctozepto> hmm
15:45:55 <yoctozepto> must be mod_wsgi version
15:45:57 <mgoddard> re other release tasks, we should probably do some stable releases soon
15:46:51 <mgoddard> #action mgoddard check open backports & propose stable releases
15:48:24 <mgoddard> #topic Xena cycle planning
15:49:20 <mgoddard> Do we need to take some time to look at open patches, and try to agree some things to focus on
15:49:23 <mgoddard> ?
15:49:47 <headphoneJames> Would like to get the Lets Encrypt integration in Xena :)
15:49:53 <mgoddard> +1
15:49:56 <headphoneJames> https://review.opendev.org/c/openstack/kolla-ansible/+/741340
15:50:02 <headphoneJames> part 1 is there
15:50:18 <mgoddard> we could borrow an idea from ironic
15:50:54 <yoctozepto> how so?
15:50:56 <mgoddard> core reviewers can sign up to review particular features
15:51:01 <yoctozepto> ah
15:51:45 <mgoddard> which means that the developer of the feature has an idea if their patch is likely to get merged, and who to ask/bug for help/reviews
15:52:00 <mgoddard> it might also help focus attention of cores
15:52:20 <mgoddard> and maybe get more cores involved in reviewing
15:52:35 <mgoddard> since the review firehose can be a bit overwhelming
15:52:45 <yoctozepto> true that
15:53:07 <mgoddard> and it is easy to focus on easy wins
15:53:24 <mgoddard> just to get them out of the queue :)
15:53:27 <mgoddard> any thoughts?
15:53:43 <yoctozepto> (sorry for mentioning but I need to before I forget, there is a thread on ml which could use mgoddard replying - some kayobe slides are gone)
15:53:56 <yoctozepto> mgoddard: I agree we could adopt this strategy
15:53:57 <mgoddard> k, thanks
15:54:09 <yoctozepto> yw
15:54:22 <yoctozepto> (the one from Amit)
15:54:54 <mgoddard> I see it
15:56:11 <mgoddard> right now we don't have a great view of features being developed
15:57:38 <mgoddard> well, something to think about
15:57:44 <mgoddard> bright ideas on a postcard please
15:57:47 <mnasiadka> Well, it would be nice to have a list of features that need reviewing on some board/etherpad
15:58:02 <mgoddard> we previously had priorities on https://etherpad.opendev.org/p/KollaWhiteBoard
15:58:11 <mgoddard> but we stopped doing priorities :)
15:58:24 <mgoddard> I guess they had two flaws
15:58:36 <mgoddard> 1. no guarantee that anyone picks up priorities
15:59:05 <mgoddard> 2. people often contribute unanticipated features that do not get added to the list
15:59:42 <yoctozepto> yeah, need to adapt that approach
15:59:45 <yoctozepto> to reality
15:59:54 <mgoddard> so maybe we need more of a separation between 'community priorities as a guide for project direction'
15:59:54 <yoctozepto> ok, we have to wrap up I guess
16:00:00 <mgoddard> vs work in progress
16:00:01 <yoctozepto> yup
16:00:34 <mgoddard> sounds like a good challenge for the next PTL :)
16:00:44 <mgoddard> on that note, thanks all
16:00:52 <mgoddard> #endmeeting