15:00:20 <mgoddard> #startmeeting kolla
15:00:20 <openstack> Meeting started Wed Nov 20 15:00:20 2019 UTC and is due to finish in 60 minutes.  The chair is mgoddard. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:00:21 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:00:23 <openstack> The meeting name has been set to 'kolla'
15:00:25 <mgoddard> ping mgoddard mnasiadka hrw egonzalez yoctozepto rafaelweingartne
15:00:29 <mgoddard> #topic rollcall
15:00:31 <mgoddard> \o
15:00:34 <mnasiadka> o/
15:00:36 <yoctozepto> o/
15:02:43 <osmanlicilegi> o/
15:02:55 <openstackgerrit> Zhuo Zhen proposed openstack/kolla-ansible master: Fix hard-coded admin project name and username in blazar task  https://review.opendev.org/695214
15:03:09 <mgoddard> #topic agenda
15:03:19 <mgoddard> * Roll-call
15:03:19 <hrw> o\
15:03:21 <mgoddard> * Announcements
15:03:23 <mgoddard> ** yoctozepto wishes to invite people to Białystok (on 2020-02-04) to talk about clouds (will explain)
15:03:25 <mgoddard> * Review action items from last meeting
15:03:27 <mgoddard> * CI status
15:03:29 <mgoddard> * Train release planning
15:03:32 <mgoddard> ** old Erlang for RMQ still scary: https://bugs.launchpad.net/kolla/+bug/1853167 - better update and take the risk of upgrade or leave it b0rken?
15:03:32 <openstack> Launchpad bug 1853167 in kolla ussuri "CentOS RabbitMQ Unsupported Erlang Version" [Medium,Triaged] - Assigned to Radosław Piliszek (yoctozepto)
15:03:33 <mgoddard> * Ussuri release planning
15:03:35 <mgoddard> * Review priorities
15:03:37 <mgoddard> #topic announcements
15:03:43 <mgoddard> #info yoctozepto wishes to invite people to Białystok (on 2020-02-04) to talk about clouds (will explain)
15:03:58 <yoctozepto> yes, exactly
15:04:05 <yoctozepto> we are organizing a little conference
15:04:06 <hrw> yoctozepto: bad timing - right after fosdem
15:04:29 <yoctozepto> not me to decide, maybe I can still get some of you to join
15:04:45 <yoctozepto> I can let bosses know about it but that's it :-)
15:04:54 <yoctozepto> the point is
15:05:02 <mnasiadka> fosdem, maybe I would finally go ;)
15:05:20 <yoctozepto> I have money to spend on one international (read: european) guest and one local guest
15:05:30 <yoctozepto> it would cover travel and accommodation costs
15:06:34 <mgoddard> Thanks for sharing yoctozepto, keep us updated with details
15:06:35 <yoctozepto> the auditorium would consist of academia and education workers (i.e. researchers and educators plus other staff), students and local IT business people
15:06:45 <yoctozepto> topics include
15:06:51 <yoctozepto> cloud for hpc
15:06:57 <yoctozepto> cloud for science
15:07:05 <yoctozepto> science for cloud
15:07:26 <yoctozepto> hence asking around ppl from the so-called stackhpc company :-)
15:07:47 <yoctozepto> I will let you know if I can amend the schedule still
15:08:01 <yoctozepto> but please let me know if you are interested at any rate
15:08:36 <mgoddard> what sort of size do you expect the event to be?
15:09:10 <yoctozepto> one day, about 6-7 presenters
15:09:31 <yoctozepto> mostly depends on who I and others manage to "hire"
15:10:44 <mgoddard> sounds like an interesting event
15:11:01 <mgoddard> I'll pass it on
15:11:06 <mgoddard> #topic Review action items from last meeting
15:11:06 <yoctozepto> thanks, mgoddard
15:11:23 <mgoddard> mgoddard to check stable/train backports
15:11:26 <mgoddard> yoctozepto: to get in touch with cloudnull and find a time to discuss kolla image consumers, e.g. next week's meeting
15:11:29 <yoctozepto> I notified cloudnull
15:11:34 <mgoddard> I did mine, although it's an ongoing effort
15:11:34 <yoctozepto> he promised to be around
15:11:41 <yoctozepto> checking mine now
15:11:43 <cloudnull> o/
15:11:47 <yoctozepto> hah :D
15:11:47 <mgoddard> hi cloudnull
15:11:53 <yoctozepto> hi-ya
15:12:02 <JamesBenson> o/
15:12:05 * cloudnull is around. though very split-brained, so situation normal :D
15:12:12 <yoctozepto> oh my
15:12:18 <yoctozepto> it can be cured
15:12:22 <mgoddard> added to agenda
15:12:41 <mgoddard> #topic CI status
15:12:46 <cloudnull> however I am here for this meeting, and will do my best to not be distracted :D
15:12:57 <yoctozepto> cloudnull: thanks
15:13:16 <mgoddard> thanks cloudnull. I added it the end of the agenda
15:13:31 <mgoddard> I've been a bit distracted from kolla this past week
15:13:51 <mgoddard> I saw there were some ubuntu CI issues, possibly mirror related. Still present?
15:14:20 <yoctozepto> still a few hours ago
15:14:38 <yoctozepto> no idea now
15:14:59 <yoctozepto> checked, last failure less than an hour ago
15:15:00 <mnasiadka> Yeah, infra has some AFS problems, I saw they mentioned some lock issues ;-)
15:15:02 <yoctozepto> so still
15:15:23 <yoctozepto> so this week is kolla-release-less definitely
15:15:48 <mnasiadka> yeah, especially we're failing on openssl update from bionic-security ;-)
15:15:54 <yoctozepto> and we drop usage of infra mirrors as a workaround if they don't fix it by Friday lol
15:17:33 <mgoddard> ouch
15:17:37 <mgoddard> hopefully they will
15:17:44 * yoctozepto hopes too
15:17:52 <mgoddard> any other CI issues?
15:17:58 <yoctozepto> nah
15:18:39 <mgoddard> cool
15:18:46 <mgoddard> #topic Train release planning
15:18:54 <mgoddard> from agenda:
15:18:58 <mgoddard> old Erlang for RMQ still scary: https://bugs.launchpad.net/kolla/+bug/1853167 - better update and take the risk of upgrade or leave it b0rken?
15:18:58 <openstack> Launchpad bug 1853167 in kolla ussuri "CentOS RabbitMQ Unsupported Erlang Version" [Medium,Triaged] - Assigned to Radosław Piliszek (yoctozepto)
15:19:12 <yoctozepto> the question is included
15:19:40 <mgoddard> this seems a bit nasty
15:19:52 <yoctozepto> upgrading would bring my plan to keep 3.7.x in train (stein?) but upgrade erlang to a supported version
15:20:13 <yoctozepto> the upgrade in stein was clumsy, discovered lated I know
15:20:19 <yoctozepto> late*
15:20:54 <hrw> so I would need to prepare erlang 22 for aarch64?
15:20:57 <mgoddard> how does one install an updated erlang?
15:21:07 <yoctozepto> mgoddard: rmq provides dedicated binaries
15:21:10 <yoctozepto> for x86_64
15:21:21 <yoctozepto> then we run their supported platform for rmq
15:21:33 <yoctozepto> hrw would need to build for aarch64 to keep it on par
15:21:38 <mgoddard> I assume this comes with all required libraries?
15:21:41 <yoctozepto> otherwise we diverge with compatibility
15:21:45 <yoctozepto> yeah
15:22:15 <yoctozepto> only hipe is gone now, from erlang vm  I mean
15:22:34 <yoctozepto> but that is upstream erlang thingy
15:22:41 <yoctozepto> not much on rmq or us
15:24:05 <mgoddard> would upgrading erlang cause any headaches on upgrades?
15:24:38 <mnasiadka> rabbitmq equals headache I guess - so we can't be certain
15:24:58 <yoctozepto> mgoddard: need to test that
15:25:17 <yoctozepto> considering it's actually erlang vm that does the heavy networking/clustering stuff
15:25:27 <mnasiadka> but the in place upgrade for rmq is 1) check that everything is stable 2) stop node 3) upgrade 4) start node
15:25:30 <yoctozepto> it could well cause some
15:26:00 <mnasiadka> https://www.rabbitmq.com/upgrade.html#rolling-upgrades
15:26:04 <mnasiadka> #link https://www.rabbitmq.com/upgrade.html#rolling-upgrades
15:26:13 <mgoddard> it sounds like we need this long term. we probably need to try it out to decide on the risk/reward for a backport
15:26:28 <mgoddard> is anyone actively working on it?
15:26:57 <yoctozepto> mgoddard: nope but I can
15:27:17 <yoctozepto> mnasiadka: I learn the chat bot collects all http stuff anyway, no #link needed
15:27:21 <yoctozepto> learnt*
15:27:24 <mnasiadka> yoctozepto: allrighty
15:27:42 <yoctozepto> I asked to know if you are not 100% against
15:27:44 <yoctozepto> :D
15:28:00 <yoctozepto> also, it means work for hrw
15:28:16 <hrw> which I would need to do in ussuri anyway
15:28:45 <yoctozepto> right you are, sir
15:28:58 <yoctozepto> just not it becomes urgent ;p
15:29:03 <yoctozepto> now*
15:29:07 <osmanlicilegi> yoctozepto: I can adopt this bug if you let me. you know I have the upgrade bp also.
15:29:55 <yoctozepto> osmanlicilegi: yeah, you are welcome to - could you deliver something by tomorrow?
15:30:38 <osmanlicilegi> I will
15:31:01 <mnasiadka> yoctozepto: around erlang - have you tried to talk with the RDO guys to bump it up in RDO? (looking at the bug it's a bit non-compatible?)
15:31:23 <yoctozepto> mnasiadka: we are using another source of rabbitmq
15:31:37 <mnasiadka> ah, packagecloud I guess
15:31:42 <yoctozepto> not checked rdo version
15:31:49 <yoctozepto> maybe they package the right combination
15:32:26 <mnasiadka> yoctozepto: well, RDO provides rmq 3.6.16
15:32:33 <mnasiadka> I doubt you are interested in that :)
15:32:50 <yoctozepto> you have answered all the questions
15:33:19 <mnasiadka> 3.6 is end of life from May 2018? huh
15:33:37 <mnasiadka> https://www.rabbitmq.com/versions.html
15:36:05 <mgoddard> ok, let's move on. Thanks to osmanlicilegi for picking it up, and yoctozepto for offering.
15:37:18 <mgoddard> Are there any other train blockers?
15:37:31 <mgoddard> Or should we consider putting out a GA release soon?
15:37:44 <mgoddard> https://launchpad.net/kolla-ansible/+milestone/9.0.0
15:38:06 <mgoddard> https://bugs.launchpad.net/kolla-ansible/+bug/1853201 is marked high
15:38:06 <openstack> Launchpad bug 1853201 in kolla-ansible ussuri "deploy fails with qinling_dev_mod is undefined" [High,In progress] - Assigned to Radosław Piliszek (yoctozepto)
15:38:25 <yoctozepto> mgoddard: already addressed
15:38:30 <yoctozepto> we add qinling in Train
15:38:38 <yoctozepto> and it seems b0rken since inception ;p
15:38:45 <mgoddard> interestingly, seems people run train on xenial
15:39:19 <yoctozepto> yeah, saw that
15:39:20 <mgoddard> well, needs to merge & backport to get into the releaase
15:39:24 <yoctozepto> though it does not break qinling
15:39:28 <yoctozepto> yeah
15:39:35 <yoctozepto> and it is still b0rken after that
15:39:35 <mgoddard> marked RP+1
15:39:36 <yoctozepto> https://storage.bhs1.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_593/695192/3/check/kolla-ansible-ubuntu-source-qinling/59377d1/primary/logs/ansible/deploy
15:39:50 <yoctozepto> so we need a patchset to get qinling to work
15:39:56 <yoctozepto> not tested = not working
15:40:05 <yoctozepto> cite me
15:40:31 <mgoddard> is that CI test new?
15:40:48 <mgoddard> added to reproduce?
15:40:53 <yoctozepto> yeah, I am trying to reproduce and fix
15:41:02 <mnasiadka> well, if we would have a support matrix for k-a, we could state qinling is not tested, and only community supported == bug priority is low
15:41:28 <yoctozepto> mnasiadka: still, I feel bad having Qinling in prelude and knowing it failing ;p
15:41:34 <mgoddard> goldyfruit_ was the author of qinling, right
15:41:38 <mgoddard> around goldyfruit_?
15:42:35 <mgoddard> are you planning to get qinling fixed before release yoctozepto?
15:42:53 <yoctozepto> yeah, that's my plan indeed
15:42:56 <mgoddard> ok
15:42:58 <mnasiadka> well, then we could have a CI with Magnum, Qinling and so on - I guess, but it wasn't done as part of pushing the actual code
15:42:59 <yoctozepto> the code looked nice overally
15:43:05 <yoctozepto> should not be broken completely
15:43:13 <yoctozepto> let it at least deploy and answer on API :D
15:43:28 <mgoddard> sure, that's a good start
15:43:35 <yoctozepto> ;P
15:43:39 <mgoddard> would be nice to get your CI test committed
15:43:56 <yoctozepto> yeah, when it is ready for it
15:44:21 <yoctozepto> now I am a bit glad that we might be bumping rmq for centos
15:44:32 <yoctozepto> it enables ipv6 as long as one does not need mariadb ha
15:44:37 <yoctozepto> ;D
15:44:53 <yoctozepto> makes our support sound much nicer
15:45:06 * yoctozepto caring for its baby
15:45:15 <mgoddard> I'll go through the in progress bugs and add RP+1 where possible
15:45:18 <yoctozepto> his*
15:45:32 <yoctozepto> geez, I write weird today
15:45:42 <mgoddard> #topic Ussuri release planning
15:45:45 <yoctozepto> mgoddard: nice, I tried keeping up with it
15:45:57 <goldyfruit_> mgoddard, o/
15:45:58 <yoctozepto> but you are welcome to fill-in the gaps :-)
15:46:05 <goldyfruit_> yoctozepto, o/
15:46:08 <yoctozepto> goldyfruit_: hi
15:46:14 <goldyfruit_> What's wrong with Qinling?
15:46:31 <mgoddard> oh hai goldyfruit_
15:46:33 <goldyfruit_> We are using the master/train code from kolla-ansible to deploy Qinling
15:46:47 <yoctozepto> goldyfruit_: oddly, it does not deploy :-)
15:46:56 <yoctozepto> unless you have the missing somewhere else
15:46:58 <goldyfruit_> Hum
15:47:04 <yoctozepto> like in your inventories :-)
15:47:17 <mgoddard> goldyfruit_: https://bugs.launchpad.net/bugs/1853201
15:47:17 <openstack> Launchpad bug 1853201 in kolla-ansible ussuri "deploy fails with qinling_dev_mod is undefined" [High,In progress] - Assigned to Radosław Piliszek (yoctozepto)
15:47:30 <yoctozepto> and after fixing that, another: https://storage.bhs1.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_593/695192/3/check/kolla-ansible-ubuntu-source-qinling/59377d1/primary/logs/ansible/deploy
15:47:37 <mgoddard> maybe you can work with yoctozepto to get it fixed
15:47:55 <yoctozepto> yeah, I appreciate being around for my questions if need be :-)
15:48:07 <mgoddard> let's move on
15:48:16 <goldyfruit_> Will do my best to be around
15:48:25 <mgoddard> #info Ussuri priority voting is finished
15:48:29 <mgoddard> #link https://bugs.launchpad.net/bugs/1853201
15:48:29 <openstack> Launchpad bug 1853201 in kolla-ansible ussuri "deploy fails with qinling_dev_mod is undefined" [High,In progress] - Assigned to Radosław Piliszek (yoctozepto)
15:48:30 <mgoddard> #undo
15:48:31 <openstack> Removing item from minutes: #link https://bugs.launchpad.net/bugs/1853201
15:48:37 <mgoddard> #link https://etherpad.openstack.org/p/kolla-ussuri-priorities
15:48:54 <mgoddard> I've ordered them, but they need some tidying up
15:49:15 <mgoddard> remove our PTG discussion, replace with owners & status etc.
15:50:03 <yoctozepto> mgoddard: will you handle this?
15:50:07 <mgoddard> yes
15:50:11 <yoctozepto> cool
15:50:22 <mgoddard> #action mgoddard to tidy up ussuri priorities and add to whiteboard
15:50:32 <yoctozepto> I added ceilosca after the voting to let people know it is a valid option
15:50:35 <mgoddard> first impressions - there are a lot of things listed
15:50:42 <yoctozepto> ;D
15:51:00 <mgoddard> maybe we should just have one liners for anything with X or less votes?
15:51:21 <mgoddard> doesn't mean we won't do it
15:51:53 <mnasiadka> well, ideally maybe not to pollute the whiteboard - we should have a bp link and an owner?
15:52:11 <osmanlicilegi> sounds better
15:52:15 <yoctozepto> +1
15:52:50 <mgoddard> means we won't have it all in one place
15:53:05 <mgoddard> will be hard to see the TODO/unassigned tasks
15:54:12 <mgoddard> I'll see how it looks after the tidy up
15:54:41 <mnasiadka> well, you click in the bp, and we can have dependencies, larger chunks of work tidied up and so on
15:54:53 <mgoddard> oh ok
15:54:59 <mnasiadka> I just feel that we are too lazy to update the statuses in the whiteboard :)
15:55:05 <mgoddard> yes
15:55:21 <mnasiadka> and if changes are bound to bp - those will be visible on the bp itself
15:55:24 <mgoddard> and blueprints will not be automatically updated :)
15:55:28 <goldyfruit_> the bug in qinling has been introduced by this: https://github.com/openstack/kolla-ansible/commit/e610a73e9863bcc63f6e6c907f7eb654aa1ee37f#diff-669ed79df1cdc81f33b7527aa8e1faba
15:56:21 <mgoddard> #topic Image consumers
15:56:23 <mnasiadka> mgoddard: well, something will be updated there :)
15:56:26 <mgoddard> cloudnull: ping
15:56:34 <cloudnull> o/
15:56:43 <mgoddard> yoctozepto: you're up
15:56:58 <yoctozepto> goldyfruit_: yeah, I felt you would not propose something broken right away :-)
15:57:07 <yoctozepto> mgoddard: sure
15:57:27 <yoctozepto> so, I had this idea that we could document kolla consumers
15:57:46 <yoctozepto> mostly about which images are used
15:58:01 <yoctozepto> consumers actually consist of tripleo and k-a
15:58:05 <yoctozepto> so we do the k-a part
15:58:11 <yoctozepto> but someone needs to do the tripleo part
15:58:33 <yoctozepto> and, more importanlty, such that it will be kept up to date
15:58:41 <cloudnull> OK. so you're looking for the list of images we use?
15:58:49 <mgoddard> https://docs.openstack.org/kolla/latest/support_matrix.html
15:59:29 <yoctozepto> list of images is a start, but we would prefer to get consensus on how to keep it current
16:00:18 <cloudnull> hum. how is that current list generated?
16:00:37 <mnasiadka> it's not, but there's room for improvement - it's a csv :)
16:00:44 <cloudnull> OK
16:01:24 <cloudnull> hum...
16:01:55 <mgoddard> well that list is our support matrix. I think we're looking at a new table on that page
16:01:59 <cloudnull> I can get you a list, and we can provide updated lists for new releases, but updating that CSV may be difficult from our point of view
16:02:25 <yoctozepto> we can do either in csv
16:02:31 <mgoddard> I think what we'd be looking at is a list of:
16:02:38 <cloudnull> in an automated fashion that is
16:02:55 <mgoddard> <service> <TripleO (Y/N)> <Kolla Ansible (Y/N)>
16:03:42 <yoctozepto> +1
16:03:48 <yoctozepto> if changes are rare
16:03:56 <yoctozepto> we can as well synchronize on our cycle end
16:04:03 <openstackgerrit> Viktor Michalek proposed openstack/kolla-ansible master: Adds support of CADF notifying for other APIs next to keystone.  https://review.opendev.org/674579
16:04:09 <mgoddard> an alternative is that tripleo provides a list of containers they use which we could link to
16:04:48 <mnasiadka> cloudnull: can't we generate a list of container images used by tripleo and then transform them into a csv that could be rendered by sphinx-doc?
16:04:59 <mgoddard> Skipping back, I think the goal here is to have a place to check which downstream projects will be affected by a change to an image
16:05:23 <cloudnull> mnasiadka you could
16:05:49 <yoctozepto> mgoddard: yeah, exactly
16:07:12 <mnasiadka> mgoddard: you were investigating the proposal bot for something similar, right?
16:07:13 <cloudnull> if the goal is to notify potential consumers, then maybe a release note is all that is really required ?
16:07:53 <yoctozepto> cloudnull: they would need aggregating
16:07:59 <mnasiadka> if somebody really reads the release notes... :)
16:08:08 <mgoddard> cloudnull: I think we were hoping to encourage feedback earlier in the cycle than that
16:08:26 <mgoddard> rather than hit you with it
16:08:30 * cloudnull reads release notes
16:08:36 <mgoddard> then find it doesn't work for tripleo and we need to revert
16:08:44 <cloudnull> ++
16:08:49 <cloudnull> this makes some sense
16:10:00 <mgoddard> anyway, we're out of time. Hopefully we've got the problem across. Implementation details can be determined down the line
16:10:11 <cloudnull> I will work on getting you a list of images
16:10:25 <cloudnull> and an easy way to pull that info
16:10:29 <mgoddard> thanks
16:10:35 <cloudnull> something that we will maintain as a downstream
16:10:51 <yoctozepto> thanks, cloudnull
16:11:10 <mgoddard> thanks all
16:11:12 <mgoddard> #endmeeting