15:01:16 <mgoddard> #startmeeting kolla
15:01:17 <openstack> Meeting started Wed Feb 12 15:01:16 2020 UTC and is due to finish in 60 minutes.  The chair is mgoddard. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:01:18 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:01:21 <openstack> The meeting name has been set to 'kolla'
15:01:32 <mgoddard> #topic rollcall
15:01:33 <hrw> /o\
15:01:37 <mgoddard> \o/
15:01:46 <yoctozepto> /o/
15:02:00 <mgoddard> \o\
15:02:01 <mnasiadka> o/
15:02:06 <well100> #info
15:02:29 <well100> #link
15:02:46 <yoctozepto> well100: chair only I believe
15:02:47 <mgoddard> well100: those commands are for the IRC meeting
15:02:50 <cosmicsound> o/
15:02:54 <hrw> well100: you are not chairing so stop
15:03:04 <mgoddard> feel free to join in though
15:03:30 <mgoddard> #topic agenda
15:03:32 <mgoddard> * Roll-call
15:03:34 <mgoddard> * Announcements
15:03:35 <well100> okay sorry
15:03:36 <mgoddard> * Review action items from last meeting
15:03:38 <mgoddard> * CI status
15:03:40 <mgoddard> * Ussuri release planning (kolla & kolla ansible)
15:03:42 <mgoddard> * Ussuri release planning (kayobe)
15:03:44 <mgoddard> * Kolla SIG (aka Kolla Klub?) https://etherpad.openstack.org/p/kolla-sig
15:03:46 <mgoddard> #topic announcements
15:04:10 <mgoddard> #info CentOS 8 CI jobs merged (mostly)
15:04:48 <mgoddard> Any others?
15:04:51 <hrw> Infra patchset passed first k-a tests. TripleO team informed that images can be renamed. Cloudnull added as reviewer.
15:05:11 <yoctozepto> \o\
15:05:15 <mgoddard> Nice
15:05:23 <mgoddard> will we be able to avoid a hard cutover?
15:05:30 <hrw> I plan to reenable all CI tests and check how it goes
15:05:43 <hrw> mgoddard: no. it is all or nothing now.
15:05:55 <mgoddard> would it be possible?
15:06:00 <hrw> mgoddard: a way to --go-old-way-without-infra-ones can be added on top
15:06:14 <hrw> it is just a matter of not calling rewrite_dockerfiles ;D
15:06:42 <mgoddard> ok. We can discuss later
15:07:00 <mgoddard> #topic Review action items from last meeting
15:07:17 <mgoddard> osmanlicilegi to fix rocky ubuntu build jobs
15:07:18 <mgoddard> mgoddard to add a TODO list to centos8 status page
15:07:20 <mgoddard> mgoddard to add features for kolla and kayobe to openstack-map
15:07:22 <mgoddard> yoctozepto to drop cinder-lvm CI and job and trigger zun job on cinder-lvm role changesy
15:07:32 <yoctozepto> done
15:07:33 <yoctozepto> ;D
15:07:36 <mgoddard> osmanlicilegi did his
15:07:39 <yoctozepto> train merging
15:07:44 <mgoddard> yoctozepto did his
15:07:52 <yoctozepto> oh yes, he did
15:07:52 <mgoddard> mgoddard did his
15:07:55 <mgoddard> \o/
15:08:00 <yoctozepto> \o\
15:08:02 <mgoddard> good actioning team
15:08:03 <yoctozepto> /o/
15:08:09 <yoctozepto> |o|
15:08:09 <mgoddard> #topic CI status
15:08:21 <mgoddard> Virtualenv...
15:08:22 <mnasiadka> kolla is bleeding red due to virtualenv mumbo jumbo
15:08:40 <mnasiadka> kolla change with workaround: https://review.opendev.org/#/c/707363/
15:08:40 <patchbot> patch 707363 - kolla - Upgrade virtualenv in pre - 6 patch sets
15:08:50 <mnasiadka> kolla-ansible change with workaround: https://review.opendev.org/#/c/707377/
15:08:51 <patchbot> patch 707377 - kolla-ansible - Upgrade virtualenv in pre - 1 patch set
15:09:19 <mnasiadka> kolla-ansible itself is fine, but when kolla launches kolla-ansible changes that build something - it fails
15:09:22 <mnasiadka> so we need to patch both
15:09:37 <yoctozepto> mnasiadka: k-a confirmed fixing?
15:10:01 <mnasiadka> yoctozepto: did a depends-on of the kolla change on the k-a change - it wouldn't work in other way :)
15:10:17 <mnasiadka> well, kolla change depends-on k-a change
15:10:41 <mnasiadka> that sounds better
15:11:00 <yoctozepto> mnasiadka: ack
15:11:10 <yoctozepto> mnasiadka: quick summary why only k is hit and not k-a?
15:12:02 <mnasiadka> yoctozepto: we don't use tox in kolla-ansible, and that's where it fails now (virtualenv 20.0.1 depends on six 1.12.0 or newer - and six is not installed in the infra centos7/8 image)
15:12:35 <mnasiadka> as long as you dont use --system-site-packages it's fine
15:12:37 <yoctozepto> mnasiadka: what about tox jobs?
15:12:43 <mnasiadka> and we only use it in k-a in one place - ceph-ansible
15:12:44 <yoctozepto> they are there in k-a as well
15:12:56 <mnasiadka> yoctozepto: the tox jobs from template run on Ubuntu
15:13:06 <yoctozepto> mnasiadka: ah, centos is the key
15:13:09 <yoctozepto> mnasiadka: thank you
15:14:05 <mnasiadka> and since all devstack jobs also run on Ubuntu - it's mainly us and other deployment projects using centos (like tripleo)
15:14:22 <mnasiadka> I guess we can end this thread
15:14:30 <mnasiadka> mgoddard: carry on!
15:14:58 <mgoddard> #topic Ussuri release planning (kolla & kolla ansible)
15:15:13 <mgoddard> hrw: want to carry on discussing infra images?
15:15:33 <hrw> what to discuss? :D
15:15:47 <hrw> basic stuff works. all-in-one can be deployed and VM started
15:15:59 <hrw> basic k-a tests passed.
15:16:35 <hrw> patch uses f-strings so py3.6+ required
15:16:56 <mnasiadka> that should be fine, we shouldn't be strict on keeping py2 compatibility
15:17:10 <hrw> it's ussuri material so I do not care about py2 anymore
15:17:27 <hrw> while we are still py2 compatible for tripleo so it has to wait
15:17:41 <hrw> I would prefer to merge infra once we have c8 working
15:18:20 <mgoddard> I'd like to know how we could avoid a hard cutover, as it would break k-a and tripleo
15:18:21 <hrw> have to check how aarch64 cope with change as it now needs to build far more images in one job
15:18:38 <mgoddard> why does it build more images?
15:18:40 <hrw> mgoddard: there is patch for both kolla and kolla-ansible already
15:18:52 <hrw> mgoddard: you need to build binary+source on one machine
15:18:59 <mgoddard> why?
15:19:16 <hrw> if you want to push then infra need to be same
15:19:56 <hrw> you run binary build. it gives you infra and binary images. then you run source. you get source ones and infra ones are recreated from local cache == same as binary. then you push (or not)
15:20:01 <mgoddard> how much longer do jobs take?
15:20:25 <hrw> on x86-64 we can ignore time
15:20:53 <hrw> build takes less than 40 minutes in total
15:21:05 <hrw> on aarch64 it was about 2h ;(
15:21:08 <mnasiadka> oh my, can't we make it simpler and just move the infra images to docker/infra subfolder, and add an option to tag them centos-infra centos-binary centos-source in the interim period?
15:21:31 <mnasiadka> (centos is only an example)
15:21:41 <hrw> mnasiadka: fetch patch, do build, take a look how it looks.
15:21:51 <yoctozepto> I am more worried about stability
15:21:57 <yoctozepto> since we fail early
15:22:13 <yoctozepto> we migth be losing precious results
15:22:17 <mgoddard> other options are 4 dependent jobs: base, infra, binary, source
15:22:17 <yoctozepto> due to binary failures
15:22:38 <hrw> for now image build logs are stored in same files
15:22:38 <yoctozepto> makes sense
15:22:46 <hrw> this is something I would like to change
15:23:03 <hrw> mgoddard: kolla does not pull other images then one used for base.
15:23:21 <yoctozepto> yes, it does not
15:23:28 <yoctozepto> and..? :D
15:23:29 <hrw> mgoddard: you suggest adding 'check is image you are about to use as parent available in registry' for each image
15:23:34 <mnasiadka> hrw: I just understood we are building twice - binary and source, and checking which one works :)
15:23:45 <yoctozepto> mnasiadka: haha
15:23:47 <hrw> mnasiadka: no.
15:24:07 <yoctozepto> is that "skip existing" already fixed?
15:24:21 <mgoddard> hrw: you can use --skip-existing
15:24:24 <hrw> yoctozepto: it works since UNBUILDABLE iirc
15:24:29 <yoctozepto> hrw: ack
15:24:36 <hrw> mgoddard: on fresh machine? good luck
15:24:41 <mgoddard> why?
15:24:46 <mgoddard> docker pull <image
15:24:47 <yoctozepto> ?
15:24:59 <hrw> ok. let me say other way
15:25:27 <hrw> do myself a favour: fetch patch, do build (binary+source), do all-in-one deploy, check how it looks.
15:26:08 <mgoddard> hrw: do us a favour and help us to discuss your feature
15:26:09 <hrw> it is very small change. things should work properly. probably upgrade from train will require some stuff
15:26:14 <hrw> moment..
15:26:24 <mgoddard> it's a small change but with big consequences
15:27:26 <mgoddard> ok, let's move on
15:27:34 <mgoddard> Any other features we want to discuss?
15:27:47 <hrw> had to handle a call
15:28:21 <mnasiadka> about features - let's focus on c8 and getting rid of ceph, once this is stable I have some patches I need reviews on :)
15:28:38 <hrw> mnasiadka: +myljon
15:29:18 <mnasiadka> mgoddard: https://review.opendev.org/#/c/707377/ care to unblock the CI? :)
15:29:19 <patchbot> patch 707377 - kolla-ansible - Upgrade virtualenv in pre - 1 patch set
15:29:31 <mgoddard> mnasiadka: was waiting for zuul
15:30:07 <mgoddard> zuul is happy, so am I
15:30:33 <mnasiadka> one ceph job failed on mariadb, second one on reconfigure (keystone auth timeout)
15:30:38 <mgoddard> ok, let's peruse the C8 status page
15:30:45 <mgoddard> https://etherpad.openstack.org/p/kolla-centos8
15:30:47 <mgoddard> now with TODOs
15:30:55 <openstackgerrit> Marcin Juszkiewicz proposed openstack/kolla-ansible master: WIP: Use 'infra' type of images  https://review.opendev.org/706982
15:31:41 <hrw> mgoddard: +2 on dropping mongodb
15:31:57 <mgoddard> should we deprecate first?
15:32:13 <mgoddard> given that ceilometer may use it
15:32:27 <mgoddard> we could just drop for centos
15:32:27 <mnasiadka> didn't RHEL8/CentOS8 remove mongodb? or just stopped pushing in new versions?
15:32:38 <yoctozepto> and even suggesting to use it in place of gnocchi
15:32:42 <hrw> debian and centos8 just lack mongodb
15:32:57 <mnasiadka> I remember there were some licensing issues
15:33:08 <mgoddard> yeah they changed their license
15:33:08 <yoctozepto> yeah, mongodb went nuts with licensing
15:33:11 <mnasiadka> so then only Ubuntu would come with MongoDB? that's a bit weird
15:33:17 <mgoddard> yeah
15:33:27 <mgoddard> we can deprecate in U
15:33:32 <yoctozepto> drop and deprecate
15:33:33 <hrw> mnasiadka: since when ubuntu cares about licenses?
15:33:38 <yoctozepto> and drop
15:33:38 <yoctozepto> :D
15:33:44 <mnasiadka> hrw: since when Ubuntu cares about anything? :D
15:33:46 <yoctozepto> hrw: openstack cares
15:33:56 <mgoddard> who wants to deprecate mongo?
15:33:57 <yoctozepto> right, openstack? ;-)
15:34:17 <yoctozepto> o/
15:34:19 <mnasiadka> mgoddard: so, deprecating means it still works in Ubuntu, and we don't deliver in CentOS/Debian, and then we remove it in V?
15:34:31 <yoctozepto> +1 to mnasiadka plan
15:34:39 <hrw> +1
15:34:39 <mgoddard> #action yoctozepto to deprecate mongodb for ubuntu, drop for centos
15:34:45 <yoctozepto> doh
15:34:48 <yoctozepto> ;D
15:34:57 <yoctozepto> all righty
15:35:08 <mgoddard> have I got that wrong?
15:36:03 <yoctozepto> mgoddard: no, it's fine
15:36:10 <mgoddard> what's the next priority for CentOS 8?
15:36:19 <mgoddard> backport to Train?
15:36:30 <mgoddard> that would allow us to get upgrade jobs going
15:36:33 <yoctozepto> prime time
15:36:37 <mgoddard> or migration from c7?
15:36:51 <hrw> mgoddard: c8/train
15:37:02 <mgoddard> I've moved onto kayobe c8 support, if anyone wants to pick up kolla that would be good
15:37:04 <hrw> as we need c8/train to think of migration from c7/train
15:37:32 <mgoddard> although we currently have c7 and c8 on master so could test migration there and then backport
15:37:58 <mgoddard> also not clear if c8 train RDO repos are available yet
15:37:59 * hrw deep in stein py2 hell
15:38:14 <yoctozepto> hrw: what's in there? ;D
15:38:31 <mnasiadka> mgoddard: https://trunk.rdoproject.org/centos8-train/current/
15:38:55 <mgoddard> cool
15:38:57 <hrw> yoctozepto: nasty kind of forwardports from oldstable to stable
15:39:18 <yoctozepto> hrw: ;-(
15:39:45 <hrw> yoctozepto: or py2 builds of py3 packages etc.
15:40:18 <hrw> yoctozepto: we (linaro) need set of stein debian/source images to migrate rocky->stein->train without staying on stein for longer than minutes
15:40:31 <mnasiadka> mgoddard: I can start the backporting, got some spare cycles this week
15:41:05 <mnasiadka> given the ceph stuff is on good way forward and we have a change even to move from ceph chacra repo to Storage SIG nautilus builds
15:41:58 <yoctozepto> very well
15:42:10 <hrw> mnasiadka: you can always start c7->c8 migration guide with 'first you need to migrate to external ceph' ;D
15:42:34 <mnasiadka> yeah, that point is awesome
15:42:44 <yoctozepto> ah, that needs docs as well...
15:42:56 <mnasiadka> and then I can write you can use ceph-ansible, although it doesn't have a bright future
15:42:57 <mnasiadka> :D
15:43:02 <mgoddard> mnasiadka: I already pushed a few backports. Need to check commit IDs as I think not all were merged at the time
15:43:18 <mnasiadka> mgoddard: good, awaiting status report where to start :)
15:43:59 <mgoddard> mnasiadka: https://review.opendev.org/#/q/status:open+project:openstack/kolla+branch:stable/train+topic:bp/centos-rhel-8
15:44:18 <mnasiadka> goodie
15:45:23 <mgoddard> mnasiadka: ceph removal status?
15:46:11 <mnasiadka> mgoddard: the change in k-a is probably in good state - given your last review, I'll double check and run a recheck if needed
15:46:41 <mgoddard> great
15:46:48 <mgoddard> #topic Ussuri release planning (kayobe)
15:46:56 <mgoddard> priteau, dougsz, Wasaac
15:47:06 <priteau> o/
15:47:10 <mgoddard> What's going on in kayobe land?
15:47:19 <dougsz> \o
15:47:20 <mgoddard> We've had quite a bit of interest recently
15:47:25 <priteau> I heard mgoddard is fixing all the c8 bugs
15:47:31 <mgoddard> lol
15:47:31 <dougsz> +1
15:47:50 <mgoddard> slowly working my way through some
15:48:26 <mgoddard> we have a perfect storm of py2 drop, poor py3 support and centos8
15:48:38 <mgoddard> trying to work out how to navigate through
15:48:49 <mgoddard> I added a section to https://etherpad.openstack.org/p/kolla-centos8 for kayobe
15:48:54 <mgoddard> (just above todo)
15:50:03 <mgoddard> priteau, dougsz: do you have any kayobe work planned?
15:50:30 <dougsz> nothing planned
15:50:40 <priteau> Nothing specific to Kayobe on my todo list for now except helping with py3/c8
15:51:14 <mgoddard> ok
15:51:19 <priteau> It would be nice to get the multiple environments support soon though
15:51:23 <mgoddard> +1
15:51:40 <priteau> That was a tie for c8 / py3 during our train planning meeting
15:51:44 <mgoddard> we should try to squash some bugs from time to time
15:52:03 <mgoddard> we have a few that should be easy enough to fix
15:52:16 <mgoddard> https://storyboard.openstack.org/#!/project/openstack/kayobe
15:53:09 <mgoddard> if you have some free time, please pick one up
15:53:11 <mgoddard> #topic Kolla SIG (aka Kolla Klub?) https://etherpad.openstack.org/p/kolla-sig
15:53:36 <mgoddard> This is an idea I've been thinking about for a while. I think it's time to make it real
15:53:45 <mgoddard> #link https://etherpad.openstack.org/p/kolla-sig
15:54:02 <mgoddard> have a read, let me know your thoughts
15:55:32 <hrw> mgoddard: kind of 'let grab some users of kolla projects without making them developers', right?
15:55:44 <mgoddard> hrw: yeah
15:55:52 <hrw> +2
15:56:23 <mgoddard> people who might not have time or interest for weekly IRC meeting, but still want a regular opportunity to meet with the community
15:56:39 <mgoddard> more about operators than developers
15:56:45 <hrw> yep
15:56:48 <hrw> kevinz: ^^
15:56:57 <mgoddard> could be called kolla operator SIG
15:57:03 <hrw> even better
15:57:07 <cosmicsound> +1
15:57:07 <mgoddard> suggestions for names welcome
15:57:28 <mgoddard> I've called it a SIG. Should it be an openstack SIG?
15:57:48 <hrw> no idea how openstack handles SIGs
15:58:10 <mgoddard> https://governance.openstack.org/sigs/
15:59:01 <hrw> thanks
15:59:23 <mgoddard> yoctozepto, mnasiadka: awake?
15:59:42 <yoctozepto> back
15:59:51 <yoctozepto> kolla sig is fine
15:59:55 <yoctozepto> great idea btw
15:59:57 <mnasiadka> Yeah, I'm awake - I don't know if a project-bound SIG would be easily accepted - but I guess why not
16:00:10 <yoctozepto> let's have a biweekly video call
16:00:11 <mgoddard> there is a baremetal sig
16:00:14 <hrw> mnasiadka: name it Operators SIG
16:00:27 <mnasiadka> It's a good idea, and lately I've seen new people coming here - so probably they would be happy
16:00:37 <mgoddard> "SIGs are a good match for an activity that centers around a topic or practice that spans all the community (developers, operators, end users…), by forming a guild of people with a shared interest."
16:01:07 <mnasiadka> it doesn't need to be even video calls - a newsletter would be nice, some more human version of release notes :)
16:01:45 <mgoddard> I'd like it to be interactive
16:02:03 <mgoddard> we could do a newsletter on the mailing list without a SIG
16:02:33 <mgoddard> but I'd like this to improve the personal side of a global community
16:02:49 <mgoddard> i.e. what might normally happen at conferences
16:03:11 <yoctozepto> mgoddard: exactly why I propose a video call
16:03:14 <mgoddard> +1
16:03:20 <yoctozepto> we might test out different solutions for conferencing
16:03:25 <mgoddard> it doesn't need to be the same format each time
16:03:28 <yoctozepto> to make vtpgs go smoothly
16:03:30 <yoctozepto> ;D
16:03:42 <yoctozepto> yeah, sometimes it's going to be a comedy, sometimes drama
16:04:39 <mgoddard> anyway, we're over time.
16:04:51 <yoctozepto> thanks mgoddard
16:05:06 <mgoddard> sounds like a positive response. Please air any thoughts on the etherpad or via IRC
16:05:24 <mgoddard> then I'll draft a proposal next week
16:05:44 <mgoddard> thanks all
16:05:46 <mgoddard> #endmeeting