15:01:33 <mgoddard> #startmeeting kolla
15:01:34 <openstack> Meeting started Wed Mar 11 15:01:33 2020 UTC and is due to finish in 60 minutes.  The chair is mgoddard. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:01:35 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:01:37 <openstack> The meeting name has been set to 'kolla'
15:01:39 <mgoddard> #topic rollcall
15:01:41 <mgoddard> \o
15:01:48 <yoctozepto> o/
15:01:50 <hrw> /o\
15:02:19 <yoctozepto> mnasiadka: ?
15:02:24 <yoctozepto> osmanlicilegi: ?
15:02:43 <yoctozepto> (sad Christian does not join us)
15:02:46 <cosmicsound> yoctozepto , heya
15:03:14 <yoctozepto> cosmicsound: hey-hey (meeting running)
15:04:08 <mgoddard> #topic agenda
15:04:15 <mgoddard> * Roll-call
15:04:16 <mgoddard> * Announcements
15:04:18 <mgoddard> ** CentOS 7 jobs dropped on master
15:04:20 <mgoddard> ** Kolla SIG/club signup: https://etherpad.openstack.org/p/kolla-sig
15:04:22 <mgoddard> * Review action items from last meeting
15:04:24 <mgoddard> * CI status
15:04:26 <mgoddard> * infra images - what to do with patches? postpone to V proposal
15:04:28 <mgoddard> * https://review.opendev.org/698753 adds support for Kolla
15:04:28 <patchbot> patch 698753 - kolla - WIP: Create 'infra' type of images - 19 patch sets
15:04:30 <mgoddard> * https://review.opendev.org/707599 adds 'non-infra-base' image (not required for ^^ but can be useful to show split)
15:04:30 <patchbot> patch 707599 - kolla - WIP: introduce non-infra-base image - 5 patch sets
15:04:32 <mgoddard> * https://review.opendev.org/706982 k-a patch
15:04:32 <patchbot> patch 706982 - kolla-ansible - WIP: Use 'infra' type of images - 8 patch sets
15:04:34 <mgoddard> * Kolla --SIG-- (aka Kolla Klub?) https://etherpad.openstack.org/p/kolla-sig
15:04:36 <mgoddard> * Ussuri release planning (kayobe)
15:04:38 <mgoddard> * Ussuri release planning (kolla & kolla ansible)
15:04:40 <mgoddard> #topic announcements
15:04:59 <mgoddard> #info CentOS 7 jobs dropped on master
15:05:22 <mgoddard> While we have not removed support for CentOS 7, prepare for it to break at any time
15:05:29 <hrw> and we have https://review.opendev.org/692450 to eradicate c7 from master
15:05:30 <patchbot> patch 692450 - kolla - WIP: Remove support for CentOS 7 - 11 patch sets
15:06:25 <yoctozepto> ++
15:06:41 <mnasiadka> yoctozepto: busy like hell
15:06:48 * hrw at kolla and horizon meetings at same time
15:06:50 <mgoddard> true. Will need something similar for kolla-ansible at some point
15:07:08 <mgoddard> #info Kolla SIG/club signup open
15:07:10 <mgoddard> #link Kolla SIG/club signup: https://etherpad.openstack.org/p/kolla-sig
15:07:14 <mgoddard> #undo
15:07:14 <openstack> Removing item from minutes: #link https://etherpad.openstack.org/p/kolla-sig
15:07:21 <mgoddard> #link https://etherpad.openstack.org/p/kolla-sig
15:08:08 <mgoddard> All kolla users/operators/developers welcome - will discuss more later
15:08:30 <mgoddard> Any others?
15:08:40 <hrw> nope
15:08:55 <mgoddard> #topic Review action items from last meeting
15:09:07 <mgoddard> dougsz to write bug report about nova SSH nproc issue
15:09:15 <dougsz> dougsz finally did it
15:09:25 <mgoddard> thanks
15:09:41 <mgoddard> #link https://bugs.launchpad.net/bugs/1866190
15:09:42 <openstack> Launchpad bug 1866190 in kolla-ansible "Number of SSH connections can be restricted by nproc" [Undecided,New] - Assigned to Doug Szumski (dszumski)
15:09:55 <mgoddard> #topic CI status
15:10:33 <mgoddard> #info Kolla CI broken due to horizon, pyscss & setuptools 46
15:10:51 <mgoddard> #link https://bugs.launchpad.net/kolla/+bug/1866961
15:10:51 <openstack> Launchpad bug 1866961 in kolla ussuri "ImportError: cannot import name 'Feature'" [Critical,Triaged]
15:10:59 <hrw> we are going to discuss it with horizon team at the end of their meeting (same time as kolla one)
15:11:06 <yoctozepto> with hrw on #openstack-meeting-alt
15:11:24 <mgoddard> joined
15:11:33 <hrw> there is a workaround for it - we can downgrade setuptools to <46 in openstack-base
15:11:44 <hrw> imho for now it's the only way
15:12:03 <mgoddard> well there is a blacklist in requirements, but it won't last long
15:12:10 <hrw> its too late
15:12:18 <hrw> virtualenv gets latest virtualenv
15:12:19 <yoctozepto> we don't read gr
15:12:23 <hrw> virtualenv gets latest setuptools
15:12:34 <mgoddard> hmm
15:12:36 <hrw> so once we use upper-constrains etc we have v46
15:12:56 <yoctozepto> we could reinstall based on gr
15:13:01 <yoctozepto> just like devstack actually does
15:13:50 <hrw> to be honest I prefer to install "setuptools<=46" rather than use GR there.
15:14:01 <hrw> GR will fail once 46.0.1 came
15:14:01 <mgoddard> <46
15:14:06 <hrw> yes. <
15:14:16 <mgoddard> it seems like the best option at the moment
15:14:37 <hrw> ok. will send patch for review after meetings
15:14:41 <yoctozepto> gr does that
15:14:44 <hrw> for master and train
15:14:44 <mgoddard> hopefully distros will not pick up 46
15:14:45 <yoctozepto> <46
15:14:53 <mgoddard> gr does !=46 last I saw
15:14:58 <hrw> yoctozepto: it does now? was !46.0.0
15:14:58 <yoctozepto> hmm
15:15:12 <yoctozepto> sorry, you could be right, I saw too many things
15:15:18 <mgoddard> can't unsee
15:15:51 <hrw> I've seen things you people would not believe... and better do not try to google for them.
15:15:55 <mgoddard> ok. Let's see what horizon say later, but sounds like we have a plan
15:16:08 <yoctozepto> yeah, it was lame
15:16:09 <hrw> next thing?
15:16:15 <mgoddard> something odd happening in Stein&Train regarding in-kolla ceph (yoctozepto)
15:16:28 <yoctozepto> mhm, observed, all jobs fail
15:16:38 <yoctozepto> not ubuntu afair
15:16:40 <yoctozepto> but centos
15:17:36 <mgoddard> doesn't look great
15:17:38 <mgoddard> http://zuul.openstack.org/builds?project=openstack%2Fkolla-ansible&job_name=kolla-ansible-centos-source-ceph#
15:18:12 <mgoddard> last pass on 4th
15:18:14 <hrw> I am not a fan of ceph recently. too many rebuilds
15:19:16 <mgoddard> cat: /var/lib/ceph/osd/a1ad5231-566a-4af1-aa47-038cbccbf2c0/whoami: Permission denied
15:19:20 <mgoddard> https://bae8a59a03658584228a-8f7750086e5a0cc034f577473702e18b.ssl.cf2.rackcdn.com/711905/2/check/kolla-ansible-centos-source-ceph/5e93b6b/primary/logs/ansible/reconfigure
15:19:24 <yoctozepto> weird, eh?
15:19:26 <mgoddard> funky
15:19:33 <mnasiadka> mgoddard: might be related to new packages from Storage SIG
15:19:36 <yoctozepto> did not have time to deal with it
15:19:40 <yoctozepto> oh right
15:19:49 <mnasiadka> raise a bug, assign to me - I'll look into that (but soonest tomorrow)
15:20:12 <mgoddard> #action mgoddard to raise a bug on Ceph job fails
15:20:23 <mgoddard> #action mnasiadka to look into Ceph job fails
15:20:35 * yoctozepto to cheer them up
15:20:45 <Fl1nt> Hi everyone, quick question, let say I want to rebuild a container run command in order to test what's going on with a failing one
15:20:45 <mgoddard> #topic infra images - what to do with patches? postpone to V proposal
15:20:56 <Fl1nt> if I want to use dumb-init
15:21:05 <mgoddard> hi Fl1nt we're just in the meeting atm
15:21:10 <mgoddard> https://review.opendev.org/698753 adds support for Kolla
15:21:10 <patchbot> patch 698753 - kolla - WIP: Create 'infra' type of images - 19 patch sets
15:21:14 <mgoddard> https://review.opendev.org/707599 adds 'non-infra-base' image (not required for ^^ but can be useful to show split)
15:21:15 <patchbot> patch 707599 - kolla - WIP: introduce non-infra-base image - 5 patch sets
15:21:23 <mgoddard> https://review.opendev.org/706982 k-a patch
15:21:23 <patchbot> patch 706982 - kolla-ansible - WIP: Use 'infra' type of images - 8 patch sets
15:21:33 <Fl1nt> @mgoddard, ok sorry, which time will be best to reach out lately?
15:21:40 <hrw> ok, infra stuff
15:21:44 <mgoddard> Fl1nt: in about 40 minutes
15:21:45 <hrw> Fl1nt: in 50 minutes
15:21:51 <yoctozepto> ;d
15:21:53 <Fl1nt> ok noticed that, thx
15:22:05 <hrw> non-infra-base is now dependency of infra patch
15:22:20 <hrw> as it gives nice separation we need
15:22:52 <hrw> it only stores INSTALL_TYPE variable basically as we do not need it in 'infra' images
15:23:05 <hrw> https://blueprints.launchpad.net/kolla/+spec/infra-images has a bit more info
15:23:28 * yoctozepto diving in blueprint
15:23:59 <hrw> with feature freeze in ~month I would prefer to postpone it to V cycle and concentrate our efforts on adding C8 and removing C7
15:24:41 <hrw> as infra images do not give us nothing new basically and require time to test, improve docs etc
15:24:52 <mgoddard> makes sense
15:25:09 <mgoddard> although there may be some small things we could do in U
15:25:18 <hrw> non-infra-base only iirc
15:25:26 <hrw> rest of stuff is self contained and simple
15:25:48 <mgoddard> do we need a non-infra-base image? It looks like it only adds ENV var
15:25:52 <hrw> we need
15:26:06 <hrw> if you have ENV in base then binary-base != source-base
15:26:15 <hrw> and we need ENV for binary/source images
15:26:25 <mgoddard> yeah, but only 2 images depend on it - kolla-toolbox and openstack-base
15:26:29 <mgoddard> we could just add ENV to those
15:26:36 <yoctozepto> +1
15:26:37 <hrw> so non-infra-base is better/easier than injecting ENV to all binary/source images like I did before
15:27:01 <mgoddard> that's not what I'm suggesting
15:27:03 <hrw> mgoddard: ok. can be done simply
15:27:37 <mgoddard> ~unrelated: should kolla-toolbox depend on openstack-base?
15:27:40 <hrw> mgoddard: there can be other binary.source images than those two
15:27:57 <hrw> monasca-thresh and openswitch-netcontrold were on list
15:28:25 <hrw> and we have to rewrite dockerfiles anyway then injecting ENV vars is simple
15:29:11 <hrw> mgoddard: there was attempt to do that
15:29:24 <hrw> mgoddard: k-t is always binary, o-b is binary/source
15:29:29 <mgoddard> netcontrold should be infra
15:30:01 <hrw> it is either binary or source
15:30:08 <mgoddard> currently, yes
15:30:08 <hrw> as 'only one way'
15:30:15 <mgoddard> but it should be infra
15:30:40 <hrw> ok
15:30:57 <mgoddard> monasca-thresh is a weird one because it depends on storm
15:31:21 <mgoddard> might need to make a special case for it
15:31:35 <hrw> there will be some exceptions probably
15:31:39 <mgoddard> yeah
15:31:42 <mgoddard> always :)
15:32:24 <mgoddard> ok, so that's non-infra-base
15:32:39 <mgoddard> there are two hard parts IIRC
15:32:46 <hrw> yes?
15:32:57 <mgoddard> 1. how to build images with a common base
15:33:06 <mgoddard> 2. how to provide a smooth transition
15:33:26 <mgoddard> let's start with 1
15:33:29 <hrw> 1. build binary and then source on same machine, then push
15:33:43 <hrw> the best option
15:33:57 <hrw> otherwise you have to build binary, push, pull, build source, push
15:34:05 <hrw> which result in huge network use
15:34:11 <mgoddard> or
15:34:13 <hrw> which is worse than we have now
15:34:18 <mgoddard> build base, push
15:34:29 <mgoddard> then 3 separate jobs to build binary, source, infra
15:34:39 <hrw> or :D
15:34:41 <hrw> right
15:34:47 <mgoddard> with pipelines
15:34:49 <yoctozepto> mgoddard: ++
15:35:09 <mgoddard> CI is one thing
15:35:15 <mgoddard> we should also think about what users will want
15:35:29 <mgoddard> generally they do not want to build both source & binary
15:35:33 <hrw> --noinfra switch gives old style images
15:35:58 <hrw> and then for k-a we need to implement "use_infra_images: boolean" in globals.yml
15:36:18 <mgoddard> this is for the transition?
15:36:43 <hrw> not only
15:37:18 <hrw> if user1 wants own images then they can do kolla-build --noinfra to get only source images and deploy from them.
15:37:33 <hrw> or builds source and gets infra+source ones
15:37:53 <hrw> you do not need to build both binary/source images.
15:37:59 <hrw> as user
15:38:07 <hrw> we as project have
15:39:26 <mgoddard> it would be good to aim to have everyone on the same pattern eventually
15:39:45 <hrw> 2. transition - now upgrade is from blablah-horizon:stein -> blablah-horizon:train right?
15:39:52 <mgoddard> i.e. remove --noinfra in the long term
15:40:13 <hrw> so it will be blablah-binary-mariadb:train -> blablah-infra-mariadb:ussuri etc
15:40:52 <hrw> when you run 'docker ps' is it still 'mariadb' container
15:41:31 <yoctozepto> I don't see why we would want to keep them separate
15:41:35 <hrw> but I never did k-a upgrade so hard to tell
15:41:41 <yoctozepto> you mean for external integrations?
15:41:53 <yoctozepto> k-a will handle it
15:42:05 <hrw> mgoddard: --noinfra was added as someone asked for old way
15:42:23 <mgoddard> +1, it will probably be required not to break tripleo
15:42:31 <hrw> they can do --noinfra ;D
15:42:34 <mgoddard> (and k-a)
15:43:00 <mgoddard> yoctozepto: "I don't see why we would want to keep them separate"
15:43:10 <mgoddard> could you expand?
15:43:26 <yoctozepto> mgoddard: infra still source/binary
15:43:36 <hrw> yoctozepto: expand?
15:43:38 <yoctozepto> mgoddard: I guess we could fix up tripleo
15:43:50 <yoctozepto> hrw, mgoddard: your discussion about --noinfra switch
15:44:22 <mgoddard> so you think we should not provide a --noinfra switch?
15:44:28 <yoctozepto> mgoddard: exactly
15:44:32 <hrw> ok
15:44:33 <yoctozepto> just my gut feeling
15:44:37 <mgoddard> our way or the highway
15:44:44 <yoctozepto> exactly
15:44:48 <mgoddard> lol
15:44:51 <yoctozepto> now switching topic a bit
15:44:58 <yoctozepto> horizon stays with pyscss for ussuri
15:45:12 <yoctozepto> so we must use the workaround forever
15:45:19 <yoctozepto> thank you for your attention :-)
15:45:39 <hrw> we will shout at then in V
15:45:45 <mgoddard> is there no way to fix pyscss?
15:45:46 <hrw> and then deprecate horizon
15:45:57 <yoctozepto> mgoddard: very easy, harder to do release
15:46:01 <hrw> mgoddard: fork, rename
15:46:02 <mgoddard> why?
15:46:13 <yoctozepto> mgoddard: abandoned :S
15:46:21 <mgoddard> then fork?
15:46:32 <hrw> mgoddard: 16:32 < rdopiera> hrw: we don't have the resources to take on more projects
15:46:34 <yoctozepto> mgoddard: horizon does not like forks :-)
15:46:44 <yoctozepto> so maybe we fork
15:46:48 <mgoddard> kolla does not like pinning setuptools
15:46:49 <hrw> it is logged channel so I will not comment
15:46:56 <yoctozepto> and propose patch to horizon
15:47:18 <hrw> yoctozepto: and will you maintain it? it has huge amount of issues and some PRs...
15:47:35 <yoctozepto> hrw: issues closed :D "final version"
15:47:36 <hrw> and once you fork people will came with ideas, changes, issues and expect you to maintain
15:47:41 <mgoddard> we only need to get it past this issue
15:47:58 <hrw> pyscss46 ;D
15:48:29 <mgoddard> tbh, I don't mind pinning setuptools but I think it will come back to bite us
15:48:38 <hrw> sooner or later but will
15:48:48 <mgoddard> devstack will break on 46.0.1
15:48:58 <hrw> anything will
15:49:01 <yoctozepto> ^ indeed
15:49:17 <hrw> requirements fix was done in a rush without realizing what needs to be done
15:49:31 <hrw> that's my personal opinion
15:49:35 <mgoddard> +1
15:49:39 <yoctozepto> +1
15:49:52 <mgoddard> anyway, suggest we pin and ignore for now :D
15:49:52 <hrw> ok, what's next on agenda?
15:49:58 <yoctozepto> it's not longer yours, nor personal now
15:50:12 <mgoddard> #topic Kolla --SIG-- (aka Kolla Klub?) https://etherpad.openstack.org/p/kolla-sig
15:50:37 <yoctozepto> almost ate you alive for calling it SIG
15:50:43 <mgoddard> indeed
15:50:58 <mgoddard> WILL EVERYONE JUST STOP CREATING SIGs
15:51:09 <mgoddard> I thought they were supposed to be a lightweight thing
15:51:10 <yoctozepto> yeah, we got the message there
15:51:16 <mgoddard> nvm
15:51:17 <yoctozepto> supposedly not
15:51:25 <mgoddard> I don't really care what we call it
15:51:29 <mgoddard> Kolla klub
15:51:33 <yoctozepto> i like "Kollaborators" for members of Kolla Klub
15:51:44 <mgoddard> yeah
15:51:55 <mgoddard> shame the etherpad is now called kolla-sig :)
15:51:56 <yoctozepto> let's do something good
15:52:04 <yoctozepto> mgoddard: oopsie :-)
15:52:29 <mgoddard> anyway, if anyone is listening/reading - do sign up https://etherpad.openstack.org/p/kolla-sig
15:53:05 <yoctozepto> mgoddard: are we listing only one role?
15:53:13 <yoctozepto> mgoddard: aren't you operators as well?
15:53:16 <mgoddard> suggest if people start discussions here we direct them to the pad?
15:53:32 <yoctozepto> mgoddard: +1
15:53:42 <mgoddard> yoctozepto: you can call yourself the fairy godmother if it pleases
15:53:43 <hrw> you can always create new etherpad and add info in old one with link to proper one
15:54:05 <yoctozepto> mgoddard: oh yeah
15:54:35 <mgoddard> I'll give it a few more days then start approaching some people more directly
15:54:37 <cosmicsound> kolla club :D
15:54:50 <mgoddard> you signing up cosmicsound ?
15:54:51 <yoctozepto> mgoddard: :O
15:54:55 <cosmicsound> nope
15:54:56 <openstackgerrit> Marcin Juszkiewicz proposed openstack/kolla master: openstack-base: pin setuptools < 46 to get horizon working  https://review.opendev.org/712485
15:55:00 <yoctozepto> assassins among us
15:55:16 <mgoddard> cosmicsound: out of interest, why not?
15:56:35 <mgoddard> So once we have a few more names on the list, how should we proceed?
15:56:45 <mgoddard> Schedule a first meeting, to discuss...
15:57:32 <yoctozepto> ...how great we are
15:57:58 <hrw> conspiracy theories about making openstack grate again
15:58:26 <mgoddard> good idea
15:58:44 <yoctozepto> it grates all the time, hrw
15:59:02 <mgoddard> I think first meeting should be a meet and greet, and work out the scope of the group, plan future meetings etc.
15:59:09 <yoctozepto> +1
15:59:13 <hrw> +2
15:59:14 <scottsol> +1
15:59:16 <yoctozepto> as we have no other idea
15:59:26 <yoctozepto> we need to gather them ;D
15:59:32 <mgoddard> get people to sign up for bios/case studies etc.
15:59:47 <hrw> Tue/Thu same time as this meeting?
15:59:55 <hrw> for start
16:00:05 <mgoddard> I have other ideas (expressed on ML), but want to hear what others think
16:00:22 <mgoddard> let's poll for times
16:00:34 <mgoddard> we have emails
16:00:47 <mgoddard> but often this time slot gets picked
16:01:06 <mgoddard> #topic open discussion
16:01:18 <hrw> none
16:01:22 <mgoddard> Any other things to cover before we close?
16:02:01 <generalfuzz> one question
16:02:04 <mgoddard> sure
16:03:23 <generalfuzz> Not sure how to follow-up with tripleo regarding making uwsgi available for install
16:03:51 <cosmicsound> mgoddard , i did not made any contribs to code for kolla
16:04:12 <cosmicsound> was thinking is more suitable for whom contributed code
16:04:12 <hrw> cosmicsound: nevermind. kolla-klub is kind of devs/users stuff
16:04:29 <hrw> cosmicsound: to gather ideas from users etc
16:04:30 <mgoddard> cosmicsound: it is primarily for operators
16:04:31 <cosmicsound> right well me i
16:04:36 <cosmicsound> +1
16:04:41 <cosmicsound> in*
16:04:58 <hrw> generalfuzz: you mean?
16:05:00 <mgoddard> other people have been confused about that, I will try to make it more clear
16:05:02 <cosmicsound> if this brings me closer to a production ready k-a deployment
16:05:13 <cosmicsound> seems best way to meet more people involved
16:05:18 <generalfuzz> https://review.opendev.org/#/c/710879/
16:05:19 <patchbot> patch 710879 - kolla - Install uwsgi for Keystone - 14 patch sets
16:05:24 <mgoddard> that is the plan :)
16:05:28 <cosmicsound> exchange ideas or share knowledge alike
16:05:49 <mgoddard> generalfuzz: there is the #tripleo channel
16:05:54 <hrw> cosmicsound: I assume that we have far more users than developers in kolla. and we implement developers' ideas rather than users' ones as we do not know them
16:06:22 <mgoddard> generalfuzz: and cloudnull is our local point of contact
16:06:54 <generalfuzz> ok, thanks
16:06:58 <hrw> generalfuzz: macros.enable_extra_repo(['epel']) is what you need btw
16:07:09 <hrw> or sth like that I named it
16:07:23 <generalfuzz> hrw: where?
16:07:32 <hrw> will comment in gerrit
16:07:40 <generalfuzz> thanks
16:08:12 <mgoddard> generalfuzz: the package would be provided by RDO rather than tripleo, but both are connected teams
16:08:28 <cosmicsound> ohmy
16:08:31 <mgoddard> generalfuzz: I think for PoC EPEL is fine
16:08:42 <cosmicsound> integrating external ceph, aint so simple as planed
16:09:16 <mgoddard> generalfuzz: have you looked at how to run uwsgi? Does it need to be a second container, or second process in a container?
16:09:33 <mgoddard> We can probably end the meeting and carry on
16:09:38 <mgoddard> Thanks all
16:09:41 <mgoddard> #endmeeting