15:02:26 <yoctozepto> #startmeeting kolla
15:02:28 <openstack> Meeting started Wed May  6 15:02:26 2020 UTC and is due to finish in 60 minutes.  The chair is yoctozepto. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:02:30 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:02:33 <openstack> The meeting name has been set to 'kolla'
15:02:37 <yoctozepto> #topic rollcall
15:02:43 <yoctozepto> \o
15:02:46 <Fl1nt> \o
15:02:48 <osmanlicilegi> o/
15:03:35 <yoctozepto> mgoddard mnasiadka hrw egonzalez yoctozepto rafaelweingartne cosmicsound osmanlicilegi: meeting running
15:03:37 <erolg> o/
15:03:47 <hrw> ~o
15:04:17 <headphoneJames> o/
15:04:38 <yoctozepto> headphoneJames: generalfuzz?
15:04:50 <headphoneJames> re-branding myself
15:04:55 <yoctozepto> headphoneJames: ack :-)
15:05:04 <osmanlicilegi> I loved the new brand :]
15:05:14 <yoctozepto> yeah, it suits James indeed :D
15:05:18 <priteau> o/
15:05:32 <yoctozepto> all right, welcome everyone
15:05:45 <yoctozepto> #topic agenda
15:05:54 <yoctozepto> * Roll-call
15:05:54 <yoctozepto> * Announcements
15:05:54 <yoctozepto> * Review action items from last meeting
15:05:54 <yoctozepto> * CI status
15:05:54 <yoctozepto> * Ussuri release planning (kayobe)
15:05:55 <yoctozepto> * Ussuri release planning (kolla & kolla ansible)
15:05:55 <yoctozepto> * Kolla Klub https://etherpad.openstack.org/p/kolla-sig
15:06:02 <yoctozepto> #topic Announcements
15:06:21 * hrw in two meetings at same time
15:07:01 <yoctozepto> #info kolla in change freeze now, please ask for FFEs, so far any internal TLS work is FFE accepted
15:07:18 <headphoneJames> groovy
15:07:32 <yoctozepto> and normally we accept bug fixes and those little things that happen to be small features but actually make the world better
15:07:39 <yoctozepto> and things like that
15:07:43 <yoctozepto> try us anyway
15:07:48 <yoctozepto> anyone anything?
15:07:54 <yoctozepto> (for advertisements)
15:07:56 <Fl1nt> rogger that
15:08:00 <osmanlicilegi> nope
15:08:13 <yoctozepto> oh, the malls are now open in Poland
15:08:24 <yoctozepto> you can visit the famous boutiques
15:08:54 <Fl1nt> Same thing in France, more or less... but you can visit us ^^
15:09:01 <osmanlicilegi> koalas have great discounts?
15:09:27 * Fl1nt hearing carefully
15:09:28 <yoctozepto> osmanlicilegi: yeah, koalas get special treatment, get to know me!
15:09:38 <yoctozepto> prize in itself
15:09:48 <osmanlicilegi> roger that
15:09:48 <yoctozepto> a'ight, let's move on
15:10:01 <yoctozepto> #topic Review action items from last meeting
15:10:24 * yoctozepto blind or there were none
15:10:44 <yoctozepto> #topic CI status
15:11:14 <osmanlicilegi> pretty green
15:11:31 <yoctozepto> all green it seems
15:11:38 <yoctozepto> agreed, osmanlicilegi
15:12:26 <yoctozepto> priteau: kayobe also green, no?
15:12:42 <priteau> Pretty much green
15:13:00 <priteau> Occasional failures due to mirrors or SSH issues in infra I think
15:13:23 <yoctozepto> priteau: ok
15:13:27 <yoctozepto> updated whiteboard
15:13:28 <yoctozepto> thanks, priteau
15:13:34 <yoctozepto> cleared issues
15:14:40 <yoctozepto> while we are on CI and stability
15:14:45 <yoctozepto> please review https://review.opendev.org/723342
15:14:46 <patchbot> patch 723342 - kolla-ansible - Coordinate haproxy and keepalived restarts - 8 patch sets
15:14:58 <yoctozepto> this seems to greatly help with upgrades stability
15:15:46 <Fl1nt> keepalived and haproxy restarts are really painful, they wait for each other and generate a lot of race condition.
15:15:50 <osmanlicilegi> another great work for kolla
15:15:56 <Fl1nt> yep
15:16:06 <yoctozepto> thanks
15:16:13 <osmanlicilegi> I have a plan to test this change with some scenarios tomorrow.
15:16:26 <yoctozepto> it would not be possible if not for Christian's fix of those MANY MANY restarts of haproxy
15:16:41 <yoctozepto> (as then we would still be doing it in a BAD WAY <TM>)
15:16:52 <yoctozepto> but that motivated me to finish the keepalived+haproxy drama
15:17:36 <Fl1nt> we just need to add active/active VIP and it will all be fun ^^
15:17:37 <yoctozepto> also, as far as CI and general stability is concerned, there is mnasiadka's work related to haproxy+mariadb https://review.opendev.org/710213
15:17:38 <patchbot> patch 710213 - kolla-ansible - Custom haproxy script for monitoring galera - 40 patch sets
15:18:03 <Fl1nt> this patch
15:18:09 <yoctozepto> Fl1nt: yeah, throw in pacemaker and we lost our last drops of sanity
15:18:12 <Fl1nt> is exactly what we did on our own
15:18:25 <yoctozepto> Fl1nt: y u no upstreamify?
15:18:28 <Fl1nt> @yoctozepto, don't please ^^
15:18:31 <yoctozepto> teamwork
15:18:47 <Fl1nt> was responding to the haproxy+mariadb
15:18:56 <Fl1nt> and we found 1 case
15:19:02 <Fl1nt> where it's not working ^^
15:19:06 <Fl1nt> which is rare
15:19:09 <yoctozepto> this patch?
15:19:14 <yoctozepto> please do comment :-)
15:19:18 <osmanlicilegi> Fl1nt: I have some works for active-active. I will try to clean my dirty patches and push to upstream for V.
15:19:19 <Fl1nt> but happens with mariadb sometimes
15:19:22 <yoctozepto> any help is appreciated
15:19:29 <yoctozepto> testing, triaging, comments, discussion
15:19:57 <yoctozepto> osmanlicilegi: please do!
15:20:03 <Fl1nt> @yoctozepto, we didn't find a way to fix it as it happen when the WSREP cluster is hanging in panic
15:20:16 <Fl1nt> so it's available but it's not
15:20:26 <yoctozepto> please also think about these little things you would like to do/coordinate with us so that we can include that in PTG
15:20:44 <yoctozepto> #info PTG is first week of June
15:20:49 <yoctozepto> (late announcement)
15:20:53 <yoctozepto> ok, let's move on
15:20:53 <Fl1nt> Yep, I'm starting to review things and need to make time for CEPH external and few virtualenv patch
15:21:10 <yoctozepto> #topic Ussuri release planning (kayobe)
15:21:32 <yoctozepto> Fl1nt: yeah, we need to switch nicely to native venv, but let's postpone the discussion till open
15:21:48 <yoctozepto> priteau, dougsz, jovial[m]: want to discuss release planning?
15:21:51 <Fl1nt> perfect
15:23:15 <yoctozepto> guess not really :-) waiting a bit more and moving on
15:24:28 <priteau> sorry, was afk
15:25:05 <yoctozepto> priteau: well, no problem, you are the only kayobian around anyways
15:25:11 <priteau> We're supposed to feature freeze kayobe, I think a few patches are waiting for review
15:26:05 <yoctozepto> priteau: seems you have quite a few half-approved ones
15:26:45 <priteau> Will (jovial) has a couple of feature patches that are not yet passing Zuul checks
15:27:19 <yoctozepto> well, then they are candidates for FFE
15:27:46 <yoctozepto> what matters is whether you can handle stabilizing and merging (or merging and stabilizing)
15:28:11 <yoctozepto> priteau: anything from kolla/kolla-ansible side to handle for kayobe?
15:28:21 <yoctozepto> (and then I move on)
15:28:31 <priteau> Not that I know.
15:28:42 <yoctozepto> ok
15:28:45 <yoctozepto> thanks
15:29:08 <yoctozepto> like someone once said, kolla is actually having a week long meeting so you can easily catch us anyway
15:29:11 <yoctozepto> even on weekends
15:29:17 <yoctozepto> (workaholics eh)
15:29:32 <yoctozepto> #topic Ussuri release planning (kolla & kolla ansible)
15:29:42 <yoctozepto> hrw, osmanlicilegi
15:30:02 <yoctozepto> so I'll repeat the stability fixes
15:30:08 <yoctozepto> as info for section
15:30:29 <yoctozepto> #info https://review.opendev.org/710213 - haproxy custom checking script for mariadb (with xinetd, active polling)
15:30:29 <patchbot> patch 710213 - kolla-ansible - Custom haproxy script for monitoring galera - 40 patch sets
15:30:52 <yoctozepto> #info https://review.opendev.org/723342 - coordinate haproxy and keepalived restarts
15:30:52 <patchbot> patch 723342 - kolla-ansible - Coordinate haproxy and keepalived restarts - 8 patch sets
15:31:28 <osmanlicilegi> and also https://review.opendev.org/#/c/723021/
15:31:28 <patchbot> patch 723021 - kolla-ansible - Check that used Ansible can see Kolla Ansible - 6 patch sets
15:31:41 <yoctozepto> #info https://review.opendev.org/724793 - nova mod_wsgi (apache) root user by headphoneJames
15:31:41 <patchbot> patch 724793 - kolla - Enable WSGI in Nova API container - 4 patch sets
15:31:48 <yoctozepto> this is part of tls work
15:31:56 <yoctozepto> thanks, osmanlicilegi, would forget
15:32:28 <yoctozepto> #info https://review.opendev.org/723021 - ansible and kolla_ansible environment coherency error out
15:32:28 <patchbot> patch 723021 - kolla-ansible - Check that used Ansible can see Kolla Ansible - 6 patch sets
15:32:42 <yoctozepto> generally all changes with RP+1 are important, a'ight
15:33:02 <yoctozepto> RP+2 is to be approved even at 2 AM (after reviewing)
15:33:16 <yoctozepto> and these are my initials for a good reason
15:35:57 <yoctozepto> I don't think we have anything with similar urgency among the other patches
15:36:04 <yoctozepto> please correct me if i'm wrong
15:36:32 <osmanlicilegi> +2
15:36:55 <osmanlicilegi> seems we will be ready for feature freeze in next few days
15:37:12 <yoctozepto> osmanlicilegi: we are in feature freeze :-)
15:37:19 <yoctozepto> just tls is excepted from freeze
15:37:37 <yoctozepto> we merged most of the other stuff
15:37:41 <yoctozepto> and majority of tls too
15:37:49 <osmanlicilegi> I have a time machine
15:37:54 <yoctozepto> again thanks headphoneJames for his hard work and responsiveness
15:38:05 <headphoneJames> pleasure
15:38:18 <yoctozepto> headphoneJames: is ours :-)
15:38:35 <yoctozepto> ok, moving on, I'm too chatty already ;-)
15:38:39 <headphoneJames> will I still be able to add more services in Ussuri? Or should next batch target the following release
15:39:23 <yoctozepto> headphoneJames: we agreed for the main ones, I guess it really depends on how clean they end up
15:39:44 <yoctozepto> if we have to rewrite half of ansible and dockerfile then it's a no-no for ussuri
15:39:50 <Fl1nt> @headphoneJames, You mean roles?
15:39:58 <headphoneJames> yes
15:39:59 <yoctozepto> but nova will likely be accepted
15:40:19 <yoctozepto> would make sense to cover barbican but it has finicky config
15:40:25 <yoctozepto> so it all depends
15:40:37 <yoctozepto> I guess it's better to stabilize and release early
15:40:44 <yoctozepto> we are already doing great with stable centos8
15:41:08 <yoctozepto> kudos to mgoddard and hrw for handling the majority of the centos8 work
15:41:13 <Fl1nt> Do you have any rule about new roles addition ? Like, if it's fix or improvement of existing one, it's OK to build R x.y.z +1
15:41:16 <yoctozepto> (and smaller to me and mnasiadka)
15:41:58 <yoctozepto> Fl1nt: feature freeze is a bad time for role addition :-) and it likely will not be backported unless life-saving
15:42:05 <yoctozepto> but please always propose
15:42:16 <yoctozepto> the worst that happens is that it gets in in the next cycle
15:42:29 <Fl1nt> yep that was my thoughts too don't worry
15:42:34 <yoctozepto> (so that you can feel your contribution is "newer" :-) )
15:42:46 <hrw> yoctozepto: c8 stuff still need testing and checking why ci jobs fail
15:43:20 <yoctozepto> hrw: is there anything in particular that you are referring to or that rbd failure with glance?
15:43:25 <yoctozepto> that's what I normally observe
15:43:31 <yoctozepto> (and I'm not talking aarch64)
15:43:38 <Fl1nt> ah ah ah ^^ My contribution will rather start with review, then patch then we will thought about services, but I'm more on the deprecation team :p
15:43:52 <yoctozepto> Fl1nt: yay for deprecations!
15:44:20 <yoctozepto> ok, this one got lengthy, we can continue in open discussion with everything else
15:44:21 <yoctozepto> #topic Kolla Klub https://etherpad.openstack.org/p/kolla-sig
15:44:45 <Fl1nt> I've finished my use-case slides
15:44:50 <yoctozepto> #info Kolla Klub meeting right tomorrow (Thursday, 07th of May 2020)
15:45:09 <yoctozepto> mgoddard: Kolla Klub tomorrow, do the write up or I'll do (yoctozepto threatened)
15:45:30 <yoctozepto> Fl1nt: oh, great, I'm really looking forward to that
15:46:31 <Fl1nt> I'm not really good at making slides, but I've try to do my best and all use-cases will be explainable, company named removed for contract obvious reason tho
15:47:16 <yoctozepto> I'll try to ping mgoddard later as well
15:47:28 <yoctozepto> Fl1nt: ack, I understand you are a consultant
15:47:33 <Fl1nt> Does anyone want to review it?
15:47:43 <Fl1nt> Yep
15:48:05 <yoctozepto> Fl1nt: if it's not a problem, I could take a look if you shared it with me on google slides
15:48:09 <yoctozepto> no promises though
15:48:12 <osmanlicilegi> Fl1nt: me too
15:48:30 <Fl1nt> sure no problems:
15:48:38 <yoctozepto> I don't see Dmitry (suff) around
15:48:44 <yoctozepto> but I hope he is doing his
15:49:07 <yoctozepto> we can certainly talk about kolla in general
15:49:11 <yoctozepto> moving on to open discussion now
15:49:13 <Fl1nt> https://docs.google.com/presentation/d/1SsqKXpbzdLE7LFAlhjRVPkhjV7hMpMJNYxmp_noJdVs/edit?usp=sharing
15:49:16 <yoctozepto> #topic Open discussion
15:50:28 <yoctozepto> Fl1nt: ok, accessible
15:50:39 <Fl1nt> perfect so
15:51:04 <yoctozepto> hrw: then what about that centos8 stability?
15:51:15 <yoctozepto> osmanlicilegi: anything for open discussion?
15:51:24 <yoctozepto> Fl1nt: you too
15:51:47 <yoctozepto> headphoneJames: and you
15:51:53 <osmanlicilegi> yoctozepto: I think the hottest topic is the brand new container image tooling :]
15:51:56 <Fl1nt> hum, except for what I've said earlier regarding kolla/openstack satellite services
15:52:18 <yoctozepto> osmanlicilegi: true, but I have not seen a reply from you on the ml about my ideas ;-)
15:52:33 <osmanlicilegi> I decided to keep silent
15:52:52 <Fl1nt> What I really like with current way to do it
15:52:54 <yoctozepto> osmanlicilegi: understood, I was out of energy for the ooo topic otoh
15:52:56 <Fl1nt> with jinja
15:53:42 <osmanlicilegi> in my opinion, this could free kolla. let's wait and see how it goes.
15:53:42 <Fl1nt> is how it is easy for companies to catch up with kolla/containers and openstack all in one as they usually have a lot of already experimented Ansible people but not so much about openstack/etc
15:54:00 <Fl1nt> but
15:54:33 <Fl1nt> as I'm not signed on the ML I didn't get the suggestion about the new process, so can't really prononce myself.
15:55:04 <yoctozepto> Fl1nt: ah, that is the issue
15:55:21 <yoctozepto> the common assumption is contributing operators are signed up
15:55:36 <yoctozepto> and then we get these misunderstandings or late reactions :-)
15:55:51 <yoctozepto> I really wished we did better with our communication channels
15:55:55 <yoctozepto> wish*
15:56:01 <Fl1nt> ^^ yeah, I'll have to sign back, I've just stepped down away a bit last year, when changed my job and opened up the company ^^
15:56:01 <osmanlicilegi> for whom missed the discussion on ML: http://lists.openstack.org/pipermail/openstack-discuss/2020-May/014591.html
15:56:07 <yoctozepto> but we (kolla) try our best I guess
15:56:23 <Fl1nt> yep
15:56:36 <Fl1nt> at least there is an active community ^^
15:56:37 <yoctozepto> #info http://lists.openstack.org/pipermail/openstack-discuss/2020-May/014591.html tripleo planning to drop kolla
15:57:21 <yoctozepto> #info http://lists.openstack.org/pipermail/openstack-discuss/2020-April/014255.html kolla builder next generation (by yoctozepto)
15:57:36 <yoctozepto> so these are the contributing threads
15:57:47 <Fl1nt> ok, not to be rude, but honestly, TripleO is doing it wrong, it's a nightmare to use/deploy and cluttered with so much moving parts that it can barely work now a day ^^
15:58:01 <yoctozepto> I really wish someone could summarize tripleo thread
15:58:41 <Fl1nt> but, that's for the technics/phylosophy, I'll have an extended read about the building process, hope that they didn't complexified things too much.
15:58:44 <yoctozepto> Fl1nt: fwiw, I think the complexity of following that tripleo thread resembles how hard it is to follow tripleo apart from basic guide but I'm obviously biased :-)
15:59:29 <yoctozepto> Fl1nt: to be honest, I think as for themselves they are doing a cleaning step, they really did rewrite like 95% of kolla templates anyway
16:00:02 <yoctozepto> and recently the patches we downstreamed to tripleo were to make more rewrites to handle new kolla cases :-)
16:00:03 <Fl1nt> yeah, I think the main difference between TO and Kolla is that KA is mainly created/maintained by operators that face customers and real life use cases where TO is mainly composed by RH Devs that tend to complexify too much because they don't confront customers issues.
16:00:43 <yoctozepto> Fl1nt: it's either tripleo or ooo, people won't easily recognize TO; similarly k-a rings the bell faster
16:00:59 <Fl1nt> noticed.
16:01:21 <Fl1nt> so, can't prononce myself for now, I'll have to read first ^^
16:01:35 <yoctozepto> Fl1nt: well, that could be it, they do a lot of great work though, pushing boundaries of things
16:02:04 <Fl1nt> yep, that why I need to have a look before just trash taking them freely.
16:02:15 <yoctozepto> oh, and they (as red hat) earn a lot of monies on the way so can't be overly wrong with the approach ;-)
16:02:58 <yoctozepto> on the other end of spectrum you have canonical ubuntu with charms for openstack (think juju)
16:03:15 <Fl1nt> Well, I'm working with RH as they're one of my main provider, but honestly, many things they "build" is overkill and overly complicated against what customer really want.
16:03:17 <yoctozepto> juju did not really catch up outside of canonical
16:03:49 <yoctozepto> but looks pretty promising when looked from far away
16:03:52 <osmanlicilegi> I was a long time charmer and I can clearly say, it's not k-a :]
16:03:54 <Fl1nt> same thing with Canonical and JuJu, it's amazing, until you actually use it in production, then it start crashing etc
16:04:12 <yoctozepto> osmanlicilegi: and how was it?
16:04:25 <yoctozepto> Fl1nt: did you use it personally? with canonical support or self?
16:04:33 <osmanlicilegi> I was happy with advanced support :]
16:04:36 <Fl1nt> canonical support, on premise
16:04:45 <yoctozepto> Fl1nt: and was it that bad? :O
16:04:53 <yoctozepto> osmanlicilegi: ah :D
16:05:00 <yoctozepto> osmanlicilegi: resigned because of monies?
16:05:14 <Fl1nt> k-a is honestly the cleanest more reliable of them all. And without making me big or something, handled really large installation.
16:05:25 <osmanlicilegi> I think the main problem for charmed openstack is, it prefers lxd
16:05:30 <Fl1nt> I was happy with canonical relationship and support quality
16:05:46 <osmanlicilegi> Fl1nt: +2 for support
16:05:47 <yoctozepto> osmanlicilegi: let x = s, then lsd
16:05:56 <Fl1nt> but they run after time and money desperatly
16:06:08 * yoctozepto hopes no children follow the discussion
16:06:42 <yoctozepto> Fl1nt: what you mean by time?
16:08:06 <Fl1nt> they seems to have pretty difficult time to construct a clean roadmap, then try to add things because customers requested it and paid for, but that then break the whole meaning of having a roadmap
16:08:29 <Fl1nt> ending up releasing a half-backed solution.
16:09:05 <Fl1nt> twist plop being, it may have improved as I didn't used canonical services since a year and a half now.
16:09:08 <yoctozepto> Fl1nt: ok, we don't have a roadmap so no such problems :D
16:09:13 <Fl1nt> yep ^^
16:09:20 <yoctozepto> just kidding, we have those every cycle
16:09:34 <Fl1nt> and it actually running more smoothly than with fixed roadmap
16:09:55 <yoctozepto> powered by community-driven engine
16:10:27 <yoctozepto> I read the slides, looks impressive
16:10:36 <yoctozepto> <envy>
16:10:38 <Fl1nt> which prove the idea is right and the good one by the amont of adoption and traction that it created for the project AND openstack in the meantime
16:10:48 <yoctozepto> looking forward to hearing the backstory too
16:10:49 <Fl1nt> thanks a lot
16:11:27 <Fl1nt> There are funny stories with each one of them, but can't publicly speak out during the meeting as it is recorded ^^
16:11:29 <yoctozepto> Fl1nt: we rely a lot on redhat and canonical and other company folks to drive the upstream projects though
16:11:57 <yoctozepto> but we also contribute to other projects, yeah
16:12:05 <yoctozepto> like mgoddard is in ironic cores
16:12:21 <yoctozepto> hrw cares for aarch64 so does party with nova from time to time
16:12:32 <Fl1nt> yep, all in all, RH and Canonical are bringing a lot of value, can't be denied.
16:12:33 <yoctozepto> and other little buildy things :-)
16:12:52 <Fl1nt> and all people in here definitely kick asses.
16:12:53 <yoctozepto> I'm the QA/CI folk around
16:13:01 <yoctozepto> moar stability
16:13:22 <yoctozepto> Fl1nt: understood, we can do a private one another time!
16:13:23 <Fl1nt> Exactly what company appreciate and found value in with kolla / kolla-ansible
16:13:51 <yoctozepto> a'ight, time to end the meeting
16:13:58 <yoctozepto> thank you for attending
16:14:06 <Fl1nt> For sure, be my guest, although my non disclosure agreement with video-game industry should end in few months ^^
16:14:07 <yoctozepto> and chatting with your chair today
16:14:16 <yoctozepto> #endmeeting