15:01:09 <yoctozepto> #startmeeting kolla
15:01:10 <openstack> Meeting started Wed Aug 19 15:01:09 2020 UTC and is due to finish in 60 minutes.  The chair is yoctozepto. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:01:11 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:01:13 <openstack> The meeting name has been set to 'kolla'
15:01:18 <yoctozepto> #topic rollcall
15:01:25 <priteau> o/
15:01:27 <yoctozepto> \o/
15:01:29 <yoctozepto> roll in
15:01:31 <osmanlicilegi> o/
15:01:33 <headphoneJames> o/
15:01:35 <JamesBenson> o/
15:01:40 <chensa> o/
15:02:00 <jovial[m]> 0/
15:03:29 <yoctozepto> guessing that's it :-)
15:03:31 <yoctozepto> #topic agenda
15:03:38 <yoctozepto> * Roll-call
15:03:38 <yoctozepto> * Announcements
15:03:38 <yoctozepto> ** Kolla Kall tomorrow (2020-08-20)
15:03:38 <yoctozepto> * Review action items from the last meeting
15:03:38 <yoctozepto> * CI status
15:03:38 <yoctozepto> * Victoria release planning (kayobe)
15:03:39 <yoctozepto> * Victoria release planning (kolla ansible)
15:03:39 <yoctozepto> * Victoria release planning (kolla)
15:03:46 <yoctozepto> #topic announcements
15:04:00 <yoctozepto> #info Kolla Kall tomorrow (2020-08-20)
15:04:07 <yoctozepto> any others?
15:05:15 <yoctozepto> guess not!
15:05:17 <yoctozepto> #topic Review action items from the last meeting
15:05:28 <yoctozepto> mgoddard to message openstack-discuss about focal & victoria upgrade
15:05:29 <openstackgerrit> Pierre Riteau proposed openstack/kayobe stable/stein: Skip broken ansible-lint  https://review.opendev.org/746952
15:05:37 <hrw> o\
15:05:41 <yoctozepto> pretty sure he did not :-)
15:05:50 <yoctozepto> #action mgoddard to message openstack-discuss about focal & victoria upgrade
15:05:57 <yoctozepto> feels like stuck in the queue!
15:06:03 <yoctozepto> #topic CI status
15:06:37 <priteau> Kayobe is RED
15:06:37 <yoctozepto> last time new ansible
15:06:42 <yoctozepto> this time new ansible-lint :-)
15:06:49 <priteau> because of ansible-lint too
15:06:54 <yoctozepto> priteau: ack
15:07:05 <priteau> I shamelessly stole your fix
15:07:08 <yoctozepto> priteau: all branches?
15:07:13 <yoctozepto> priteau: I may forgive you
15:07:24 <priteau> I submitted all the way back to stein
15:07:29 <yoctozepto> ok
15:07:38 <priteau> ansible-lint is not in upper-constraints unfortunately
15:07:54 <yoctozepto> priteau: yeah :-(
15:08:01 <yoctozepto> we could pin it completely
15:08:23 <yoctozepto> kolla-ansible master got fixed
15:08:39 <openstackgerrit> Pierre Riteau proposed openstack/kayobe stable/rocky: Skip broken ansible-lint  https://review.opendev.org/746955
15:08:43 <yoctozepto> ussuri blocked by a funny situation that we branched off ussuri during victoria cycle
15:09:01 <yoctozepto> and reqs-check for ussuri never checked its deps and now it got stuck due to yamllint not being ignored :-)
15:09:11 <yoctozepto> hopefully it's only yamllint...
15:09:21 <yoctozepto> anyhow, all fixes checking/gating
15:09:31 <yoctozepto> kolla master and ussuri green
15:09:49 <yoctozepto> older red; I guess mnasiadka won't be fixing that too soon :-)
15:11:05 <yoctozepto> hrw: care to massage train and older kolla branches to make them GREEN? (or any other volunteers for that matter)
15:11:39 <hrw> -ENOTIME a bit
15:11:53 <yoctozepto> hrw: ack, no problem
15:12:31 <yoctozepto> #topic Victoria release planning (kayobe)
15:12:47 <yoctozepto> priteau, jovial[m]: anything to discuss regarding kayobe?
15:13:37 <priteau> Nothing big going on at the moment.
15:13:47 <jovial[m]> nothing from me I'm afraid
15:13:57 <yoctozepto> sure, we are all waiting till the end of cycle :-)
15:14:10 <priteau> I could highlight a few patches that I submitted recently:
15:14:11 <priteau> https://review.opendev.org/#/c/746459/
15:14:11 <patchbot> patch 746459 - kayobe - Add support for custom Aodh configuration - 1 patch set
15:14:13 <priteau> https://review.opendev.org/#/c/746465/
15:14:13 <patchbot> patch 746465 - kayobe - Support setting ethtool options on network interfaces - 2 patch sets
15:14:14 <jovial[m]> exactly - it's the only way to work ;-)
15:14:27 <priteau> (no need to W+1 them yet, we need to fix CI first)
15:14:40 <priteau> But having another +2 would be nice
15:16:16 <yoctozepto> priteau: enjoy my +1s, changes look sane from far perspective
15:17:05 <yoctozepto> #topic Victoria release planning (kolla ansible)
15:17:09 <priteau> jovial[m]: I said no need to W+1 :P
15:17:21 <yoctozepto> anyone willing to discuss kolla-ansible?
15:17:31 <jovial[m]> too busy looking at your patch to see that :D
15:17:38 <yoctozepto> haha
15:17:54 <hrw> I was on holidays
15:19:09 <yoctozepto> guess not much about kolla-ansible without mgoddard and mnasiadka :-)
15:19:46 <yoctozepto> #topic Victoria release planning (kolla)
15:19:55 <yoctozepto> and for kolla itself?
15:20:09 <hrw> no one looks at infra so nothing from my side
15:20:19 <hrw> and no one started tier stuff
15:20:41 <yoctozepto> yeah, lack of time, low priority, always something more urgent to deal with
15:20:49 <yoctozepto> the big little issues
15:21:06 <yoctozepto> (and I don't mean endianess)
15:21:15 <hrw> yoctozepto: do not mention big.little
15:21:25 <yoctozepto> xD
15:21:37 <yoctozepto> I felt late it might be hrw-triggering
15:22:01 <hrw> good side: disallowed in servers ;D
15:22:15 <yoctozepto> hrw, in Polish: duże problemiki :-)
15:22:32 <yoctozepto> sad that English does not use proper diminutives
15:22:45 <yoctozepto> anyhow, not much to discuss
15:22:55 <yoctozepto> #topic Open discussion
15:23:13 <headphoneJames> I started considering how to integrate let's encrypt into openstack.
15:23:34 <headphoneJames> It doesn't seem trivial, and maybe warrants its own separate meeting
15:24:09 <chensa> i've got some bugs that happend to me with centos and rhel distributions if its relevant now
15:24:25 <yoctozepto> headphoneJames: we could use tomorrow's Kall; but without mgoddard and mnasiadka I would have to proxy the thoughts to them :-)
15:24:29 <hrw> I think that we need to train some new cores
15:24:36 <yoctozepto> headphoneJames: yeah, it's not trivial
15:24:48 <yoctozepto> headphoneJames: anyway, how do you imagine it?
15:24:58 <yoctozepto> I can tell you my expectation
15:25:12 <yoctozepto> at least manual refresh of certs via commands
15:25:18 <yoctozepto> which could be run from some simple cron
15:25:30 <yoctozepto> premium version that refreshes by itself
15:25:50 <yoctozepto> but for that we need some reload machinery
15:25:50 <hrw> once you have a way to replace them automation will be easy
15:25:57 <yoctozepto> we seem to be approaching this topic from another angle as well (the reload thingy)
15:26:08 <wuchunyang> hi , the octavia bp really really need to review
15:26:32 <yoctozepto> chensa: it's relevant, please speak up; our launchpad also always welcomes bug reports :-)
15:26:36 <hrw> headphoneJames: do you have a way to replace certificates with new files?
15:27:11 <hrw> headphoneJames: 1. do a way to replace certificates 2. create a way to refresh LE certs 3. automate LE refresh 4. use 1st step after 3rd one
15:27:45 <headphoneJames> Not yet to all
15:27:48 <chensa> in centOS 8 I had a bug that kolla-toolbox did not install and it affected mariaDB by not configuring the db users
15:27:57 <hrw> headphoneJames: if Henry comes from MoneyCorp and they have own SSL certs then he would use 1st to refresh certs with his own ones
15:27:58 <chensa> couldn't find any fix
15:28:16 <hrw> headphoneJames: at same time Bob will use LE
15:28:29 <yoctozepto> chensa: odd, this is the most basic and the most well tested part of kolla-ansible :-)
15:28:43 <headphoneJames> My impression is that the ansible host will need to redistribute certificates and inform containers to pick up fanook certificates
15:28:44 <yoctozepto> chensa: was it ussuri release with ussuri kolla-ansible run from centos 8 against centos 8?
15:28:59 <yoctozepto> chensa: did you report that to launchpad? it's best documented well
15:29:01 <hrw> headphoneJames: so small steps
15:29:35 <yoctozepto> headphoneJames: the brutal way is to keep doing restarts but that's normally not so feasible
15:29:41 <yoctozepto> we need reloads also for regular refreshes
15:29:44 <chensa> I don't know the protocols i'm very new to kolla (2 weeks) so i didn't report anything
15:30:21 <yoctozepto> chensa: ok, no problem; the gates to bug reporting machine are here: https://bugs.launchpad.net/kolla-ansible
15:31:23 <headphoneJames> In the scenario that ansible distribute certificates, also indicates that that certbot would run on the ansible host
15:31:44 <yoctozepto> hrw, priteau, jovial[m]: would you be joining tomorrow's Kall? we could use it to review kolla/kolla-ansible bugs - the more, the merrier (as you might recognise the symptoms)
15:31:50 <yoctozepto> wuchunyang: still in my queue
15:32:08 <wuchunyang> yoctozepto thanks
15:32:24 <yoctozepto> wuchunyang: I am not knowledgeable about octavia so it gets postponed pretty much every time, hence the delay, I am really sorry about that
15:32:40 <priteau> yoctozepto: if not too busy I'll join
15:32:52 <hrw> yoctozepto: ok
15:32:53 <chensa> yoctozepto second bug was when I tried deploying openstack on rhel 8.2 ansible reports ansible_distribution variable as RedHat and not RHEL which breaks deployment
15:33:27 <chensa> its only FYI i'll report them to launchpad
15:33:42 <yoctozepto> if there is anyone around familiar with octavia, then please take a look at wuchunyang's set of patches: https://review.opendev.org/#/q/project:openstack/kolla-ansible+topic:bp/implement-automatic-deploy-of-octavia
15:33:56 <wuchunyang> i have updated octavia docs,  https://review.opendev.org/#/c/746409/ , you can refer to this docs
15:33:57 <patchbot> patch 746409 - kolla-ansible - update octavia doc - 3 patch sets
15:34:11 <yoctozepto> chensa: that could be a real issue - we don't test against rhel because it's not freely available
15:34:16 <johnsom> I can also raise this at the Octavia meeting later this morning so Octavia folks can also help review.
15:35:03 <yoctozepto> johnsom: thanks, that would be awesome!
15:35:07 <yoctozepto> wuchunyang: ^^
15:36:02 <wuchunyang> johnsom thanks
15:36:03 <yoctozepto> the goal is to make kolla-ansible really deploy ready-to-use octavia rather than just throwing basic config and containers around :-)
15:36:03 <johnsom> On the agenda. Thank you for the work!
15:36:40 <yoctozepto> yes, thanks wuchunyang for making this happen
15:36:50 <hrw> chensa: and we check for RHEL?
15:36:56 <hrw> chensa: then send a patch
15:37:10 <yoctozepto> hrw: we do have some RHEL conditionals
15:37:18 <JamesBenson> side note:  before meeting I asked exactly about the certs, so def. +1 on getting this integrated
15:37:25 <yoctozepto> hrw: there were some users running centos7 containers on rhel7
15:37:56 <chensa> hrw sorry I am very new, what does it mean to send a patch?
15:38:33 <hrw> chensa: git clone, do a change, test it, git commit changed-file;git review
15:38:42 <yoctozepto> chensa: please get acquainted with https://docs.openstack.org/kolla-ansible/latest/contributor/index.html
15:38:48 <hrw> o! better
15:39:03 <yoctozepto> it explains some of the things we already said and draws the whole picture linking to other relevant docs
15:39:20 <JamesBenson> I've found a bug also with enable_cinder: "yes" & enable_cinder_backend_*: "no"
15:39:26 <yoctozepto> if anything is unclear, then please reach out to us
15:39:30 <JamesBenson> the prechecks will fail stating it needs a backend
15:39:47 <JamesBenson> this is on centos distro
15:39:59 <hrw> chensa: you mean RHEL in ansible/roles/prechecks/vars/main.yml file?
15:40:05 <yoctozepto> JamesBenson: ah, yeah; it prevents doing a out-of-kolla backend config as a sole one
15:40:24 <JamesBenson> yeah
15:40:25 <yoctozepto> JamesBenson: it might have been reported; please report to launchpad if not
15:40:32 <yoctozepto> JamesBenson: or ping in the current one
15:40:41 <JamesBenson> I'll double check, I didn't see it earlier when I check
15:40:43 <openstackgerrit> Michal Arbet proposed openstack/kolla-ansible master: Fix kolla-ansible not reflect environment changed  https://review.opendev.org/746965
15:41:14 <openstackgerrit> Pierre Riteau proposed openstack/kolla-ansible master: Add workaround for keystonemiddleware/neutron memcached issue  https://review.opendev.org/746966
15:41:16 <headphoneJames> regarding certs / letsencrypt - perhaps first step is simply just developing a certificate distribution command that is separate from deploy. Deal with lets encrypt second
15:41:21 <yoctozepto> JamesBenson: it might have been told me on irc, I have weird memory
15:41:27 <hrw> headphoneJames: yes.
15:41:37 <hrw> headphoneJames: look at Henry usecase I gave before
15:41:42 <yoctozepto> hrw: that totally makes sense
15:41:43 <JamesBenson> headphoneJames: totally agree
15:41:50 <chensa> @hrw yes, but I think they changed it only for the latest versions of RHEL.. might need to accept both
15:41:55 <yoctozepto> totally totally
15:42:00 <hrw> chensa: sure
15:42:23 <hrw> chensa: at sme time it is trivial enough change for new contributor to learn how stuff works
15:42:36 <JamesBenson> regarding certs:  will there have to be a usecase for when the certs are pulled into the images as well?  "kolla_copy_ca_into_containers"
15:43:11 <yoctozepto> chensa, hrw: I guess it could be ansible that changed things; or it was just broken and rhel users just brute-patched it for themselves
15:43:12 <yoctozepto> :-)
15:43:19 <JamesBenson> yoctozepto: I think I might have mentioned it earlier too, I've been a bit scattered as of late, juggling a ton atm.
15:43:32 <hrw> JamesBenson: I think it depends on how many certs you plan to have.
15:43:33 <yoctozepto> JamesBenson: welcome in the club
15:44:03 <hrw> some systems will want 1 cert for horizon frontend and other for infra. some may want cert per service?
15:44:15 <hrw> some will just use one for everything including frontend
15:44:24 <hrw> just to have TLS on connections
15:44:32 <JamesBenson> hrw: Plan for everything, that's my saying...
15:45:00 <hrw> plan for simple. implement. plan bigger. implement
15:45:17 <hrw> How to eat an elephant?
15:45:21 <yoctozepto> plan for planning... oh wait
15:45:26 <hrw> Piece by piece.
15:45:47 <headphoneJames> Would there have to be a mapping file to indicate where certs should be distributed? Could it be done by directory? Is it just a new config property per service?
15:46:42 <yoctozepto> headphoneJames: would the current deploy approach be bad? just focused on getting certs though
15:47:29 <JamesBenson> Could the certs just be in a directory like on the deploy node /kolla/certificates and have the containers pull them in on start?
15:47:36 <JamesBenson> then it could just be a simple restart the container?
15:47:51 <yoctozepto> (and reload in the future)
15:47:53 <openstackgerrit> Doug Szumski proposed openstack/kolla-ansible master: Add workaround for keystonemiddleware/neutron memcached issue  https://review.opendev.org/746966
15:47:57 <headphoneJames> Just pull all certs over to each service?
15:48:10 <hrw> "docker run -v/etc/kolla/certs:/etc/kolla/certs nova-compute"
15:48:32 <hrw> headphoneJames: what is already implemented?
15:48:42 <JamesBenson> yeah, distribute certs to folder on nodes, then issue the command above from hrw
15:49:19 <headphoneJames> currently, we search for a cert and copy it modifying the name
15:49:22 <JamesBenson> maybe 2 certs folders, one for previous to roll back if issues, and another for current
15:49:27 <headphoneJames> https://www.irccloud.com/pastebin/kdUOL362/
15:50:00 <JamesBenson> Also with my deploy yesterday/today I noticed that if certs are there, they wont overwrite.
15:50:17 <headphoneJames> ansible/roles/service-cert-copy/tasks/main.yml
15:50:23 <JamesBenson> And that certs are deleted upon destroy.
15:55:11 <yoctozepto> I guess the discussion stalled with that sad news :-)
15:55:51 <openstackgerrit> Pierre Riteau proposed openstack/kolla-ansible master: Add workaround for keystonemiddleware/neutron memcached issue  https://review.opendev.org/746966
15:56:05 <yoctozepto> thank you all for participating; and remember about the Kolla Kall tomorrow: same time, different place: https://wiki.openstack.org/wiki/Meetings/Kolla/Kall
15:56:11 <yoctozepto> #endmeeting