15:00:19 <mgoddard> #startmeeting kolla
15:00:22 <openstack> Meeting started Wed Mar 24 15:00:19 2021 UTC and is due to finish in 60 minutes.  The chair is mgoddard. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:00:23 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:00:26 <openstack> The meeting name has been set to 'kolla'
15:00:28 <mgoddard> #topic rollcall
15:00:44 <hrw> ]°[
15:00:45 <osmanlicilegi> o/
15:01:01 <headphoneJames> o/
15:01:04 <mgoddard> \o\ |o| /o/
15:01:08 <parallax> \o
15:01:40 <yoctozepto> o_
15:02:57 <mgoddard> #topic agenda
15:03:07 <mgoddard> * Roll-call
15:03:09 <mgoddard> * Announcements
15:03:11 <mgoddard> ** PTG 19th - 23rd April, registration open | https://april2021-ptg.eventbrite.com | https://www.openstack.org/ptg/
15:03:13 <mgoddard> * Review action items from the last meeting
15:03:15 <mgoddard> * CI status
15:03:17 <mgoddard> * Review requests
15:03:19 <mgoddard> * PTG team signup http://lists.openstack.org/pipermail/openstack-discuss/2021-March/020915.html
15:03:21 <mgoddard> * Quay.io
15:03:23 <mgoddard> * Wallaby release planning
15:03:25 <mgoddard> #topic announcements
15:03:28 <mgoddard> #info PTG 19th - 23rd April, registration open
15:03:35 <mgoddard> #link https://april2021-ptg.eventbrite.com
15:03:40 <mgoddard> #link https://www.openstack.org/ptg/
15:04:07 <mgoddard> #link https://etherpad.opendev.org/p/kolla-xena-ptg
15:04:18 <mgoddard> Please add your name to ^ if you plan to attend
15:04:29 <mgoddard> We will discuss PTG more later
15:05:03 <mgoddard> #info Kolla feature freeze next week
15:05:19 <mgoddard> Any other announcements?
15:06:27 <mgoddard> #topic Review action items from the last meeting
15:06:39 <mgoddard> yoctozepto try out quay.io
15:06:45 <mgoddard> he most certainly did
15:06:49 <mgoddard> will discuss later
15:06:55 <mgoddard> #topic CI status
15:07:14 * hrw 
15:07:40 <mgoddard> lots of nice notes on debian issues, thanks hrw
15:07:53 <hrw> and we need all of them merged
15:09:30 <mgoddard> this one makes x86 zuul pass: https://review.opendev.org/c/openstack/kolla/+/782606
15:09:33 <hrw> then would love to get some help on checking do things work
15:09:50 <yoctozepto> I've added a note on rc -13
15:10:39 <hrw> mgoddard: so s/-1/+2/ ;D
15:11:09 <yoctozepto> +23
15:11:13 <yoctozepto> oops
15:11:48 <hrw> I would need to check Debian in victoria, ussuri, train
15:12:01 <mgoddard> yoctozepto: how often is ubuntu cephadm failing
15:12:20 <yoctozepto> quite often but not 100% I think
15:12:23 <yoctozepto> let's see the recent runs
15:13:16 <yoctozepto> https://zuul.openstack.org/builds?job_name=kolla-ansible-ubuntu-source-cephadm&branch=master
15:13:22 <yoctozepto> seems not horribly often
15:13:51 <yoctozepto> or perhaps it was not always ubuntu
15:13:53 <yoctozepto> let me see
15:14:41 <yoctozepto> yup, bingo
15:14:50 <yoctozepto> it just I recognised ubuntu as happening more often
15:14:52 <yoctozepto> could be
15:14:55 <yoctozepto> but it's on both
15:15:16 <mgoddard> ok, sounds like more investigation needed
15:15:23 <yoctozepto> I wonder if it's not rc -13 in disguise
15:15:44 <yoctozepto> perhaps the block storage is unreliable or something
15:16:05 <yoctozepto> I think it would make sense to decrease the amount of retries
15:16:11 <openstackgerrit> Pierre Riteau proposed openstack/kayobe master: [DNM] Set environment for testing  https://review.opendev.org/c/openstack/kayobe/+/773411
15:16:11 <yoctozepto> especially for non-voting jobs
15:16:24 <yoctozepto> as we sometimes wait for several runs
15:16:50 <yoctozepto> wonder what happens there
15:16:55 <mgoddard> I don't think we're going to solve this here. Let's move on
15:17:06 <mgoddard> #topic Review requests
15:17:36 <mgoddard> You know the drill. One review per person
15:17:43 <yoctozepto> I will be back on masakari
15:17:54 <yoctozepto> nothing new today :-(
15:18:27 <hrw> I do not have one. Debian needs 3 ;D
15:18:52 <hrw> found one: https://review.opendev.org/c/openstack/kolla/+/782386 - ussuri backport
15:19:08 <mgoddard> I will choose https://review.opendev.org/c/openstack/kolla-ansible/+/781062
15:19:21 <hrw> as we still use Ussuri and want upgrade to newer qemu to test SVE guests
15:19:47 <mnasiadka> yoctozepto: https://review.opendev.org/c/openstack/kolla-ansible/+/761872 - that one is for you ;)
15:19:52 <headphoneJames> I have some Questions about test case requirements for Let's Encrypt
15:20:11 <headphoneJames> https://review.opendev.org/c/openstack/kolla-ansible/+/741340
15:20:34 <mgoddard> headphoneJames: go for it
15:21:32 <headphoneJames> Currently, I use "pebble" to create the TLS certificate. Then I distribute that TLS certificate to all haproxy
15:21:42 <yoctozepto> mnasiadka: why me?
15:22:09 <mnasiadka> yoctozepto: well, find another active core in k-a that does not work in StackHPC :)
15:22:24 <yoctozepto> mnasiadka: wuchunyang
15:22:53 <mnasiadka> is he on the meeting? nope
15:23:11 <headphoneJames> The certificate is not valid, because pebble is a testing product. Therefore, all subsequent requests to the OpenStack deployment would need to ignore the insecure certificate.
15:23:22 <yoctozepto> mnasiadka: ok
15:23:47 <mgoddard> headphoneJames: why isn't it valid? is there no CA certificate available for it?
15:23:47 <headphoneJames> Since I don't have access to the certificate authority for the certificate generated by pebble, I added a boolean to allow for insecure curl method executions to get around this for now.
15:25:02 <headphoneJames> The valid CA cert is generated by pebble -I have not determined a way to pull that certificate out of pebble / docker volume distribute it to the executor that's running the test
15:26:07 <headphoneJames> Note, when I run this with let's encrypt proper (not functional test with pebble), the certificate generated is valid and trusted
15:28:02 <headphoneJames> My first question is just validating the logs for certbot (That a certificate was properly generated) and that the certificate is distributed to all HAProxy enough for a test case?
15:28:05 <yoctozepto> we should be able to get the CA cert from pebble
15:28:24 <yoctozepto> mnasiadka: y no healthchecks?
15:28:33 <wuchunyang> yes
15:28:48 <mnasiadka> yoctozepto: where?
15:29:29 <yoctozepto> mnasiadka: left a comment
15:29:36 <mgoddard> headphoneJames: there's not much point in running a full test suite with insecure mode. We need to either do a more targeted test of the letsencrypt code, or somehow get hold of the CA cert
15:29:44 <yoctozepto> mnasiadka: on ovn-octavia
15:30:20 <mgoddard> headphoneJames: e.g. using something like the openssl suite to grab the cert from haproxy and check that it has come from pebble
15:30:39 <mnasiadka> yoctozepto: actually we can disable distributed FIP, but where do we communicate what CI does nowadays? :D
15:30:57 <yoctozepto> mnasiadka: what about the commit message for the starters? :P
15:31:08 <mnasiadka> commit message sounds good :)
15:31:11 <mnasiadka> will update
15:31:15 <yoctozepto> let's do the distr fip
15:31:23 <headphoneJames> mgoddard: the openssl suite approach would be doable if that feels acceptable
15:31:23 <yoctozepto> sounds fancier
15:31:36 <yoctozepto> and get those healthchecks in
15:32:59 <mgoddard> headphoneJames: that would be better than nothing
15:33:27 <mgoddard> and probably necessary in any case to verify that the cert has been rotated
15:33:42 <mgoddard> I think we've derailed a bit
15:33:46 <mgoddard> Let's move on
15:33:57 <mgoddard> #topic PTG team signup
15:34:05 <mgoddard> #link http://lists.openstack.org/pipermail/openstack-discuss/2021-March/020915.html
15:34:18 <mgoddard> Tomorrow is the deadline to choose time slots
15:34:35 <mgoddard> I didn't get any responses regarding using earlier slots
15:34:53 <mgoddard> so I propose we stick to the usual plan
15:35:07 <mgoddard> 13:00-17:00 UTC on Monday and Tuesday for Kolla and Kolla Ansible
15:35:17 <mgoddard> 13:00-15:00 on Wednesday for Kayobe
15:35:28 <mgoddard> #vote
15:35:31 <hrw> +1
15:35:57 <wuchunyang> 13:00-17:00 UTC is good for me
15:36:34 <yoctozepto> +1
15:36:43 <mnasiadka> +1
15:37:23 <mgoddard> done
15:37:37 <mgoddard> please add your names to https://etherpad.opendev.org/p/kolla-xena-ptg
15:38:26 <mgoddard> please also add discussion topics!
15:38:42 <hrw> * deprecate 'base' image
15:38:48 <hrw> ops, wrong window
15:39:23 <yoctozepto> let's deprecate something :-)
15:40:10 <mgoddard> deprecate yoctozepto
15:40:31 <yoctozepto> :-(
15:41:04 <hrw> mgoddard: you want to be PTL for rest of your life?
15:41:23 <mgoddard> of course
15:41:27 <hrw> +1
15:41:33 <yoctozepto> <3
15:41:52 <hrw> uf. nova change which makes Debian work just got +w ;D
15:42:05 <hrw> or rather s/work/fail in known place/
15:42:18 <yoctozepto> good enuff
15:42:42 <mgoddard> #topic Quay.io
15:43:15 <mgoddard> yoctozepto has started a nice PoC of using Quay.io in CI
15:43:32 <mgoddard> #link https://review.opendev.org/c/openstack/kolla/+/781130
15:43:41 <mgoddard> #link https://review.opendev.org/c/openstack/kolla-ansible/+/781546
15:43:43 <yoctozepto> thanks, I've pushed all master and victoria images there
15:43:51 <mgoddard> #link https://review.opendev.org/c/openstack/kolla/+/781899
15:43:51 <yoctozepto> except for centos binary which was failing at the time
15:44:07 <yoctozepto> but it's no biggie
15:44:13 <mgoddard> so I think we have two things to discuss
15:44:19 <mgoddard> 1. any concerns
15:44:25 <mgoddard> 2. plan
15:44:31 <mgoddard> yoctozepto: any concerns?
15:44:38 <yoctozepto> ~> https://review.opendev.org/q/topic:%22quay.io%22+projects:openstack/kolla
15:44:51 <yoctozepto> there is one limitation
15:45:00 <yoctozepto> in that new repositories get pushed as private
15:45:19 <yoctozepto> quay.io is "actively investigating" how to improve this
15:45:42 <yoctozepto> I have a script that fixes it for all repos
15:45:49 <yoctozepto> but it has to be run with human user credentials
15:46:14 <yoctozepto> otoh, we don't create new images that often
15:46:20 <yoctozepto> and quay.io might fix it sooner or later
15:46:35 <yoctozepto> other than that, I am quite happy with it
15:46:46 <yoctozepto> (not to mention having total control over it now)
15:46:50 <yoctozepto> (though I can share)
15:47:02 <yoctozepto> as for the plan
15:47:17 <yoctozepto> I would consider adding daily quay.io publishing jobs
15:47:30 <yoctozepto> leaving dockerhub ones in place to run their weekly sacred dance
15:47:46 <yoctozepto> then switching kolla-ansible to test from quay.io
15:48:15 <yoctozepto> we can run back to dockerhub if it proves worse
15:48:18 <yoctozepto> ;p
15:48:38 <mgoddard> so we would keep publishing to dockerhub for the time being
15:48:52 <mgoddard> it probably makes sense
15:49:01 <mgoddard> less disruption to users
15:49:09 <mgoddard> no need to clean up
15:49:26 <mgoddard> although we would have no way to test the images
15:49:36 <mgoddard> perhaps a weekly test pipeline
15:49:56 <hrw> mgoddard: we build weekly and publish to dockerhub and quay.io in same job?
15:50:18 <mgoddard> yoctozepto suggests publishing to quay.io daily
15:50:25 <yoctozepto> yes, quay.io more often
15:50:35 <yoctozepto> to get on track like we had it before ;p
15:50:47 <hrw> daily job on mon-sat to quay, weekly on sun to quay,docker?
15:51:09 <mgoddard> it would probably be simpler to just have separate publishing jobs
15:51:13 <hrw> sure
15:51:42 <openstackgerrit> Radosław Piliszek proposed openstack/kolla-ansible master: [CI] Drop the workaround in Masakari client calls  https://review.opendev.org/c/openstack/kolla-ansible/+/777182
15:51:50 <mgoddard> although potentially we could more easily test and promote to dockerhub
15:52:22 <yoctozepto> well, we can publish to dockerhub daily
15:52:28 <yoctozepto> it was not publishing that was broken
15:52:30 <yoctozepto> it was pulls
15:52:31 <yoctozepto> (and is)
15:52:38 <mgoddard> I think weekly is fine
15:52:50 <yoctozepto> "is enough"
15:52:59 <hrw> is both
15:53:07 <mgoddard> and doesn't double our CI load
15:53:23 <yoctozepto> well, we can publish from the same jobs
15:53:24 <hrw> we may write in docs "please use quay" and keep dockerhub as source for those who still use it
15:53:29 <yoctozepto> but I guess we could timeout
15:53:34 <yoctozepto> and not know what to blame
15:53:37 <openstackgerrit> Pierre Riteau proposed openstack/kayobe master: [DNM] Set environment for testing  https://review.opendev.org/c/openstack/kayobe/+/773411
15:53:45 <yoctozepto> hrw: yes
15:53:48 <yoctozepto> for the time being
15:53:55 <yoctozepto> let's see
15:53:59 <mgoddard> so we have a rough plan
15:54:05 <yoctozepto> I will enact it
15:54:12 <yoctozepto> will make me happy
15:54:14 <hrw> we can also deprecate dockerhub in Xena and do quay only in Yeti
15:54:17 <mgoddard> wonderful
15:54:40 <mgoddard> what about account credentials for quay.io
15:54:57 <mgoddard> currently I don't think any of us have credentials for dockerub
15:55:06 <mgoddard> which you might argue is a security feature
15:55:22 <yoctozepto> I can give you admin access
15:55:30 <yoctozepto> as you are PTL
15:55:33 <mgoddard> that's the opposite of what I'm suggesting :)
15:55:39 <yoctozepto> and I'm just a humble CI servant :D
15:55:50 <yoctozepto> oh, someone has to have them
15:56:01 <yoctozepto> someone from the previous team generated them for dockerhub
15:56:02 <mgoddard> if any person has credentials, it would allow them to compromise the images
15:56:05 <yoctozepto> they did not appear magically
15:56:11 <yoctozepto> mind you
15:57:03 <yoctozepto> it all boils down to trust
15:57:09 <mgoddard> indeed
15:57:12 <yoctozepto> I can't give you a better answer
15:57:24 <yoctozepto> I trust myself
15:57:28 <yoctozepto> I trust the PTL
15:57:37 <mgoddard> but quite a lot of effort goes into trust in zuul
15:57:43 <mgoddard> this could effectively side step that
15:57:50 <yoctozepto> we can perhaps write something down who is moderating the images
15:58:02 <yoctozepto> zuul has it encrypted
15:58:07 <yoctozepto> as it always had
15:58:08 <hrw> I think that opendev infra should have it somewhere
15:58:12 <yoctozepto> and we trust it a lot
15:58:19 <yoctozepto> as well as all its admins
15:58:26 <hrw> in case of bus incident happening with yoctozepto and mgoddard
15:58:32 <yoctozepto> yeah, they can decryp the secrets
15:58:44 <mgoddard> potentially, a zuul job could rotate the password, encrypt it, and print the encrypted result
15:58:58 <mgoddard> for example
15:59:03 <mgoddard> then no human would have the password
15:59:31 <hrw> I know that I do not want to know it
15:59:33 <mgoddard> but infra could access it
15:59:42 <yoctozepto> I would worry about that thing getting broken in the middle
15:59:49 <yoctozepto> new password and it being nowhere
16:00:00 <mgoddard> it's possible
16:00:20 <mgoddard> we'd probably need a safety access account until we know it works
16:00:29 <mgoddard> anyway, we're out of time
16:00:30 * hrw out
16:00:41 <yoctozepto> thanks mgoddard
16:00:48 <mgoddard> I think this is a concern though, and we shouldn't assume anything about what happened with dockerhub
16:00:59 <mgoddard> perhaps we should put something on openstack-dicuss
16:01:19 <yoctozepto> yes, please do; sometimes we get really nice insight
16:01:40 <mgoddard> #action mgoddard email openstack-discuss about quay.io credentials
16:01:44 <mgoddard> #endmeeting