15:00:10 #startmeeting kolla 15:00:11 Meeting started Wed Dec 4 15:00:10 2019 UTC and is due to finish in 60 minutes. The chair is mgoddard. Information about MeetBot at http://wiki.debian.org/MeetBot. 15:00:12 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 15:00:15 #topic rollcall 15:00:15 The meeting name has been set to 'kolla' 15:00:18 \o 15:00:24 o/ 15:00:26 o/ 15:00:30 \o 15:00:30 o/ 15:03:30 #topic agenda 15:03:41 * Roll-call 15:03:43 * Announcements 15:03:45 * Review action items from last meeting 15:03:47 * CI status 15:03:49 * Train release planning 15:03:51 * Ussuri release planning 15:03:54 * Review priorities 15:03:56 ** CI scenarios aka ETOOMANYJOBS 15:03:58 #topic announcements 15:04:01 Anyone have any? 15:04:41 #topic Review action items from last meeting 15:04:44 mgoddard to update kayobe train release etherpad 15:04:48 I did not 15:04:51 #action mgoddard to update kayobe train release etherpad 15:04:59 #topic CI status 15:05:15 Is kolla blocked by the tripleo job? 15:05:24 unfortunately 15:05:30 other than rmq? 15:05:47 no, rmq only 15:05:58 hmm, maybe not 15:06:00 ok 15:06:02 good :) 15:06:09 Any issues? 15:06:40 nope 15:06:49 ubuntu binary looks a bit broken 15:06:51 http://zuul.openstack.org/builds?project=openstack%2Fkolla&branch=master&job_name=kolla-build-ubuntu-binary 15:07:32 Unable to locate package senlin-health-manager 15:07:35 busted 15:07:49 we had this recently merged, yeah 15:08:21 should we disable these images for ubuntu binary for now then? 15:08:26 anyone want to pick that up? 15:08:37 o/ 15:08:48 thanks 15:09:10 #action osmanlicilegi to mark senlin images as unbuildable for ubuntu binary 15:09:28 #undo 15:09:29 Removing item from minutes: #action osmanlicilegi to mark senlin images as unbuildable for ubuntu binary 15:09:32 #action osmanlicilegi to mark senlin images as unbuildable for ubuntu & debian binary 15:09:41 debian busted too 15:09:51 mgoddard, osmanlicilegi ideally ask Erik about it 15:09:59 https://review.opendev.org/#/c/692948/3/docker/senlin/senlin-conductor/Dockerfile.j2 15:10:00 patch 692948 - kolla - Added senlin-conductor and senlin-health-manager (MERGED) - 3 patch sets 15:10:06 I think he wanted good :D 15:10:32 yeah, we can add him to reviews. We need green CI though 15:10:34 got it 15:11:22 #topic Train release planning 15:11:31 RMQ/erlang 15:11:36 So here's a funny thing 15:11:39 pinging him on #senlin 15:11:57 I looked at the erlang version page, and the version we have already is fine for 3.7.10 15:12:11 added this comment to the bug report: 15:12:13 I think we are reading this wrong. RabbitMQ 3.7.10 supports 19.3.x to 21.x according to https://www.rabbitmq.com/which-erlang.html. So the versions we use are compatible. 15:12:59 mgoddard: the problem is that we can not go beyond 3.7.10 15:13:00 I think this removes the argument for pushing an erlang upgrade in train 15:13:14 mgoddard: I think we don't even have 19.3 on centos 15:13:22 the real reason behind this change 15:13:24 hrw: that's right, but that's not a problem for train and earlier 15:13:35 erlang-19.3.6.4-1.el7.x86_64 15:13:36 (no point staying too low though witj just 19.3) 15:13:42 hmm, odd 15:13:57 let me check again 15:14:32 INFO:kolla.common.utils.rabbitmq: erlang aarch64 19.3.6.4-1.el7 delorean-master-testing 32 k 15:14:35 IMO, we should leave this alone for train and upgrade erlang at the same time as rmq 3.8 15:14:49 mgoddard: they seem to have changed the support table 15:14:49 +1 15:15:01 (3.8 in ussuri) 15:15:08 they probably had a typo there 15:15:15 that's why it was installable 15:15:28 yoctozepto: what did it say before? 15:15:33 20.3 15:15:38 hmm 15:15:41 like the row above 15:16:20 I remember it was 20 too 15:16:46 all the more reason to dump rmq and go qpid ;D 15:16:55 anyways 15:17:02 this makes this commit irrelevant for train 15:17:05 so no blockers 15:17:21 well, Joseph had some issue with rmq 15:17:32 but maybe something else was awry, who knows 15:17:42 https://github.com/rabbitmq/rabbitmq-website/commit/d5c37acfe04876d5cdd62f3fc623fa35bb37601f#diff-4ae651de32ebda8d248ca50c4d6d0fb9 15:18:04 looks like you're right 15:18:18 mgoddard: thank you 15:18:20 the patch is ready for 3.8 too, so lets go for ussuri 15:18:37 Viktor Michalek proposed openstack/kolla-ansible master: Support for custom api-paste files https://review.opendev.org/695054 15:18:44 so what's left for train? 15:18:47 yeah, let's leave it because we are already lagging the release a bit 15:18:51 hrw: k-a patches 15:18:54 almost all are in 15:19:19 https://review.opendev.org/#/c/694476/ 15:19:19 patch 694476 - kolla-ansible - Default to etcd3gw driver for etcd-based coordination - 4 patch sets 15:19:23 https://launchpad.net/kolla-ansible/+milestone/9.0.0 15:19:24 https://review.opendev.org/#/c/697088/ 15:19:24 patch 697088 - kolla - Install etcd3gw to fix Ubuntu binary tooz coordina... - 1 patch set 15:19:29 ah so it's kolla too 15:19:31 I forgot 15:20:05 masakari not going to happen 15:20:08 so unpin 15:20:34 I need to look at that etcd business 15:21:04 we're going back and forth in this area a bit 15:21:05 mgoddard: finally made it better, not cryptic 15:21:22 do we know what other deployment projects use? 15:21:36 mgoddard: how so? I digested your point and did some tests 15:21:46 either etcd historically for v2 15:21:51 or etcd3gw for 3 15:22:00 Chandan Kumar (raukadah) proposed openstack/kolla master: Added python3 packages for mellanox agent container https://review.opendev.org/697236 15:22:15 and we have etcdgw in centos images already? 15:22:19 otherwise redis or consul 15:22:25 mgoddard: yeah 15:22:30 ok 15:23:09 python2-etcd3gw-0.2.4-6.el7.noarch 15:23:13 probably as a dep 15:23:27 I wonder how compatible these drivers are 15:23:36 python2-cinder and openstack-cinder depend on it 15:23:48 no idea, etcd3 looks lame 15:23:49 i.e. if you had etcd3 then we switch you to etcdgw, will it work? 15:24:02 mgoddard: well, it does work finally in CI 15:24:05 python-etcd3 maintainer used to sit next to me in my office 15:24:14 mgoddard: what can I say 15:24:20 maybe tooz part is b0rken 15:24:22 small world :) 15:24:26 indeed 15:25:12 our-project-wise they are interchangeable 15:25:19 both are limited to one endpoint 15:25:23 I'm not seeing any hits for etcd3gw for deployment projects in http://codesearch.openstack.org/?q=etcd3gw&i=nope&files=&repos= 15:25:29 so we have to be lame and point to [0] 15:25:48 mgoddard: *coughs* devstack *coughs* 15:25:58 also, try etcd3+http 15:26:04 though codesearch fails with it 15:26:20 I did http:// 15:26:27 and then browser for etcd3+http 15:26:37 ok 15:26:49 http://codesearch.openstack.org/?q=etcd3%3A%2F%2F&i=nope&files=&repos= 15:26:57 now only tooz, us and vitrage (lol) 15:27:47 let's end the etcd drama :-) 15:27:54 ok. I'll look at the reviews 15:28:07 if we document, people can choose their own adventure 15:28:32 mgoddard: we have it kind-of documented in code and docs for designate 15:28:42 the bug for coordination is open for all improvements :-) 15:28:43 ok 15:29:00 Shall we aim for an RC2 by the weekend? 15:29:07 +1 15:29:09 +! 15:29:14 =1 15:29:16 I saw some 'bump versions' patches go by. Was there one for train? 15:29:17 mhm 15:29:35 is that a yes? 15:29:52 yes 15:29:58 cool 15:29:58 train/stein/rocky 15:30:02 #topic Ussuri release planning 15:30:18 we had user asking for monasca-ui bump so I did all for 3 branches 15:30:29 Scott Solkhon proposed openstack/kolla-ansible master: Add also_notifies to Infoblox backend for Designate https://review.opendev.org/697267 15:30:34 hrw: much appreciated 15:30:39 +1 15:30:40 and for Ussuri we want a bot 15:30:45 who imitates hrw 15:31:01 or at least the version bumps... 15:31:05 hrw: centos8 & RDO - you said they changed approach. What's going on? 15:31:11 :O 15:31:42 for br in rocky stein train;do git checkout stable/$br; git up; ./tools/version-check.py;git commit -a -m"Bump versions for Openstack ${br}";git review;done 15:32:10 mgoddard: no buildroot for centos8 so they started building all their deps in copr. to not waste time waiting and waiting 15:32:40 what a mess 15:33:28 any ETA from them 15:33:30 ? 15:33:42 no 15:34:16 ok 15:34:36 heh. they had meeting an hour ago. 15:34:41 moment, looking at log 15:34:58 14:09:29 #topic CentOS8 Updates 15:34:59 14:09:30 i think ykarel added it 15:34:59 14:09:33 but anyway 15:34:59 14:09:46 #info We are continuing building deps in copr 15:34:59 14:09:49 yes i did add 15:35:01 14:10:01 #info issues reported in https://review.rdoproject.org/etherpad/p/rebuild-deps-centos8 15:35:04 14:10:10 Soniya Vyas created rdo-infra/ci-config master: Changed the look of 'Compare jobs' button https://review.rdoproject.org/r/23933 15:35:07 14:10:20 #link https://review.rdoproject.org/etherpad/p/rebuild-deps-centos8 15:35:10 14:12:18 #info If someone wants to try deps early can use http://38.145.34.66/test-el8/test-repos.repo 15:35:13 14:13:19 #info building DLRN builder for CentOS8 with master is in progress 15:35:16 14:14:03 #info we expect to start RDO Trunk CentOS8 bootstraping next week 15:35:36 nice 15:35:37 http://eavesdrop.openstack.org/meetings/rdo_meeting___2019_12_04/2019/rdo_meeting___2019_12_04.2019-12-04-14.00.log.html 15:36:11 thanks, good info 15:36:21 hopefully getting closer then 15:36:52 I wonder how much rdo-deps would help source builds 15:37:16 https://copr-be.cloud.fedoraproject.org/results/%40openstack-sig/centos8-deps/centos-stream-x86_64/ 15:37:26 mgoddard: and how much for infra ones 15:37:49 might be worth trying to point to that repo and seeing if it unlocks any more images 15:38:08 Scott Solkhon proposed openstack/kolla-ansible master: Generate HAProxy configuration when upgrading from Rocky to Stein https://review.opendev.org/697275 15:38:30 we shouldn't have too many deps for source images, would be nice to get them building 15:39:21 yep 15:39:28 anyone want to try using that repo? 15:40:44 (venv3) 16:40 (0s) hrw@puchatek:kolla$ time ~/devel/moje/docker-utils/kolla-build-images.sh centos "source binary" "" rdo-test 15:40:47 started 15:40:56 nice 15:41:19 on the k-a side, I started a discussion on the ML about migration 15:41:23 http://lists.openstack.org/pipermail/openstack-discuss/2019-November/011130.html 15:41:43 had a good response from mwhahaha about the tripleo approach 15:41:58 I believe the plan is to have a Train version on CentOS8 after all the 15:42:00 things get bootstrapped. Unfortunately the current target is trying to get 15:42:02 master on centos8 with the time frame currently TBD. I'm personally hoping 15:42:04 really soon. 15:42:09 yea still waiting :( 15:42:56 yeah hrw just checked rdo meeting logs, sounds like they're making some progress on master 15:43:25 yea last i heard was issues with a centos8 build root in cbs or something. So they wre going to bootstrap with copr or something 15:43:37 yep 15:43:38 mwhahaha: it is in progress 15:44:16 so I think it makes sense for us to follow the tripleo approach here, which would require a slight modification of our proposal 15:44:49 we would deploy centos 7 train containers on centos 7 hosts, and centos 8 train containers on centos 8 hosts 15:45:23 so we'd need to support building containers for both releases in train 15:45:41 and k-a would need to check the host's OS version and use the appropriate image ta 15:45:44 *tag 15:45:54 how does that sound? 15:45:57 I just hope that it will not migrate to s/train/ussuri/ in plan 15:46:12 we need train on 8 15:46:13 hrw: it can't really 15:46:15 period 15:46:32 ussuri won't have centos 7 containers 15:46:56 so far we do not have anythinf for c8 anyway 15:47:18 it's a slightly awkward timing of kolla release, centos 8 release, and dropping py2 15:47:28 sure, it is easier to migrate in one version of openstack 15:47:32 very awkward indeed 15:47:37 anyways, let's press on 15:47:41 but py2 kina correlates 15:47:45 kinda* 15:47:46 #topic Review priorities 15:47:47 in worse case it would be centos7/train -> ubuntu/train -> */ussuri -> */v 15:47:56 lol 15:48:00 hrw: lol 15:48:07 let's pretend you didn't say that 15:48:09 :) 15:48:20 mgoddard: it is worst case scenario 15:48:33 I could think of worse 15:48:37 anyway... 15:48:44 I think we know priorities 15:48:50 train, then ussuri priorities 15:49:00 #topic CI scenarios aka ETOOMANYJOBS 15:49:03 train etcd3 15:49:07 and that's it 15:49:11 oh yeah 15:49:15 * yoctozepto adding moar 15:49:57 I saw the new senlin job proposal and it was the threshold for mw :) 15:50:01 *me 15:50:12 :D 15:50:15 it's great that we are adding more CI jobs 15:50:35 too bad that infra starts to hate us due to this? 15:50:36 we have qinling, senlin and swift from me 15:50:41 all conflicting due to obvious reasons 15:50:45 but I think we need to start using more scenarios 15:50:47 hrw: indeed 15:51:03 where each scenario tests multiple services 15:51:32 otherwise we will have 50 per-service jobs with 3 distro variants 15:51:41 and we will not be invited back to opendev 15:51:44 mgoddard: that's my worry 15:51:52 so we need to group things 15:51:53 though it might still not reduce that number too much 15:52:09 we are proposing to test zun with cinder 15:52:12 so the same job 15:52:19 but where the other would fit? 15:52:28 everywhere and nowhere :D 15:52:35 along the same lines as the NFV scenario 15:52:44 test multiple services in one job 15:52:54 yeah, zun and nfv are straight 15:53:11 try to group them in a sane way 15:53:12 but it's the case that you need mistral and stuff for tracker 15:53:16 tacker* 15:53:20 (what a bad name) 15:53:31 sure 15:54:23 point acked, had the same feeling so I strongly sympathize 15:55:21 what's the best way to test two unrelated services in one job? 15:55:43 mgoddard: random match, most just way :D 15:55:44 just enable them all and test sequentially? 15:55:55 but not fail immediatelly 15:56:01 still could fail right on deploy 15:56:02 or setup one, test, destroy, setup two, test 15:56:08 and getting a job that runs 3 hours 15:56:20 mgoddard: then have separate jobs lol ;p 15:56:47 my ideas 15:56:50 I think we spend a lot of time in the setup though 15:56:51 test what is popular 15:56:54 test what we use 15:56:58 test what is new 15:57:04 try to combine in real scenarios 15:57:16 then enable tls :) 15:57:20 mgoddard: which setup? 15:57:29 generalfuzz: yeah, it's in my review queue 15:57:33 bootstrap servers etc 15:57:50 haven't really profiled a job, that would be interesting to see where the time goes 15:57:52 just afraid it eats my dreams (encrypts them) and provides me nigthmares (wrong key for decryption) 15:58:10 mgoddard: so maybe that's the next step? :D 15:58:22 if we spend most in deploy/upgrade/reconfigure+tests 15:58:25 maybe 15:58:31 then nothing to do 15:58:38 I'm mostly going on the time taken for different jobs 15:58:54 a job that deploys one service still seems to take a long time 15:59:30 mgoddard: hmm, well, it takes some time to do common and friends 15:59:31 anyways, we're out of time 15:59:37 to check 15:59:48 thanks, mgoddard 15:59:55 thank you all 16:00:00 #endmeeting