15:00:58 <mgoddard> #startmeeting kolla
15:00:59 <openstack> Meeting started Wed May  8 15:00:58 2019 UTC and is due to finish in 60 minutes.  The chair is mgoddard. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:01:00 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:01:02 <openstack> The meeting name has been set to 'kolla'
15:01:07 <mgoddard> #topic rollcall
15:01:19 <mgoddard> Hi o/
15:01:19 <hrw> o/
15:03:59 <mgoddard> quiet one today it seems
15:04:14 <mgoddard> #topic agenda
15:04:24 <mgoddard> * Roll-call
15:04:27 <mgoddard> * Announcements
15:04:29 <mgoddard> ** Virtual PTG on 28th May & 29th May @ 12:00 - 16:00 UTC
15:04:30 <hrw> looks like
15:04:31 <mgoddard> * Review action items from last meeting
15:04:33 <mgoddard> * Kolla whiteboard https://etherpad.openstack.org/p/KollaWhiteBoard
15:04:35 <mgoddard> * Stein release status
15:04:37 <mgoddard> * Train summit summary
15:04:39 <mgoddard> * Virtual PTG
15:04:41 <mgoddard> * Kayobe seeking official status
15:04:43 <mgoddard> #topic announcements
15:04:52 <mgoddard> #info Virtual PTG on 28th May & 29th May @ 12:00 - 16:00 UTC
15:05:24 <mgoddard> We'll use a google hangout/meet for the call, maybe with video we'll see how well it works
15:05:24 <hrw> I added two entries to agenda
15:05:33 <hrw> ok
15:05:47 <mgoddard> #link http://lists.openstack.org/pipermail/openstack-discuss/2019-May/005928.html
15:06:00 <mgoddard> ^ has PTG meeting link
15:06:05 <mgoddard> thanks hrw
15:06:16 <mgoddard> any other announcements?
15:06:55 <mgoddard> #topic review action items from last meeting
15:07:04 <mgoddard> there were two for me:
15:07:07 <mgoddard> mgoddard
15:07:09 <mgoddard> mgoddard to try again to disable ocata publishing jobs
15:07:11 <mgoddard> mgoddard to add dates to the PTG poll further out
15:07:32 <mgoddard> I didn't do 1, but did do 2, and we now have a date for the PTG
15:07:46 <mgoddard> #action mgoddard to try again again to disable ocata publishing jobs
15:08:03 <mgoddard> #topic Kolla whiteboard https://etherpad.openstack.org/p/KollaWhiteBoard
15:08:19 <hrw> updated my tasks there
15:09:19 <mgoddard> thanks
15:11:09 <mgoddard> CI jobs look mostly ok, except for bifrost on rocky and all ansible jobs on stable/stein
15:11:21 <mgoddard> #action mgoddard to review CI failures
15:11:38 <hrw> bifrost is something I ignore all the time
15:11:58 <hrw> it is breaking too easy
15:12:14 <mgoddard> yeah :(
15:12:19 <mgoddard> would like to kill it
15:12:34 <hrw> or atleast drop from CI
15:12:57 <mgoddard> we use it in kayobe, so test it there too
15:13:39 <mgoddard> it's only periodic or run when you change bifrost, so shouldn't affect most people. Good to be able to test changes though
15:13:56 <mgoddard> #topic Stein release status
15:14:08 <hrw> people asked why there was no 8.0.0 tag
15:14:20 <hrw> 15:03 < StingRayZA> out of interest, when does kolla-ansible 8.0.0rc1 become 8.0.0 ?
15:14:21 <mgoddard> I've been a bit out of the loop due to the summit, but I expect we're in the same position as before
15:14:34 <mgoddard> mainly we're waiting on centos RDO repo
15:14:50 <mgoddard> #link https://review.opendev.org/651601
15:15:33 <mgoddard> looks like it might be out: https://cbs.centos.org/koji/buildinfo?buildID=25934
15:15:38 <mgoddard> I'll recheck that job
15:15:43 <hrw> ok
15:15:55 <hrw> can you add that info to whiteboard?
15:16:11 <mgoddard> yeah
15:16:37 <hrw> thx
15:17:30 <mgoddard> done
15:17:35 <hrw> coolio
15:17:58 <mgoddard> Any other questions on the release?
15:18:03 <hrw> no
15:18:32 <mgoddard> Should probably do a review of recent bugs to see if there are any blockers as some people have been testing the RC
15:19:16 <mgoddard> I wonder if we ought to tag images as stein-rc1 rather than stein? I think it might confuse people into thinking they're ready
15:19:42 <hrw> that's for train
15:19:52 <mgoddard> not an easy way to do an atomic switch of image tags though
15:19:55 <mgoddard> yeah
15:20:08 <mgoddard> #topic Train summit summary
15:20:23 <mgoddard> #link http://lists.openstack.org/pipermail/openstack-discuss/2019-May/005923.html
15:20:30 <hrw> nice slides
15:20:38 <mgoddard> Sent a summary to the ML, think you already saw it hrw
15:20:40 <mgoddard> thanks
15:20:46 <hrw> yes
15:21:22 <hrw> next topic?
15:21:36 <mgoddard> Overall seemed to go well, quite a few people interested in using kolla and/or kayobe
15:21:54 <mgoddard> #topic Virtual PTG
15:22:08 <mgoddard> Date & time has been agreed
15:22:19 <mgoddard> We have some ideas in the etherpad
15:22:25 <mgoddard> #link https://etherpad.openstack.org/p/kolla-train-ptg
15:22:43 <mgoddard> I think we have a habit of talking about things that don't end up getting done
15:22:54 <mgoddard> Any ideas for focussing on things we will actually do?
15:23:14 <hrw> I need to get that py3 done
15:23:29 <mgoddard> Alternatively, making sure we get the important things done
15:23:37 <hrw> that's number one for me this cycle
15:23:38 <mgoddard> yeah, I think that's #1 for Train
15:23:55 <hrw> aarch64 ci will then just 'jump in'
15:23:58 <mgoddard> great
15:24:10 <mgoddard> sounds like that's coming on well too
15:24:31 <hrw> next topic?
15:25:26 <mgoddard> #info Please come to the PTG with an idea of priority of features for Train - both for you and kolla overall
15:25:41 <mgoddard> hopefully someone will read the meeting notes :)
15:25:47 <hrw> ;D
15:25:52 <mgoddard> #topic Kayobe seeking official status
15:26:10 <hrw> what is kayobe in 20 words?
15:26:26 <mgoddard> Bare metal provisioner & host configurer for Kolla Ansible
15:27:08 <hrw> the missing part between 'I got rack of servers' and 'deployed OpenStack with k-a'?
15:27:16 <mgoddard> we're open to how to approach this - whether a separate project or under kolla governance
15:27:20 <mgoddard> exactly
15:27:46 <mgoddard> I don't want to use PTL powers to force it in unless the team has bought in
15:27:51 <hrw> sure
15:28:02 <hrw> that's good topic for vPTG
15:28:16 <hrw> sell it to us ;D
15:28:29 <hrw> how ready for use, how many devs etc
15:28:42 <mgoddard> #info Kayobe seeking official status during Train cycle. Please have a think about whether kolla should own it or it should remain separate. Will discuss at vPTG
15:28:54 <mgoddard> sure, it's on the list
15:29:01 <mgoddard> #topic Debian/Ubuntu Python 3 status (hrw)
15:29:07 <mgoddard> #link https://review.opendev.org/642375
15:29:08 <hrw> tada!
15:29:10 <mgoddard> hrw: you're up
15:29:38 <hrw> so, we are at step when images build and then ansible fails somewhere somehow
15:29:57 <hrw> horizon, placement-api, cinder-api behave like apache2 is not even starting
15:30:26 <mgoddard> do you have a local dev environment?
15:30:41 <hrw> built images today, will to all in one to check
15:30:48 <hrw> and then some multinode if needed
15:31:11 <mgoddard> it's a lot easier to iterate and debug on ansible there
15:31:17 <hrw> yep
15:31:40 <hrw> stopped counting patches to other projects since that whole py3 stuff started
15:31:58 <mgoddard> I remember we had issues with the wsgi script having a different name under py3
15:32:26 <hrw> mgoddard: checked. in debian config files are same for py2/3 but need to investigate
15:32:52 <hrw> as CI runs ubuntu so ;d
15:33:09 <mgoddard> could be that apache is immediately exiting?
15:33:17 <hrw> config error
15:33:18 <mgoddard> any way to crank up the debug logs for apache?
15:33:22 <mgoddard> yeah
15:33:26 <hrw> apache2: Syntax error on line 80 of /etc/apache2/apache2.conf: DefaultRuntimeDir must be a valid directory, absolute or relative to ServerRoot
15:33:37 <hrw> which is weird as it is set by envvars
15:34:06 <hrw> mgoddard: debugs are one of things to try for sure
15:34:45 <mgoddard> looks like the deploy reached the end
15:34:46 <hrw> I dislike size of that patch but it could not be smaller ;(
15:34:53 <hrw> and test-openstack failed
15:35:03 <hrw> openstack server create errors out
15:35:15 <mgoddard> probably due to placement container restarting
15:35:23 <hrw> bingo!
15:35:42 <hrw> that's why it is good to discsuss with someone ;D
15:36:31 <mgoddard> here's the apache config: http://logs.openstack.org/75/642375/21/check/kolla-ansible-ubuntu-source/cb08acd/primary/logs/kolla_configs/placement-api/placement-api-wsgi.conf
15:36:51 <mgoddard> this log is always useful: http://logs.openstack.org/75/642375/21/check/kolla-ansible-ubuntu-source/cb08acd/primary/logs/system_logs/docker-info.txt.gz
15:37:24 <hrw> ;)
15:37:48 <hrw> python-path=/var/lib/kolla/venv/lib/python2.7/site-packages!
15:38:16 <hrw> thanks!
15:38:20 <mgoddard> not sure about L80 of apache.conf - I think that's referring to a default config file rather than one we've written out
15:38:34 <mgoddard> this one only has 25 lines: http://logs.openstack.org/75/642375/21/check/kolla-ansible-ubuntu-source/cb08acd/primary/logs/kolla_configs/placement-api/placement-api-wsgi.conf
15:38:38 <hrw> api-wsgi.conf will rpobably be main issue
15:38:53 <hrw> needs to be fixed anyway
15:39:12 <mgoddard> crack open the image and see what configs it has. Sometimes we have to remove defaults in kolla IIRC
15:39:53 <hrw> "docker run -it -u root CONTAINERID bash" is now far too often in history
15:40:18 <mgoddard> I add --rm :)
15:40:44 <hrw> indeed
15:41:02 <hrw> 'docker image prune' is quick
15:41:06 <mgoddard> maybe also see what keystone is doing since it uses apache and seems to be working
15:41:15 <hrw> reclamed 182GB in 2s
15:41:33 <mgoddard> nice
15:42:06 <mgoddard> any more on this?
15:42:15 <hrw> next topic
15:42:24 <mgoddard> #topic AArch64 CI status (hrw)
15:42:30 <mgoddard> #link https://review.opendev.org/557659
15:42:39 <hrw> it is more or less ready for use
15:43:04 <hrw> I went through list of images it builds and removed those which do not build under Debian/aarch64
15:43:24 <hrw> also removed whole monasca due to pypi dependencies which take huge amount of time to build
15:43:39 <hrw> and we were crossing 3h time which is maximum available
15:43:46 <mgoddard> ouch
15:43:54 <mgoddard> are you planning to keep it experimental?
15:44:04 <hrw> it depends on py3 thing
15:44:27 <hrw> so once py3 gets in I will change it to normal non-voting
15:44:46 <mgoddard> oh ok
15:44:51 <hrw> now it is easier to run recheck only on it
15:45:16 <hrw> as I can compare local job on aarch64/amd64 with CI
15:45:55 <hrw> machine is hosted in London so networking etc is not an issue
15:46:09 <hrw> have to check did infra added one change into zuul config or not yet
15:46:26 <hrw> https://review.opendev.org/#/c/557659/44/tests/templates/kolla-build.conf.j2 has info what was removed
15:46:40 <hrw> https://review.opendev.org/#/c/557659/44/docker/base/Dockerfile.j2 has that infra hack
15:47:24 <hrw> questions?
15:48:26 <mgoddard> Not really, will be a good thing for us to have
15:48:45 <mgoddard> So dependencies are infra zuul config and kolla py3. Any others?
15:48:56 <hrw> https://review.opendev.org/#/c/657370/ but you +2 it already
15:49:06 <hrw> qemu-kvm thing
15:49:07 <mgoddard> ok
15:49:26 <mgoddard> Are you going to chase up infra?
15:49:36 <hrw> rechase
15:49:45 <hrw> will probably just open a bug for it
15:49:53 <mgoddard> they're probably still settling after the summit
15:49:56 <hrw> and add it into blueprint
15:49:59 <hrw> exactly
15:50:10 <mgoddard> If we're blocked on py3 anyway I guess it's not a huge rush
15:50:21 <hrw> yes
15:50:44 <mgoddard> ok. I guess that's all unless you have anything else?
15:50:49 <hrw> nope
15:51:12 <mgoddard> #topic OpenStack Ansible collaboration
15:51:16 <mgoddard> One last thing :)
15:51:34 <mgoddard> had a session with the OSA folks at the summit which was interesting
15:51:36 <hrw> something to share k-a with and vice versa?
15:51:36 <mgoddard> #link http://lists.openstack.org/pipermail/openstack-discuss/2019-May/005925.html
15:51:52 <mgoddard> hopefully we can work together on some things this cycle
15:52:07 <hrw> and kayobe into it
15:52:16 <mgoddard> #info Kolla planning collaborate with OSA project during the Train cycle where possible
15:52:22 <mgoddard> yeah, maybe
15:52:26 <mgoddard> that's all
15:52:40 <hrw> thanks for chairing and hints
15:52:47 <mgoddard> thanks for attending hrw, would have been lonely otherwise :)
15:52:57 <mgoddard> np
15:52:58 <hrw> :)
15:53:08 <mgoddard> #endmeeting