15:59:18 <inc0> #startmeeting Kolla
15:59:19 <openstack> Meeting started Wed Jun  7 15:59:18 2017 UTC and is due to finish in 60 minutes.  The chair is inc0. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:59:21 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:59:23 <openstack> The meeting name has been set to 'kolla'
15:59:30 <inc0> #topic w00t for Kolla
15:59:32 <hrw> o/
15:59:34 <duonghq> o/ wo0t || t0ow
15:59:35 <jascott1> o/
15:59:35 <spsurya_> w0t
15:59:40 <Jeffrey4l> w00t
15:59:41 <spsurya_> w00t
15:59:42 <inc0> woot:)
15:59:49 <krtaylor> o/
16:00:10 <inc0> let's give few minutes for everyone to join
16:00:14 <rwellum> o/
16:00:42 <egonzalez> woot!
16:00:52 <daidv_> o/
16:01:43 <pbourke> o/
16:03:15 <inc0> ok let's move on
16:03:20 <inc0> #topic Announcements
16:03:30 <inc0> 1. we release 4.0.2 and 5.0.0b2 this week
16:04:15 <hrw> 4.0.2=ocata, 5.0.0b2=pike?
16:04:17 <inc0> 2. image publisher guidelines TC resolution is coming along https://review.openstack.org/#/c/469265/
16:04:23 <inc0> hrw: right
16:04:58 <inc0> any community announcements?
16:05:10 <krtaylor> I'm building on Power (ppc64le) now thanks to hrw 's changes for multi-arch,  had to change a couple of things but its all working
16:05:27 <krtaylor> finally able to get back at it
16:05:42 <inc0> coo
16:05:44 <inc0> cool
16:05:53 <krtaylor> ubuntu, centos next
16:05:54 <inc0> we need a demo in Sydney guys;)
16:06:00 <inc0> just sayin'
16:06:07 <krtaylor> you know, I was thinking the same thing
16:06:14 <hrw> krtaylor: any extra changes to share?
16:06:31 <krtaylor> hrw, just base image stuff, I'll push a patch
16:06:44 <inc0> ok, let's move to agenda it might be busy
16:06:46 <hrw> krtaylor: for centos/binary you need to get some help from RDO as they do not build for ppc64le yet
16:06:56 <inc0> #topic Gating discussion - multinode, upgrade, scenarios (inc0)
16:07:03 <krtaylor> maybe gema/hrw and I could do a talk on multi-arch kolla?  :)
16:07:19 <inc0> krtaylor: guys, let's move this to open discussion plz
16:07:23 <krtaylor> hrw, good info, I'll ping you when I get there
16:07:27 <krtaylor> yes
16:07:37 <inc0> We're getting small overhaul of gating
16:07:43 <inc0> gates are weakest point of Kolla
16:07:47 <inc0> let's change that
16:07:49 <inc0> so few things
16:08:12 <inc0> #link https://review.openstack.org/#/c/466007/
16:08:25 <inc0> #link https://review.openstack.org/#/c/466815/
16:08:39 <inc0> first one enabled multinode gates and moved lots of gate logic to ansible
16:09:26 <inc0> that in turn allows us to create new gates easily as we'll have templating and stuff like that at our disposal
16:09:58 <inc0> I would like us to increase number of scenarios we gate on
16:10:29 <inc0> so to summarize my plan
16:10:37 <inc0> 1. enable multinode gates (done)
16:10:44 <inc0> 2. make voting publisher pipeline
16:11:09 <inc0> so this is something we work with infra for publishing on dockerhub
16:11:32 <inc0> idea goes like taht: every night we'll run periodical job to build images - for example ubuntu-source
16:11:58 <inc0> after that we'll run ubuntu-source gates on these images in votting manner, and if and only if all of gates pass
16:12:06 <inc0> we publish them to tarballs the way we did
16:12:26 <Jeffrey4l> sound cool.
16:12:28 <inc0> then we run periodical job to pull from tarball and push to dockerhub/quay.io
16:12:41 <inc0> that will also mean that all images are deployed, which means if we use it
16:12:48 <inc0> we can make ansible gates voting.
16:13:20 <inc0> 3. another thing is we'll have tested and confirmed working imagers for both ocata and master
16:13:27 <inc0> so we can create gates that will perform full upgrade
16:13:40 <inc0> pull ocata images/kolla-ansible
16:13:52 <inc0> deploy ocata, test
16:13:57 <inc0> spawn vm and such
16:14:05 <inc0> pull pike images, pike kolla-ansible
16:14:10 <inc0> perform an upgrade
16:14:28 <inc0> and test again, see if vm that was deployed is still alive and accessible
16:15:17 <inc0> so that's a plan, it's going to be quite a lot of work, so any help would be greatly appreciated
16:15:34 <duonghq> I think it's just fraction test, just nova, keystone validation
16:15:56 <duonghq> kind of unit test
16:16:02 <inc0> well, so if we spawn vm out of volume and ssh to it and save sth to disk
16:16:06 <inc0> that also tests neutron, cinder
16:16:23 <inc0> that's quite extensive test
16:16:27 <duonghq> ya, at least we should make sure compute-kit work
16:16:44 <inc0> we can increate test coverage by using tempest
16:17:00 <duonghq> +1
16:17:04 <inc0> or perform more interesting persistence tests (check if evetyhing keeps working after upgradE)
16:17:12 <inc0> that's space where I could definetly need help
16:17:37 <duonghq> roger
16:17:41 <krtaylor> would multi-arch test (vote?) via running/reporting like a third-party CI test (except periodic)?
16:17:44 <spsurya_> inc0: roger
16:18:01 <inc0> krtaylor: 3rd party CI can't be voting
16:18:05 <inc0> and for good reason
16:18:18 <inc0> if something happens with this we couldn't merge anythign to Kolla any more
16:18:20 <krtaylor> not for gate, just +1
16:18:29 <inc0> ah, that's not voting, that's fune
16:18:38 <krtaylor> yeah, just thinking out loud
16:18:42 <inc0> voting means "if I fail, you can't merge"
16:19:00 <inc0> adding +1 to change and showing as regular gate is fine
16:19:09 <inc0> and I'd love to have it
16:19:27 <krtaylor> yes, I of all people should know not to use the word voting  :)
16:19:40 <inc0> I'm nto sure if we're ready to push non-amd64 images to dockerhub, but that would be good in Queens
16:20:20 <hrw> docker image manifest supports architecture field iirc
16:20:31 <hrw> just do not know yet how it is used before push ;D
16:20:47 <krtaylor> and how? that would be something new for openstack, not infra based hardware unless it was donated, hm
16:20:48 <inc0> well, I'm more concerned about testing images
16:20:56 <inc0> we won't have testing pipeline
16:21:18 <inc0> anyway, we can discuss that on PTG
16:21:20 <krtaylor> we don't need to solve that right now, just keep it in mind
16:21:22 <krtaylor> yes
16:21:35 <inc0> we'll know more as I hope to have amd64 images pushed by then
16:21:41 <krtaylor> +1
16:22:31 <inc0> duonghq: do you have capacity to take on increasing test coverage (or covfefe) in gates?
16:22:37 <inc0> as in running tempest and such?
16:23:05 <duonghq> I'll try but need one more hand
16:23:11 <inc0> or anyone really, that would be awesome change
16:23:17 <inc0> so, looking for volunteers;)
16:23:19 <Jeffrey4l> i have some thing which try to run tempest in gate. https://review.openstack.org/402122
16:23:20 <jascott1> +1 for increasing covfefe
16:23:25 <duonghq> iirc, vhosakot want to do that
16:23:39 <inc0> yeah, Vikram mentioned he wanted to do upgrade gates
16:23:55 <inc0> but that's big effort, let's make it team effort
16:24:05 <duonghq> sure
16:24:24 <inc0> also check if anyone in non-US-friendly timezones would like to work on that
16:26:03 <inc0> so, bottom line, keep an eye on gate changes and ask friends for volunteers to make them better;)
16:26:09 <inc0> let's fix this problem once for all
16:26:17 <inc0> that's it from me
16:26:28 <inc0> anyone else whats to add anything or have any questions?
16:26:29 <duonghq> I doubt that we can fix gate problem once and forever :P
16:26:38 <duonghq> nothing atm
16:26:41 <inc0> well, make it reasonable at least
16:26:47 <inc0> it was bad, now let's make it good enough
16:26:53 <duonghq> roger
16:27:06 <Jeffrey4l> seem the patch break kolla's gate.
16:27:17 <spsurya_> inc0: i will try, but need little better understanding of gating
16:27:29 <inc0> spsurya_: I'll be happy to help
16:27:47 <inc0> I think I have fair grasp on how they works by now
16:28:00 <inc0> Jeffrey4l: which patch?
16:28:10 <Jeffrey4l> kolla-ansible gate patch.
16:28:23 <Jeffrey4l> i pushed a test patch to test kolla
16:28:39 <Jeffrey4l> seems all deploy jobs are red.
16:28:49 <inc0> I'll take a look and fix it then
16:29:10 <inc0> probably lack of registry makes it bad
16:29:15 <inc0> but I'll just add push then
16:29:20 <Jeffrey4l> have to fix it before tag B2.
16:29:45 <Jeffrey4l> maybe. let's check it lather.
16:29:45 <inc0> kk, I'll fix it today so you'll have it ready for reviews tomorrow for you
16:30:32 <Jeffrey4l> cool.
16:31:01 <inc0> ok, let's move on then
16:31:16 <inc0> #topic how to manage api-paste.ini file in kolla ( jeffrey4l ) https://review.openstack.org/462535 and https://review.openstack.org/469939
16:32:04 <Jeffrey4l> it is mine.
16:32:17 <Jeffrey4l> thanks inc0
16:32:36 <Jeffrey4l> it is very common to introduce something to fix a bug.
16:32:50 <Jeffrey4l> like this https://review.openstack.org/462535
16:33:11 <Jeffrey4l> a copy of keystone-api-paste.ini is introduced to fix admin_token issue.
16:33:40 <Jeffrey4l> and now, it is hard to remove. seems we have to provide another way to custom keystone-api-paste file.
16:33:59 <inc0> merge_ini doesn't work there?
16:33:59 <Jeffrey4l> do we have any policy to handle this cause? inc0
16:34:14 <Jeffrey4l> inc0, merge_config doesn't.
16:34:49 <inc0> hmm
16:35:04 <inc0> maybe we need some way to add optional configs
16:35:06 <inc0> ?
16:35:09 <inc0> I don't know really...
16:35:17 <Jeffrey4l> because, api-paste.ini must be provide all it value ( it do not have a default value i code base)
16:35:37 <Jeffrey4l> add thing is very easy, and drop is hard ;(
16:36:00 <Jeffrey4l> the only one solution i am thinking is wath "with_first_found" does
16:36:03 <inc0> keep it in sync with upstream api-paste ?
16:36:19 <inc0> yeah but there has to be something found out there I think
16:36:24 <inc0> or it will be empty file
16:36:26 <inc0> which is bad
16:36:56 <Jeffrey4l> inc0, it is hard and we do not have such mech to make sure it is synced.
16:37:26 <Jeffrey4l> if custom provide api-paste file, use it. ( override the code base ones'), other wise do nothing.
16:37:33 <inc0> Jeffrey4l: start ML thread? maybe keystone community could help
16:37:34 <Jeffrey4l> but this is more like a bp ;(
16:37:57 <vhosakot> o/ sorry for being late, was in a meeting.
16:37:59 <Jeffrey4l> inc0, barbican role have almost the same issue like keystone.
16:38:03 <inc0> I'm ok with bp if we figure out good solution
16:38:24 <inc0> vhosakot: np, check log later
16:38:33 <Jeffrey4l> yes.
16:38:34 <vhosakot> inc0: yep, doing it now
16:38:51 <inc0> Jeffrey4l: so start ML thread please with tags of both keystone and barbican
16:39:00 <egonzalez> instead of with_first_found create a task to look for a custom file, and if a file exists copy it
16:39:12 <Jeffrey4l> egonzalez, yes.
16:39:13 <Jeffrey4l> inc0, will do .
16:39:14 <egonzalez> sth similar to what is done with policy.json
16:39:25 <inc0> egonzalez: we'd also need to allow optional copy by set_configs inside container
16:39:30 <Jeffrey4l> yes. these two file is almost the same.
16:40:02 <Jeffrey4l> oslo.policy is trying to support default value in code base, iirc.
16:40:13 <Jeffrey4l> but not sure which project is using this feature.
16:41:09 <egonzalez> inc0, set_configs already allow optional files
16:41:19 <Jeffrey4l> anyway, i will start a ML and try to registe an bp.
16:41:28 <inc0> sure
16:42:22 <Jeffrey4l> what i am confused is: how to fix the keystone paste file ASAP before the bp is landed.. 1. just remove it? or sync with upstream .
16:42:42 <inc0> sync with upstream would be my answer
16:42:48 <Jeffrey4l> ok.
16:42:50 <inc0> keeping it synced is hard
16:43:04 <inc0> but we'll have time to get proper solution
16:43:11 <egonzalez> +1
16:43:43 <Jeffrey4l> roger. will update the patch to sync with upstream.
16:43:56 <Jeffrey4l> ok. this is what i want to say. thanks ;)
16:43:58 <inc0> thanks
16:44:17 <egonzalez> maybe add a release note to allow us to remove the file when the bp lands
16:44:36 <egonzalez> *deprecation release note
16:44:47 <duonghq> egonzalez, +1, but do we need depreration policy for that
16:45:12 <inc0> duonghq: no, no need for deprecation as removing it will be optional
16:45:21 <egonzalez> the bug fix has become in a feature users are using, see osprofile ML thread http://lists.openstack.org/pipermail/openstack-dev/2017-May/117607.html
16:45:24 <duonghq> inc0,  roger
16:45:26 <inc0> you don't have to do it, it's safe to do it now
16:46:09 <Jeffrey4l> yes. how the api-paste.ini file is used will be change and a release note must be provided.
16:47:37 <inc0> ok, anything else on this topic
16:47:39 <inc0> ?
16:47:56 <Jeffrey4l> no from me.
16:48:05 <inc0> ok, let's move on
16:48:10 <inc0> #topic open discussion
16:48:16 <inc0> anyone?
16:48:19 <duonghq> inc0, there is one topic from me
16:48:23 <inc0> go ahead
16:48:27 <duonghq> * bp/mixing-binary-and-source-image status (duonghq)
16:48:31 <spsurya_> one from my side too inc0
16:48:35 <duonghq> #link https://blueprints.launchpad.net/kolla-ansible/+spec/mixing-binary-and-source-image
16:48:55 <duonghq> as I see, daidv_ has implemented all ps of this
16:49:00 <duonghq> can we close the bp?
16:49:09 <inc0> yes, if it's done, close it
16:49:10 <vhosakot> duonghq: I'll catchup with you about upgrade gates..
16:49:21 <duonghq> inc0, thanks
16:49:47 <inc0> duonghq: anything else?
16:49:49 <duonghq> vhosakot, thank you :D
16:49:53 <duonghq> inc0, no from me
16:49:55 <Jeffrey4l> (marked as implemented)
16:49:58 <inc0> spsurya_: go ahead
16:50:09 <vhosakot> inc0: does https://review.openstack.org/#/c/402122/ need more contributors?
16:50:19 <spsurya_> inc0: started triaging of bug for kolla and kolla-ansible
16:50:22 <spsurya_> https://www.irccloud.com/pastebin/tlT8mQPf/
16:50:28 <inc0> vhosakot: if you want to resurrect this change, would be awesome
16:50:39 <spsurya_> these are in progress
16:50:44 <spsurya_> for this week
16:51:02 <spsurya_> So on weekly basis
16:51:13 <spsurya_> i will be triaging the bugs
16:51:22 <inc0> that would be awesome
16:51:24 <spsurya_> and will be reporting the same in meeting
16:51:31 <inc0> let me check if you're in drivers team
16:51:45 <spsurya_> inc0: yes i am in driver teram
16:51:48 <spsurya_> team*
16:51:57 <spsurya_> thanks for checking
16:51:58 <inc0> cool, thank you spsurya_, that's totally awesome
16:52:03 <inc0> also prioritize bugs
16:52:12 <spsurya_> inc0: yes
16:52:14 <duonghq> spsurya_, awesome
16:52:20 <spsurya_> doing the same
16:52:21 <inc0> I hereby nominate you our bug sheriff;)
16:52:39 <spsurya_> inc0: thanks
16:52:55 <spsurya_> that is all from my side for today
16:53:01 <spsurya_> thanks
16:53:18 <inc0> ok, still few minutes left
16:53:22 <inc0> anyone?
16:53:33 <hrw> krtaylor?
16:53:51 <Jeffrey4l> inc0, my test patch for kolla gate. https://review.openstack.org/432901
16:54:01 <inc0> yeah I have it open
16:54:04 <inc0> I'll work on fix
16:54:04 <krtaylor> hrw, I didnt have anything else
16:54:06 <inc0> today
16:54:12 <Jeffrey4l> thanks.
16:54:35 <inc0> ok, let's wrap up:)
16:54:40 <inc0> thank you all for coming!
16:54:44 <duonghq> thank guys
16:54:49 <inc0> #endmeeting kolla