16:01:03 <adrian_otto> #startmeeting containers
16:01:04 <openstack> Meeting started Tue Dec  6 16:01:03 2016 UTC and is due to finish in 60 minutes.  The chair is adrian_otto. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:01:05 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
16:01:08 <openstack> The meeting name has been set to 'containers'
16:01:08 <adrian_otto> #link https://wiki.openstack.org/wiki/Meetings/Containers#Agenda_for_2016-12-06_1600_UTC Our Agenda
16:01:12 <adrian_otto> #topic Roll Call
16:01:14 <adrian_otto> Adrian Otto
16:01:21 <Drago1> o/
16:01:22 <jvgrant> Jaycen Grant
16:01:27 <tonanhngo> Ton Ngo
16:01:31 <mjura_> Michal Jura
16:01:33 <hongbin> o/
16:01:33 <strigazi> Spyros Trigazis
16:01:36 <randallburt> o/
16:02:25 <yatin> Yatin Karel
16:04:33 <adrian_otto> hello Drago1 jvgrant tonanhngo mjura_ strigazi randallburt and yatin
16:05:29 <adrian_otto> #topic Announcements
16:05:31 <adrian_otto> 1) Reminder, there will be no meeting on 2016-12-27
16:05:39 <adrian_otto> are there any other announcements form team members?
16:05:44 <adrian_otto> s/form/from/
16:06:02 <adrian_otto> #topic Review Action Items
16:06:03 <adrian_otto> (none)
16:06:13 <adrian_otto> #topic Blueprints/Bugs/Reviews/Ideas
16:06:18 <adrian_otto> Essential Blueprints
16:06:33 <adrian_otto> #link https://blueprints.launchpad.net/magnum/+spec/flatten-attributes (strigazi)
16:06:36 <adrian_otto> this is (1/4)
16:07:12 <adrian_otto> strigazi: summary of progress from this week?
16:07:36 <strigazi> stable progress, I haven't finished yet. The only issue is that I do tests manually
16:08:14 <adrian_otto> I recall you raised this concern in our last team meeting, about testing the before/after
16:08:32 <adrian_otto> any further thoughts on how to address that?
16:09:10 <strigazi> We could do something similar to ironic, (other projects might also do it)
16:09:50 <adrian_otto> are you referring to grenade, or something else?
16:10:17 <jvgrant> we discussed the need for this testing for both this and other up coming changes. and the possiblity of using grenade for testing
16:10:24 <Drago1> There's a file he'll link, I think. It's not grenade
16:10:33 <strigazi> I don't know if it's called grenade
16:11:03 <Drago1> https://github.com/openstack/ironic/blob/master/ironic/tests/unit/db/sqlalchemy/test_migrations.py
16:11:11 <randallburt> IIRC, yes, grenade tests test db upgrades etc
16:11:34 <adrian_otto> we don't use sqlalchemy, we use alembic, but it has an equivalent of that
16:12:07 <randallburt> alembic should be the standard so there should be things to crib from in other projects
16:12:17 <strigazi> It's a separate task if someone wants to work on it
16:12:18 <jvgrant> swatson had started looking into adding some of this kind of testing, he is on vacation this week but he was interested in working on it
16:12:43 <adrian_otto> ok
16:13:06 <adrian_otto> let's open a bug ticket for that work item and link it to the blueprint
16:13:19 <adrian_otto> that way we can keep it tracked
16:13:33 <adrian_otto> strigazi: would you agree to creating that?
16:13:42 <strigazi> I'm creating it :)
16:13:59 <adrian_otto> ok, so I will skip recording that as an action item then.
16:14:12 <adrian_otto> any more remarks on flatten-attributes?
16:14:33 <adrian_otto> ok next is (2/4):
16:14:33 <adrian_otto> #link https://blueprints.launchpad.net/magnum/+spec/nodegroups (drago)
16:14:39 <strigazi> https://bugs.launchpad.net/magnum/+bug/1489908
16:14:39 <openstack> Launchpad bug 1489908 in Magnum "Tech-Debt: Add tests for DB migration" [Undecided,Won't fix]
16:14:45 <strigazi> we have one already
16:14:46 <adrian_otto> thanks strigazi !!
16:15:07 <Drago1> No update on NodeGroups, unless jvgrant has an update
16:15:24 <adrian_otto> #action adrian_otto to ask swatson about claiming https://bugs.launchpad.net/magnum/+bug/1489908
16:15:38 <jvgrant> just continued discussion the specs around it
16:16:33 <adrian_otto> ok, thanks jvgrant and Drago1
16:16:37 <adrian_otto> next is (3/4):
16:16:39 <adrian_otto> #link https://blueprints.launchpad.net/magnum/+spec/template-versioning (jvgrant)
16:16:59 <adrian_otto> This one definitely has discussion in it that's beyond the scope of the blueprint
16:17:14 <jvgrant> lots of discussion in this review.  Most of it is really around cluster upgrade now
16:17:33 <hongbin> it is not really out of the scope since those bps are inter-related
16:17:38 <adrian_otto> ok, so my advice on this one is to table any action on it for now.
16:18:15 <jvgrant> we made this a dependency for the upgrade spec but it is really isn't
16:18:28 <randallburt> jvgrant:  agreed
16:18:33 <adrian_otto> jvgrant: would you be willing to mark this one as workflow-1?
16:18:55 <adrian_otto> I don't want to stop advancing our plans for improving Magnum
16:18:58 <jvgrant> the template versioning is more about the user experience for operators
16:19:06 <adrian_otto> but I also want to be careful not to go too far into a rabbit hole here
16:19:06 <strigazi> and users
16:19:36 <tonanhngo> maybe we should move some of the discussion to cluster upgrade
16:19:49 <randallburt> if we decouple it (by saying so) from upgrades, it won't hold up that train and can continue to be debated in isolation, sounds like a win to me
16:19:59 <adrian_otto> and my expectation is that our team will continue the discussion on a solution to the fundamental question of how to control policy with relation to acting on clusters and their templates
16:20:06 <jvgrant> let's figure out upgrade first and then we can figure out separately how to improve the experience after
16:20:12 <randallburt> jvgrant:  +1
16:20:16 <adrian_otto> which may adjust the direction of the spec
16:20:17 <hongbin> i am ok to move the discussion to cluster upgrade, however, it will be the same discussion as before
16:20:42 <randallburt> hongbin:  but might be more productive without the template versioning muddying the waters
16:20:44 <adrian_otto> hongbin: acknowledged
16:20:59 <jvgrant> hongbin: the discussion is a very valid one. just want to remove part so it can be more focused
16:21:06 <adrian_otto> right, I'm seeking to optimize reviewers time around the concepts that we are clear on now.
16:22:01 <hongbin> then, the template version spec need an revision to decouple from upgrade
16:22:09 <tonanhngo> Template versioning is also required for nodegroup?
16:22:16 <randallburt> hongbin:  +1
16:22:17 <hongbin> e.g. the upgrade use cases should be removed
16:22:24 <jvgrant> not required either. It is a nice to have
16:22:48 <adrian_otto> So we did mark this blueprint as Essential, but I think we should revisit that, as we can decouple it. We are trying to address a legitimate use case, but it's not a critical dependency for cluster upgrade.
16:22:59 <jvgrant> i'll make it more generic and we can lower the priority and come back to it after upgrade and nodegroup are finalized
16:23:11 <adrian_otto> any objections to making this as a medium?
16:23:17 <jvgrant> nope
16:23:30 <yatin> adrian_otto, +1
16:23:32 <hongbin> not from me
16:23:36 <strigazi> ok
16:23:43 <adrian_otto> ok, I'm going to adjust it, and if anyone has a concern with it, please see me and we'll work through it together, thanks!
16:24:08 <tonanhngo> +1
16:24:21 <adrian_otto> done
16:24:34 <yatin> Same question as ton, Template versioning is also required for nodegroup?
16:24:44 <Drago1> No, it is not required
16:25:07 <adrian_otto> next is (4/4):
16:25:07 <adrian_otto> #link https://blueprints.launchpad.net/magnum/+spec/secure-etcd-cluster-coe (yatin)
16:25:08 <strigazi> no
16:25:12 <adrian_otto> yeah, related but not in the critical path
16:25:13 <tonanhngo> Wonder if there is a use case left if we remove both dependencies
16:25:21 <yatin> Drago1, For what you referenced about reply about jvgrant then
16:25:36 <yatin> I have patches for k8s and swarm for review
16:26:20 <adrian_otto> ok, I marked the BP accordingly.
16:26:24 <Drago1> yatin: I am not sure what you meant in your message to me just now
16:26:31 <strigazi> Can you add some examples on how to test
16:26:40 <strigazi> yatin ^^
16:26:58 <yatin> Drago, No update on NodeGroups, unless jvgrant has an update
16:27:13 <yatin> strigazi, Should i add examples in BP
16:27:18 <strigazi> yes
16:27:24 <yatin> strigazi, OK
16:27:38 <strigazi> thx
16:27:51 <adrian_otto> thanks yatin
16:28:00 <adrian_otto> any more remarks on this work item?
16:28:19 <yatin> No
16:28:42 <adrian_otto> ok, any other work items for team discussion today? Bugs, Reviews, etc.?
16:28:49 <strigazi> yes
16:29:05 <randallburt> would appreciate some eyes on this patch chain: https://review.openstack.org/#/c/396781
16:29:40 <strigazi> We check the k8s functional tests with Mathieu and found out that we don't wait for pods to reach status Running
16:30:34 <strigazi> we modify them to do the right thing but this change will impact the speed and stabillity of ours gates
16:30:39 <Drago1> You mean we just check for existence?
16:30:40 <strigazi> fetching the review
16:30:43 <strigazi> yes
16:30:57 <adrian_otto> randallburt: I will review 396781 this morning.
16:31:06 <randallburt> thanks adrian_otto
16:31:10 <strigazi> #link https://review.openstack.org/#/c/404253/
16:31:13 <dirk> I would like to RFC on https://review.openstack.org/#/c/390668
16:31:33 <dirk> is bashate something worthwile for magnum? should I file bug/blueprint first?
16:31:57 <strigazi> in many cases the tets succeed but in others don't.
16:32:16 <strigazi> Should we procceed with it?
16:32:31 <randallburt> tech debt bug would be a good idea IMO dirk just for tracking and input
16:32:53 <adrian_otto> strigazi: I would rather have reliable, consistent gates rather than ones that fail faster intermittently
16:33:27 <strigazi> but we don't tests a real result
16:33:27 <adrian_otto> strigazi: can we quantify the delay that the currently contemplated fix may introduce?
16:33:37 <randallburt> strigazi:  I think our tests should ensure we're functioning correctly. I think we can sort gate lag after
16:33:43 <Drago1> I think that properly testing the functionality of clusters is important
16:33:45 <mvelten> the problem is that we actually don't test k8s properly
16:34:24 <mvelten> we are checking for the creation of the pod at the api level but not wait for it to actually be scheduled
16:34:25 <Drago1> Third party gate!
16:34:33 <adrian_otto> Drago1: agreed
16:34:54 <randallburt> well, we shouldn't be testing k8s per se, but we should be testing that our drivers result in a functioning clusters and imo, that includes checking that pods are up and running
16:35:00 <randallburt> Drago1:  lol
16:35:04 <adrian_otto> yes, ideally we'd have a third party gate per COE driver
16:35:09 <Drago1> randallburt: Not a joke!
16:35:25 <randallburt> as long as they're voting
16:35:28 <mvelten> is there a way to force our job to a provider ?
16:35:29 <strigazi> only OSIC resources would work
16:35:32 <randallburt> if its in our tree, it *must* work
16:35:52 <adrian_otto> mvelten: not with OpenStack CI, but with 3rd party gate yes
16:36:00 <yatin> May be we can create custom image for gate tests that have already hyperkube images then gate time can be shorten up
16:36:10 <randallburt> yatin:  +1
16:36:24 <Drago1> It depends on how large the hyperkube image is. I think it is small, so it should be fine
16:36:33 <strigazi> 150MB
16:36:33 <yatin> 187 MB
16:36:35 <mvelten> yatin we thought about it, it is not that simple, there is currently no easy way to do that
16:36:51 <Drago1> Large images still take forever to `docker load` and cause systemd to fail it
16:36:57 <mvelten> exactly
16:36:58 <strigazi> ok, I'll have a look in tarballs again
16:37:06 <mvelten> so embedding a tar will not help
16:37:21 <Drago1> 150 MB is probably too big still
16:38:07 <adrian_otto> dirk: we will get to your remarks about https://review.openstack.org/390668 in just a moment
16:38:18 <yatin> mvelten, Drago1 we had some images over 500MB in tar but that didn't failed at docker load
16:38:21 <Drago1> Unless it could be pre-loaded into docker too, then it's fine
16:38:27 <mvelten> if we want that we need to package /var/lib/docker directly so it is already imported
16:38:49 <Drago1> yatin: Did you test that in the gate? I did too but in the gate it fails
16:39:07 <yatin> Drago1, Not in gate
16:39:08 <mvelten> +1 the problem is again no kvm nested :)
16:39:17 <adrian_otto> mvelten: this should become a non-issue in 3rd party CI
16:39:50 <randallburt> right, if we can do third party voting, then I don't see why we wouldn't optimize that way
16:41:00 <hongbin> i want to bring up one thing if nobody else has any item to discuss
16:41:03 <mvelten> ok yes we need nested somehow in CI or it will come back again and again
16:41:04 <Drago1> Okay, so how important to us is getting a 3rd party gate
16:41:14 <Drago1> Because we keep finding reasons it's useful
16:42:16 <adrian_otto> let's do this... Let's set 3rd party gate testing as a top level discussion topic for next week
16:42:33 <adrian_otto> and we can invite representation form our infra team to help us make an informed decision about it
16:42:42 <adrian_otto> s/form/from/
16:43:08 <adrian_otto> one downside I see is that someone needs to provide the infrastructure to run the driver CI
16:43:33 <adrian_otto> so we'll need to find parties willing to do that
16:43:34 <hongbin> i think tonanhngo got some machines from CNCF, we could leverage those servers
16:43:42 <hongbin> tonanhngo: ^^
16:43:55 <tonanhngo> hongbin:  that was only for a short time
16:44:08 <hongbin> tonanhngo: bumper
16:44:08 <randallburt> adrian_otto:  sounds good
16:44:27 <strigazi> adrian_otto: sounds good
16:44:43 <jvgrant> short term was there a decision on the review strigazi brought up? should we added waiting for the pods?
16:44:44 <adrian_otto> and we can start an ML discussion too so that we have some background set before coming back to the topic
16:44:54 <mvelten> for the record we dont have nested for now. We are trying to switch it on for new HV
16:44:54 <adrian_otto> I can take that action item
16:44:55 <randallburt> jvgrant:  IMO, yes
16:45:11 <strigazi> +1 ^^
16:45:19 <tonanhngo> I think we should wait for the pods, otherwise it's not really a valid test
16:45:24 <randallburt> agreed
16:45:28 <jvgrant> +1
16:45:29 <adrian_otto> agreed
16:45:42 <adrian_otto> ok, dirk.
16:45:50 <strigazi> thanks
16:45:51 <adrian_otto> hongbin, I noted you have a topic too.
16:45:53 <yatin> tonanhngo, +1
16:46:07 <dirk> adrian_otto: yes :)
16:46:10 <hongbin> i suggested to bump the priority of the cluster-upgrade bp to essential
16:46:29 <mvelten> and according to spyros the rackspace vms in the common CI pools doesnt have it either
16:46:32 <adrian_otto> dirk: besides indentation, is there some other motivation to introducing a pop8 equivalent for shell scripts?
16:46:34 <hongbin> the cluster upgrade feature is a blocker for magnum to be production-ready imo
16:46:35 <adrian_otto> pep8
16:46:53 <randallburt> hongbin:  agreed
16:47:03 <hongbin> it should be classified as essential so that we could discuss it every week at the team meeting
16:47:16 <yatin> hongbin, +1
16:47:17 <dirk> adrian_otto: consistency. one formatting style, less nitpicks over formatting issues for future changes..
16:47:18 <adrian_otto> ok, I am happy to elevate set cluster upgrade to essential
16:47:28 <strigazi> ok
16:47:34 <Drago1> I'm happy to ride herd on it
16:47:34 <mvelten> I have a problem with the indentation patch. It can break stuff quite easily
16:47:47 <adrian_otto> dirk: from what I can see there is no test for style besides indentation. Did I miss something?
16:47:52 <mvelten> I think
16:47:56 <Drago1> Oh, nevermind, strigazi is already assigned
16:48:26 <dirk> adrian_otto: I didn't want to enable all bashate tests as that made it unreviewable. I can do followups that enable more tests
16:48:39 <randallburt> I think that's just the first check to keep it from being too disruptive too early
16:48:41 <dirk> adrian_otto: just wanted to see as a test balloon whether that route would be acceptable at all
16:48:51 <mvelten> if a tool or a cat or anything expect good tabbing / new lines it would break it
16:49:04 <randallburt> mvelten:  that's kinda the point IMO
16:49:21 <randallburt> mvelten:  run tox before you post a review to catch things like that
16:49:57 <randallburt> if we have a significant number of artifacts in bash (and we do), then it should be linted IMO
16:50:00 <adrian_otto> hongbin: I set https://blueprints.launchpad.net/magnum/+spec/cluster-upgrades as Essential
16:50:08 <hongbin> adrian_otto: thank you
16:50:09 <adrian_otto> hongbin: did you also have an additional topic, or was that it?
16:50:18 <Drago1> Linting for heat templates would be nice too
16:50:19 <hongbin> no. that is all from me
16:50:23 <vijendar> I have one topic. I would like to get  a spec reviewed https://review.openstack.org/#/c/391537/ to proceed with implementation
16:50:32 <adrian_otto> thanks hongbin
16:50:33 <randallburt> Drago1:  indeed. don't know if we ever built something like that
16:50:39 <mvelten> linting python is fine, bash which rely on calling other binaries is weid to me
16:50:40 <Drago1> Yes, let's discuss that, vijendar
16:51:12 <adrian_otto> vijendar: we will get to that in one moment
16:51:23 <vijendar> adrian_otto: ok
16:51:41 <jasond> mvelten: yeah, if it did break something, we probably wouldn't see it in the gate
16:51:52 <jasond> mvelten: at least, not always
16:52:00 <randallburt> this is checking style/formatting, not functionality
16:52:33 <yatin> Is this earlier bug related: https://bugs.launchpad.net/magnum/+bug/1561232
16:52:33 <openstack> Launchpad bug 1561232 in Magnum "bashisms found in /bin/sh scripts" [Low,Fix committed]
16:53:18 <mvelten> the stuff is that formatting can matters to underlying tools. if bashate now how to handle that fine
16:53:39 <adrian_otto> so is shell script style an actual problem for our team? If so, I have not seen the evidence of it yet.
16:54:09 <adrian_otto> I'm reluctant to shake things up unless we are sure this is solving a problem
16:54:27 <mvelten> lets say I write a yaml file like we do in k8s, does the bashate breaks the indentation ? it would break the fonctiunality
16:54:28 <hongbin> the check of any script could be helpful imo
16:54:30 <dirk> adrian_otto: e.g. devstack uses it
16:54:48 <adrian_otto> the bug 1561232 happened because of upstream packaging tools had problems with bash code in shell scripts.
16:54:48 <openstack> bug 1561232 in Magnum "bashisms found in /bin/sh scripts" [Low,Fix committed] https://launchpad.net/bugs/1561232
16:55:00 <adrian_otto> dirk: that makes sense, as devstack is a shell script
16:55:04 <dirk> adrian_otto: inconsistent indenting can hide bugs (e.g. 2 spaces / 4 spaces in the same file)
16:55:30 <Drago1> It's also annoying
16:55:30 <randallburt> IMO, if it doesn't break things and can help increase code quality, there's no reason not to
16:55:48 <yatin> randallburt, hongbin +1
16:55:50 <adrian_otto> I agree, but the worry is that it may break things that are not apparent until later
16:56:13 <randallburt> scripts in yaml probably won't be checked which is good, but then we could use that as the impetus to move those scripts out of the templates
16:56:25 <Drago1> Then we need to write tests that would cover those things, so we obviously need a 3rd party gate even more!
16:56:34 <mvelten> I mean we have scripts that write yaml through echo
16:56:41 <mvelten> sh scripts
16:56:46 <randallburt> mvelten:  oh, gotcha
16:56:52 <mvelten> :)
16:56:53 <dirk> adrian_otto: ok, then how about we try the consistent-indenting rule first and see whether it breaks anything?
16:57:06 <randallburt> dirk:  +1
16:57:08 <dirk> adrian_otto: I don't have a lot of experience with bashate but from all usages I had so far I had zero issues due to it
16:57:14 <adrian_otto> ok with me. Let's take this discussion to the review thread
16:57:23 <adrian_otto> we don't have strong objections to it right now
16:57:24 <dirk> great, thanks for the time to discuss this!
16:57:32 <adrian_otto> vijendar: you asked for reviewers to have a look at https://review.openstack.org/391537
16:57:34 <dirk> yep, thanks all
16:57:43 <vijendar> yes
16:58:27 <vijendar> so far I got couple of +1s on that, but need more reviews to get it merged
16:58:37 <adrian_otto> I'll look at the recent revision this morning
16:58:47 <adrian_otto> only one minute left for open discussion, sorry
16:58:53 <adrian_otto> #topic Open Discussion
16:59:35 <yatin> [openstack-dev] [Kuryr][Magnum] How stable is kuryr-kubernetes?
16:59:50 <yatin> There is an item that should be looked up
17:00:02 <yatin> May be ton is working on the integration part
17:00:12 <tonanhngo> It's still in development
17:00:12 <adrian_otto> Thanks for attending. Our next team meeting will be at 1600 UTC on 2016-12-13. See you all then!
17:00:19 <adrian_otto> #endmeeting