15:59:20 #startmeeting kolla 15:59:21 Meeting started Wed Apr 12 15:59:20 2017 UTC and is due to finish in 60 minutes. The chair is inc0. Information about MeetBot at http://wiki.debian.org/MeetBot. 15:59:22 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 15:59:25 The meeting name has been set to 'kolla' 15:59:35 #topic w00t for Kolla 15:59:38 O/ 15:59:39 o/ 15:59:40 o/ 15:59:45 woot 15:59:50 w00t o/ 16:00:02 o/ 16:00:17 oOo/ 16:00:22 o/ 16:00:27 o/ 16:00:30 o/ 16:01:13 o/ 16:01:15 o/ 16:01:21 o/ 16:01:22 o/ 16:01:42 o/ 16:01:57 since we have busy schedule I'll move on 16:02:06 #topic announcements 16:02:20 I have one "help your project" request 16:02:48 we'll hold hands on workshop for deploying kolla-ansible in Boston 16:03:02 and we'll need mentors in room to help workshopers deal with issues 16:03:18 so if anyone is interested in helping, please let me know after meeting 16:03:21 inc0 who is leading? 16:03:25 I do 16:03:34 your leading the workshop? 16:03:37 yes 16:03:39 is it the 2 hour lab thing? 16:03:42 yes 16:03:45 cool count me in iif my schedule doesn't conflict 16:03:50 thanks 16:04:08 ok moving on 16:04:18 im in as well 16:04:24 barring schedule conflicts 16:04:31 #topic protocore (an onramp to core reviewer) (pbourke) 16:04:34 agg 16:04:38 sorry one more announcement 16:04:45 very important to me 16:05:11 recently it came to my attention that some of us have issues with visa entry to US for summit/ptg 16:05:24 if you are affected, please let me knw on email 16:05:37 inc007@gmail com 16:05:50 +1 16:06:12 I'm trying to asses scale of this issue to make sure we're not locking out anyone from discussion 16:06:19 ok, let's get back to meeting 16:06:27 so protocore, pbourke sdake you have the floor 16:06:43 so, sdake had asked me to cover this, but it seems like he's around? 16:06:56 pbourke go ahead an dcover it - i'm in training sort of 16:07:02 ok 16:07:27 so sdake just brought this up earlier today, we dont have all the details but, 16:07:41 I can do it if you guys want 16:07:41 o/ 16:07:46 and sdake you can correct me 16:08:10 the idea came about of having an onboarding program for new core reviewers 16:08:29 so we wanted to see if people think this is a good or bad idea 16:08:45 Good one. 16:08:58 pbourke i dont know that everyone has heard the idea ;) 16:08:58 +1 for idea 16:09:02 so I think it's wonderful idea, question I'm still strugglign with is how exactly will that look like 16:09:18 summary is people who would like to be core reviewers will be added to a group. During a certain period a +1 from members of this group will count as a +2 16:10:05 if a trial core +1's and a current core disagrees, it is the responsibility of the core to explain why in additional detail 16:10:30 i agree with it 16:10:33 I think the idea is they count as 1 of the +2's needed to wf. but can't be both. 16:10:42 kfox1111 right 16:10:55 kfox1111 also trials don't have capability to +2 or +w a review 16:10:58 they are in "training" 16:10:59 apparently nova are currently doing this, so it would be good to get some detail from them 16:11:01 my question for this would be - if core seconds with +2 do we merge with single +2? I'm on the fence with this 16:11:30 I think we should not remove the second +2. 16:11:33 inc0 yup it is a reasonable concern because it places more responsibility on core reviewers then theey already have 16:11:39 personally, I'm not sure this is actually necessary. That said, I think it could be a nice entry path for people, as kolla and kolla-ansible in particular are short on cores lately imo 16:11:42 +1 not remove second +2 16:12:01 at least at the beginning 16:12:02 i think it would be best to have an "agreement" on those protocores that they do not do +w 16:12:04 but they can do +2 16:12:10 even if it doesn't remove the second +2, it might alow existing cores to prioritize those reviews, 16:12:27 getting them in faster, and building up the protocore's knowlege to the point where more cores are grown faster. 16:12:28 nova doesn't permit protocores to +w or +2 a review 16:12:28 so a protocore +2 + core +2 can only be wf'd by that 2nd core (or a 3rd core) 16:12:28 yes, in this case that's totally fine, we keep 2*+2 and just put more pressure on explaining why we didn't +2 16:12:31 they just arne't in the gerrit g roup 16:12:39 Is it not possible to have a group that can give +2 without being able to approve a review? 16:12:48 have we seen a case were a new core reviewer performed sub standardly? I would say no... 16:13:11 berendt I am not sure if that is possible but it may be 16:13:15 no, but that's because we have high standards for cores 16:13:22 berendt that would be a great solution to the 2 +2 problem :) 16:13:36 so I have 2 different approaches to this, one is already being in nova for some time 16:13:36 pbourke: the bar has been high for cores in the past, so few mistakes are made. 16:13:46 that being = mentorship 16:13:51 ok makes sense 16:13:55 kfox1111 the thing is protocores aren't voted on 16:14:07 sdake: I think the main idea is not looking for a short cut to 2x+2 issue but to motivate people to be more active 16:14:12 kfox1111 however we expect people to learn how to become a core reviewer without training them 16:14:18 this magic incantation stage - is very diffficult 16:14:24 hence mentorship 16:14:43 every core can become a mentor for non-core, or protocore 16:14:45 sbezverk, +1 16:14:46 sdake: yeah. this is a way to help train reviewers to be cores. it makes sense to me. 16:14:50 yup - and the wayt o be mentored is to be excited that your review actually matters 16:15:06 so... 16:15:19 sdake: but I'm not ready to remove need for 2*+2 16:15:21 mentorship is a two way street :) and requires interest form both aprties 16:15:24 shoudl each protocore be assigned to 1+ core officially? 16:15:35 inc0: +1 for not removing it 16:15:45 we could offer protocorss +2 but not +w 16:15:50 then this mentor core can just +2 his mantees change 16:15:51 protocores rather 16:15:52 inc0: yeah, I think we have enough cores now ,that a core can ask another core for a priority review to deal with the +2. 16:16:14 the nultimately teh responsiblity of the core reviewer is preserved 16:16:27 and mantee will have trusted core to either ask for review and get +2 or ask for feedback 16:16:34 lets ask the quesiton 16:16:42 who would be interested in participating as a protocore 16:16:44 protocors with +2 but not wf sounds very much like what google does. 16:16:44 can we get a liset? 16:16:51 they have cores, and approvers. 16:17:04 sdake: let me ask this clearly - are you suggesting merging changes with just single +2 and +1 from protocore? 16:17:10 cores make sure the code is up to snuff. aprovers make sure the code fits in the overall archetecture properly. 16:17:26 no, I'm suggesting adding +2 ability to a protocore, and merging based upon 2 +2s and a +w from a core reviewer 16:17:59 so it doesn't changing anything really, we still merge with 2*+2 from cores 16:18:01 in fact ithink berendt suggested that :) 16:18:07 sdake: but if you give +2 protocore it is not the same as 2x+2 from cores 16:18:19 so you relaxing quality checks 16:18:29 sbezverk: right. 16:18:38 that is the risk in it. 16:18:46 kfox1111: I do not think it is right 16:18:50 sbezverk i think the tradeoff is worth it to train the next gen of core reviewers 16:18:54 can two protocores +2 a single issue? 16:19:00 jascott1_ nada 16:19:10 jascott1_ they could but always takes a core reviweer to +w any change 16:19:12 sbezverk: yeah, I get the risk. the risk in not having enough cores is also a problem. 16:19:14 sdake the system prevents it? 16:19:16 ok...so thing we want to solve is training 16:19:20 it seems like the short term solution and long term solution may be different. 16:19:23 ah ok 16:19:27 and I'd suggest more official mentorship program for that 16:19:35 at least for start to see if that's work 16:19:37 this is that inc0 16:19:44 no sdake, not exactly 16:20:06 nova is anal anal retentive about reviews 16:20:06 by mentorship I mean each protocore will have mentor (known by name) 16:20:11 yet they permit a protocore to count as a +2 16:20:16 are we more anal then nova about reviewing? 16:20:33 sdake: we are different project and we deal with our policies differently 16:20:39 inc0 i get that 16:20:51 inc0 we generally have pretty lax policies in comparison to other projects 16:20:54 not more stringent 16:21:02 what I *do not* want to do is to rush issue that could potentially lower our quality 16:21:09 the one issue I see is, 16:21:26 nova's pretty stable at this point. not so many breaking architectural changes. 16:21:26 sdake: maybe because nova is not ivolving that rapidly as we, they can afford it 16:21:34 I'm not saying it's bad idea, I'm saying let's spend some time exploring and thinking about problem 16:21:36 kolla-kubernetes is changing a lot. 16:21:43 so the risk of paches going in that break things are higher. 16:21:57 yeah 16:21:57 so do cores tap protocores that they see potential in or do protocores sit in the snow outside temple for period of time? 16:22:01 kolla-ansible is somewhere in between 16:22:01 we have until september to shake out kolla-kubernetes kfox1111 16:22:03 kfox1111: +2 that is why we need to be extra cautios 16:22:15 since nova still have hard spec process 16:22:25 which means changes have lots of context, we don't follow specs 16:22:35 I think kolla-images is quite stable in arch 16:22:35 so our volitality is higher for new changes 16:22:52 lets have a summary of where we agree and disagree 16:22:56 inc0 can you do that plz? 16:23:16 not yet, I feel we didn't explore different ideas to solve the problem 16:23:26 can we start ML thread? 16:23:43 indeed 16:23:49 thanks 16:23:55 I'm not saying it's bad idea 16:24:03 in fact I totally support idea in general 16:24:13 yeah. that would give other projects that may be struggling the ability to discuss too. 16:24:16 it's details of implementation that I want to get sorted out before we agree/disagree 16:24:32 your focused on implemetnation - I have one quesiton 16:24:38 do we agree to the following 16:24:52 1) we have folks that want to be core reviewers 16:25:00 2) they dont know how to become core reviewers 16:25:20 yeah. 16:25:23 totally 16:25:23 3) the core reviewers will mentor these protocores on how to become core reviewers 16:25:31 yup 16:25:34 4) they should be incentivized in some wya to know that their voice matters 16:25:41 (vs a drive by reviewer) 16:25:57 soooo here's where I want to stop (4) 16:26:02 sdake: incentive should not impact the quality 16:26:04 *everyones* voice matters 16:26:06 i am actuallly done 16:26:07 :) 16:26:18 sup rwallner 16:26:22 rwellum that is :) 16:26:23 not just core or protocore 16:26:59 sbezverk the incentive is to learn how to become a core reviewer 16:27:02 I want to break this perception that you need to be core for your voie to matter 16:27:06 part of that is taking the responsibility of reviewing properly 16:27:19 and adding new layer of "priviledged" will not help imho 16:27:26 sbezverk the incentive isn't the ability to +2 a review 16:27:26 but again, let's have ML discussion 16:27:41 sdake: I think the way to become core is to be more active 16:27:51 I'm in full support of having cleared path to core 16:27:57 hi sdake sorry about my lateness, it's at the wrong time on my calendar 16:27:58 if you do not want to be active then, you just not ready for the core 16:28:01 sbezverk we need to WRITE DOWN how to beomce a core reviewer 16:28:08 there are people that actively want it yet dont know how to obtain it 16:28:19 sdake: easy more reviews and more contribution :) 16:28:20 rwellum DST FTL :) 16:28:27 sdake agree. documenting expectations would be nice as well 16:28:30 sbezverk how do you do a good review? 16:28:35 sdake: I think the main disconnect is, 16:28:41 people need to revew stuff, 16:28:50 kfox1111 right - and they dont know *how* 16:29:02 and cores need to be able to notice enough quality reviews from a person are happening to suggest promotion. 16:29:08 sdake: I tihnk it's less about how and more about "voice matters" 16:29:17 sdake: frequency helps with that. 16:29:32 ok, I'll cut us here 16:29:33 if they reveiw something, and say +1, and then a core -1's it, there's a learning opertunity. 16:29:44 this is important and we need discussion across community 16:29:45 eventually the reviewers learn what to look for. 16:29:51 so please, let's start ML thread 16:29:51 inc0: yeah. 16:29:53 yes although we are squandering it by not teaching the new potential core reviweeer 16:30:06 ml it is 16:30:08 sdake: I don't disagree. 16:30:11 thank you 16:30:13 moving on 16:30:16 #topic kolla devstack (pbourke) https://review.openstack.org/#/c/454690/ 16:30:22 kfox1111 if you dont disagree, you could say ou agree ;) 16:30:40 so, just wanted a brief discussion on this 16:30:42 sdake: at this point it's not that we agree or not, it's how we implement it 16:30:55 sdake: assumed boolean. ;) 16:31:04 right. 16:31:05 thanks pbourke :) 16:31:16 it came up quite a while ago that people would like to see kolla as a viable replacement for devstack 16:31:18 ahh, we're here at last 16:31:35 it was lingering idea around 16:31:43 an interesting idea. 16:31:52 I've put up a patch for a possible implementation, but would like to get some ideas on whats needed and whether this can even work 16:31:54 there is few immmediate benefits for that 16:32:10 1. multinode is assumed from start so will allow people to do better testing of multinode code 16:32:41 2. you'll be able to run upgrades from latest master to your codebase to ensure upgrades aren't breaking 16:32:51 #link https://review.openstack.org/#/c/454690/ 16:32:55 I can think of 1 major benifit and 1 major drawback right away from the idea. 16:33:08 kfox1111: the benefit is you dont have to use devstack :p 16:33:25 the beinifit is its closer to what operators would want to use, so devs will see operator problems sooner. 16:33:54 the drawback is, it will potentially be more work for devs to do quick dev changes repeatedly to work on a ps. 16:34:10 kfox1111: second one is fixable 16:34:15 by making it easy 16:34:21 I've been using the above patch the past day or two to work on a heat change and its quite easy to work with 16:34:34 inc0: yeah. its solvable. but non trivial. 16:34:40 you make a change and restart the container, same as you'd restart a process 16:34:45 will require code and good ideas 16:34:51 there one issue in this implementation. what if the end-user changed the database schema, or add a new python dependency? 16:34:52 and I think this is what this change is about 16:35:25 Jeffrey4l: rebuild of container should be way to go 16:35:27 others LGTM 16:35:32 inc0: you dont need to rebuild 16:35:39 inc0: in most cases 16:35:42 pbourke: can you put in a document describing how that flow works into the ps? 16:35:45 I think that might help a lot. 16:35:48 kfox1111: sounds good 16:35:50 also, this will require a lot of howtos 16:35:54 and some code in or codebase 16:36:08 it may be easy. its just not obvious to me. 16:36:50 devstack for the time of its existance, produced lots of useful scripts developers got used. absense of these scripts on kolla side might delay adoption it as devstack replacement.. 16:36:53 I starred this review, will throw some ideas I've been playing with over the time 16:37:12 sbezverk: but also has a lot of baggage 16:37:23 for example it's not easy to test upgrades 16:37:26 i haven't used devstack in a long time so if people are used to it I'd be interested in what parts of the workflow they need the most 16:37:36 and it will take over your node 16:37:43 where kolla can be quickly removed 16:37:44 inc0: right but when you do not have a spare t-shirt it is not very conveneient ;) 16:37:49 devstack can't 16:37:50 right now I can guarantee there are people execing into containers and making changes 16:37:57 so anything is better than that 16:38:22 as a certain project developer, for example horizon, who can use kolla to set up whole stack quickly. 16:38:40 and tear it down quickly 16:38:41 better than devstack. 16:38:44 yep. 16:38:48 and deploy it prod-like 16:39:06 this direct is meaningful ;) 16:39:46 #action check pbourke's review and throw ideas to the basket 16:39:54 thanks inc0 sounds good 16:40:01 thats it from me 16:40:09 thank you Paul for taking this work, it's been whispered around for few years already;) 16:40:26 ok moving on 16:40:31 #topic tagging of images for automatic pushes to dockerhub (inc0) 16:40:45 soo..we are close to create auto publisher of images 16:40:53 after merge image will be pushed to dockerhub 16:41:21 issue is tagging these images 16:41:33 today we tag with github tag 16:41:44 and I think this is good and proper behavior 16:41:57 so 4.0.0 image tag corresponds with 4.0.0 git tag 16:42:25 but having images from HEAD of each stable branch+master will not follow this tagging 16:42:31 should not at least 16:42:53 so I see few options here, none of which are perfect 16:43:02 We could simply use the first part of the git hash. 16:43:08 1. we create whole new set of tags, for example nova-api:ocata 16:43:36 berendt: negative, if someone would like to keep upgrading their env to HEAD of stable, which is good use case 16:43:41 that would be horrible experience 16:44:11 plus you'd end up with thousands of tags very soon 16:44:12 berendt, git hash will generate lots of useless images. 16:44:16 on a semi-related note, i had an interesting discussion with dmsimard a while back about this about tagging and he mentioned that it would be interesting to look at the image tag to be a version that corresponds to the version of the software in the container (e.x: nova-api:14.0.0) rather than nova-api:4.0.0 (which is the kolla version) 16:44:45 Using hashs in CI currently, there it is working pretty good. 16:44:48 mnaser: how do you then map that to the appropriate kolla version though 16:44:54 mnaser: so if we use release codename that would be quite easy to figure out 16:45:24 berendt: because nobody needs to upgrade their globals.yml every time hash changes;) 16:45:35 mnaser, for current implementation in kolla, this is hard ( kolla build all images rather one of images) 16:45:44 yeah, release codename is really easy too, i agree inc0 .. pbourke, interesting issue but the idea was: kolla 4.0.0 bits were used to generate nova-api 14.0.0 16:46:05 so the bits themselves are those of nova-api 14.0.0, the code used to build it was 4.0.0 16:46:11 why cant we use the branch name again? 16:46:17 problem with release name would be automatic tagging detection aka pbr 16:46:36 I think the 4.0.0 thing is not quite right. 16:46:39 Using the release number of Nova does not currently work because we only define one version in the globals file. 16:46:41 I think it should tag based on updates. 16:46:43 i do not think use the nova-api:14.0.0 is better. it cause another issue that which cinder version should i use? 16:46:44 on the bright side...globals.yml openstack_release will point to actual release;) 16:46:54 so, 4.0.0-1, 4.0.0-2, 4.0.1-1, etc. 16:46:59 Jeffrey4l the appropriate cinder version for that release. but branch names works for me except i think that will generate a huge churn 16:47:12 aliases can be made for 4.0.0-2 to 4.0.0 in case people don't want to care about atomic updates. 16:47:23 i.e.: is nova-compute:ocata really stable/ocata? will we rebuild on every merge of every project? 16:47:27 kfox1111: 4.0.0-2 would mean second commit after tag? 16:47:46 mnaser: no, dailt 16:47:47 inc0: I'm talking about stable not trunk. 16:47:48 daily 16:47:49 what issue are we talking and solving ;( 16:47:55 tags would be for unversioned changes. 16:48:05 i think i've created a different conversation, sorry 16:48:10 so I guess for trunk, yeah, it would be incrementing the release, not the version. 16:48:25 going back to topic, nightly builds tagged with branch name for kolla seem reasonable 16:48:31 but for stable, anything that causes any changes to the container that arn't a version change need to bump revision. 16:48:34 kfox1111: well we still keep tagged images 16:48:43 as in once we tag, we push 16:48:46 mnaser, yep, for branch image is useful. 16:48:48 so, for example, shared libraries needing to update. 16:49:13 inc0: versions are for bunches of patches "released". 16:49:21 so, kolla 4.0.1 is a bug fix release to 4.0.0. 16:49:26 now what if a change was introduced in stable/ that needed something in $your_fav_kolla_deployment_tool ? 16:49:34 kfox1111: but we also rebuild at that time 16:49:44 now you deploy from the tagged branch but things break 16:49:58 but 4.0.1 built at 4/12/2017 is different then 4.0.1 built at 5/12/2017. 16:50:14 due to security updates of base packages. 16:50:15 kfox1111: agree, but we can't solve this in kolla 16:50:24 in dockerhub at least 16:50:24 so, the first should be 4.0.1-1 and the second 4.0.1-2. 16:50:26 in dockerhub. 16:50:33 kfox1111, this is not big issue, too. 16:50:42 ^ i think that is the more fundamental issue which will help us answer the part for inc0's question 16:50:49 Jeffrey4l: it is a big issue to folks wanting security updates. 16:50:59 well, it's going to be big issue if we want to have same experience while pulling from dockerhub and building locally 16:51:06 if you want security updates, just update all your containers. 16:51:09 not being able to tag revisions means its hard to know whats updated and whats not. 16:51:16 Jeffrey4l: thats a very costly thing to do with openstack. :/ 16:51:34 i disagree, as an operator, if there was some sort of really big change that happened from 14.0.1 to 14.0.2 in nova and 4.0.0 happens to change on that deploy 16:51:35 I want the ability to update only the stuff needed, when needed. 16:51:37 $bad_stuff 16:51:38 I agree with kfox1111 ops need to be able to selectively pick and chose containers 16:51:42 otherwise, how to get the updates? 16:52:10 kfox1111, it is hard to define *which* stuff need to be update. 16:52:17 what I don't agree with you kfox1111 really is that ops that wants this type of control would use dockerhub... 16:52:27 comparing whats running on your system, vs whats in the repo's is step 1 to solving it. 16:52:32 I'd rather suggest keeping local registry and control your builds 16:52:33 if the names are the same, you have no idea whats going on. 16:52:43 if you wanna just update nova-api from 4.0.1 to 4.0.2, then just update nova_api_tag: variable 16:52:48 if you see 4.0.0-1 is on your sytem, but 4.0.0-2 is released, now hyou know things. 16:53:11 and can start digging into why the change was made, and why you may or may not want to apply it. 16:53:17 hrm are u talking daily build for kolla tag? 16:53:20 kfox1111: that could maybe solved with labels 16:53:28 also I don't want us to become effectively repository mgmt team for openstack 16:53:29 Jeffrey4l: kind of. security updates. 16:53:32 no other projects on dockerhub use that kind of tagging 16:53:40 shouldn't the operator use self-build images? use hub.docker.com is danger. 16:53:41 waaay beyond our capability 16:53:49 at least, i will not do this.. 16:53:54 pbourke: and many projects on docker hub are riddled with seciruty problmes. :/ 16:54:06 inc0: its an autobuild thing. its not hard. 16:54:11 kfox1111: im mainly talking about the official images 16:54:17 kfox1111: which should serve as a reference 16:54:22 pbourke: I am too. 16:54:27 the reference images shoudl be secure. 16:54:37 ok...so I suggest something different for now 16:54:39 should not need to build the images themselves. 16:54:43 we need to have both conversations 16:55:00 but short term, let's figure out autobuild not-fully versioned 16:55:16 and have discussion on version mgmt in general 16:55:26 the revision's not hard to do. you just have to know what version was built last, 16:55:31 and increment it by one before pushing. 16:55:43 the infra for pushing is the hard bit. 16:56:01 I agree, but doing this on buildtime locally will be fun code to write 16:56:22 and we need to do this on build time locally to have similar experience on local registry and dockerhub 16:56:28 which for me is important 16:56:34 +1. 16:56:44 for folks that want to ci/cd it themselves, 16:56:46 because again, we really suggest to build your own images 16:56:47 they will need the same functionality. 16:56:52 and again, will need revisions. 16:57:00 this is how I'd run my prod anyway 16:57:07 just driven by jenkins or whatever. 16:57:18 ok, we're running out of time 16:57:43 I'll start ML thread about both issues - 1. autotagging for HEAD for daily pushes to dockerhub 16:57:50 and 2. general revisiion mgmgt 16:57:52 mgmt 16:58:05 or rather, do we agree with nova-api:ocata approach? 16:58:20 for HEAD builds? 16:58:21 for stable or just trunk? 16:58:26 both 16:58:32 re nova-api:ocata, agree, but need tell the end-user, this is not stable. 16:58:40 it's stable 16:58:42 and not recommended to use directly. 16:58:46 anyway, we're out of time 16:58:50 why not just nova-api:trunk? 16:59:03 thank you all, let's move it to #openstack-kolla 16:59:06 #endmeeting kolla