16:00:26 #startmeeting Kolla 16:00:27 Meeting started Wed Apr 5 16:00:26 2017 UTC and is due to finish in 60 minutes. The chair is inc0. Information about MeetBot at http://wiki.debian.org/MeetBot. 16:00:29 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 16:00:31 The meeting name has been set to 'kolla' 16:00:39 first 16:00:41 o/ :) 16:00:44 #topic rollcall 16:00:48 w00t! 16:00:53 o/ 16:00:53 o/ 16:00:57 repeat after me 16:01:01 w-o-o-t 16:01:11 w-O-0-t 16:01:13 o/ 16:01:19 o/ 16:01:21 w00t 16:01:39 o/ 16:02:17 o/ 16:02:17 o/ 16:02:22 o/ 16:02:27 o/ 16:02:32 w00t o/ 16:03:10 #topic announcements 16:03:30 I've got nothing, but maybe someone from communi? 16:03:42 hmm 16:03:49 deadline for kolla-kubernetes is 4/15/2017 16:03:56 for 0.6.0 16:03:59 that is all :) 16:04:04 kk 16:04:11 o/ 16:04:25 #topic Non-x86 support patchset (hrw) https://review.openstack.org/430940 16:04:29 oy 16:04:32 Marcin, you have the floor 16:04:37 so... 16:04:45 base_arch variable got merged last week 16:04:47 I sent an email to the ML, but would like to announce our project kayobe which we've recently open sourced. https://github.com/stackhpc/kayobe 16:05:03 https://review.openstack.org/#/q/owner:%22Marcin+Juszkiewicz%22+status:open lists my patches 16:05:19 provides some additional automation around kolla-ansible with a focus on bare metal compute 16:05:21 most of them are now easy to merge stuff which will help non-x86 targets 16:05:36 mgoddard: cool, let's talk about it later in the meeting 16:05:39 from selecting repos to taking care of missing packages 16:05:45 inc0: sure 16:06:03 http://people.linaro.org/~marcin.juszkiewicz/kolla/ shows state of aarch64/ppc64le/x86-64 builds from some time ago 16:06:03 mgoddard ++ :) 16:06:29 I am now rebuilding all images on those 3 architectues 16:06:51 in other words: I need reviewers ;D 16:07:22 hrww: since you guys want to re-introduce debian 16:07:26 I'll want gates for it 16:07:42 this is only way we can ensure that it works 16:07:46 hrww, iirc, you want to bump debian to stretch? 16:07:52 gnocchi, ceph, ironic-pxe, openstack-base, nova-libvirt, kubernetes are easy to check simple patches 16:08:12 hrww thats an awesome setup of tables, great to track progress 16:08:13 inc0: debian. that's next part of agenda 16:08:18 kk 16:08:27 O/ 16:08:32 let me know when you want to move on to it:P 16:08:41 https://review.openstack.org/430940 patch adds base of non-x86 support - repos etc 16:08:52 review and lgtm :D 16:08:59 * hrww done on non-x86 - questions? 16:09:10 akwasnie? 16:09:32 hrw winning - keep at it dude :) 16:09:36 hrww ^ 16:09:47 ok, let's move on to debian 16:10:01 #topic Debian 'stretch' support patchset (hrw) https://review.openstack.org/434453 16:10:07 ok...so, again 16:10:09 gates 16:10:16 debian died because of lack of thereof 16:10:54 inc0: gates, yes. how can I check do they work in other way then 'someone enable and then recheck recheck recheck'? 16:10:59 inc0 it really died becasue of lack of a maintainer 16:11:05 a gate is a plus for additional distros 16:11:30 hrww peopel are workingon the recheck problem 16:11:37 so far all I know about gates/CI is "sorry, unable to make it at home" 16:11:41 hrww: workgin with gates unfortunately is pita 16:11:43 hrww we know its a problem - it annoys everyone in the community not just you 16:11:54 sdake: no, I do not mind rechecks 16:12:02 way to do it is to create experimental gates 16:12:06 well i mind rechecks - its slopy :) 16:12:16 sdake: the problem for me is how to work on gates without checking how it works 16:12:21 so it won't affect other changes before we can get them to healthy state 16:12:24 inc0: experimental gates sounds good to me 16:12:27 +1. rechecks are a pain and not the right solution. 16:12:34 hrww your next mission - if you choose to accept it - is to make an experimental gate of debian image building 16:12:40 it means the 'commons' isn't getting enough love. 16:12:49 ok, ok 16:12:58 imho debian should start as non voting check, then once things are stable, you can remove non voting 16:13:00 hrww we have a whole lot of people that can helpy ou get going 16:13:00 guys - please - patches first, ok? 16:13:12 mnaser: totally starts from non voting:P 16:13:16 without patches merged we have nothing to work on 16:13:18 fyi, i made this as well, it may help finding out how stable things are, we can adjust it for debian - http://grafana.openstack.org/dashboard/db/kolla-failure-rate 16:13:21 mnaser we start with experimenal checks here 16:13:41 Debian support is 2-3 patches 16:13:50 1. https://review.openstack.org/432787 - enable all ubuntu images for Debian 16:13:52 hrww is there an action requested? 16:14:01 sdake: english? 16:14:04 sup mandre 16:14:13 hrww are you requesting something of someone in the commnity? 16:14:16 need a review or somehting? 16:14:18 order of business is, 1. merge basic support for debian 16:14:29 sdake: yes, I do need reviews. 16:14:38 2. in opesntack infra introudce debian jobs (I can help with that) 16:14:43 experimental 16:15:00 hrww ok - lets get ya unblocked - i dont have bandwidth befor efriday to review but others may 16:15:04 2nd patch is https://review.openstack.org/450805 which takes care of rtslib naming 16:15:11 then you publish patch with gates and do recheck experimental to also run debian gates 16:15:13 and then the problematic one goes 16:15:21 iterate until gates are ready 16:15:39 https://review.openstack.org/434453 - move to Debian 'stretch' (aka 'testing') instead of Jessie (current stable) 16:15:41 after that we remove experimental and keep them healthy 16:16:19 inc0: to tell the truth I never used kolla ON debian. 16:16:25 I'll review them today, hrww you can start working on gate in infra in the meantime 16:16:35 it doesn't matter 16:16:42 what matters is whats inside container 16:16:45 yep 16:16:52 ubuntu is similar enough to debian to make it work 16:16:57 indeed 16:17:06 ubuntu on centos used to be problematic, but this should work 16:17:13 it works fine 16:17:24 I built on centos 7 and fedora 25 16:17:33 also I don't think infra even has debian images for gates 16:17:33 hrww I will also review it 16:17:42 akwasnie: thx 16:17:49 so ubuntu it is anyway 16:17:51 inc0 we ca nuse ubuntu images for gates 16:17:53 hrww np 16:18:02 yeah, that's what I'm saying 16:18:19 inc0: ok. gates are defined in openstack-project-config repo, right? 16:18:27 yes 16:18:51 ok, I have it cloned 16:19:09 cool, let me know if you run into any prblems 16:19:12 https://github.com/openstack-infra/project-config/blob/master/nodepool/nodepool.yaml#L25-L40 16:19:14 and duonghq asked why I moved to debian/stretch 16:19:19 also people on #openstack-infra are super helpful 16:19:19 it does look like there is jessie images 16:19:41 in case that helps make things easier 16:19:44 mnaser: let's keep it ubuntu regardless 16:20:06 im okay with either, just wanted to provide insight :) 16:20:09 mnaser: it would need to be jessie-backports probably 16:20:12 hrww: anything else or can we move on? 16:20:15 move on 16:20:27 #topic Upgrade gates (inc0) 16:20:37 ok, so I want to start this effort 16:20:57 few things: https://review.openstack.org/#/c/401003/10 16:21:07 way I see it working is 16:21:19 once we get pushable registry 16:21:32 we will push our built images up 16:21:47 2. introduce new gate for upgrades 16:22:12 inc0: will registry allow non-x86 images? 16:22:12 that will 1. install latest stable, do smoke test, upgrade, redo tests 16:22:28 registry can save any sort of images, just a blob store 16:22:30 aarch64/kolla/*:4.0.0 etc 16:22:33 hrww: registry won't mind either way, but we'll need to fingure out different name/tag for them 16:22:59 inc0: I like upgrade testing idea 16:23:20 so that's for today, when zuulv3 lands we make it 5 node multinode 16:23:33 in effect OpenStack gets proper upgrade gating 16:23:40 fyi im sure inc0 had a look at it but it would be nice to have a look at the grenade project and how they do upgrade tests 16:23:50 it's bad 16:24:06 oh :p 16:24:12 inc0 always the diplomat :) 16:24:36 lol 16:25:13 well grenade will never give us multinode, based of devstack, test single service not proper upgrade path that people will actually use 16:25:19 and is a shellscript 16:25:24 inc0: hold on 16:25:30 devstack... argh. 16:25:30 oh i just mentioned that it woudl be a 'source of inspiration' 16:25:30 :D 16:25:31 first of all grenade already does multinode upgrade testing 16:26:01 but also I'm not sure what zuulv4 and 5 node testing and upgrade testing all have to do with each other (they should be orthogonal) 16:26:06 er v3 16:26:07 clarkb is it based upon devstack? 16:26:22 sdake: yes it uses devstack's deployment mechanisms to turn on the cloud 16:26:24 we cant run grenade (for the most part, i think?) -- but we can at least look at how they solved certain arch. issues we'll run into inevitebly 16:26:37 example, things like this: https://github.com/openstack-dev/grenade#theory-of-upgrade 16:26:39 well, grenade is orch of upgrade (sort of) 16:26:44 we already have orch of upgrade 16:26:46 clarkb i see - i think becaseu fo that, wecan't directly reuse grenade, however, we could learn from it with kolla 16:26:54 in kolla-ansible 16:27:09 sdake: thats fine, my contention is not with you reusing it or not, its with the incorrect data being reported here 16:27:18 (and with tying unrelated items together that shouldn't be) 16:27:19 thats exactly what i meant to say: it's something to learn from 16:27:29 clarkb ya - which is why I asked inc0 to expand with "the alwaasy the diplomat" response :) 16:27:47 ok, I'm sorry then 16:27:51 clarkb not sure how zuulv4 or v5 got into discussion? 16:27:51 what I'm saying 16:27:55 we do upgrade with kolla-ansible 16:27:58 clarkb I may have missed that 16:28:07 zuulv3 and 5 node gates 16:28:12 that was typo sdake 16:28:17 inc0 now while i think upgrade gates are good, we need voting deploy gates beforehand 16:28:19 inc0 | so that's for today, when zuulv3 lands we make it 5 node multinode 16:28:26 oh - clarkb that is a dep for kolla to do multinode gating 16:28:31 sdake: why? 16:28:34 and it got into discussion because I mentioned it as plan forward 16:28:43 well multinode is our main use case 16:28:45 you can do multinode gating today 16:28:47 clarkb because inc0 says we can't get multinode until zuulv3? 16:28:49 and many people do it 16:28:58 single node is little more than science project as far as I'm concerned 16:29:01 i see - sound slike we have alot to learn about multinode gating 16:29:04 clarkb: 2 node 16:29:08 inc0: and 3 and 4 16:29:17 can ou do 3 node? I didn't know 16:29:20 the ask ahs been that you start smaller and ramp up 16:29:20 well why are we waiting for zuulv3 then? 16:29:24 have to go - starts raining and I am at the park now 16:29:27 sdake: you shouldn't be 16:29:37 thanks for all and see you later 16:29:37 clarkb right -t hat was always my understanding 1->2->3 16:29:51 well 3 was for mariadb 16:29:54 clarkb however inc0 indicated we can't ramp up 16:29:56 if you want to wait thats fine, I just don't want you to think we are blocking you 16:29:57 node count 16:30:04 regardless, when zuulv3 lands it's going tob e different mechanism 16:30:13 because we aren;t and many other groups are very successful doing multinode testing today 16:30:17 clarkb: no, it wasn't about blocking 16:30:32 clarkb the general understanding in the kolla project is zuulv3 is blocking our gating - I think we need t of ix that perception 16:30:35 it was abot not spending time on something that will be obsolete in 2 months 16:30:40 clarkb rather our multinode gating 16:30:51 inc0 we don't knwo when zuulv3 will be available 16:30:55 well kolla-k8s has multinode gating 16:31:07 inc0: I really doubt that it will be obsolete (you will learn lots and 95% fo what you do should be reusable) and I also doubt zuulv3 will be available in 2 months 16:31:24 right 2 nodes - one of the infra core said 2 nodes was the max, although in previous conversations i had heard if we go to 2, we can have 3 16:31:38 clarkb: but it will be ansible based so we could reuse lots of it 16:31:42 like inventory 16:31:51 sdake: yes what we ask is thatyou start small because most of the problems present themselves at 2 nodes and are easier to debug there. Then when tahts working ramp up 16:31:53 yes, lets stop sending the message in our project "we are waiting on zuulv3" because we don't knwo when zuulv3 will be deployed 16:31:55 it could be years 16:32:06 i think the idea here is infra wants to make sure that we're carefully using resources that we're given 16:32:07 clarkb: 3 nodes are for galera, that's why we wanted 3 16:32:16 clarkb kolla-kubernetes deliverable has 2 nodes already - could we get 3, and if so , can we get coaching on how to get to 3 nodes? 16:32:20 (if you guys dont mind, i think we're getting a bit off track) 16:32:21 but that was my understanding too - 3 is issue for infra 16:32:25 clarkb we can teach the rest of the cores 16:32:44 sdake: yes, if the 2 node tests are running succesfully we can bump up to 3. You just have to change the node type in the job definition to the 3-node variant 16:32:57 going back to upgrade gates, i think its great, but i think we need deploy gates to become voting first before we do taht 16:33:09 clarkb cool - i'll see about doing that when i return on the 15th 16:33:26 mnaser: I would agree with that, if the base test set doesn't work then upgrades likely not to work either, you'd build one on the other 16:33:32 clarkb our 2 node tests run fantastically well in that particular deliverable - when kubernetes itself isn;t broken 16:33:44 well our deploy gates does work... 16:33:51 they're nv but that's simple change 16:34:24 do work* 16:34:26 ok - well i think the answer is "zuulv3 isn't blocking, proceed with multinode gating *now*, port to zuulv3 whenever it comes out" 16:34:33 also, if i can suggest this review which i built of Jeffrey4l work -- run tempest instead of some shell scripts: https://review.openstack.org/#/c/402122/ 16:34:39 and ramp up 1 node, to 2 nodes ,to 3 nodes, etc 16:34:47 we can tempest to run post deploys, we switch deploys to voting, i think thats a great first step before starting upgrade gates 16:34:53 clarkb sound good to you? 16:34:55 inc0 ^^? 16:35:15 yes, sounds good, we need to do aio upgrade regardless 16:35:20 of multinode 16:35:25 agreed 16:35:33 my initial thought was to wait with just multinode piece for zuulv3 16:35:33 ++ 16:35:35 so next step is 2 nodes - to prove that 2 node gating is stable 16:35:38 if we want to add it earlier 16:35:40 (for kolla-ansible deliverable) 16:35:40 sure 16:35:53 no need to do that, we can do it once our 2 node stuff is happy 16:35:56 that could go to deploy gates 16:35:59 downside of waiting is we have no clue with zuulv3 will be ready 16:36:07 adn msot o the work is in our repos 16:36:16 let's do 2 pararell efforts 16:36:18 changing around the gate scripts 16:36:21 1. aio upgrade 16:36:27 2. multinode deploy 16:36:39 once both are ready multinode upgrade should be super easy 16:36:42 and possible 3. make deploy gates voting? 16:36:44 2. 2 node deploy :) 16:37:03 sounds good to me inc0 16:37:04 mnaser: yeah, we should be good now 16:37:13 although if we move to docker01 16:37:30 that will change logic a bit so maybe lets wait for that for votin 16:37:32 g 16:37:58 cool, as usual, more than happy to help with ci issues... i have a few ideas surrounding scenario based testing (ala puppet-integration) 16:38:03 docker01 - infra hosted docker registry 16:38:34 I also have few friends working on persistence testing (spawn a vm, test if it works, upgrade, test if it keeps working) 16:38:58 but, one step at the time plz 16:39:06 ok let's move on 16:39:21 #topic kolla-kubernetes launchpad trcking (sdake) https://launchpad.net/kolla-kubernetes/+milestone/0.6.0 16:39:40 indeed 16:39:46 if folks could ope nthat up 16:39:48 plz 16:40:21 these are the blueprints for april 15th deadline 16:40:36 thi should give us a pretty solid imrpovement over 0.5.0 16:41:05 if anyone wants to change blueprint prioiritie feel free 16:41:11 please keep blueprints up to date 16:41:20 and use release notes so people know what we actually release in an update 16:41:31 any quesitions? 16:41:58 cool progress :> hopefully we can see it come more alive soon 16:42:01 these are all on teh march to a 1.0.0 release 16:42:20 mnaser rwellum got kolla-kubernetes running today and eanlin has had it running for afew days based upoon the deploy doc 16:42:30 awesome #progress 16:42:37 mnaser there are 20 reviewers on the deploy doc - so seems like there is interest :) 16:43:00 ok i'll give it 60 seconds then we can move on 16:43:04 in case pople hae further questions 16:43:10 (or inc0 can you give it 60 secondds) :) 16:43:36 april 15th - lets hitit :) 16:43:56 inc0 can move on now thanks 16:44:01 kk) 16:44:03 inc0 - mgoddard probably first up on opens 16:44:07 #topic open discussion 16:44:18 yes, I wanted to talk about this, looks awesome 16:44:25 great :) 16:44:31 same 16:44:46 so few things 16:45:01 1. let's examine how much of this automation can be brought to kolla-ansible 16:45:10 since I bet lots of it can be reused 16:45:23 2. I propose new section in docs - scenerios 16:45:37 also one thing i like about this from a deployer perspective is like 16:45:40 where we'll explain how to deploy certain opesntacks for various use cases 16:45:50 this seems to be an excellent example of .. how do i reuse kolla for my specific usecase 16:45:53 scientific being one of them:) 16:45:56 1: agreed. 16:45:57 but also making it re-usable 16:46:54 for 1, kolla-ansible is currently not doing any significant configuration of the hosts. I thought that was initially a design goal? 16:47:00 for all of our deployments, we want to maintain a set of baseline rules, and then we modify a certain few things on top for kolla. the code that does this causes a lot of extra duplication. this seems like a good way of reusing kolla with a common set of values 16:47:05 mgoddard there is kolla-ansible bootstrap-server 16:47:13 but that assumes an existing server 16:47:17 aka installed os, etc 16:47:19 mgoddard: yeah, but we can extend our config of the hosts significantly 16:47:27 as long as it's decoupled from actual deployment 16:47:31 mnaser: true, although it's optional and incomplete 16:47:45 I'd keep it optional 16:47:47 inc0: ok, sounds good 16:47:47 yep 16:47:55 but incomplete is fixable 16:48:03 inc0: sure :) 16:48:07 maybe even separate it to new directory 16:48:17 not sure 16:48:22 just roles is also fine 16:48:31 but we can extend it, I'm all for it 16:48:43 one issue is that it does tend to get a bit opinionated and site-specific 16:49:12 mgoddard: that's why we need to get only non-specific to main bootstrap 16:49:29 I've tried to keep it flexible but things are inherently less consistent outside of container-land 16:49:30 and rest can live in kaobe or kolla-ansible/contrib and/or docs 16:49:46 ofc 16:50:04 ok, so what can I do to help facilitate all of this, or at least kick it off? 16:50:15 I'd start with docs 16:50:25 add new doc section for use cases 16:50:36 document scientific use case using kaobe 16:50:38 inc0: +1 16:50:40 presumably we need to work out what's in scope for kolla-ansible 16:50:57 and we'll go from there 16:51:12 together examine what can be reused and maybe create different use case docs 16:51:16 mgoddard we have a bit of capacity problem atm with kolla 16:51:20 mgoddard everyone working their tials off 16:51:37 inc0: ok. I'll try to put something together 16:51:40 yes, but this will be extreamlly appreciated 16:52:00 mgoddard yup I think what you have would be a really good fit inside kolla - although dont feel compelled to do it that way :) 16:52:03 sdake: I'm sure kolla-k8s is keeping you busy :) 16:52:04 real life guides are what operators love 16:52:29 mgoddard indeed i hae little itme to contribute to the part i'm most proud of - teh ansible implementation 16:52:34 we have broaded community to help:) 16:52:54 sdake: getting some or all of our work into kolla would be a great result for us 16:52:55 also another thing is if we could get at least docs done before summit 16:53:08 we can talk to operators there to share their love in form of docs;) 16:53:40 no point in having separately maintained projects if not necessary 16:53:47 mgoddard: think what you could move to kolla-ansible, you know both projects 16:53:59 inc0 don't force it hto :) 16:54:00 ok, I'll keep boston as a target in mind 16:54:04 its really up to you mgoddard 16:54:09 one potential wold be another repo split for all the host-based stuff 16:54:21 inc0++ repo splits are so much fun :) 16:54:23 but that's different discussion 16:54:43 mgoddard as far as mission goes, it does fit in the mission 16:54:53 "deployment tools for openstack clouds" 16:55:09 deploying the bare metal is part of that 16:55:13 we have some host scripts already 16:55:17 they need some love 16:55:41 from my perspective it's only a separate project because we have customers with requirements 16:56:00 yeah it's easier to move with local repo 16:56:13 but we can move it to kolla in our own pace 16:56:50 better to do it the right way than to rush 16:56:56 ok, few minutes left 16:57:04 anything else we want to mention? 16:57:10 that was a great reception of the project, thanks all! 16:57:20 thank you Mark for awesome work! 16:58:00 standing on the shoulders of giants - thanks to you all 16:58:09 ok, I'll wrap up meeting then:) 16:58:20 thank you all for coming and let's get back to making cool things work 16:58:30 #endmeeting kolla