15:59:10 #startmeeting kolla 15:59:10 Meeting started Wed Sep 20 15:59:10 2017 UTC and is due to finish in 60 minutes. The chair is inc0. Information about MeetBot at http://wiki.debian.org/MeetBot. 15:59:12 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 15:59:14 The meeting name has been set to 'kolla' 15:59:17 #topic w00t! 15:59:24 welcome back everyone! 15:59:25 o/ 15:59:37 wo0T! 16:00:10 o/ 16:00:21 o/ 16:00:26 WoOt! 16:00:50 wW0o0otT 16:01:11 o/ 16:01:16 o/ 16:01:54 Hey - I know many faces now since the PTG. 16:02:09 Maybe 1 or 2 I'd like to forget. jk. 16:02:35 hehheh 16:02:40 rwellum: lol 16:02:47 o/ 16:03:12 inc0: anywhere we can get the pics in higer res than the one on the mailing? 16:03:19 o/ 16:03:24 lol 16:03:24 these are only ones I got:( 16:03:37 I think they are just this way 16:03:43 inc0: just so that I can tell hrw next week who is who, ok, I will ask the foundation guys 16:03:50 o/ 16:03:55 and pass it on if they are on some google drive somewher 16:04:06 rwellum: I envy you then 16:04:27 hrw: yeah sucks you couldn't make it :( 16:04:31 #topic Announcements 16:04:45 PTG is over, we have heads full of plans 16:05:03 any community announcements? 16:05:22 We discussed 1.0 for kolla-k8s 16:05:28 PTG was awesome, great chats and ideas exchanged! 16:05:30 inc0: we'll be working hard towards adding aarch64 to the gates 16:05:30 * hrw waits for public docs about plans to write mail to bosses about next ptg 16:05:37 5.0.0 is released lol 16:05:37 Kolla will certainly send man to pluto ;) 16:05:51 gema: +100! 16:05:59 cool gema! 16:06:12 Jeffrey4l_: that's awesome, and thoughts on dockerhub image push? 16:06:15 #topic PTG summary 16:06:16 I'm testing egonzalez's vault review. 16:06:17 o/ 16:06:22 #link https://etherpad.openstack.org/p/kolla-queens-ptg-schedule 16:06:27 I am already afraid of amount of work gema will put on my plate ;D 16:06:39 lol 16:06:41 rwellum, we need vm to push the images 16:06:51 hrw: you won't be bored 16:06:57 Jeffrey4l_: negative, we have better plan:) 16:07:10 inc0, what's it? 16:07:36 let's start with docs, I'll get to dockerhub 16:08:20 #link https://etherpad.openstack.org/p/kolla-doc-restructure 16:08:38 with Alex and docs team we started to work on new table of contents for docs 16:08:54 ok. 16:09:01 I pushed a patch that started with the new format, but do we want to use the old on einstead? 16:09:25 krtaylor: I'd be ok with starting fresh with new one 16:09:50 well, as egonzalez pointed out, they are similar 16:09:52 krtaylor: new one all the way. 16:09:58 I would like this to be full community effort this release 16:10:07 can you link to the patch krtaylor? I'd like to help 16:10:16 yeah, me too 16:10:20 ditto 16:10:22 I saw both reviews from krtaylor. As egonzalez commented, they do look similar. 16:10:24 its in the restructure etherpad 16:10:40 https://review.openstack.org/#/c/504801 16:10:57 I'm fine either way, the older one -> https://review.openstack.org/#/c/486423 16:11:29 we can decide and then I'll start pushing other sections 16:11:58 ok let's review both after meeting 16:12:04 I stopped on mine until decided, there are a couple of fixes left 16:12:11 sure, that'll work 16:12:14 krtaylor: the old review does not have comments/inputs from PTG? 16:12:33 right, slight tweaks to that one would be necessary 16:12:38 besides new ToC and directory structure we'd love to create use-case based docs too 16:12:41 I see. 16:12:47 something I heard over and over again 16:12:50 o/ 16:12:59 hey sdake 16:13:07 yes and some for kolla-ansible toc as well 16:13:08 we can plan around that once we get basic structure in place 16:13:19 the old one was since the doc restructure started, but was in queue for so long 16:13:37 I see, cool.. that is what I thought. 16:13:55 For the use-cases - did we decide where they are to live? 16:14:02 krtaylor: can you lead this effort? I mean it's full community but it'd be great to have you keeping track on all this 16:14:08 * rwellum reading doc 16:14:08 If the new one matches what we came up with PTG, seems like we should take that one forward and incorporate old one into that? 16:14:13 yep, I'm trying :) 16:14:30 and you're doing awesome job, just wanted to make it a bit more official here;) 16:14:40 krtaylor: if there is consensus, abandon old review to avoid confusion :) 16:14:42 ^^ 16:14:51 vhosakot: +1 16:14:56 let's move forward with the new one 16:14:59 inc0, hehheh, well, I'll have to beat back downstream pressures, but sure 16:15:14 yes, new has old in it anyway ;) 16:15:22 so, if anyone wants to help, please talk to Kurt to figure out best ways to contribute in organized manner 16:15:39 ok, if thats consensus, I'll finish it this afternoon 16:15:41 inc0: yeah, I'd like to help too... will ping krtaylor after meeting. 16:15:53 that'll work! 16:15:59 krtaylor: maybe start an etherpad with tasks 16:16:04 and let us pick from there? 16:16:05 cool, thanks, we need to start by organizing new dir structure 16:16:08 or a BP 16:16:20 I'll set up an etherpad for us to track work over the cycle 16:16:20 gema: BP better 16:16:26 so everyone please review notes from PTG and reviews linked 16:16:32 BP can have tasks, acn have link to etherpad too 16:16:35 great idea krtaylor 16:16:45 I like the BP because we can use it to show docs team and they might promote our project as a fore-runner a bit.. 16:16:52 let's make it full topic on next meeting too 16:16:53 ok, will do, BP that is, and etherpad 16:16:57 and BP can be mentioned in commit logs, sorted in gerrit etc 16:17:25 we should use this week to work out ideas and then, next meeting, kick off actual work full time 16:18:15 yeah we're a bit light on process where it comes to bps, but obviously we should create one, thanks everyone 16:18:16 should we create new etherpad for idea <-> bp mapping or use old one? 16:18:25 I hope that linaro connect will have boring keynotes on wednesday so I can attend meeting rather than keynotes ;d 16:18:32 * gema and hrw will likely skip next week but be back after that (due to travel to conference) 16:18:38 I like bp to point to etherpad and etherpad to be actual work notes 16:18:44 it's less effort to work with it 16:18:53 inc0: I'd like to help with the DockerHub publishing task too. After chatting with the infra guys at PTG, were there any notes taken? 16:19:05 I'd like to start a new etherpad, keep that one for PTG reference 16:19:12 vhosakot: we'll get to that 16:19:25 krtaylor: link both etherpads by adding links 16:19:27 actually, let's move to that 16:19:29 I'll set that up this afternoon 16:19:36 sounds good, I'll ping yall for help 16:19:40 #topic PTG summary - Images 16:19:59 so we have few big items regarding images 16:20:05 1. DockerHub publishing 16:20:28 current idea is to start by deploying registry in Infra 16:20:36 to get rid of tarballs 16:20:54 inc0: cool 16:21:10 registry in infra is good idea as it allows to put whatever got built for tests and publish only tested images 16:21:21 I'm leading this effort (whole dockerhub thingy) and like Kurt, I'll get etherpad with work items up and running 16:21:24 today 16:21:30 yes 16:21:40 Infra will support he registry? or we'll spin up a registry as part of our gate? 16:21:59 we'll have single registry in infra, master registry you could say 16:22:06 cool! 16:22:26 gate-to-gate registry spinup was discussed but that's a bit far away, we'll figure it out when we get there 16:22:59 This will apply to both orchestration methods correct? 16:23:00 so we'll have job to populate infra-registry and another job to pull from it and publish on dockerhub 16:23:30 rwellum: yes, images will be available to k8s too...or anything else for that matter, like homebrew solutions 16:23:33 Yes, getting registry in infra is long awaited and good start to publishing 16:23:35 is there credential solution 16:23:39 inc0: will the registry contain a single version of kolla images (every night's images pushed)? or container many versions of kolla images? 16:23:44 Jeffrey4l: will be addressed in zuulv3 16:24:04 is zuulv3 released? 16:24:07 vhosakot: good qn 16:24:10 zuulv3 is days away afaik, I'll double check with them today 16:24:11 Jeffrey4l: 9/25 16:24:17 thanks. 16:24:28 vhosakot: all stable + master for sure 16:24:30 Because current solution in k8s is a registry which downloads one big tarball 16:24:36 i heard zuulv3 since last year ;D 16:24:46 Sep 25 according to that team 16:24:48 yes, it's a huge amount of work there 16:24:49 and multiple architectures I hope 16:25:00 inc0: ah, registry has stable too.... then, infra registry sounds like a duplicate of DockerHub? 16:25:12 hrw: for that we'll need to wait for non-x86 gates 16:25:17 inc0: sure 16:25:21 * inc0 looking at gema 16:25:25 hrw, there should be no arch issue for docker registry. it just a bunch of tar files. 16:25:37 but once that's done, I see no reason why not publish these too 16:26:00 vhosakot: yes with better bandwidth:) 16:26:00 issue will be building and testing, so we need region of infra with arm 16:26:37 we discussed that with infra and I think gema has good idea about that, right? 16:26:39 inc0: we'll see once we've first 16:26:39 sorry had to step out for a sec, reading 16:26:42 vhosakot: when you do builds and testing inside infra why waste bandwidth/time on sending files here and there? 16:26:43 hrw is building 3rd part build jobs i think. once it is done, he can push the image into master registry. 16:26:51 Registry 16:27:45 vhosakot, docker hub is used for kolla tag image. infra registry is mainly used for branch images. i think. 16:28:04 right, to close up this topic, I'll make summary etherpad, work items and if anyone wants to help, please talk to me 16:28:11 I'll be contact point in dockerhub things 16:28:12 cool inc0 Jeffrey4l, that is what I thought. 16:28:26 inc0: yep, no prob, I have already agreed on what cloud to use and just need to find time to add the 2 projects to it and tell infra about it 16:28:28 inc0: I'd like to help. 16:28:31 then we'll be able to schedule jobs 16:28:31 next big topic in image pipeline was dealing with plugins (again)( 16:28:33 inc0: so is zuul3 a blocker? 16:28:37 rwellum: yes 16:28:39 will coordinate with Jeffrey4l once I get there 16:28:43 got it 16:28:45 but there are things we can do before it landa 16:28:46 lands 16:29:18 3rd party CI will take a bit longer 16:29:21 so scenario we were dicussing was - currently I have neutron-server-odl and neutron-server-ovn 16:29:22 but it is also on the making 16:29:33 inc0: did the infra team say they do no want the infra registry to carry images of plugins? 16:29:33 how to make neutron-server-odl-ovn if I want multi-backend 16:29:45 vhosakot: that's besides the scope 16:29:49 cool 16:29:59 there will be some plugins, whatever we decide 16:30:05 right 16:30:21 let's talk about this later, first, basic images 16:30:39 hrw: can you make sure our images also end up there? 16:31:07 gema: pushing images is a matter of auth+address iirc 16:31:10 there are few ideas in this etherpad: 16:31:17 #link https://etherpad.openstack.org/p/kolla-queens-ptg-images 16:31:24 hrw: and naming, we need to agree with everyone on the right one 16:31:41 (so the arch is parsable from the name) 16:31:45 (I think) 16:31:46 I'll need a name if someone wants to take on this effort, otherwise we'll wait till our time gets freed up 16:32:16 Also does this address kfox1111 point about 'micro builds'? 16:32:17 inc0: maybe hrw can start looking into it 16:32:23 * rwellum ducks instinctively 16:32:31 shrinking images: that was cool because we wrote new docker-build 16:32:34 I am not sure how much work is involved nor how much time he is going to have 16:32:36 gema: images may be multiarch handled by registry too. "docker pull centos" works on aarch64 now 16:32:53 that was fun experiment;) 16:33:10 hrw: ok, cool 16:33:11 hrw that's cool. 16:33:18 and finally, we discussed what we're doing with Ceph 16:33:21 this is important 16:33:36 long story short, right now we bump ceph to L 16:33:44 so we need upgrade and containers 16:34:08 and we need to get ceph/L packages for debian/stretch, centos/7 and ubuntu/xenial 16:34:12 but we also discussed with leseb (ceph community) about moving out of our ceph deployments to ceph-ansible 16:34:41 during Queens timeframe I'd like us to think how it's possible if it is 16:34:54 without sacreficing our nice deployment experience 16:34:56 ubuntu/xenial has ceph/L in ubuntu-cloud repo 16:35:07 hrw: ^ this means we need to start adding aarch64 images to ceph-ansible in prep for the later move 16:35:10 krtaylor: ^ 16:35:11 hrw we have pin it to J now. 16:35:11 if we decide that we want to move out of kolla-deployed ceph 16:35:21 Jeffrey4l: in pike, yes 16:35:21 Jeffrey4l: I know 16:35:26 Queens can be L 16:35:50 if we want to move out of kolla deployed ceph and always assume external ceph 16:35:51 Jeffrey4l, ubuntu install L in pike 16:36:09 we'll need good guide, migration path and deprecation of it over Rocky release 16:36:20 which means next ceph LTS would be outside of Kolla 16:36:22 egonzalez, iirc, didn't you push a patch to pin the ceph version. 16:36:22 centos is on jewel, debian on jewel iirc too 16:36:28 leseb said they'll help 16:36:30 Is ceph documented in etherpad inc0 ? 16:36:41 inc0: I'm working on a shell/ansible script that uses multinode file to deploy ceph.... 16:36:45 no:( 16:36:48 Jeffrey4l, didnt work, package conflicts, my .deb knowledge is limited for that 16:36:56 jamesbenson: with ceph-deploy? 16:37:01 inc0: yes 16:37:03 ok. thanks 16:37:05 that'd be good 16:37:10 inc0: It's a wrapper around ceph-deploy 16:37:20 ... I like wrappers... 16:37:31 anyway, that's outcome from discussion 16:37:44 we have 6 months to explore and make decision 16:37:54 so let's make sure it's good and informed one 16:38:08 any other comments regarding image part of PTG? 16:38:37 #topic PTG summary - Ansible 16:38:41 #link https://etherpad.openstack.org/p/kolla-queens-ptg-ansible 16:38:52 so one of biggest ones was devmode 16:39:12 we talked to different communities and people are really excited about prospect of ditching devstack 16:39:18 Jeffrey4l, egonzalez: I may take a look on getting ubuntu images stick to 10.2.7 ceph/j 16:39:34 btw, devmode is marked as implemented in pike branch. and i re-opened it 16:39:49 right, also we'll need to change it slightly 16:39:51 +1 inc0 - the kolla+cinder_ironic meeting has 25+ people attend with good feedback. 16:39:56 because some good feedback was provided 16:39:57 hrw, thanks. i will check this later. 16:40:07 about how our current approach won't exactly work for everyone 16:40:19 but we have ideas so let's make it work this release in it's full potential 16:40:26 here it is the origial bp https://blueprints.launchpad.net/kolla/+spec/mount-sources 16:40:47 we have volunteers in Ironic and Cinder that wants to work with us on it and move their dev to it early 16:41:14 right now I'm experimenting with mounting full venvs;) 16:41:22 (with mixed results :() 16:41:38 anyway, it's a good effort that will improve OpenStack en large so let's make it great 16:41:52 pbourke: do you want to be leader of this effort? 16:42:05 sbezverk: scroll up a bit and you'll see quite a bit of discussion about images fyi 16:42:20 (if not, I can do it, but it's your child:)) 16:42:47 did we decide which solution we will use? 16:43:06 no, not really 16:43:14 we have ideas and problems to solve 16:43:25 roger. 16:43:47 ok, for now I'll be contact point for devmode 16:43:54 next one, gates 16:44:14 we discussed 2 big thigns regarding gates 16:44:18 1. run tempest scenarios 16:44:26 +1 16:44:49 rwellum: I do not see the scroll back since session was disconnected. 16:44:57 since we want to build scenario-based gates (framework for that is already implemented) 16:45:25 we can create sets of globals.yml/overrides + tempest confs for per-scenario testing 16:45:33 that will also work as documentation later 16:45:52 2. upgrade gates 16:46:07 I, for one, really really (really) want to see full release upgrade tested in gates 16:46:09 sbezverk: https://etherpad.openstack.org/p/kolla-queens-ptg-images 16:46:17 also with different scenarios 16:46:52 ah one more thing we discussed, I don't think there is huge value in testing deployment of all binary+source in kolla-ansible gates 16:47:09 one issue for kolla right now is: we support multi disro * multi type. this will generate lots of jobs :( 16:47:16 we could free up these resources and test more scenarios I think 16:47:29 right, that's why ^ 16:47:38 then which one will be dropped? 16:47:58 we can do just ubuntu and centos source 16:48:03 so 2 jobs per scenario 16:48:15 (have oracle and binary in kolla images) 16:48:46 then does this mean kolla claims that it don't support binary anymore? 16:49:01 since there is no difference whatsoever in deployment logic per binary/source, I don't see value in having it in kolla-ansible gates 16:49:18 I'd rather have daily jobs to test sanity of it 16:49:22 instead of per commit CI 16:49:38 inc0, things changes between binary and source, distros sometimes drops things we use 16:49:53 inc0: binary tests daily should be fine as those packages do not change more often rather 16:49:56 in example apache when we use serverlet 16:50:00 that's monitoring/periodic sanity testing not CI 16:50:03 So you're saying keep the multi distro * type in kolla gate, but simplify kolla-ansible gate to just ubunut + centos source? 16:50:08 CI testing is to check if patch is correct 16:50:30 and since kolla-ansible doesn't have any different deployment logic for binary vs source or cenots vs oracle 16:50:47 I'd like to stick to CI jobs that are relevant for kolla-ansible code changes 16:50:58 right britthouser 16:51:06 inc0, it has, especially in kolla's Dockerfile. but not much 16:51:20 Dockerfiles are in kolla, not kolla-ansible 16:51:26 britthouser, is a not idea. 16:51:27 that's what I'm sayin 16:51:29 g 16:51:34 britthouser, is a good idea. 16:51:48 britthouser is a great idea :) 16:52:14 is not my idea. =) Just restating inc0 to make sure I understood. =) 16:52:15 moreover, we could have very detailed testing in periodic job and have dashboard rather than per-commit test 16:52:25 so, I like the idea of this type of gating... helps monitoring/health chacking 16:52:48 bottom line is, from kolla-ansible gates we drop binary and other stuff like that 16:53:00 we limit gates to centos-source and ubuntu-source 16:53:09 and increase number of different scenarios instead 16:53:20 it's going to be more valuable 16:53:24 +1 16:53:46 last but not least, stability testing 16:53:59 seems reasonable. Make kolla-ansible choose a smaller number of distro/types, but more permutations of scenarios 16:54:00 I mentioned scenario-based docs and scenario-based gates 16:54:09 now scenario-based manual testing before release 16:54:38 we need to create a matrix of different scenratios (globals.yml+tempest) 16:54:40 Would it also be reasonable to have kolla gate include a single scenario? 16:54:54 but run that scenario on all the distro/types? 16:54:54 britthouser: it does now and it will 16:55:01 we need deployment smoketest for sure 16:55:11 we can have periodic 16:55:12 too 16:55:16 for detailed 16:55:21 I'll ask infra about that 16:55:30 anyway 16:55:44 yeah having a bunch of stuff coded up we can run manually on RC would be good. 16:55:54 s/run/trigger/ 16:56:04 with release testing goal is to lower barrier of testing for people who aren't super faimilar with kolla 16:56:08 inc0: periodic meaning, experimental scenario-based gates? 16:56:16 which means, pre-defined configs 16:56:44 vhosakot, it means run daily or weekly automatically. 16:57:06 inc0: one scenario can be the linaro reference architecture 16:57:14 I was planning to do this anyway for us 16:57:17 and our members 16:57:28 Jeffrey4l: how can we do that? using flags in gates (experimental?) 16:57:29 something they can easily grab and stand a cloud from for testing 16:57:38 will the test run before/after pushing to docker? if it fails, could we stop it from pushing? 16:57:56 zuul supports periodic pipeline vhosakot 16:58:04 jamesbenson: different topics 16:58:09 Jeffrey4l: ah, zuul magic.. cool 16:58:12 roger 16:58:45 anyway, we didn't get to kolla-k8s, we'll make summary on next meeting 16:58:53 inc0: we could do a basic cloud (no HA) and an HA simple cloud, with/without ceph 16:59:02 gema: right, that's the idea 16:59:03 then run tempest on it 16:59:08 inc0: I really like to add a simple L2/L3 scenario first in our gate other than jus booting a nova VM. We should also create a FIP, SSH into the VM, and ping the internet (google) to make sure L3 end-to-end works. 16:59:12 maybe not all tempest, just interop tests 16:59:14 a subset 16:59:21 yes 16:59:22 very easy to do with refstack client 16:59:25 love the idea and been trying to do tempest on my stuff, but no experience with it. so it's hard. 16:59:28 again, that's the idea 16:59:31 once we have a nice tempest template conf 16:59:45 anyway, I need to cut it short 16:59:49 we ran out of time 16:59:51 jamesbenson: we are working on an automatic config generator 17:00:01 thank you all for coming! let's move to Kolla channel 17:00:12 #endmeeting kolla