16:00:05 #startmeeting kolla 16:00:07 Meeting started Wed Jun 29 16:00:05 2016 UTC and is due to finish in 60 minutes. The chair is sdake. Information about MeetBot at http://wiki.debian.org/MeetBot. 16:00:08 #topic rollcall 16:00:08 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 16:00:11 The meeting name has been set to 'kolla' 16:00:12 \o/ ;) 16:00:13 o/ 16:00:22 ola 16:00:23 o/ 16:00:30 \o/ 16:00:33 SergeyLukjanov, nice to see you:) 16:00:54 Hi 16:01:14 Hi 16:01:20 o/ 16:01:32 o/ 16:01:44 #topic announcements 16:01:54 1. midcycle is July 12, 13th at Ansible HQ 16:01:58 make sure to book travel asap 16:02:13 under 14 days the fares go up 16:02:19 o/ 16:02:35 any other announcements? 16:02:38 o/ 16:02:59 most of the images are pushed to kolla repo on docker hub 16:03:08 Barcelona hotels are available...just FYI. 16:03:17 #topic review backlog / bug backlog / blueprint backlog 16:03:37 we have a tremendous amount of work in our backlog 16:03:47 most of the reviews without -1's seem to be moving quckly 16:04:13 we have approximately 200 bugs outstanding 16:04:24 struggling to handle the workload myself - need help with triage 16:04:42 we have about 80 blueprints outstanding 16:04:52 anyone have solutions to these problems? :) 16:05:06 sdake: I'll triage them today and pick some 16:05:53 we need some more active participation during reviews 16:05:57 i was looking more for solutions to managing the workload 16:06:14 if you see the bug change it there itself 16:06:28 other than spending time doing code review and triage bugs, I don't see how to decrease the backlog 16:06:38 sdake: we're talking about managing n2 workload right? https://launchpad.net/kolla/+milestone/newton-2 16:06:52 vhosakot yup but there are a slew of bugs in n1 that i haven;t moed over yet 16:07:05 because clicking launchpad sucks 16:07:06 and there are hundreds of them 16:07:08 we need to move from n1 16:07:19 i do about 30 a d ay ;) 16:07:34 sdake, i take the action to triage all n1 bus 16:07:36 bugs 16:07:38 wht i'm getting at is i'd like all the core reviewers to be active in hepling out on all this activity 16:07:40 and move them appropriately 16:07:42 ok, let us first move high-priority bugs from n1 to n2 and fix outstanding n1 bugs... we can share this effot 16:07:58 open bugs should not be in n1, n1 is closed 16:08:20 the reality is this project is very large now and too much work for one person to handle lal of launchpad 16:08:29 we have 300+ people contributors in the last year 16:08:54 if there is anything you cn do to help out on a continual basis, that owudl be great 16:08:57 one tie mwork is helpful 16:09:00 dont get me wrong 16:09:07 but i'm looking for a permanent solution :) 16:09:37 i was also thinking it might be time to have the non-nuclear spec option (only require 2 +2s) for new work - we can discuss more at midcycle 16:09:45 but start thinking about it 16:09:54 sorry got disconnected 16:09:56 i think we need people to constantly do this 16:09:59 rather than doing it once 16:10:25 coolsvap_ ++ 16:10:25 Create a day-by-day rotation where members of core pick a day to triage like your life depended on it? 16:10:48 i think if we just accept triage as part of our workload it would help me alot ) 16:11:05 ok enough compaining 16:11:09 if you can help - please do ;) 16:11:12 if you can't, I understand 16:11:16 I've not been the most active contributor lately... sorry for that. 16:11:25 sdake, ++ 16:11:45 SergeyLukjanov you want to go next or shoudl we do our midcycle brainstorming next 16:11:48 SergeyLukjanov up to you 16:11:58 sdake as you wish 16:12:06 #topic universal containers 16:12:27 so the idea of universal containers - I think the core team is all in agreement that we want that model 16:12:35 that is why we have na abi today 16:12:37 SergeyLukjanov, you're up 16:12:43 ithink if I read the review correctly the problelm is around separte repos 16:12:46 Can you define that? 16:12:57 of which ther ehasn't been a real good explination of the 5 ws of that 16:13:12 based on the current discussion it seems like one of the most arguable topics is repo per component and so I think it should be moved out of current spec 16:13:12 (what/where/when/how/why) 16:13:17 especially the why 16:13:34 the main points from my side are about containers - deployer separation and bootstrap separation 16:13:39 SergeyLukjanov, agree, so as for universal containers, that was one of things we tried to do from day 1 16:14:00 technically current containers should be deployer-agnostic 16:14:09 which was partially proven by both tripleo and mesos 16:14:24 define deployer separation SergeyLukjanov 16:14:31 inc0_ yeah, I understand, it's as always about different point of views 16:14:50 bootstrap separation - I'd leave this out till we figure out what would be architecture of separated one 16:14:51 you mean extracting the ansible code to a separate repo? 16:15:01 also we need to keep upgrades working 16:15:14 sdake #1 separate repos and #2 no ansible bootstraping 16:15:18 if keep it short 16:15:30 separate repo is doable 16:15:32 Ansible is not required for bootstrapping 16:15:36 i dont understand the no ansible bootstrapping 16:15:39 we dont rqeuire that at all 16:15:53 sdake but it's embedded into containers? 16:16:02 yeah, for convenience 16:16:05 ansible is embedded in olla-toolbox true 16:16:10 so you start container with BOOTSTRAP env 16:16:15 other then that it is not 16:16:28 and kolla-toolbox for stuff like creation of users 16:16:32 yeah. there is no ansible in the containers other then the kolla-toolbox container. 16:16:51 but if you want to do this all on your own, totally doable, just don't spawn container with BOOTSTRAP env 16:16:53 which you shouldn't need if you are doing orchestration with puppet. 16:17:20 SergeyLukjanov but yuor goal isn't puppet orchestration, right? 16:17:20 I've manully spawned a ceph with kolla containers and horizon. :) 16:17:25 okay, but still containers contains bootstraping 16:17:38 it does contain bootstrapping 16:17:40 bootstraping is just run a one-shot container then remove it. it is nothing to do with ansible. 16:17:41 You can disable that 16:17:42 SergeyLukjanov, optional...why is that wrong? 16:17:43 yeah. and that boostrap shoudl go I think. 16:17:47 in order to use it, you have to specify BOOtSTRAP env var 16:17:49 With one env var 16:17:50 if you do not it does nothing 16:18:07 sorry for the bad wording - I mean not *Ansible* bootstraping but bootstraping itself 16:18:10 kfox1111_ have to tihnk about upgrades in that model 16:18:26 while different tools used for deployment - different bootstraping approaches used 16:18:34 as well as different topologies and configurations 16:18:35 i am not keen on external bootstrapping or database management 16:18:37 too easy for schemas to get out of sync 16:18:40 SergeyLukjanov, and it's totally possible even now 16:18:43 sdake: sure. I'm not sure bootstrapping is the best way to do upgrades anyway though. 16:19:01 inc0_ it's possible but mixed with containers itself 16:19:03 as I said, dont run containers with BOOTSTRAP env 16:19:06 I think the bootstrap thing shoudl be a different container. like a k8s job. spawn, let it run, then destroy. 16:19:08 Yeah, to look at some of the bootstrapping I’ve helped make work in kolla-kubernetes, I’m not sure you can treat bootstrapping as a separable issue. 16:19:20 SergeyLukjanov, so let's say mariadb 16:19:29 you can just run mariadb with predefined volumes 16:19:38 but you'll still have bootstrap scripts in containers and they are located in the same folder with Dockerfile and managed together 16:19:39 kolla_bootsrap would create root user and such 16:19:43 It’s significantly safer to do it in a consistent fashion from inside of the containers than it is to insist that it meet some magical definition of separable. 16:20:05 SergeyLukjanov are you concerned that someone would trigger that bootstrap code? 16:20:23 hey 16:20:34 Hey Ryan 16:20:35 e.g. MariaDB containers probably need the same consistent set of CLI tools to provision themselves. 16:20:35 SergeyLukjanov, so all you guys have to do in separate project is to make sure you're aligned with bootstrap method out there 16:20:45 wirehead_: I'm more worried from the standpoint of, something desides to do something overly clevor and reinints something it shouldnt. 16:20:57 also I think you can't really bootstrap mysql outside of container 16:21:02 as no root user exists 16:21:03 sdake my concern is that it's a deployment tool specific code inside containers, it isn't most critical thing, but IMO it makes containers more deployment tool agnostic 16:21:08 I had a horible experience with a mythtv client desiding the server database should be upgraded once. :/ 16:21:27 Its not ansible specific though 16:21:35 it isn't deployment tool sepcific, its part of the universal container model 16:21:46 SergeyLukjanov: agreed. but, maybe the solution is, if enough deployment tools add bootstrap code to the containers, it get really generic. 16:21:51 SergeyLukjanov, since it's possible to use different deploy, let's just revisit separation of bootsrap later ok? 16:21:56 I mean, it works , it's there 16:21:57 Sergey why do you say it is 16:22:03 and whatever you want to do, is possible 16:22:09 Well, let’s be clear. OpenStack has historically been hard to ugprade. With the icehouse and Havana releases floating around being good examples of how bad it can be. 16:22:14 inc0 mr positive :) 16:22:26 well, in scope of external bootsrap;) 16:22:33 pbourke-mobile it is b/c it supports only topologies and configs supported by specific deployment tool 16:22:36 wirehead_: I've upgraded clouds from Essex all the way up to Mitaka. Yes, its really hard. :) 16:22:40 unless it isn't (like mariadb) then you have perfectly working bootsrap for you;) 16:22:49 Its bash env vars 16:22:56 pbourke-mobile it couldn't be supporting all cases (and shouldn't IMO) 16:23:03 Could not be more agnostic 16:23:18 I'm for external bootstrap as much as possible. there may be cases where internal might make sense. 16:23:32 to be clear though, like in the spec, external orchestration. internal binaries. 16:23:33 SergeyLukjanov, do you have any use-case in mind that is not achievable with our current arch? 16:23:35 SergeyLukjanov, how to init the mariadb /var/lib/mysql file without the container, without the bootstrap script? 16:23:52 actually the mariadb case is good one - it's a situation when some bootstraping embedded is good 16:23:55 Jeffrey4l: ceph for example with k8s. 16:24:08 like users creation, it's more about container bootstrap 16:24:11 the bootstrap code currently assumes it will make new id's with every run. if you run it in a k8s job and it fails, 16:24:13 not service/component bootstrap 16:24:24 it will keep creating osd id after id after id. 16:24:36 ok, I'm lost, what's container bootsrap? 16:25:16 https://github.com/openstack/kolla/blob/master/docker/heat/heat-api/extend_start.sh#L5-L17 16:25:21 I'm not saying about removing code that creates user for example or code that is needed each time you running container 16:25:46 you mean the code that parses the json file 16:26:14 sdake good sample - I mean such kind of code - https://github.com/openstack/kolla/blob/master/docker/heat/heat-api/extend_start.sh#L5-L17 16:26:55 SergeyLukjanov, so heat-manage db_sync has to be in container right? 16:26:59 yes, but if KOLLA_BOOTSTRAP is not deffined, nothign happens 16:27:05 so triggering db sync externally? 16:27:13 I dont think it's possible rhallisey 16:27:23 its possible if you make one bootstrap contianer 16:27:26 I mean, we could do docker exec -it heat-api heat-manage db_sync 16:27:32 inc0_ yup 16:27:34 i agree. we should move the heat related use create to ansible. but the heat-manage db_sync should leave there. 16:27:41 I think what I'd like to see there, is KOLLA_BOOTSTRAP section moved to its own shell script. 16:27:41 SergeyLukjanov, so I'm ok with tht 16:27:59 then you can docker run CONTINER /bootstrap 16:28:00 inc0_, ansible docker could handle that 16:28:06 it could 16:28:10 we do it already 16:28:12 inc0_ or run container that will have that entrypoint will be needed command 16:28:19 there are several ways to do it 16:28:21 or that 16:28:24 reasonable idea kfox 16:28:29 inc0_, we even has no heat-api container, before bootstrap. how to to use docker exec ? 16:28:31 then puppet or ansible or k8s jobs can run it. 16:28:39 Jeffrey4l, we do... 16:28:52 heat_bootstrap is started from heat_api contianer 16:28:54 anyway, we can have bootstrap container as sdake said 16:28:58 there are ways 16:29:07 we can run a separated container "heat-bootstrap" with entrypoint command "heat-manage db_sync" 16:29:13 personally i'd prefer bootstrap happen all at once 16:29:21 or, yeah. split it to a different container. 16:29:24 SergeyLukjanov, one thing tho..instead of having separat econtainer 16:29:27 sdake, idk if it matters 16:29:28 we have container with or without env 16:29:38 doesn't make us build one additiona container 16:29:42 that will be run once 16:29:55 since all the tools are there 16:29:57 yes it doesn't matter. the bootstrap container will be removed after it exit. 16:30:03 yup 16:30:05 i'd prfer to keep things simple nad not require docker exec to bootstrap containers tho 16:30:09 inc0_: take the heat example again, heat-manage db_sync is a thing that happens during upgrades as well as bootstrap. 16:30:18 the rest of it shouldn't ever be called outside of a bootstrap though. 16:30:19 yeah, that too 16:30:28 kfox1111_, yup. 16:30:36 so thats where mixing orchestration with bootstrapping kind of worries me. 16:30:37 sdake, yes but the counter to that is removing it from the extend start makes things simpler 16:30:45 SergeyLukjanov, how critical this issue is for you 16:30:46 ? 16:30:53 can we revise it on summit or sth? 16:30:58 it's not the highest for sure :) 16:31:10 i guess i dont understand what the pain is about haing a BOOTSTRAP environment variable conditionallly execute code 16:31:12 cool, I'm cool with any improvement proposals 16:31:21 sdake ++ 16:31:23 less automagic i suppose 16:31:28 the general issue of having ansible and containers in a same repo is more important for me 16:31:44 SergeyLukjanov, it's a topic for the midcycle 16:31:57 the reason it is like that fwiw, is to facilitate backporting 16:32:17 SergeyLukjanov, reason we didnt separate yet is simply because it's irreversable 16:32:27 just a random thing - what's about making option no not "COPY bootstrap.sh" in Dockerfile? 16:32:30 but moment we arrive to prod-readiness of kolla-k8s 16:32:35 if the repos are split backporitng becomes extremely dificult 16:32:36 we are revising this issue 16:32:46 SergeyLukjanov, negative 16:32:48 sdake: there is also a general op thing where I want to make sure what I deploy isn't going to do something unexpected. 16:33:00 you would create 2 different docker images 16:33:03 just looking at the code, shows up the bootstrap stuff which looks very dangerious. (which it is) 16:33:11 so I have to audit the scripts much more carefully. 16:33:12 remember build is separate from deploy 16:33:22 if they were seperate, then its easier to ignore. 16:33:28 kfox1111_, technially you would have to always audit something 16:33:56 yes, but the amount is differnet. for example, I deploy horizon containers directly on one of my clouds today. 16:33:57 SergeyLukjanov, so differet question, why you don't want ansible in tree specifically? 16:34:06 what use case it prevents you from achieving? 16:34:25 if the bootstrap scripts were seperate from the containers I'm running, I wouldn't have to review them at all. 16:34:41 kfox1111_ +1 16:34:43 kfox1111_ they could still be broken if separate 16:34:46 and would require review 16:34:46 kfox1111_, what about review the ansible code 16:34:53 so, I do really like the idea of making, say, a heat-bootstrap container that all it is is heat-api with the bootstrap code added. 16:35:11 rhallisey: yeah. that scares me a bit too. thats why I'm working on kolla-k8s. :) 16:35:20 ha 16:35:30 there's a lot of code in the ansible stuff, and I have a hard time knowing if its safe or not. 16:35:40 while k8s provides a layer of isolation. 16:35:41 inc0_ it's about need to dig into the ansible code in addition to containers itself and too coupled reference implementation 16:35:44 kfox1111_, I have bad news for you then man... we're looking at ansible in k8s;) 16:35:52 kfox1111_, are u meaning the heat-bootstrap container? or heat-bootstrap images? 16:35:58 hense the, "I want to run kolla-ansible genconfig as non root" for example. 16:36:05 SergeyLukjanov, so code is logically separate alread 16:36:06 y 16:36:17 the only thing that is not separate is the json file 16:36:18 kfox1111_: sry. ment images I think. 16:36:22 one of reason we keep it there is, as you've said, reference implementation 16:36:30 sdake rhallisey folks, I'll not be able to be on the midcycle, will be there anny posibility to join remotely? 16:36:38 inc0_: yeah, but I expect that to be a lot simpler as k8s does alot of lifting itself. 16:36:40 so far ansible is only way to deploy kolla in prod ready way 16:36:43 ya we will have remote participation 16:37:14 sdake cool 16:37:16 kfox1111_, idk if you've seen my new spec 16:37:20 base on the poc 16:37:26 I'm ok with ansible orchestrating things. It just needs to be simple enough I can get my head around it. 16:37:28 ok so in conclusion 16:37:32 rhallisey: yeah. I have. I commented I think. 16:37:33 once we'll have alternative on that front, we agreed to keep things as they are 16:37:38 kfox1111_, ok cool 16:37:41 there are two issues - bootstrap is integrated into the containers - ansible is in repo 16:37:43 to limit deployers confusion 16:38:01 ansible in or out of repo has been hotly debated in the pasat 16:38:05 i dont care at this point 16:38:07 in the past i did 16:38:15 i do have concerns about backports and maintenance in general 16:38:30 SergeyLukjanov, long story short, still open depate and I think it's rather matter of when than if 16:38:37 as for bootstrapping - if someone invents a beetter bootstrap mechnaism and makes it work throughout the current code base wfm :) 16:38:48 I think the contianers themselves should be at a point where what you expect is what you get. Having flags in the scripts it doesn't 100% guarantee that 16:38:51 sdake agree about backports, such changes can make it unmaintainable 16:38:53 * rhallisey can write a spec 16:38:54 I think if it prevents another kolla competetor from spawning, moving ansible code to kolla-ansible is a reasonable thing, since we've been debating it for a while now anyway. 16:39:03 but since it's only way that works today, we keep it for now 16:39:20 kfox1111_ it doesn't stop any competitor 16:39:29 SergeyLukjanov, if you guys would like to help with kolla-k8s and accelerate it's way to success 16:39:31 kfox1111_, I think the technically reasons be enough tbh 16:39:35 stackanetes took kolla containers and merged them into their project 16:39:56 providing valid alternative to ansible 16:40:04 this discussion can be accelerated 16:40:45 ok well we need to get to brainstomring - since midcycle is only 2 weeks out 16:40:59 inc0_ we're going with own k8s based impl and I want mainly to use kolla images in future as I believe it's very good to have universal containers and not duplicate work on them. we can review further what will be achieved, let's say on summit 16:41:42 ya duplicating the containers is a huge waste of time 16:41:54 SergeyLukjanov: why not contribute to kolla-k8s? 16:42:10 there's still a lot of room to participate there. 16:42:37 I think some of the tripleo folks are too. 16:42:42 could build the same logic into kolla-kube 16:42:53 so it could be consumed by any project 16:43:07 kfox1111_ from my understanding mirantis has already made up its mind on this topic 16:43:07 yeah. 16:43:10 but kolla-kube is the central hub prof development 16:43:38 sdake kfox1111_ that's correct, it was done significantly before the last summit 16:43:43 it's just olive branch. The community is open for collaboration :) 16:43:47 sdake: sure, but I want to have the discussion out in the open to ensure other folks picking something to contribute to, know the difference. 16:44:50 i think people willl natureally be drawn to kollla in this case, as this is our core mission 16:44:59 not afraid of competition 16:45:00 yup. kolla-k8s just wants to produce the best, open solution to openstack on containers using k8s. everyone's welcome. 16:45:14 ya dont see need for competition either fwiw :) 16:45:36 +1. I can't stop them. Just incuraging otherwise. 16:45:55 ok brainstomring - running out o time :) 16:46:05 #topic midcycle planning 16:46:06 #link https://etherpad.openstack.org/p/kolla-newton-midcycle 16:46:15 does kolla-kube have a slot? Or did I miss it? 16:46:35 everyone - please add items, i will schedule as best as possible 16:46:41 thx folks for the discussion, very appreciate 16:47:04 SergeyLukjanov, thanks! 16:49:13 btw, I want to spend a moment to inform folks of what is happening related to governance of the project 16:49:42 i have been a ptl for 5 years of my life, and think kolla is in pretty fantastic shape, so I am not running for ptl again (unless nobody else does) 16:50:13 Ive been training 3 people who may or may not intend to run - who nkows :) 16:50:24 plan to stay on core for 1-2 years 16:51:05 ! 16:51:06 wow sdake 16:51:08 any questions 16:51:34 I'll just say...thanks for all the work and getting us here! 16:51:42 yeah...huge thanks! 16:51:43 you guys did all the work 16:51:51 i did a bit myself 16:51:54 Thanks sdake! 16:51:57 but definately team thing 16:52:01 ! 16:52:05 anyway - im not gone yet - 3 month left in the cycle 16:52:14 sdake, ya thanks for the leadership 16:52:15 but I dont want people thinking I'm flaking out when I'm training people 16:52:30 i intend to finish strong this cycle ;) 16:52:42 sdake, I remember when the project was just us 2 :) 16:52:51 well there was daneyon :) 16:52:52 3 :) 16:52:53 sort of 16:52:55 3 16:53:03 meetings were lonely 16:53:33 sdake, wow! thanks for all the hard work 16:53:35 sdake, did kolla-kube already happen as a topic? 16:53:42 rhallisey no 16:53:47 #topic kolla-kube 16:53:52 i thought you meant midcycle not meeting :) 16:54:33 So, we’ve got a patch up that we’ve been playing with that installs enough bits of kolla-k8s to run Horizon and Keystone and do it multi-host. 16:54:50 And I spent some time yesterday deleting nodes and watching as everything magically came back up. 16:55:03 sdake: thanks for all the hard work. 16:55:11 wirehead_, mariadb and rabbitmq behave? 16:55:13 thank you sdake 16:55:17 kfox1111_ again, styaing on the core team for 1-2 years :) 16:55:27 MariaDB behaves. I haven’t done RabbitMQ yet. https://review.openstack.org/#/c/335215/ 16:55:49 wirehead_, so for action items, we need a doc to show how to accomplish this 16:55:51 Basically, first thing I did was kill MariaDB a few times. 16:55:53 idk if there is one yet 16:56:08 since this requires ceph/shared storage + skydns 16:56:44 maybe we need to start a doc for what's expected of a k8s cluster. 16:57:03 k8s installed, skydns setup, ceph deployed (possibly with kolla-ansible) 16:57:16 Yeah, sbezverk and I have been chatting about SkyDNS in particular. 16:57:18 (and maybe pv's precreated) 16:57:40 whatever it is, we need to have everything laid out with explanations 16:57:43 As to me, the pv precreation is a k8s provided thing. 16:57:48 yeah. 16:58:01 another thing, a spec I wanted to bring upo 16:58:01 I agree. dcwangmit01 posted that mostly as a “Hey, is this the right direction” 16:58:13 https://review.openstack.org/#/c/335279/1 16:58:19 t-1 minute 16:58:24 sdake, roger 16:58:35 that spec is to orchestrate k8s using anisble 16:58:52 it's something the comunity needs to debate 16:58:59 read through and drop some comments pls :) 16:59:02 done! 16:59:08 rhallisey: The comment I made about the cli stuff still existing is inline with what I said above in this channel about wanting to be able to manually orchestrate as needed. 16:59:24 kfox1111_, I'll get to it later 16:59:28 k. 16:59:38 kfox1111_, actually I think I did respond to your comment 16:59:57 I take that back I missed it 17:00:04 draft? :) 17:00:21 I draft things more then I'd like to admit. :/ 17:00:29 ya let me revisti :) 17:00:31 meeting time over :) 17:00:34 #endmeeting