16:30:20 #startmeeting kolla 16:30:21 Meeting started Wed Feb 17 16:30:20 2016 UTC and is due to finish in 60 minutes. The chair is sdake. Information about MeetBot at http://wiki.debian.org/MeetBot. 16:30:22 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 16:30:23 #topic rollcall 16:30:24 The meeting name has been set to 'kolla' 16:30:27 o/ 16:30:28 o/ 16:30:30 o/ 16:30:34 o/ 16:30:36 in before stan [o] / 16:30:49 o/ 16:30:56 o/ 16:31:01 pbourke: ping 16:31:21 o/ 16:31:29 SamYaple: thanks :) 16:31:50 hi 16:32:06 #topic announcements 16:32:19 hello meeting bot? 16:32:39 o/ 16:32:47 he didn't even acknowledge topic rollcall 16:32:56 o/ 16:33:04 your right 16:33:14 welll, not sure what to do about that atm, i'll just proceed :) 16:33:22 freenode has some maintenance reboots, so 16:33:54 1. midcycle was really productive, I'd recommend people hae a look at the midcycle meeting notes if you weren't able to make it 16:34:26 #link etherpad.openstack.org/p/kolla-mitaka-midcycle 16:34:55 2. meeting change about to enter the irc meetings repo - hopefully today 16:35:07 what that means is we will have a meeting at 23:30 and 16:30 UTC alternating 16:35:13 this was by vote 16:35:33 I believe 23:30 was the highest vote, I need to double check - I had a patch in progress prior to midcycle but its not handy atm :) 16:36:15 3. speaking with the foundation, we will get plenty of sessions (more then before) 16:36:36 we dont want to waste slots, but if there is stuff we need to discuss, we can do that 16:37:15 #topic liberty-3 milestone 16:37:20 mitaka? 16:37:26 ya mitaka ;) 16:37:30 busy busy morning sorry guys 16:37:51 so there were some risks identified at the midcycle related to kolla development 16:37:54 you can see those here: 16:38:36 #link https://etherpad.openstack.org/p/kolla-mitaka-midcycle-roadmap 16:38:43 line 12 16:38:55 if your looking for work to do, I highly recommend working on those items 16:39:07 can folks open that up real quick 16:39:31 i want to touch on each one and get a status of where things stand on those items 16:39:31 looking at it 16:39:52 diagnostics 16:40:01 interested to know more about that 16:40:03 elemoine_ - akwasnie is not present 16:40:30 this is on akwasnie? 16:40:32 just to be clear, these are things we identified as critical for mitaka 16:40:50 I can definitely talk to her about that 16:40:51 elemoine_ whoever is working on diags which atm is you and elemoine_ 16:41:02 i dont know if its at risk or not 16:41:05 we had a session scheduled 16:41:18 but the timezones and both of your availability didn't match up 16:41:49 sdake i don't see 2-VIP support or TLS support for external network. are they planned for m-3? 16:41:51 does "diagnostics" include "logging"? 16:41:57 dave-mccowan: yes 16:42:07 dave-mccowan: i said I would handle them, but you are welcome to 16:42:13 dave-mccowan i wanted to ad them to the tracking at risk but sam said they are not at risk :) 16:42:13 its a few minutes worth of work 16:42:42 dave-mccowan mind taking that off SamYaple's plate entirely? He is super overloaded 16:42:55 seems like he is good with the idea :) 16:43:02 ill make sure to submit a patch if its not there by Mar 1st 16:43:17 so i would like to go in order tho :) 16:43:35 elemoine_ can you give us a snapshot of the status of diagnositcs and if anything is at risk or if this item should just be dropped from the at risk list 16:43:57 SamYaple i'm happy to help. let's chat some when you have time. i'll need some pointers. 16:44:06 dave-mccowan: sure thing 16:44:26 nice 16:44:43 ok elemoine_ may be having lag problems i'll move on and come back 16:44:45 reconfigure 16:44:49 sdake: my patch set is very close to be ready 16:45:04 does it integrate with awkasnie's patch set? 16:45:12 no 16:45:27 I'd like them to be merged separately 16:45:34 i would agree with that too 16:45:39 as my patch set is already quite big 16:46:02 after they are both merged, there is integration work to be done? 16:46:22 sdake: yes, but that should be pretty straightforward 16:46:41 and awkasnie and I agreed to collaborate on that 16:46:59 we discussed it during midcycle 16:47:00 ok 16:47:11 where was I? :) 16:47:22 during YOUR midcycle 16:47:25 they were in the channel 16:47:29 oh 16:47:32 i didn't have irc open my bad 16:47:43 I just want to add one commit to the patchset: removal of Rsyslog 16:47:43 busy taking notes :( 16:48:31 cool 16:48:32 and I want to postpone logrotate, Elasticsearch integration, and doc changes (minor) until after the merge of the current patches 16:49:07 so you want to go first on the patch merge you mean 16:49:44 yes, merge what I have today (+ Ryslog removal), after final reviews obviously 16:49:44 yea Heka first, then logrotate, then kibana/elastic search 16:49:54 agreed 16:49:57 exactly how I see things 16:49:57 ok lets get that in the etherpad 16:50:03 if that makes the most sense 16:50:12 typically we give priority to first to write a patchset 16:50:18 but if that makes more sense technically then lets do that 16:50:29 line 15 please 16:50:33 add what you think should be done 16:50:43 and i'll run it by ak* when she is about 16:50:51 I will 16:51:25 inc0 infrastructure service upgrades 16:51:28 tomorrow I'll test Swift 16:51:41 this is the last thing I need to do/test 16:51:41 inc0 iirc you were going to do a haproxy upgrade 16:51:43 I'll push haproxy upgrade today 16:51:45 yeah 16:51:55 as soon as I rebase+test nova 16:51:56 inc0 can you add that to the etherpad please 16:52:13 line 19 16:52:32 I think after this meeting i'll create a new at risk etherpad - if I do tha tI'll link from this etherpad 16:52:45 any objections? 16:52:59 so we can keep the roadmap and mitaka items separate 16:53:32 no problem from me 16:53:50 elemoine_ please put your data in the etherpad during this meeting or shortly thereafter thanks :) 16:54:02 inc0 we have a slew of other services for infrasturcture 16:54:16 so we need to find folks to work on those 16:54:17 yeah, we had separate session for that 16:54:21 right 16:54:28 will do after the meeting 16:54:30 i'll send an announcement to hte ml looking for contributors 16:54:42 FYI, i assigned the bp upgrade haproxy to me. Feel free to change it inc0 16:54:56 migration path for data volumes is at risk 16:54:59 Jeffrey4l, if you want to do it, it should be as simple as redeploy of containers 16:55:02 because nobody is working on it 16:55:35 sdake: I'm trying to become a contributor.. what other areas in the kolla infra need work ? 16:55:44 samyaple re functional tests, is line 24 accurate? 16:56:02 vhosakot, let's talk after meeting ok? 16:56:03 vhosakot i'll send a note to the ml for upgrades, which are 9.2 on our priority list ;) 16:56:04 inc0, I have not started it. 16:56:06 sdake: line 24 is haproxy 16:56:16 ok, I'll do it then 16:56:16 samyaple popele are inserting :) 16:56:19 oh 16:56:27 inc0, sdake : cool! 16:56:27 you mean the functional test vm boot/ping? 16:56:28 thanks. 16:56:34 SamYaple the part vm booting 16:56:35 yes 16:56:40 vms are booting with my patch 16:56:43 no ping yet 16:56:49 cool so no longer at risk 16:56:50 but it validates the instance is ACTIVE 16:56:52 oh ping remians 16:57:03 well tbh I think that is a really rockin solid start 16:57:55 #topic openstack service upgrades 16:58:03 I think the meeting bot died, don't see in channel 16:58:29 sdake: I'm done editing the etherpad, I will run this by awkasnie 16:58:45 elemoine_ for the moment i am going to leave the atrisk items whre thy are 16:58:52 so people ren't confused by a new etherpad 16:59:03 after mitaka-3 i will cut them into a new etherpad so the roadmap doc is tidy 16:59:20 because mitaka-3 is a roadmap item ;) 16:59:25 ok upgrades 16:59:43 at the midcycle we had a super productive session on openstack service upgrades 16:59:53 I htink we have patches for every service in the queue but heat 17:00:00 heat is on me 17:00:13 literally and figuratively ;) 17:00:26 does anyone have questions or concerns about openstack service upgrades? 17:00:35 ive still got neutron 17:00:50 thanks SamYaple i'm confident you can knock that out 17:01:03 if that changes let me know :) 17:01:09 less question more request, we need good multinode testing of this stuff, so if anyone have access, can you plz run plays? 17:01:36 inc0 my gear is impaired atleast until mitaka-3 17:01:48 I *might* be able to get access to a lab to test with 17:01:56 my suggestion is test single node, and merge if it works 17:02:06 sort out multinode in the rcs 17:02:16 wfm 17:02:19 i dont want to block on hardware availaility 17:02:29 sdake: on that note 17:02:39 please let me have a moment to the end of the meeting 17:02:46 sure SamYaple 17:02:47 toward* 17:03:02 actually its open agenda, since nobody added agenda items :) 17:03:13 so SamYaple if you want, you have the floor 17:03:13 topic? 17:03:19 multinode 17:03:24 and testing 17:03:27 #topic multinode testing 17:03:44 so the main issue ive heard from alot of people is the inability to test multinode 17:03:54 just lack of hardware to do so on 17:04:21 Ive recently moved so I was down for a bit, but I have a decent amount of hardware in an openstack cluster running here at the house 17:04:37 there was talk of thirdparty ci during the midcycle 17:04:45 (I've seen it, it is a decent amount) 17:05:14 wanna run 3rd party CI in your room Sam?;) 17:05:26 my first though was to use the equipment as sort of a third party ci, but i think it may be better suited if i gave access to some of those heavier contributors to run multinode vm for kolla testing and development 17:05:42 what does everyone think would benefit the community more? 17:05:47 SamYaple the second 17:06:16 third party ci should be company sponsored imo - although if someone sets up third party ci to comment on patches its not like wed stop them even if they were running AIO third party cis :) 17:06:17 I'd say first 17:06:19 thats what I was leaning towrad too, but im not sure if everyone has access to multinode and just arent using it 17:06:30 the second option that is 17:06:48 anyone can get vms at rax or the like but its pretty expensive 17:06:54 right 17:06:55 non-heavy developer can break multinode just as much 17:06:55 when i talked about 3rd party ci, i meant the one runned by mirantis 17:06:56 how many vms can your environment support sam? 17:07:06 personally I have access to multinode, but it doesn't mean I have time to multinode every PS I review 17:07:14 but i'm not sure whether you're talking about my idea of this ci ;) 17:07:14 but that's a different topic 17:07:23 my hardware can do ~90 8GB vms 17:07:56 actually a really solid idea SamYaple 17:08:05 * nihilifer has the same situation as pbourke 17:08:17 im not hearing a concensus though. it soundsl ike everyone has multinode and doesnt use it 17:08:28 I wouldnt say everyone 17:08:37 most heavy contributors 17:08:43 can we get a poll 17:08:49 who doesn't have access to multinode, please speak up 17:09:05 ( I am, only a laptop :( ) 17:09:10 well, i have one multinode env, but used only for mesos - that's why it's hard to me to test quickly kolla-ansible stuff 17:09:14 that's what i meant 17:09:18 Jeffrey4l GFC may be difficult for you 17:09:37 Jeffrey4l: my stuff is hosted in US, is that an issue? 17:10:00 SamYaple, absolutely no. 17:10:23 yeah i generally don't do multinode 17:10:31 so atm my gear is afu and I don't have multinode but I think I can get that rectified 17:10:33 so we have Jeffrey4l for multinode access rather than CI (right?) 17:10:38 jpeeler: but do you have access? 17:11:03 SamYaple: not really 17:11:03 ya redhat has an internal development openstack environment SamYaple 17:11:17 it's not very stable last i checked 17:11:19 so i don't use that 17:11:49 so really what this comes down to is do I volunteer my equipment to people or Jenkins. 17:11:58 you could do both? 17:12:01 not really 17:12:05 jenkins will tear it up 17:12:08 ok 17:12:17 I only a simple ( three vms running my laptop) env now. can not launch one more vms. 17:12:23 only have 17:12:48 SamYaple: I think you said rhallisey and inc0 were candidates 17:12:51 so with Jeffrey4l that's 3 17:12:55 there's probably more 17:13:03 I can spawn more vms 17:13:12 nah inc0 is covered now he said 17:13:16 but what I was getting at is path of least resistence means people generally dev AIO unless they really have to 17:13:20 SamYaple if you can get it setup and want to maintain openstack hosting for kolla devs, its up to you :) 17:13:23 sounds painful personally 17:13:31 I'll have much better gear soon so I'm covered 17:13:32 so Im not sure it will magically get people doing more multinode testing 17:13:34 pbourke agreed 17:13:45 i qualified this with heavt contributors for a reason 17:13:53 without some sort of coercion :) 17:13:54 that's why I'd rather go for 3rd party CI 17:14:03 yea :/ 17:14:04 so we can at least catch stuff that breaks multinode 17:14:13 really we should all be testing multinode primarily 17:14:24 we can figure out ways to promote that 17:14:24 i thought it was resource contention 17:14:33 so my take on this is we really wan tto try using the openstack infrasturcture for multinode rather then rolling our own third party ci 17:14:37 it sounds like its laziness/time constraints 17:14:37 Im not sure I see the benfit of 3rd party CI 17:14:40 I'd say it's rather efficiency thing 17:14:46 CI is necessary. 17:14:54 im really effiecnt on multinode ic 17:14:55 Jeffrey4l nobody denies that 17:14:55 pbourke, you'll see "failed" gate job on CI 17:14:56 inc0: 17:15:01 that's alerting 17:15:05 its no slower for me to test multinode than aio 17:15:09 inc0: I dont think its a good idea for it to do something infra cant 17:15:16 inc0: so with that said its just another env 17:15:19 pbourke totally agree 17:15:28 pbourke, infra is heavily limited 17:15:36 we can barely run aio on infra 17:15:43 we should fix that then 17:15:48 im not disagreeing but... 17:15:51 infra has repeadily said they will fix whatever we can't get to work 17:15:52 I don't think we can currewntly 17:15:53 we cant gate on sams house 17:15:57 but we haven't really tried multinode 17:16:13 fyi with my gate patches we have ~3gb of ram before VM launch 17:16:18 sdake, we needed to cut down services to boot single vm in infra 17:16:27 inc0 yes i understand 17:16:45 i am nervous about gating on mirantis gear, much less sam's personal lab ;) 17:16:59 lets figure out gating in infra 17:17:01 im also nervous about my equipment for gating 17:17:08 so imho if we lose this gate, that's exactly what we have now 17:17:25 it's a protesis before we can get normal infra to work with us 17:17:25 inc0 except we have to develop the gating, which we could do in openstack-infra 17:17:40 you mean prototype? 17:17:51 alright well to close out this topic ill say this. Im going to be setting up my lab for use by Kolla devs, if anyone wants to use it please contact me 17:18:06 openstack-infra said repeatedly they would give us as many vms as we need, we just need to prove tha twe ccan for example do "two" vms 17:18:12 I mean artificial leg that will keep us standing for now;) 17:18:13 and then 3 17:18:17 SamYaple: I think you'll see up take on that offer. If not you can reconsider for other purposees 17:18:36 cool we have 12 mins 17:18:41 pbourke: did you not have access to multinode? 17:18:41 #topic open discussion 17:18:45 this is whats confusing me 17:18:56 SamYaple people dont test multinode because they are -ELAZY 17:19:02 SamYaple: I do but Id be surprised if others didnt find it useful 17:19:08 can I talk about what I'm working on as a new contributor ? 17:19:22 vhosakot: yes 17:19:25 go for it! 17:20:09 Updating quickstart, hit keystone errors when cross-building (centos on Ubuntu), hence filed https://bugs.launchpad.net/kolla/+bug/1546652, will submit the fix soon 17:20:09 Launchpad bug 1546652 in kolla "Add "--cross-build" argument to kolla-build" [Undecided,In progress] - Assigned to Vikram Hosakote (vhosakot) 17:20:25 SamYaple: I think it would be useful to know/document how you set up you multinode env, so that people can mimic that and do the same in their labs 17:20:41 vhosakot: would you explain what --cross-build is for the team? 17:20:45 sure 17:20:48 elemoine_ d_code has a single node centos setup script coming 17:20:55 By default, "kolla-build" builds all containers using CentOS as the base image. 17:21:03 ue to this, kolla-build fails to build many containers on Ubuntu due to the following errors. 17:21:07 Due* 17:21:17 INFO:kolla.cmd.build:keystone:Error unpacking rpm package httpd-2.4.6-40.el7.centos.x86_64 17:21:17 INFO:kolla.cmd.build:keystone:error: unpacking of archive failed on file /usr/sbin/suexec: cpio: cap_set_file 17:21:17 INFO:kolla.cmd.build:keystone:error: httpd-2.4.6-40.el7.centos.x86_64: install failed 17:21:20 if you run centos containers on ubuntu host, you will have problems 17:21:22 a lot of them 17:21:26 Hence, a new argument "--cross-build" must be added for kolla-build. When the user builds CentOS images on Ubuntu and vice-versa, the "--cross-build" argument must be explicitly specified. If not, kolla-build must throw as error like "Your host OS is Ubuntu and you are building CentOS containers. Please use the "--cross-build" flag to do this" so that the user knows what he/she is doing. 17:21:34 this is anti-pattern 17:22:07 I'm -1 to that, you can print warning 17:22:19 inc0: vhosakot spent 3 days building the wrong images 17:22:23 it needs a hard stop 17:22:24 vhosakot i think with our architecture this coudl be dangeorus to build on a distro with a different systemcall interface 17:22:31 agreed 17:22:43 this is the whole reason we have multi-distro support 17:22:44 what vhosakot purposes is _blocking_ cross building by default 17:22:46 i that case, how about kolla-build preventing what I did the past 3 days :) 17:22:50 add the printed red capitalized warning "you are building wrong images, you will have issues" 17:22:56 vhosakot that wfm 17:23:09 inc0: no it should block. why continue if ther eare known issues? 17:23:19 we currently block containers that we know dont work 17:23:24 not just warn 17:23:44 I just don't want to add new cli options 17:23:51 inc0 me either 17:23:56 cool, so, if host OS and container image type do not match, throw an error and exit... is this right ? 17:23:57 fine well make it a conf only option 17:23:58 and here I think it's redundant 17:24:01 i ws pretty clear bout that in liberty ;) 17:24:08 conf only option is fine 17:24:14 vhosakot return i think 17:24:20 but byy default it should block builds on cross-dristros 17:24:24 vhosakot we want to amke the build.py work as a library 17:24:29 even though it doesn't totally today 17:24:30 ah 17:24:41 one behaviour I would be more ok with is to raise on every container we know will break 17:24:51 ok, throwa good error and return and do nothing.. cool 17:24:52 and add --force to override all raises 17:25:01 inc0: we do not currently track containers. I dont think its time we start 17:25:07 but build ubuntu images on centos works so maybe option would be good some --force 17:25:17 --forece is good.. ok 17:25:29 btw, i am using Archlinux and I can build centos and ubuntu image successfully. 17:25:32 SamYaple, you just want to start to some extend 17:25:43 thats interesting I thought the whole point of docker was you could build any base on any host (within reason) 17:25:43 lets focus on simplicity first folks 17:25:53 adding option per every case we run into is not scallable 17:25:56 pbourke that is dangerous 17:26:17 one more thing, I have a Python script that automates all the steps in the quickstart for Ubuntu. works great and the user need not have to run those commands manually to install kolla.. Can I commit this... 17:26:19 pbourke docker may claim it works, but it does not 100% of the time safely] 17:26:21 ok despite where this conversation is going, here is what we need. Hard block when you are building on the wrong familiy unless some option is specific (cli or conf only i dont care) 17:26:27 sdake: i see 17:26:39 SamYaple: sounds good to me 17:26:49 i think we dont need an option 17:26:53 im surprised this is an issue honestly 17:26:53 it should just block the build 17:27:00 sdake: no because then it would block on archlinux 17:27:02 sdake: some may want to force? 17:27:05 we dont want to limit people 17:27:13 SamYaple i see 17:27:20 --force-cross-build maybe 17:27:21 ok option in this case makes sense then 17:27:22 do we agree that hard block + --force 17:27:30 inc0: yes 17:27:31 ? 17:27:31 * sdake groans at options 17:27:33 yes, --force-cross-build makes sense 17:27:50 it doesnt need to be an option guys 17:27:50 vhosakot i like that option 17:27:51 I'm -1 to for creating option specifically for this case 17:27:54 it can be a config 17:28:00 this is all stuf fwe could talk about in launchpad tho :) 17:28:11 inc0: im -2 on tracking per container what builds 17:28:18 SamYaple, never said I want to 17:28:25 you would have to though 17:28:26 vhosakot: thanks for bringing this up and proposing a solution 17:28:31 no, not really 17:28:32 sure, np 17:28:33 yes thanks vhosakot 17:28:34 vhosakot nice job ;) 17:28:38 we are about to run over time 17:28:38 :) 17:28:41 if we know of issues (like now) we add hard block 17:28:44 we should flow back to #kolla 17:28:46 did you guys see me other comment ? :) 17:28:48 one more thing, I have a Python script that automates all the steps in the quickstart for Ubuntu. works great and the user need not have to run those commands manually to install kolla.. Can I commit this... 17:28:51 we have 1 minute left 17:28:57 any one have something to bring up? 17:29:03 vhosakot: submit for review 17:29:07 cool.. thanks! 17:29:14 sdake: i think we have 5 secodns left.... 17:29:15 vhosakot don't ask for permission ;) 17:29:19 sdake: nope 17:29:19 :) 17:29:24 bye.... 17:29:24 ok folks 17:29:26 thats it :) 17:29:27 bye 17:29:28 #endmeeting