Friday, 2015-11-06

asalkeldother option is to break bps into bugs00:00
asalkeld(just saying)00:00
sdakeasalkeld too lazy for that :)00:00
sdakewe are talking like 30-40 containers00:00
*** jasonsb has quit IRC00:01
sdakethis is a perfect test of work items for distributed work - if your willing to distribute we can give it a go - i'll get blueprint setup00:02
*** jasonsb_ has quit IRC00:02
sdakeSamYaple ^^00:04
SamYaplesure, thats fine00:06
sdakecool00:06
sdakegot a sample of what yu have been prototypingg for the structure of work items?00:06
SamYaplelaunchpad sets the strucuture00:09
SamYapletask: STATUS00:09
sdakewe can't put a name next to the task?00:10
asalkeld"(asalkeld) do this thing": DONE00:10
sdakeare we going to drop permissions inside contianer or outside container00:11
sdakesamyaple ^^00:11
SamYaplewe can00:11
*** jasonsb has joined #kolla00:12
sdakeeither i or SamYaple  is lagged ;-)00:13
*** tpot has joined #kolla00:15
sdakewell i guess we will experiment  with outside the container privilege drop00:16
asalkeldthat means I have to solve this seperately?00:16
asalkeld(for mesos case)00:17
sdakei think thats what it means00:19
asalkeld:-(00:19
sdakeits just a docker flag00:19
asalkeldok00:19
sdakethe big challenge for you will be adding the docker flag to mesos00:19
SamYaplewhat is "outside the container priviliege drop"?00:19
SamYaplejust running ansible as non-root?00:19
sdakedocker run   -u, --user=                     Username or UID (format: <name|uid>[:<group|gid>])00:20
SamYapleno we would bake that into the Dockerfile00:20
sdakeyou mean inside the contianer priv drop?00:20
SamYapleno i mean set USER in hte dockerfile00:21
sdakecool didnt know that was a flag00:21
*** vilobhmm11 has quit IRC00:21
SamYaplecheck out yaodu00:21
SamYaplethis work is all done00:21
sdakeso asalkeld this solves your case as well00:21
sdakenice00:21
asalkeldyeay00:21
SamYaplesdake: was talking in #infra and im moving forward on the true multinode gating stuff00:22
SamYaplestart with two nodes then move on00:23
sdakednd didn't work?00:23
SamYapleit does00:23
SamYaplebut the OOM is a killer00:23
sdakeright00:23
sdakewhat did infra say that changed your mind00:23
SamYaplenothing i changed it yesterday when i figured out how to bypass the registry00:23
sdakecool00:24
SamYaplewith stripping down services i still only have 1G usable ram00:24
SamYapleso its just not viable i dont think00:24
sdakeright00:24
SamYaplei mean we could really limit the nodes,but then is that a real test?00:24
sdakewell booting 1 vm would be hugely beneficial to what we have now00:24
sdakeit would test out compute kit in a dead chicken sort of way00:24
sdakebut gree we want real tests not dead chicken tests00:25
SamYaplethe fact is we don't even validate the containers are running yet, we need to do that00:27
SamYaplepbourke: any thoughts on the tempest stuff00:28
*** diogogmt has joined #kolla00:29
asalkeldsdake: that demo/admin thing is more about the internal/external address than the user/project right? cos' that's confusing00:29
diogogmtsdake: do you guys have a continuos integration pipeline where you can automate the build and push of the images to docker hub?00:30
SamYaplediogogmt: no00:30
SamYapleand we likely won't until dockerhub is better00:31
diogogmtSamYaple: better in which sense?00:31
SamYapleright now we end up pushing the same layer n-number of times and the push takes 12 hours00:31
SamYaplethere are several critical bugs with docker 2.0 and pushing00:31
sdakedocker registry 2.000:34
SamYapleright that00:34
diogogmtSamYaple: but even if it takes a while to push the images, by having the pipeline in place whenever docker hub improves their layer caching machanism the pipeline would already be in place to take the benefits from the speed improvement00:34
SamYaplei agree diogogmt. we are time contrained00:34
sdakediogogmt infra wont let us hve a job run longe then 90 minutes00:34
diogogmtwhat i’m trying to say is that by having official images for the docker services would reduce the barrier of people trying kolla00:34
sdakei may be able to negotiate 120 mins00:34
sdakebut thats maxed oout00:34
sdakediogogmt nothing can be done on our end to solve the problem but i agree with your statement00:35
diogogmtsdake: oh i see, so would it be 120mins to build and push all images?00:35
sdakethat is our limit00:35
sdakeactually our limit is 90 minutes00:35
sdakeit takes me 5 hours00:35
diogogmtcan’t you parallelize the jobs for each image individually? how many nodes can you scale horizontally?00:35
asalkeldsdake: what do you mean by "CR"?00:36
sdakecr = core reviewer i think00:36
sdakeasalkeld depends on context00:37
sdaketypically whe n I say cr, I mean core reiewer00:37
asalkeldyeah, never heard that00:37
asalkeldbrb rebooting00:37
*** asalkeld has quit IRC00:37
sdakediogogmt we could push sevearl images in parlale with docker in docker00:38
sdakebut each new push would push all the same old data00:38
sdakebecause docker is broke00:38
*** asalkeld has joined #kolla00:39
sdakehttps://github.com/docker/docker/issues/1401800:39
*** mbound has quit IRC00:40
diogogmtfair enough, so the plan is to wait for docker to fix the issue with the layer checksum and later implement a ci pipeline?00:40
sdakeadding to the ci pipeline a push operation is a 1-2 liner patch00:41
sdakeso yes blocked on docker00:41
sdakethen blocked possibly on security00:41
sdakebecaue our credentials would have to go in the gate00:42
sdakethsi is a more complex problem to solve00:42
sdakethe infra guys gave me hope this can be solved00:42
sdakeor infra individuals rather00:42
openstackgerritSam Yaple proposed openstack/kolla: Add new gate for multinode  https://review.openstack.org/24228200:43
diogogmti’m not very familiar with the internal processes to build and release openstack services but i see what you are saying00:43
diogogmtjust by curiosity, who runs the infrastructure for openstack?00:43
sdakedocker requires credentials to push to hub00:43
sdakediogogmt openstack00:43
sdakewe need those credentials in the gate job00:43
sdakewe dont just want to check those straight into our repo00:44
sdakethey need to come from a secure server that only infra can access00:44
sdakeanyway as I said this problem is unsolved00:44
sdakebut solveable00:44
sdakeblocked on docker atm ;)00:44
SamYaplewe can push several images in parallel right now00:44
SamYapledockerhub has no problem with that00:44
diogogmtwould be nice if docker hub had access keys in place to segregate access in a rbac model00:45
sdakeSamYaple when I do that the push happens completely for all layers00:45
sdakeso it doesn't actually help00:46
sdakebut ya, it does work00:46
openstackgerritSam Yaple proposed openstack/kolla: Add new gate for multinode  https://review.openstack.org/24228200:46
SamYaplesdake: it does work you can do it in a way that works00:47
SamYaplei typically do00:47
sdakehmm would thatwork for us in 90 mins then?00:47
SamYapleno we cant push from the gate00:48
sdakebecause?00:48
sdakeSamYaple on a orthogonal topic why not adopt this push method in --push?00:49
SamYaplesdake: it only works when pushing to the dockerhub00:50
sdakeroger00:50
SamYaplesdake: it still takes longer than 90 minutes and the disks are slow in te gate00:50
*** dims has joined #kolla00:52
SamYapleok infra helped me out01:02
*** tfukushima has joined #kolla01:03
SamYaplewe should have an experiemental 2-node gate :)01:04
SamYaplesdake: https://review.openstack.org/#/c/242282/ if you want multinode gating, lets get this reviewed faster than infra01:08
*** kjelly has joined #kolla01:12
*** ssurana has quit IRC01:16
*** ssurana has joined #kolla01:16
sdakethere ya go sam, experiment launched :)01:16
*** dims has quit IRC01:17
sdakeSamYaple i gotta run out for 1 hour to pick up my kids glasses, but I'll be back to look at that patch you requested above01:17
sdakebbiaf01:17
*** ssurana has quit IRC01:21
*** sdake has quit IRC01:22
*** sdake has joined #kolla01:24
sdakegroan sounded like an idiot in my email01:25
sdakeoh well not the first or last time ;)01:25
*** jasonsb has quit IRC01:26
britthouserRead the email, but didn't see where you sounded like an idiot sdake?01:44
*** tobe has joined #kolla01:55
sdakebritthouser i repeated the deadline twice02:07
SamYaplesdake: youre right you sounded like an idiot. is that what yo uwant to hear? are you happy now?02:09
sdakehuh?02:09
britthouserAh...I just took that as being emphatic. =)02:09
SamYaplesdake: your emails are fine02:10
sdakethanks02:10
*** tzn has quit IRC02:10
sdakei like the etherpads because they have underline and crossout and colors and stuff02:10
*** bmace__ has quit IRC02:10
sdakebut we can always combine if needed02:11
sdakefor something simple like this work items should do the trick02:11
*** bmace__ has joined #kolla02:11
SamYapleim not quite sure we had the same vision here02:11
sdakefeel free to change asy ou like02:12
sdakewhich part did you not like?02:12
SamYaplei was expecting a larger blueprint with large workitems (drop root user: TODO, drop CAPs: TODO, etc) rather than broken out per service like this02:12
sdakei se02:13
*** stvnoyes has quit IRC02:13
SamYaplei dont neccessarily think this is the right way t ouse work items02:13
sdakepart of this blueprint is to recruit new dev02:13
sdakei hear what your saying02:13
SamYaplesure im just letting you know02:13
sdakewe can adopt that model going forward02:13
sdakeand kick off dev02:13
* britthouser took the bait02:13
sdakemost of the core team sleeping atm :)02:13
sdakeneed to wake the dogs :)02:13
SamYaplemwhahaa02:13
*** stvnoyes has joined #kolla02:14
sdakeSamYaple once every 11 years cisco pays 27 times per year rather then 2602:15
sdakeand i'm so lucky it happend my first year02:15
* sdake groans02:15
sdake-ETOOMUCHTAXES02:15
kjellyNow I awake :)02:16
SamYaplesdake: thats.... wierd02:16
*** unicell has quit IRC02:17
britthouserIts just that 14 doesn't divide 365 evenly.02:17
britthouserI think cisco was late to catch it this time.  Last time they gave us the heads up MUCH sooner in the year.02:18
sdakefinance guy A to finance guy B: "this is going to be a bad quarter."02:18
sdake"Why?"02:18
sdake"We ehave to make payroll one more time this year"02:18
sdake"groaaan."02:18
SamYaplemy companies pays twice a month02:19
sdakeya so did red hat02:19
sdakewhere my wife works ansible paid twice a month too02:19
sdakei like tiwce a month an every 2nd friday02:19
sdakebecause my paycheck rotates aroudn and robyns stays the sme02:19
sdakeits pretty sweet02:19
SamYaplei just budget so i use last months pay for this months bills02:20
SamYaplei always know how much i have t osepnd02:20
sdakeya my dad is on a yearly budget using that technique02:20
sdakehe pays hismelf once a year02:21
sdakeon the 31st02:21
sdakeof dec02:21
britthouserno tax pre-payments that way?02:21
sdakeroger02:21
sdakeno estimated taxes ftw02:21
britthouserYup!02:22
sdakei am suppsoed to make estimated tax payments but too lazy to sort that crap out02:22
sdakedoing my taxes one a year is trying02:23
sdakeocne a yer that is02:23
sdakecan't imgine having to do it 4 times a year02:23
britthouserYeah no thanks!  once a year is just frequent enough that I've forgotten how I did it last year.02:23
sdakei am joe turbotax ftw02:24
britthouserme too, but I forget how to use turbotax. =P02:24
sdakei can't take any deductions or i have to pay amt02:24
sdakeso wife (we are not actually married) takes all the deductions02:24
britthouserEw02:24
britthouserEw to the AMT, not the wife. =)02:25
sdakeya02:25
sdakei actualy can take 1 deduction02:25
sdakebefore amt kicks in02:25
sdakeamt = lets make it harder02:25
sdakePage 35, "How many physical compute nodes do OpenStack clouds have?"02:30
sdake10-99:    42%02:30
sdake1-9:      36%02:30
sdake100-999:  15%02:30
sdake1000-9999: 7%02:30
sdakewe hit 80% of the openstack deployments yay ;-)02:31
britthouserI'm sure within that 100-999 range, 90% are 100-110 =P02:32
SamYapleand 80% of those "plan on having 100 nodes, but we just have a devstack lab right now"02:33
bmace__scaled deploy should be an interesting problem to work out02:34
sdakebmace__ define scaled deploy02:37
*** tfukushima has quit IRC02:42
*** tfukushima has joined #kolla02:43
sdakesam did you watch your video of ur talk02:45
sdakeSamYaple ^^02:45
SamYaplea bit of it02:45
SamYapleim quite attractive02:46
sdakelol02:46
sdakewe did a solid job02:46
sdakesurprised that demo worked :)02:46
SamYaplei think so02:47
SamYaplewould prefer config prep ahead of time02:47
SamYapleand more questinos02:47
sdakeya needed more time for q&a02:47
sdakeyou took longer then i expected to do the config expination02:47
sdakebut it is what it is ;)02:47
SamYaplewhat can i say, i hadnt practiced it one02:48
SamYapleonce*02:48
SamYapleits fine02:48
sdakeya like i said solid job02:48
SamYaplei want a lab02:48
sdakeya a lab would rock02:49
SamYaplethe novnc stuff borked the nova.conf02:56
SamYapleto many rebases02:56
SamYaplepatch incoming02:56
sdakethe one I did?02:56
SamYapleno me02:57
SamYaplewhen i added spice02:57
sdakethis is where a boot gat would help ;)02:58
SamYapleit wouldnt have caught this02:58
SamYapleit would have had to try to hit the console02:59
sdakeoh02:59
sdakeYOU HAVE FAILED ME FOR tHE LAST tiME02:59
sdakeTHATS IT03:00
SamYapleno multinode gate for you!03:00
sdakeTHE REBELS ARE THERE03:00
sdakei have a darth vadar destiy decider03:00
SamYaplehttp://darthjarjar.com/original-theory-by-lumpawarroo/03:01
SamYaplei want you to read that03:01
SamYapleironically or non-ironically you need to read that03:01
sdakehttp://shop.hallmark.com/gifts/darth-vader-destiny-decider-1SHP3003.html03:01
SamYapleyou will be a believe in the end03:01
SamYaplefirst you will laugh. then you will believe03:01
sdakesounds viable03:07
sdakei could se e george chicke nout ;)03:08
sdakebecause boy, jar jar wa annoying03:08
sdakemy favorite character in all tar wars was "dac"03:08
sdake"I feel like I could take on the entire empire"03:09
sdake5 minutes later -> dead03:09
sdakeso epic03:09
sdakeand his name03:09
sdakedigital analog converter03:09
SamYapleyea03:11
SamYapleman though03:12
SamYapleif that really was jar jars original role, that could have been epic03:12
SamYaplewould explain the horribleness of some of the original triology03:12
SamYapleprequel*03:12
SamYaplesdake: i think I am going to convert an existing ceph cluster into the docker version i have in kolla03:22
*** cloudnautique has joined #kolla03:22
SamYaplethat ceph role is almost standalone03:22
SamYaplei may push it to the galaxy of ansible03:22
*** tzn has joined #kolla03:25
*** tzn has quit IRC03:30
*** tzn has joined #kolla03:31
openstackgerritMerged openstack/kolla: Allow disabling of sysctl values  https://review.openstack.org/24185603:31
openstackgerritMerged openstack/kolla: bootstrap takes place on the wrong node  https://review.openstack.org/24198903:32
*** sdake has quit IRC03:36
SamYaplea huge chunk of my spice patch disappered in a rebase it looks like :(03:39
*** Jeffrey4l has joined #kolla03:45
openstackgerritSam Yaple proposed openstack/kolla: Finish implementation of spice  https://review.openstack.org/24232003:46
openstackgerritSam Yaple proposed openstack/kolla: Finish implementation of spice  https://review.openstack.org/24232003:48
bmace__i hope it is an important service, because he who controls the spice, controls the galaxy03:59
bmace__actually, i think it was universe..04:01
openstackgerritSam Yaple proposed openstack/kolla: Make the database json variable more readable  https://review.openstack.org/24202404:01
SamYaplebmace__: actually that was never in the books!04:02
SamYaplebmace__: for shame!04:02
bmace__hey, i read all the books! not the drivel written by his son though04:02
bmace__but i still enjoyed the movie04:02
bmace__even the scifi mini-series04:02
SamYaplegod emperor was pretty drivel04:02
SamYapleand he wrote that04:02
SamYaplesooooo04:02
bmace__well, the first couple were the best, and it went downhill, but i enjoyed them all to varying degrees04:03
SamYaplefirst one i read several times a year04:03
SamYaplethe second one ive read a few times04:03
SamYaplethen the third.... i think ive only read it through once04:03
bmace__i think you have an extra endif in your spice change, though i'm no jinja expert.04:04
openstackgerritSam Yaple proposed openstack/kolla: Make the database json variable more readable  https://review.openstack.org/24202404:06
bmace__i hate to -1 it since i am not 100% sure.. but if it isn't then jinja syntax is odd :)04:06
SamYaplebmace__: i ran it through04:06
SamYapleit works04:06
SamYaplemy guess is you arent tracking the first if clause (enable_nova)04:06
bmace__sure.. so maybe it can work with a spurious endif?04:06
bmace__ah.. maybe that is hidden.. so it is way up there04:07
SamYaplein this patchset i and and if/elif/endif block04:07
SamYaplethats all i add, so anything that was this is already there04:07
bmace__having all this logic shifted to the left makes for terrible nesting context04:07
SamYapleyea thatss why we dont do it in the templates04:08
bmace__yup, found it, looks good04:08
SamYaplefor the docker file04:08
SamYaplehaproxy has just sorta grown04:08
SamYapleits also format dependantant to a certain degree04:08
SamYaplechange the spacing changes the output04:08
*** tfukushima has quit IRC04:28
*** sdake has joined #kolla04:35
sdakeSamYaple could yoou run ceph plan by me again04:35
sdakeyou want to break it out separately?04:35
sdakeor you want to take someone elsess and put it in?04:36
SamYaplesdake: im not quite sure04:39
SamYapleit is pretty much standalone right now04:39
SamYapleit requires nothing from the other services or openstack04:39
sdakei am not keen on taking someone elses code and putting it in04:39
SamYaplenope04:39
sdakeif yu want to break it out separately that wfm04:39
sdakethe license perits anyone to do that04:40
SamYaplewell see. i think it might be better suited as a galaxy role that we consume04:40
SamYapleright04:40
sdakei dont know endough about galaxy consumption to know if that would be good for us or not04:40
SamYapleyea me either04:40
SamYaplejust a thought04:40
SamYapleive found myself using just the ceph role for things outside of kolla04:40
sdakebut making it a sparate galaxy thing couldn't hurt the codebase04:40
sdakebut depending on it is another q :)04:41
SamYapleyea im not quite sure how i want to go about this to be honest04:41
SamYapleit just seems a waste to appear so tied to kolla when its really not04:41
sdakei am not keen on introducing deps04:42
sdakebut having a duplicate is not harmful to kolla04:42
SamYaplei know04:43
SamYaplebut im not keep on duplicates :P04:43
SamYaplekeen*04:43
SamYaplesdake: you notice the centos-source gate is busted?04:44
sdakei saw this afternoon before i went to pic up kids glaases04:45
sdakebut then just woke up from power nap04:45
sdakeso i guess i'll ook at it04:45
SamYapleitss been a few days04:45
SamYaplei think it was a key issue04:45
openstackgerritMerged openstack/kolla: Pin the liberty branch to the liberty RPM repositories  https://review.openstack.org/24222904:46
sdakewhat key?04:46
SamYaplelike a pgp key04:47
SamYapleidk i only looks for a few seconds04:47
sdakei think source and binary centos ar ethee ssame04:48
sdakeryn changed something recently with centos04:49
SamYapleyou import the percona key in openstack-base04:50
SamYapleaccording to the logs04:51
SamYapleidk anything about centos04:51
*** sdake has quit IRC04:53
mandreit looks like it tries to install percona-server-devel instead of mariadb-devel04:54
mandrehttp://logs.openstack.org/29/242229/1/check/gate-kolla-build-images-centos-source/aa0eef6/console.html#_2015-11-06_04_21_51_45304:54
mandreand later fails because it conflicts with installed files: http://logs.openstack.org/29/242229/1/check/gate-kolla-build-images-centos-source/aa0eef6/console.html#_2015-11-06_04_22_02_91004:54
*** diga has joined #kolla04:54
SamYaplei am here to approve patches04:55
*** sdake has joined #kolla04:58
sdakedamn build.py is usted04:59
*** vilobhmm11 has joined #kolla04:59
*** vilobhmm11 has quit IRC05:02
SamYaplehow so?05:06
SamYaplei just used it05:06
SamYapleon master05:06
sdakebuild.py -b rhel -t rdo fails05:06
sdakeryan definately broke it05:07
SamYaplethe gate built what ryan had05:07
SamYaplei made sure05:07
SamYaplebut how is _build.py_ broken05:07
sdakerun the command above05:07
sdakehe made -t be rhel instead of -b be rhel05:07
SamYaplebased on the patch he added, there is no way it was working before05:08
SamYapleso if you thought ti was working, it wasn't05:08
SamYaplehttps://github.com/openstack/kolla/commit/0340bc4043bd65fbc975c11a86ba89e20293be0a05:08
sdakehe just didn't know how to run it apparently05:08
SamYaplewhile i agree rhel is wrong there, there is no way this worked before05:09
sdakeit was working when i commited it05:09
SamYaplei dont see how05:09
SamYaplelook there is no rhos tag05:09
sdakethe rhos tag is in the bae image05:10
sdakeit uses mixed iirc05:10
SamYapleright but look there is not rhos code in build.py05:10
SamYaplethis couldnt have worked05:10
sdakeprofiles patch was prior05:10
SamYapleyou broke this with the install type and metatype patch sdake05:11
SamYaplehttps://github.com/openstack/kolla/commit/0e99b69de48c1deb7f6f1f707e78f445caa5c3d005:11
sdakethat is where i made it work  actually05:11
SamYaplenot possible05:11
SamYaplethere is no rhos info05:11
SamYaplefeel free to look at the code05:11
SamYaplethere would be no metatype05:12
sdakeya i am looing05:12
sdakeperhaps i left out a commit05:12
SamYaplelooks like you need to do -t rhel-rdo05:13
SamYapleyou split on -05:13
sdakeright05:13
sdakethat needs to go in docs05:13
SamYapleanyway, i dont htink ryan broke rhel05:13
SamYaplebut i could be wrong, im not rhel head05:14
sdakegood news nobody broke anything :)05:14
sdake[sdake@bigiron kolla]$ tools/build.py -t rhel-rdo05:16
sdakealso fails05:16
sdake-t rhel fails badly05:17
sdake-b rhel -t rhel is what is needed05:17
mandreso replacing mysql-devel with Percona-Server-devel-55 make centos source openstack-base image build again05:19
sdakemandre cn you geta commit up plz05:20
*** vilobhmm11 has joined #kolla05:20
mandresure, i haven't yet build all images but the gate will probably build faster than I can05:21
SamYapleyou guys really should be using mysql-devel or percona-devel05:23
SamYapleyou should be using mariadb-devel05:23
sdakehuh05:23
sdakeyou mised an n't somewhere in those last two sentences05:23
mandremariadb-devel is already there, i have no idea why mysql-devel was installed05:24
sdakeuse backport tag please05:24
SamYaplemandre: its old cruft05:24
SamYapleremove it05:24
*** tfukushima has joined #kolla05:24
mandrei will05:24
SamYaplesdake: first sentence shouldnt05:24
SamYapleyuo should NOT be using mysql-devel or percona-server-devel05:25
*** tfukushima has quit IRC05:27
*** tfukushima has joined #kolla05:28
openstackgerritMartin André proposed openstack/kolla: Remove mysql-devel cruft  https://review.openstack.org/24233605:40
mandre^ this should fix the centos source gate05:42
*** dayou has joined #kolla05:47
openstackgerritLin Hua Cheng proposed openstack/kolla: fix typo in README.rst  https://review.openstack.org/24233805:47
*** tpot has quit IRC05:48
openstackgerritJeffrey Zhang proposed openstack/kolla: Use the absoluate path  https://review.openstack.org/24233905:56
*** sdake has quit IRC06:00
*** sdake has joined #kolla06:00
openstackgerritMartin André proposed openstack/kolla: Remove mysql-devel cruft  https://review.openstack.org/24233606:08
mandreSamYaple, sdake ^^ go crazy on the +206:09
openstackgerritSteven Dake proposed openstack/kolla: Make RHEL build properly  https://review.openstack.org/24234306:13
openstackgerritMerged openstack/kolla: fix typo in README.rst  https://review.openstack.org/24233806:13
*** vilobhmm111 has joined #kolla06:18
*** vilobhmm11 has quit IRC06:19
*** Jeffrey4l has quit IRC06:24
*** unicell has joined #kolla06:24
*** tfukushima has quit IRC06:31
*** openstackgerrit has quit IRC06:31
*** openstackgerrit has joined #kolla06:31
*** tfukushima has joined #kolla06:31
*** tfukushima has quit IRC06:34
*** tfukushima has joined #kolla06:35
*** Jeffrey4l has joined #kolla06:36
*** vilobhmm111 has quit IRC06:54
*** unicell1 has joined #kolla07:03
*** unicell has quit IRC07:04
openstackgerritSteven Dake proposed openstack/kolla: Make RHEL build properly  https://review.openstack.org/24234307:09
*** Jeffrey4l has quit IRC07:09
openstackgerritJeffrey Zhang proposed openstack/kolla: Use the absoluate path  https://review.openstack.org/24233907:11
*** sdake has quit IRC07:14
*** sdake has joined #kolla07:14
*** Jeffrey4l has joined #kolla07:21
SamYaplemandre: never!07:22
openstackgerritMerged openstack/kolla: Remove mysql-devel cruft  https://review.openstack.org/24233607:24
*** Jeffrey4l has quit IRC07:26
*** sdake has quit IRC07:26
mandrethanks SamYaple, since you're a good guy I approved your ntp doc change07:27
*** sdake has joined #kolla07:27
openstackgerritMerged openstack/kolla: Add additional ntp information around ceph  https://review.openstack.org/23675807:28
SamYaplewhat the what07:28
SamYapleno ways07:28
SamYaplemandre: on the spice on yes huge portions of the implementation were missing07:28
SamYaplethe only thing that survived was some nova.conf and group_vars/all.yml changes07:29
SamYaple_and_ it broke deploys07:29
SamYapleso all around, yay!07:29
SamYaplei could live in tokyo07:30
SamYaplei could do that07:30
mandrewhat do you mean it broke deploys?07:32
openstackgerritSteven Dake proposed openstack/kolla: Make RHEL build properly  https://review.openstack.org/24234307:33
openstackgerritSteven Dake proposed openstack/kolla: Make RHEL build properly  https://review.openstack.org/24234307:33
sdakewhat broke, is stable busted?07:37
*** sdake_ has joined #kolla07:40
*** Jeffrey4l has joined #kolla07:40
kjellyHi, I want to work on the bug. https://bugs.launchpad.net/kolla/+bug/1513018  murano ui is needed to be installed extra. where is the palce to put install script? dockerfile or ansible ?07:40
openstackLaunchpad bug 1513018 in kolla "murano gui does not work" [Critical,New] - Assigned to Kuo-tung Kao (jelly) (coding1314)07:40
sdake_kjelly no idea we haven't sorted out plugins yet07:41
kjellyit seems that ansible is good way07:41
sdake_feel free to tackle it ;)07:42
kjellysdake_: Ok.07:42
*** sdake has quit IRC07:42
openstackgerritMerged openstack/kolla: Use the absoluate path  https://review.openstack.org/24233907:43
sdake_mandre can you ack https://review.openstack.org/#/c/242282/07:48
*** Jeffrey4l has quit IRC07:49
*** shardy has joined #kolla07:49
openstackgerritMerged openstack/kolla: Add new gate for multinode  https://review.openstack.org/24228207:52
SamYaplekjelly: sdake_ the murano thing is part of the bigger "horizon external plugins" stuff08:00
openstackgerritSam Yaple proposed openstack/kolla: Remove unused setup_fedora.sh  https://review.openstack.org/24236408:05
openstackgerritSam Yaple proposed openstack/kolla: Adjust gate btrfs setup  https://review.openstack.org/24236508:05
openstackgerritSam Yaple proposed openstack/kolla: Restrucutre gate scripts  https://review.openstack.org/24236608:05
openstackgerritSam Yaple proposed openstack/kolla: bootstrap takes place on the wrong node  https://review.openstack.org/24236708:05
openstackgerritSam Yaple proposed openstack/kolla: Add new gate for multinode  https://review.openstack.org/24236808:05
openstackgerritSam Yaple proposed openstack/kolla: Allow specifying erasure as the pool type  https://review.openstack.org/24237108:09
openstackgerritSam Yaple proposed openstack/kolla: Allow ceph pool names to be configurable  https://review.openstack.org/24237208:09
openstackgerritSam Yaple proposed openstack/kolla: Automate setting up a cache tier  https://review.openstack.org/24237308:09
openstackgerritSam Yaple proposed openstack/kolla: Add new gate for multinode  https://review.openstack.org/24236808:13
openstackgerritSam Yaple proposed openstack/kolla: Restrucutre gate scripts  https://review.openstack.org/24236608:13
openstackgerritSam Yaple proposed openstack/kolla: bootstrap takes place on the wrong node  https://review.openstack.org/24236708:13
openstackgerritSam Yaple proposed openstack/kolla: Remove unused setup_fedora.sh  https://review.openstack.org/24236408:14
openstackgerritSam Yaple proposed openstack/kolla: Adjust gate btrfs setup  https://review.openstack.org/24236508:14
openstackgerritSam Yaple proposed openstack/kolla: HAProxy does not have murano-api configuration  https://review.openstack.org/24237908:20
openstackgerritSam Yaple proposed openstack/kolla: Add missing murano-api to site.yml  https://review.openstack.org/24238008:20
openstackgerritSam Yaple proposed openstack/kolla: Download, install, and enable rabbitmq_cluster  https://review.openstack.org/24238108:20
openstackgerritSam Yaple proposed openstack/kolla: Use rabbitmq-clusterer when deploying  https://review.openstack.org/24238208:20
openstackgerritSam Yaple proposed openstack/kolla: Small refactor for site.yml  https://review.openstack.org/24238308:20
openstackgerritSam Yaple proposed openstack/kolla: Remove vip for rabbitmq  https://review.openstack.org/24238408:20
openstackgerritSam Yaple proposed openstack/kolla: heat containers depends on 'common' (rsyslog, log_data, kolla-ansible)  https://review.openstack.org/24238508:26
*** sdake_ has quit IRC08:33
*** zigo has quit IRC08:37
*** zigo has joined #kolla08:42
*** openstackstatus has quit IRC08:42
openstackgerritMartin André proposed openstack/kolla: Remove mysql-devel cruft  https://review.openstack.org/24239408:42
*** akwasnie has joined #kolla08:47
*** kproskurin has joined #kolla08:50
*** mbound has joined #kolla08:59
*** mbound has quit IRC09:02
*** mbound has joined #kolla09:02
kproskurinHi all09:03
kproskurinmandre: Mind to disscuss your comment on https://review.openstack.org/#/c/240827  ?09:03
mandrehey kproskurin09:04
SamYapleany cores around ecept mandre?09:05
SamYaplepbourke: coolsvap jpeeler harmw ?09:05
kproskurinHi! I agree on var and dir vs file, but debian thing… well, it’s a patch to support *Ubuntu* and, hopefully, not break other things. May be it’s same for debian… I not sure09:05
SamYapleguy we are slower than infra!09:06
mandreyeah, I understand you're trying to fix ubuntu, but if we can generalize and fix all debian-based distros I'd much prefer that09:07
kproskurin“all debian-based distros” is more than Debian + Ubuntu :-) Oh, I could try to test Debian, but just my “patch scope”09:09
*** tzn has quit IRC09:10
mandrekproskurin, if you want you can do it in a follow up patch09:10
mandrei don't want to block your patch09:10
kproskurinI just not sure if Debian is supported by Kolla at all. :-)09:11
*** cemmason has joined #kolla09:11
kproskurinAnd openstack in general09:11
mandreI suspect ubuntu just install pip packages in the same location as debian does, it would be good to confirm09:12
nihiliferyep, agree, probably all "debian family" distros have the similar policy regarding pip. but I also agree that Debian should be covered in the separate patch09:12
nihiliferbecause probably no one here has environment with Debian09:13
*** diga has quit IRC09:16
kproskurinYep09:16
SamYaplei use debian....09:18
cemmasonyou would!09:18
SamYaplego back to your hole cemmason!09:18
cemmasonfreenode is my hole!09:18
cemmason=P09:18
SamYaplei thought you decided to put yourself into a coma until fallout 4 came out09:19
SamYaplewaiting was to hard09:19
cemmason*cries*09:19
*** achanda has quit IRC09:21
*** jmccarthy has quit IRC09:22
mandreSamYaple, where are your pip packages installed?09:22
kproskurinasking the right question!09:23
SamYapledebian based systems use /usr/local/09:23
kproskuringit clone kolla09:23
kproskurinas expected09:23
*** jmccarthy has joined #kolla09:24
SamYapleif you look at that crazy sed in docker/base/Dockerfile.j2 for ubuntu it actually changes that for our images09:24
SamYapleunrelated just thought you should know09:24
kproskurinSo guys, sould I add Debian to a list of checked OS in my patch for /usr/local/? Or make it a separate patch?09:24
mandreisn't there a generic way to find the distro family?09:25
mandreor you can check for existence of /usr/share/kolla or /usr/local/share/kolla, so that it will always find the right place09:28
*** gfidente has joined #kolla09:32
*** gfidente has joined #kolla09:32
kproskurinOh well, then I start to write this patch I was suprissed what there is no easy and one way on every linux distro to actualy tell what is this distro name. Personaly, I think checking /usr/local/../kolla vs /usr/…/kolla is bad way to do things. It’s not really relaible.09:38
*** tzn has joined #kolla09:48
*** tzn has quit IRC09:54
*** tobe has quit IRC09:55
SamYaplekproskurin: /win 1809:57
kproskurinwhhhaaat?..09:58
SamYapleclearly i was asking you to join me in /win 1810:00
SamYaple:)10:00
SamYaplekproskurin: so anyway i dont agree with checking /usr/local either10:00
SamYaplebut we can throw in a small debian check to make everyone happy10:00
kproskurinLet’s make it separate patch then. I really  want to merge this and commit fixes to new doc after. And I still don’t get “/win 18” joke(?) %-) Sry!10:02
britthouserSo I"m going through the Kolla Dev Quickstart guide.  It says to use docker 1.8.2 b/c 1.8.3 is busted.  But I know 1.9.0 just came out.  Anybody tried it with kolla yet?10:03
SamYaplebritthouser: anything past 1.8.2 will not work10:04
SamYaplenot until ansible 2.010:04
SamYaplekproskurin: /win 18 was a typo. i use irssi and i was trying to change windows10:05
kproskurinoh %-)10:05
britthouserthanks SamYaple.  is the details of the incompatibility documented somewhere?  Just want to satisfy my curiousity10:07
SamYaplebritthouser: docker-devs in thier infinity wisdom changed the docker api between version without bumping the api version10:08
SamYaplethe ansible module blew up10:08
SamYapleansible modules are fixed now, but ansible wont cut another 1.9.x tag10:08
SamYapleso we have to wait for 2.0 and then upgrade all the playbooks10:08
openstackgerritProskurin Kirill proposed openstack/kolla: Few improvements in Kolla tools. Added Ubuntu support.  https://review.openstack.org/24082710:09
*** dayou has quit IRC10:09
SamYaplebritthouser: https://github.com/ansible/ansible-modules-core/pull/225810:09
*** athomas has joined #kolla10:09
britthouserOk cool.  We hit some issues with docker1.9 API changing as well.10:10
SamYaplebritthouser: i may or may not be rolling an ansible docker module for kolla to control10:11
SamYapleansible has proven unreliable here10:11
SamYaplesince the docker module is critical to us....10:11
*** Slower has quit IRC10:11
britthouserYeah if everyone rolls forward and nobody backports, then its just throwing darts. =)10:12
SamYaplenope10:12
SamYapleits backported10:12
SamYaplemade sure of that myself10:12
SamYapleansible wont snap a new 1.9.x tag10:13
SamYaplehard no10:13
britthouserbackported but never released?10:13
*** mfalatic has joined #kolla10:13
SamYaplebingo10:13
britthouserbrilliant10:13
*** Slower has joined #kolla10:13
SamYapleits not the first docker module problem we have had either10:13
SamYapleso rolling our own, how ever non-opensource that is, is the safest option for the stability of kolla10:14
*** mfalatic_ has quit IRC10:15
*** slotti has joined #kolla10:17
openstackgerritBritt Houser proposed openstack/kolla: Update Centos7 docker commands  https://review.openstack.org/24242510:25
kproskurinOh, I actually got same fixes in the incoming doc update :-)10:32
britthouserIs it up for review kproskurin?10:33
kproskurinnot yet, it’s waiting for other commit as a dependency10:33
SamYaplekproskurin: you can do dependant reviews10:34
britthouserOk.  I can abandon mine if you want.  I was also noticing some other inconsistencies around when/how pip is installed.   I was gonna fix that next, but can wait if you've got that covered.10:35
*** tfukushima has quit IRC10:35
kproskurinbritthouser: “inconsistencies around when/how pip is installed” care to eleborate? I actually working on that too10:35
kproskurinSamYaple: what tag do I need use for that?10:36
SamYaplekproskurin: just make a second patch ontop of the other one10:36
SamYaplethen git review10:36
SamYapleit will ask you if you want to push both, say yes10:36
nihiliferI have one silly question regarding keepalived container - was some magic done to succesfully "modprobe ip_vs" in start script?10:37
britthouserWell, e.g. the doc says to pip install requirements at the top, but then lower it says to install pip.  Also it says to easy_install pip for Centos, but to install pip from packaging for ubuntu10:37
kproskurinoh, I thought it’s some kind of tag like “Dependend-on: #number”10:37
britthouserI was gonna add "yum install -y epel-release" followed by "yum install -y python-pip" for centos10:37
britthouserBut if you have that covered, go for it.10:38
SamYaplekproskurin: depends-on is for cross-repo stuff (like when nova requires a patch to merge in neutron)10:38
kproskurinbritthouser: “ pip install requirements” this will go away in new doc.10:38
SamYaplenihilifer: what do you mean, magic?10:38
kproskurin“Also it says to easy_install pip for Centos, but to install pip from packaging for ubuntu” this is actually ok, no?10:38
SamYaplenihilifer: we modprobe in all kinds of places10:38
britthouserWell just not consistent.  Why not install pip from packaging on centos?10:38
kproskurinit requires EPEL10:39
kproskurinafaik10:39
nihiliferSamYaple: is making mount /lib/modules:/lib/modules:ro enough to do "modprobe" inside container without any errors?10:39
britthouserYeah..."yum install epel-release" and then "yum install python-pip"10:39
SamYaplenihilifer: da10:39
britthouseror is epel bad news here?10:39
kproskurini tjink it’s bad idea to install epel only for pip. And yes, it’s bad in general(personal opinion)10:39
britthouserOk.  Its lesser of two evils in my opinion, but its just that, my opinion. =)10:40
britthouserkproskurin: are you also covering the fact that gcc needs to be installed before doing "pip install -U python-openstackclient" ?10:41
kproskurinyes :)10:41
britthouserOk...you've got all my bases covered then. =)10:42
kproskurinI could send you full diff if want10:42
kproskurinmay be you’ll find something else10:42
britthouserno I'll just wait for review to post10:42
SamYaplekproskurin: just post the patch so we can all review it :)10:42
kproskurinok, ok. :-)10:43
kproskurinI just wasent sure about dependencies. New doc need my tools patch to be merged and I didnt know how to show it10:43
britthouserso if you've made them as two sequential commits in your same topic branch, then you can just git review on the second commit, and both will be pushed with the correct dependacy10:45
*** shardy has quit IRC10:46
britthouserif you've made them different patches in unique topic branches, then you could rebase the tools patch into the doc branch (or vice versa) and then submit the review.10:46
kproskurinwell, it’s a different branches in my git10:46
britthouserjust make sure you get the ordering right.10:46
kproskuringit is scary10:47
britthouserYou can rebase the patch from one branch to another.10:47
britthousersdake just taught me this a few months ago.10:47
britthouser=)10:47
britthouserso in your docs branch, do a 'git log'10:48
britthousercopy down the sha of your docs commit10:48
britthouserthen switch to bug/151230210:48
britthousergit rebase -i origin/master10:48
kproskurinhttp://imgs.xkcd.com/comics/git.png10:49
britthouseryou'll get an editor that says something like "pick 12312 Few improvements...."10:49
britthouserunder that line add "pick <sha>"10:49
britthouserthen git review10:50
britthouserand it'll ask you about posting both.10:50
britthouserlol kproskurin10:50
pbourkeSamYaple: are we backporting clusterer to liberty?10:51
pbourke(or have we)10:51
britthouserpbourke: it was on the list at the bottom of: https://etherpad.openstack.org/p/kolla-mitaka-upgrade10:51
pbourkebritthouser: k thanks10:52
kproskurinmandre: oh, you right, I add func and never use it. What is wrong with me…10:59
kproskurinDo you thin it should be a func, tho’? Oneliner…10:59
mandreno worries kproskurin, it happens10:59
mandreup to you, I wouldn't bother writing a function for it personally11:00
SamYaplepbourke: for 1.1 we will be, yes11:01
pbourkeim raging at gerrit right now11:01
SamYaplepbourke: https://review.openstack.org/#/c/242364/ <-- that patch series needs to merge. its holding up infra11:02
SamYapleit doesnt include ceph or rabbitmq11:02
pbourkei just moved to firefox and cant get my filters to work http://imgur.com/qted3mU11:02
pbourkedoes anyone have any useful gerrit filters other than status:open11:02
pbourkeSamYaple: can I workflow that patch given the gates are red11:03
openstackgerritMerged openstack/kolla: Update Centos7 docker commands  https://review.openstack.org/24242511:03
SamYaplepbourke: thats unrelated entirely11:03
pbourkei know but...11:03
SamYapleyes you can11:03
openstackgerritProskurin Kirill proposed openstack/kolla: Few improvements in Kolla tools. Added Ubuntu support.  https://review.openstack.org/24082711:03
mandrepbourke, the gate will be back to green after you +a https://review.openstack.org/#/c/242394/11:04
SamYaplemandre: i think he already did11:04
mandrehint hint11:04
mandrehe then retracted :)11:05
pbourkeyeah i was concerned about +a with red gates11:05
pbourkegot shouted at for that before11:05
openstackgerritMerged openstack/kolla: Remove unused setup_fedora.sh  https://review.openstack.org/24236411:06
SamYaplepbourke: in this case we are literally the ones holding up infra11:06
openstackgerritMerged openstack/kolla: Remove mysql-devel cruft  https://review.openstack.org/24239411:06
SamYaplepbourke: all the patches up to https://review.openstack.org/#/c/242368/ have to merge for the infra patch to merge11:07
SamYaplethey are cherry-picks, but feel free to -1 if you have a problem11:07
SamYaplewith the red gate i mean11:07
SamYaplei dont want to push you if you dont feel comfortable11:08
openstackgerritMerged openstack/kolla: Adjust gate btrfs setup  https://review.openstack.org/24236511:09
openstackgerritMerged openstack/kolla: Restrucutre gate scripts  https://review.openstack.org/24236611:10
openstackgerritMerged openstack/kolla: bootstrap takes place on the wrong node  https://review.openstack.org/24236711:10
openstackgerritMerged openstack/kolla: Add new gate for multinode  https://review.openstack.org/24236811:10
openstackgerritMerged openstack/kolla: heat containers depends on 'common' (rsyslog, log_data, kolla-ansible)  https://review.openstack.org/24238511:11
SamYapleso with those patches in line, we will now have an experimental 2 node gate11:13
SamYaplei can do actual multinode gate testing11:13
britthouserw00t!11:14
*** pbourke has quit IRC11:15
*** pbourke has joined #kolla11:15
*** achanda has joined #kolla11:22
*** achanda has quit IRC11:26
*** dims has joined #kolla11:27
*** gfidente is now known as gfidente|lunch11:40
*** dayou has joined #kolla11:42
*** akwasnie has quit IRC11:50
*** tzn has joined #kolla11:51
pbourkemandre: did you ever have errors around ip6tables with vagrant-libvirt ?11:54
*** tzn has quit IRC11:56
mandrepbourke, nope12:10
mandrewhat kind of issue do you have/12:10
mandre?12:10
pbourkemandre: http://paste.openstack.org/show/478175/12:13
pbourkeI haven't figured out why its using ipv6 yet12:13
pbourkeor why it thinks it needs to use ip6tables rather than iptables12:14
pbourkeipv6 should be disabled12:14
mandredo you have the right kernel modules loaded?12:16
*** rhallisey has joined #kolla12:16
mandrelsmod | grep ip612:18
openstackgerritMerged openstack/kolla: Few improvements in Kolla tools. Added Ubuntu support.  https://review.openstack.org/24082712:22
kproskurintx guys!12:23
openstackgerritProskurin Kirill proposed openstack/kolla: Few improvements in the quickstart doc  https://review.openstack.org/24246712:24
*** achanda has joined #kolla12:24
mandrekproskurin, you're now ATC, congrats!12:25
kproskurinbritthouser: check this docs update, I hope I covered your things12:25
kproskurinmandre: ATC?...12:25
mandreactive technical contributor12:25
kproskurinsounds awesome %-)12:26
mandreit means you have at least 1 commit in a project in the openstack namespace12:26
kproskurinhey, it’s my second merged commit! -)12:26
kproskurinnot first :-)12:26
mandrewhat it really means is you will have a free ticket to the openstack summit :)12:26
mandrecool, your other patch took less time to land12:28
kproskurin“ATC codes provide a discount of $600 USD off all ticket prices” - not really free if you’re slowpoke for buying tickets actually. :-)12:29
*** achanda has quit IRC12:30
mandreit's free for early bird registration12:31
kproskurinGuys, just so you know, I started working on adding binary images for Ubuntu. If anyone doing the same, we could split our work. -)12:31
mandreI believe nobody else is working on it at this moment12:34
mandreor if they do they didn't say so12:34
kproskurinWell, you never know. I think we(me + my collegue) could do it during next week or so12:36
*** zigo has quit IRC12:36
*** zigo has joined #kolla12:37
*** akwasnie has joined #kolla12:40
britthouserkproskurin: I added a couple of comments12:43
britthousergnocchi has no RPMs available...this is expected, right?12:51
*** tzn has joined #kolla12:52
*** tzn has quit IRC12:56
kproskurinbritthouser: tx! Looking into it13:08
kproskurinbritthouser: “Not sure this section is necessary since we had them install gcc prior to installing the openstackclients.” Well, install of openstackclients is optional, so it could be skiped. I’ll try to move this section a bit13:12
britthouserYeah that's true.13:13
SamYaplekproskurin: binary packages for ubuntu are pretty much a no go13:30
SamYaplekproskurin: ubuntu doesnt provide a delorean repo at all13:30
SamYapleso they have liberty packages _after_ liberty is released13:31
kproskurinSamYaple: yeah, just discuss it like 1 min ago with collegue…13:31
kproskurinThat is sad13:32
SamYapleits ok, te cloud-archive packages are always bad and broken13:32
SamYaplejust like delorean13:32
SamYaplesource is far superior for this type of thing13:32
kproskurinBuilding from source it’s like… I dunno, “bad”, for me, personaly13:33
britthouserSo we could only have ubuntu binary in the stable branch, but not master, right?13:35
SamYapleright, but we arent merging all new code into a stable branch like that13:35
SamYaplekproskurin: its build from source once, then its packaged (docker container)13:36
britthouserYeah that makes total sense.13:36
*** akwasnie has quit IRC13:38
SamYaplekproskurin: i dont know how much youve done with openstack from source, but when done right its far and away superior to the packaged versions of openstack. for all kinds of reasons13:41
*** diogogmt has quit IRC13:49
SamYaplebritthouser: if you notice I actually _do_ include cloud-archive for packages13:50
SamYaplewe take advantage of the newer qemu and rabbitmq and ceph etc13:50
SamYaplecananical has commited to maintaining security patches for those packages and they are newer than the old 14.04 packages13:51
SamYapleso its actually a pretty good balance13:51
britthouserSource for openstack, cloud-archive for the supporting packages?13:51
SamYaplecorrect13:51
SamYaplecloud-archive pulls from sid _or_ packages head of upstream for the project at the time of repo creation13:52
SamYapleso its actually really new stuff13:52
*** tzn has joined #kolla13:52
*** diogogmt has joined #kolla13:53
britthouserMakes sense.  In the ops meetings the past couple of summits there has been a "lets get together and build packages together" initiative.  Nobody has actually stepped up todo anything yet.  But if that does get off teh ground, perhaps there would be packages for ubuntu prior to release.13:53
britthouserI'm not holding my breathe, but something to keep tabs on at least.13:54
SamYapleops? build packages? you let me know when that happens! (I am an ops guy btw ;)13:54
britthouserI have a (probably dumb) questions about https://blueprints.launchpad.net/kolla/+spec/drop-root13:55
SamYapleshoot13:55
britthouserSo I'm gonna make the container run as a different user13:55
britthouserdifferent user = process I'm running (eg. keystone container runs as keystone)13:56
britthouseris that right?13:56
SamYapleyea13:56
britthouserbut if my base OS has no keystone packages installed.....I have no keystone user?13:56
britthouserso this will require creating the user as well?13:56
SamYapleoh well we create the keysont/glance/neutron users already13:56
SamYaplealthough keystone is a bad example, since that runs as apache2 (and apache2 runs keystone process as user keystone)13:57
SamYaplebritthouser: https://github.com/openstack/kolla/blob/master/docker/glance/glance-base/Dockerfile.j2#L3913:57
*** tzn has quit IRC13:57
britthouserOH...so this is all within the container?13:58
SamYapleyes13:58
SamYaplethe containers themselves run as root on the host (docker only runs as root)13:58
britthouserOk I get it now.  I was looking in /etc/passwd on the host for user keystone.13:58
britthouserI'm on the same page now I think.13:59
britthouserand I see what you're saying no in the keystone container.  httpd running as apache (except for PID1), and keystone running as keystone.14:00
britthouserSo the only diffrence I would expect to see when I add USER to the keystone container is that PID1 woudl change from root to apache, right?14:01
*** absubram has joined #kolla14:01
SamYaplefor keystone containrer it will be different, not quite sure how it will look14:02
SamYaplefrom a glance perspective youll see pid 1 be running as glance14:02
SamYaplelook at what i have set in the yaodu project14:02
SamYaplethats the final peice yoadu has to offer kolla14:02
britthouserYeah I see the USER {{ container }}14:03
britthouserok, when I login the glance container today, I see all the glance-api running as root.  That's the part we're trying to fix.14:04
* britthouser may try an easier container than keystone first.14:04
*** absubram_ has joined #kolla14:06
*** absubram has quit IRC14:06
*** absubram_ is now known as absubram14:06
*** slotti has quit IRC14:06
openstackgerritProskurin Kirill proposed openstack/kolla: Few improvements in the quickstart doc  https://review.openstack.org/24246714:15
kproskurinbritthouser: I hope it’s better now14:15
*** sdake has joined #kolla14:16
sdakemorning14:17
SamYaplemornign sdake14:17
sdakewow sam stlil up?14:17
SamYapleinfra approved our multigate before we got the patches backported :P14:17
SamYaplewe lost14:17
sdakewe los twhat14:18
SamYaplethe race to approve a patch faster than infra14:19
*** diogogmt has quit IRC14:21
SamYaplebtw sdake i put a whole bunch of backports in the queue14:24
SamYaple-1 now or forever hold your peace14:24
sdakecool did yu backport everything that is required for 1.0.1?14:25
SamYaplei dont know what was required14:25
sdakei guess anything with a backport tag14:26
*** diogogmt has joined #kolla14:26
SamYapleidk i only dug through a few patches14:27
sdakei've been saking people ot add a backport tag tothe patch if I think it is backport material14:27
britthouserovsdb data container, rabbit clusterer, no-preempt for keepalived, pine ceph to specific release, write a version to /etc/kolla/release14:27
SamYaplemy oneline cleared from my history :(14:27
*** athomas has quit IRC14:27
SamYaplebritthouser: those are things for 1.114:27
britthouseroh right.14:27
sdakealthough I may not review every change14:27
SamYaplebackporting does not boost commiting stats people14:28
SamYapleplease do it14:28
sdakebackporting is such a pain in teh ass if we rely on one person to do the job14:28
sdakebecause one person doens't know what another person should have backported14:29
sdakepleae take the time to backport folks14:29
SamYaplei backport everything non-docs and non-breaking14:29
*** severion has joined #kolla14:31
sdakeSamYaple can you figure out what your one liner ws to backport and sbmit a backport patch into the repo so anyone can backport?14:34
SamYaplei have unfortunately lost that oneliner and must rebuild it14:35
SamYaplei may or may not have rekicked my nodes14:36
SamYapleturns out bash hsitory is not a great archive for things like that14:36
sdakeroger - can yu rebuid it is whati was asking14:37
SamYapleprobably14:38
SamYaplebut this is more art than science14:38
sdakei woudl wirte one but i'm not sur eehow to get the hsh id based upon the ocntents of the change14:38
SamYapleno two oneliners are alike14:38
britthouseris it just a matter of cherry-picking to stable/liberty?14:38
SamYapleit is!14:38
britthousergerrit has a button for that.14:39
SamYapleyou go click through 200 patches14:39
SamYaplei dare you!14:39
britthouserI thought sdake was asking each person to backport as they patch?14:39
*** athomas has joined #kolla14:39
britthousermaybe I misunderstood14:39
sdakethe foolrpoof way is for the person who +a's a backport change to be rewponsible for the backport14:40
kproskurinpbourke: Mind to disscuss your comment in https://review.openstack.org/#/c/242467/ ?14:42
pbourkekproskurin: yes does it not make sense?14:42
kproskurinWe disscuss it with britthouser before, well thing is we tell to install gcc in *optional* section, which may be skiped14:43
kproskurinI would like to hear how to make it more clear in this docs, if you have any idea14:44
dmsimardrhallisey: just noticed I didn't actually merge that change from yesterday, just did14:44
SamYaplepython-dev on ubuntu pulls in gcc fyi14:45
kproskurinActually, we could make it mandatory and add some openstack cli examples… but it may make things more complicated14:45
SamYaplesdake: i like that14:45
rhalliseydmsimard, kk14:45
SamYaplesdake: +a must backport14:45
*** achanda has joined #kolla14:46
pbourkekproskurin: ok I guess it wasn't too obvious the first section was optional. No big deal I can +214:47
*** rbradfor has joined #kolla14:47
kproskurinpbourke: I agree it a bit confusing and we repeat ourselfs… I just not sure how to fix it14:48
kproskurinOn the other hand it’s really not that critical14:48
pbourkekproskurin: yeah its not a big deal14:49
pbourkeimo14:49
kproskurinty14:49
*** achanda has quit IRC14:51
*** ashishjain has joined #kolla14:59
*** dayou has quit IRC15:02
*** blahRus has joined #kolla15:24
*** cloudnautique has quit IRC15:24
sdakeSamYaple that aid, we need to sort ou tthe past15:29
sdakesamyaple i15:29
sdakei'll float the idea of +a by the cr team individually15:29
sdakepbourke are you willing to participate?15:29
sdakerhallisey ^^15:29
sdakejpeeler ^615:29
sdakemandre ^^15:29
sdakeharmw ^^15:30
pbourke?15:30
sdakethe basic idea folks is that if you approve a change that has a backport tag, (meaning you +a it) you become responsible for backporting said change15:30
pbourkeseems ok15:32
jpeeleruh15:32
*** diogogmt has quit IRC15:32
jpeeleri think maybe whoever +a should make sure the submitter has properly submitted the backport or something like that15:32
sdakethat realisticallly just doesn't happen15:33
sdakeright now nobody is backporting15:33
sdakeexcept here and there15:33
sdakewhich is creating a chaotic situation where not all the stuff is being backported15:33
jpeelerit's a new practice, i'm sure if you hammer the point in the meetings people will get it15:34
sdakejust want to make sur ecr team is game to participate15:34
sdakethe idea that th person who submits the patch also backports - gets lost in teh shffule15:34
ashishjainhello.15:34
ashishjainI look forward to setting up my development environment for kolla.15:35
sdakethe worst result it could have is people will be slow to approve backkports becuse they wont want to do the backport work15:35
sdakein which case it will end up on my plate15:35
sdakeor pbourkes15:35
ashishjainI would prefer to use the bare metal deployment http://docs.openstack.org/developer/kolla/quickstart.html15:35
sdakeor SamYaple  since they have agreed to do the work :)15:35
ashishjainI got sufficient RAM & HDD but I got only one network card.15:35
ashishjainHow can I go about having another interface?15:36
ashishjainPlease advice.15:36
sdakeSamYaple can you help ashishjain wit hthe magic incantation to create a virtual nic?15:36
jpeelersdake: either way, this is a large enough change for the mailing list or a meeting at least15:36
sdakeor pbourke  and let me know too, so I'll document it15:36
sdakejpeeler will do mailing list agreed15:36
sdakejust want ot contact devs indivdually15:36
sdakeso you understand the requeted change15:37
sdakewe can have a fuller discussion on the l15:37
sdakewhat it would mean is if yu dont agree to backporting, dont +a backports ;)15:37
jpeelermaybe i'm not understanding. a review goes into master and then if it's +a, that person submits a backport?15:38
sdakeya lets say sam submits a backport15:38
*** diogogmt has joined #kolla15:38
sdakeand I +a his backport15:38
sdakerather sumbit a patch with backport: iberty15:38
sdakeand i +a his patch15:38
sdakeI do the backport15:38
sdakethis unloads sam from doing the backport15:40
sdakeand makes sure the backport actually gets done syncrhounously15:40
sdakeif I dont have immediate time to do the backport I dont +a the change15:40
sdakethen they pile up,adn I can complain all at once15:40
sdakeand people can backport in chunks or whatever15:40
rhalliseykk15:41
jpeeleri guess the biggest thing is that the author is always going to be the expert in backporting. maybe since kolla branches are so similar it won't matter all that much15:41
sdakehow about this, if  the backport is a hugef cf the author is responsible for backporting15:44
sdakeif its a simple cherrypick then the +a is responsible15:45
sdakeand the person that +as is responsible or indicating to the developer and riding herd on him/her to do the backport if a siple cherry-pick doesn't get the job done15:45
jpeelerthat sounds like a good compromise15:47
sdakerhallisey or jpeeler  can one of you ack https://review.openstack.org/#/c/242343/15:48
sdakethis gets rhel rolling15:48
sdakei will backport it15:48
sdakethis time15:48
sdakebecause it has a dependency that needs a backport15:48
sdakerhallisey or jpeeler  do you guys know how to createa a virutal interface for use with kolla15:49
sdakeor pbourke15:49
jpeelerit's documented isn't it?15:49
pbourkeits not unfortunately15:49
pbourkeim not sure15:49
sdakeshould be :(15:49
jpeelerwhere did i read it from then...15:50
jpeeleroh i must have looked at the gate patch15:50
sdakei think it was docmented15:50
sdakebut maybe not15:50
sdakeoh is it in the gate patch?15:50
pbourkeSamYaple creates a dummy interface in the gate but that different I think15:50
sdakethat is wht i want15:50
jpeeleroh i thought that's what you meant15:50
jpeelerhttps://github.com/openstack/kolla/blob/master/tests/deploy_aio.sh#L2815:50
*** cloudnautique has joined #kolla15:51
sdakeashishjain there ya go ^15:51
sdakecan you document that with a change to quickstart.rst?15:51
sdakegood way to learn about the workflow15:51
pbourkecurrently the doc says " In the case of a single interface on a machine, a veth pair may be used where one end of the veth pair is listed here and the other end is in a bridge on the system."15:51
sdakedoc/quckstart.rst15:51
*** cloudnautique has joined #kolla15:51
sdakepbourke ya i dont know how to do that15:52
pbourkethat's different to https://github.com/openstack/kolla/blob/master/tests/deploy_aio.sh#L2815:52
sdakea dummy interface hsould be good enough for dev i think15:52
sdakeash also note the config option https://github.com/openstack/kolla/blob/master/tests/deploy_aio.sh#L2415:53
sdakeashishjain15:53
ashishjainsdake: hi15:53
sdakeashishjain pbourke has a ink to how to creat the dummy interface15:54
ashishjainreading through15:54
*** tzn has joined #kolla15:54
sdakewould recommend changing the doc he pointed out above to the dumy_interface line15:54
sdakeother then that, Id ont nkow how to setup the bridge15:54
pbourkeive set up a bridge with pairs like that before but cant remember off the top of my head15:54
pbourketry the dummy interface, Im not sure what that gives you15:55
ashishjainsdake : sure I will make the doc change15:55
openstackgerritMerged openstack/kolla: Make RHEL build properly  https://review.openstack.org/24234315:55
*** vilobhmm11 has joined #kolla15:57
pbourkeashishjain: sdake: id prefer you dont doc the dummy interface without advice from SamYaple15:58
sdakei thin we get that in a review15:58
pbourkeashishjain: that said feel free to put up a patch and he can comment on that15:58
sdakeright15:58
sdakeashishjain set workflow -115:58
*** tzn has quit IRC15:58
ashishjainpbourke:agreed15:58
sdakei dont know if sam knows how to do the bridge thing either15:59
pbourkeok thanks!15:59
pbourkebet he does15:59
sdakei think we all sort of made that up :)15:59
pbourkei would try it now but my todo list is just too long :/16:00
sdakerogeer16:01
kproskurinGuys, just got a “Build timed out (after 60 minutes). Marking the build as failed.” in gate-kolla-build-images-centos-source16:01
*** sdake_ has joined #kolla16:03
*** cemmason1 has joined #kolla16:04
*** cemmason has quit IRC16:05
*** sdake has quit IRC16:06
sdake_kproskurin got a link?16:07
sdake_i'd try a recheck16:07
sdake_sometimes the builds just dont work16:07
kproskurinhttp://logs.openstack.org/67/242467/2/check/gate-kolla-build-images-centos-source/709c8e2/console.html16:07
sdake_hesenbug ftw16:07
*** gfidente|lunch is now known as gfidente16:07
sdake_this is worrisome16:08
sdake_2015-11-06 15:01:23.277 | INFO:kolla.cmd.build:nova-compute:Failed to get D-Bus connection: No connection to service manager.16:08
sdake_oh that is part of glusterfs16:09
sdake_I think that happens all the etime16:09
sdake_kproskurin not sure just try a recheck16:09
sdake_it looks like things were just too slow to make the 60 minute window16:09
kproskurinyep16:09
kproskurinjust so you know...16:09
sdake_i knwo the gate is flakey :)16:10
sdake_what I'd like to know about is a degradation in teh gate16:11
sdake_right now it should be failing about 1 in 20 times16:11
sdake_if it fails like 1 in 3 times ,that would be something i'd like to know about16:11
sdake_rather it shouldn't be failing at all16:11
sdake_but doe so16:11
sdake_does so16:11
sdake_we got the gate all fixed up last night16:11
sdake_mandre fixed one problem yesterday and i fixed another problem the day before16:12
sdake_so the budl gates should be 1005 pass rate minus wierd 1 of ffailures16:12
*** cemmason1 is now known as cemmason16:14
sdake_folks - please take a moment to take a couple work items from https://blueprints.launchpad.net/kolla/+spec/drop-root16:14
sdake_these are easy patches - mostly meant to test how well distributed work operates with work-items16:15
britthouserI have tried adding user to both glance and keystone.  But both deploys bombed with container logs that looks like:16:16
* britthouser tries to figure out pastebin16:16
sdake_if you on centos run yum install fpaste16:16
sdake_then docker logs xyz | fpaste16:16
sdake_i haven tactually tried that USER thing16:17
sdake_so ymmv :)16:17
britthouserhttp://paste.fedoraproject.org/287688/4468265916:17
sdake_i'll be getting to it in about an hour after this backport is done16:17
britthouserOk...Yeah I'm scratching my head.  Just don't know the code well enough yet.  But I guess this is my opportunity! =)16:17
sdake_ok, during  the docker build you need to chown /etc/keystone to the keystone user16:17
britthouseryeah there is this line (above my USER statement)16:18
britthouserchown -R keystone: /etc/keystone /var/www/cgi-bin/keystone /var/log/apache2 /home/keystone16:18
sdake_so add a chown keystone: /etc/keystone to docker/keystone/Dockerifle.j216:18
sdake_can you paste your dockerfile.j2?16:18
britthouserhttp://paste.fedoraproject.org/287691/2676214416:19
britthouserOh...I guess b/c I said use apache (since that is what is running)16:19
sdake_fpaste pretty handy huh? :)O16:19
britthouserbut /etc/keystone is owned by keystone.16:19
britthouseryes it is!16:19
*** achanda has joined #kolla16:20
sdake_your USer is in the wrong place16:20
britthouserI had same problem with glance thought (I was just trying that b/c I thought it woudl be more straighforward)16:20
sdake_put on line 5416:20
sdake_ahead of run statemnt16:20
britthouseroh16:20
britthouserok16:20
britthouserlemme give that a whirl and I'll report back16:21
sdake_user has to run ahead of the run operatio essentially16:25
sdake_makes ure not to put user ahead of the chown operation thoguh116:25
*** sdake_ is now known as sdake16:28
openstackgerritSteven Dake proposed openstack/kolla: Fix RHEL builds  https://review.openstack.org/24255316:31
openstackgerritSteven Dake proposed openstack/kolla: Make RHEL build properly  https://review.openstack.org/24255416:31
sdakerhallisey jpeeler  can you guys start looking at the backports so I can cut a 1.0.1 release today - there a re about 10 in teh queue16:31
sdakepbourke as weol16:31
sdakethey are all rubber stamps16:32
sdakebritthouser btw I have glance assigned to me, did you want  ot take eownership of it?16:33
*** vilobhmm11 has quit IRC16:35
britthouserNo....I was just trying it in addition to see if I could get it to work easier than keystone.16:35
*** vilobhmm11 has joined #kolla16:36
sdakecool well your free to take it over if you want another easy one :)16:36
sdakebritthouser I think the advice I offered earlier was wrong16:42
britthouserYeah....now they don't build16:42
sdakewhat is happening is the base script is being run in the base container16:42
sdakethis sometimes calls kolla_extend.sh16:42
sdakewe need to somehow encode the user and pass it to the bae case16:43
sdakebase script16:43
sdakehttps://blueprints.launchpad.net/kolla/+spec/drop-root16:44
sdakewrong paste16:44
sdakehttps://github.com/openstack/kolla/blob/master/docker/base/Dockerfile.j2#L17316:44
britthouserso we need USER to be set before that CMD, right?16:44
sdakei am not quite sur ewhat we need16:45
sdakecan you past eyour error gain?16:45
*** achanda has quit IRC16:45
sdakemy scorllback is gone16:45
rhalliseyanyone dealt with an issue when you deploy and somehow you images always try to pull16:45
sdakei ha to restart my client16:45
rhalliseyeven with the pull policy=missing16:45
rhalliseyand the image existing16:45
britthouserhttp://paste.fedoraproject.org/287688/4468265916:45
sdakei have noticed pull policy=missing always trie to pull16:45
rhalliseymine is stuck16:45
rhalliseyit fails to pull16:46
rhalliseyso blows up16:46
rhalliseywhen I have the image locally16:46
pbourkerhallisey: that is a known issue16:46
rhalliseypbourke, is this ansible?16:46
rhallisey1.9.216:46
pbourkeI dont think anyone got to the bottom of it16:46
pbourkeits with the data image right?16:47
rhalliseyyes16:47
rhalliseyit's driving me crazy16:47
pbourkeyeah :/16:47
sdakebritthouser can you hack your base script to ls -l /var/lib/kolla/config_files/keystone.conf16:47
* rhallisey will hack around it16:47
sdakeyou will have to do it inside kolla_set_configs16:47
sdakeI want to see what permissions it has16:47
sdakei think your having errors on read not write16:47
sdakealthough it says permission denied on /etc/keystone16:48
sdakemaybe its actually a perission error on read16:48
sdakenah its definately /etc/keystone16:48
sdakecan you ls -l /etc/keystone as well16:48
pbourkesdake: do you want to wait for the gate before +a on these backports16:48
sdakemight as well16:49
pbourkei'll look over them again in about 30 mins16:49
*** Hui has joined #kolla16:49
*** vilobhmm11 has quit IRC16:49
*** vilobhmm11 has joined #kolla16:49
Huihi, is the kolla master branch installing mitaka openstack binary?16:50
sdakeshould be doing so hui16:50
rhalliseyHui, yes16:50
sdakenot sure about from source16:50
Huithanks, sdake and rhallisey16:50
britthouserok sdake so I'll edit docker/base/set_configs.py and add 'ls' in there, and the rebuild the world, right?16:50
rhalliseyomg what is going on with this pull lol..16:50
sdakejust rebuild keystone16:51
sdakebuild.py flags keystone will do it16:51
Huihow can I tell from the kolla code which rdo will be installed? mitaka or liberty?16:51
sdakehui black magic in the base file16:51
sdakediff stable/liberty from master16:51
rhalliseypbourke, do I need to downgrade ansible or something16:52
rhalliseyor comment out everything in common16:53
rhalliseyand start them by hand16:53
Huisdake, I saw this line in base file RUN yum -y install http://rdoproject.org/repos/openstack-liberty/rdo-release-liberty.rpm16:53
pbourkerhallisey: I dont think so - I tried all sorts of combinations of ansible and docker-py and couldnt solve it16:53
*** jmccarthy has quit IRC16:53
pbourkerhallisey: in the end ended up pushing to a local registry and just using that16:53
Huisdake, is this the magic?16:53
rhallisey:/16:53
sdakethat it16:53
*** vilobhmm11 has quit IRC16:54
*** absubram has quit IRC16:55
sdakehui thats it16:55
Huisdake, thanks.16:55
*** achanda has joined #kolla16:56
sdakeboy regv2 pile of shit - 90 minute push times totally unacceptable17:04
*** unicell1 has quit IRC17:04
*** unicell has joined #kolla17:05
*** kproskurin has quit IRC17:08
diogogmti just noticed after enabling debug logs that almost half of the services don’t have a source url configured in the kola-build.conf file — is that because their sources are not being hosted in the openstack http://tarballs.openstack.org repo or something else?17:09
sdakediogogmt only base imgefs have uris associated17:10
sdakethe other imges have info but not uris17:10
diogogmtsdake: so the other images are built from binaries instead?17:10
sdakeprovide example17:10
*** mfalatic has quit IRC17:11
diogogmtsdake: starting on line 94: http://pastebin.com/1ivKeBpW17:11
diogogmtit says some services don’t have a source url17:11
diogogmtbut since i set the build flag to build from source how does it build if it can’t find the source file?17:12
sdakeexampl enova-scheduler17:12
sdakehas a nova-base17:12
sdakenova-base contains the uri17:12
sdakenova-sscheduler depends on nova-base17:12
sdakenova-base has the uri17:12
sdakethe source file is inherited from the nova-base image in dockerfiles17:13
sdakethe log notices are confusing - just ignore them17:13
sdakeor file a wishlist bug and maybe someday we will clean that up a bit17:13
diogogmtsdake: kk, so if i understand correctly the nova-scheduler image will inherit from the nova-base image, right?17:14
sdakeroger17:14
diogogmtand the nova-scheduler source is being referened by the nova-base image?17:14
sdakenova-scheduler is part of the nova tarballs17:14
sdakeec17:14
diogogmtoh ok, makes sense17:14
sdakehttps://github.com/openstack/kolla/blob/master/etc/kolla/kolla-build.conf#L12217:15
sdakethat file sets the uri17:15
sdakeyuo see there is no nova-scheduler17:15
sdakehttps://github.com/openstack/kolla/blob/master/docker/nova/nova-scheduler/Dockerfile.j2#L117:15
sdakethis line inherits from nova-base17:15
sdakehttps://github.com/openstack/kolla/blob/master/docker/nova/nova-base/Dockerfile.j2#L3717:16
sdakethis line is what installs the source tarball in nova-base17:16
*** ssurana has joined #kolla17:16
*** daneyon has joined #kolla17:18
diogogmtok, got it — as far as the image building orchestration, is there a particular order the images are built, where in the code is that order defined?17:18
dmsimardHui: the master branch installs trunk RDO (Mitaka), the stable liberty branch installs liberty RDO as of https://review.openstack.org/#/c/242229/17:18
Huidmsimard, thanks17:19
Huithe neutron server on the master branch keeps failing today17:19
dmsimardHui: actually I might have sort of lied, hang on17:20
Huiok17:20
dmsimardok I lied17:22
dmsimardsdake: This bit is bad for the time being https://github.com/openstack/kolla/blob/master/docker/base/Dockerfile.j2#L52-L5317:22
sdakewhat do you mean time being17:22
dmsimardWe don't have proper CI in place for Mitaka yet so we don't actually promote packages right now. The current-passed-ci repo was the last tested repo before we cut liberty17:23
dmsimardWhich was 09-2817:23
sdakeso this isn't master?17:23
sdakewe want master in theere, preferrably a ci tested one but if not - master will do17:23
dmsimardWe really only have one master, which is http://trunk.rdoproject.org/centos7/current/ - but, when CI is in place, we promote tested builds to current-passed-ci17:24
dmsimardWe do not encourage using current as it can and will break (untested right now)17:24
sdakewe expect our master to break17:24
sdakewe are good wit hthat17:24
sdakecan you submit a change to fix it to master?17:24
dmsimardBy untested I mean we still go through unit/package/delorean tests but there is no integration tests so to speak so no tempest, installation, etc.17:24
sdakeI've been around and aroudn bout the proper upstream repo to use17:24
dmsimardYeah, and I'll point it back to the CI repo once we have the CI in place - should be two weeks or so17:25
sdakesounds good17:25
dmsimardDo you want a bug ?17:25
dmsimardor a good commit message is enough17:25
sdakeUse "TrivialFix" flag17:25
sdakeand good commit message17:25
Huisdake, dmsimard, so you think using stable/liberty is more stable than master?17:26
sdakehui absolutely yes17:26
dmsimardsdake: hm, how do I set that flag ?17:26
sdakeif your actually deploying please use stable/liberty17:26
sdakedmsimard add to the commit log TrivialFix17:26
sdakeit just goes in the commit log - git log | grep TrivailFix for examples17:26
dmsimardsdake: like, below the Id ?17:26
sdakethat way we aren't expecting a bu gid17:26
sdakeahedof change id17:27
sdakethe reason we have a flag at all is we are not mind readers if the author thinks its a trivial fix or not17:27
sdakeand sometimes it not trivial to the reviewer :)17:27
dmsimardok got it17:27
sdakeso to clarify we have the trivialfix flag which can always be used in palce a bug, but may notbe accepted by the reviewer at their preference17:27
sdakei haven't seen a trivialfix flag been rejected so far17:27
sdakepeople sem to be using it for the right things \o/17:28
sdakepbourke do you know where hte continers are stored in the registery v1 code bsae in the registryv1 container?17:28
sdakeor rhallisey  or jpeeler ?17:28
sdakeit isn't /var/lib/registry17:29
openstackgerritDavid Moreau Simard proposed openstack/kolla: Switch to RDO untested master repository  https://review.openstack.org/24257917:30
sdakereal26m32.687s17:30
*** tzn has joined #kolla17:30
sdakedocker v1 push time - 89 containers17:30
diogogmtsdake: do you know who is the maintainer for the Vagrant file? can i ask you a question about it?17:31
dmsimardHui: with that commit in, it'll really use the up-to-date master repositories ^17:31
dmsimardHui: but like sdake said, you probably really want to use the stable/liberty branch, especially as we're early in the mitaka cycle17:31
Huiwhich commit? sorry I miss some message, dmsimard17:32
dmsimardHui: https://review.openstack.org/24257917:32
sdakediogogmt hit up mandre or pbourke17:32
jpeelerdmsimard: when would CI be added for mitaka packages?17:32
diogogmtok17:32
Huidmsimard, i see. thanks17:33
dmsimardjpeeler: I'm hoping within the next 2 weeks17:33
rhalliseysdake, I don't think rhel is 100% working yet17:33
sdakewell it builds properly ;-)17:33
rhalliseynova-compute image doesn't install the nova package atm17:33
dmsimardjpeeler: we need to finish some liberty CI refactor/migration but once that is done, Mitaka CI is largely a copy/paste17:33
sdakeit worked when i tessted it with rhos a bit some time ago17:33
rhalliseyeh ok17:33
rhalliseyI will re try it17:34
sdakerhallisey eel free to please fix it17:34
rhalliseywhat flags did you use17:34
rhalliseyrhos rhel17:34
sdaketools/build.py -b rhel -t rhos17:34
rhalliseykk17:34
sdakeusing master17:34
sdakemy patch just hit the repo17:34
sdakewhich fixes that17:34
sdakeif you don't have rhos license use -t rdo17:34
sdakewe want rhos/rdo/source to work on rhel17:35
rhalliseysdake, ya it may just be rhos nova-compute17:39
rhalliseywill look into it17:39
sdakewhat iw ould do is run the built container17:39
sdakedocker run -it rhel-rhos-nova-compute bash17:39
sdakeand have al ook around17:39
rhalliseyit never installs nova17:39
rhalliseyya will do a yum in there17:39
rhalliseysee what happens17:39
sdakewhat i mean by have a look around is try yum -y install nova-compute17:40
*** mbound has quit IRC17:40
*** unicell has quit IRC17:42
rhallisey332 deps O.o17:44
sdakeya its a beast17:44
sdakesurprised it works tbh ;-)17:44
sdakemost of those deps are unneeded in a container deployment17:44
sdakebut that is the beuty of containers - who cares ;)17:44
*** mfalatic has joined #kolla17:44
sdakehey mfalatic17:45
rhallisey[Errno -1] Package does not match intended download. Suggestion: run yum --enablerepo=rhel-7-server-htb-rpms17:45
rhallisey clean metadata17:45
rhalliseyya this is where I left off17:45
sdakewtf is htb-rpms?17:45
rhalliseyI have no idea17:45
sdakethis is what i get17:46
sdakeINFO:__main__:openstack-base:https://cdn.redhat.com/content/dist/rhel/server/7/7Server/x86_64/os/repodata/productid: [Errno -1] Metadata file does not match checksum17:46
sdakeINFO:__main__:openstack-base:17:46
sdakeINFO:__main__:openstack-base:Trying other mirror.17:46
sdakeINFO:__main__:openstack-base:17:46
sdakeINFO:__main__:openstack-base:17:46
rhalliseyya there are a few of them17:47
sdakehui master works for me17:49
sdakeSTDAKE-M-J2VL:demo sdake$ nova list | grep ACTIVE | wc -l17:49
sdake      1217:49
sdakeinit-runonce follows by demos/heat/launch17:49
sdakehui just built about 30 mins ago17:50
sdakeand deployed in 3 node ha config17:50
sdakerhallisey i just built rhel + rhos and got following image name - perhaps this is the problem17:52
sdakekollaglue/rhel-binary-nova-compute                                      latest              74329d59e709        2 minutes ago       984 MB17:52
sdakeile or directory17:52
sdake[root@08e54900f685 /]# rpm -qa | grep nova-compute17:52
sdakeopenstack-nova-compute-2015.1.1-3.el7ost.noarch17:52
*** vilobhmm11 has joined #kolla17:54
Huisdake,  (_mysql_exceptions.ProgrammingError) (1146, "Table 'neutron.ml2_vlan_allocations' doesn't exist")17:54
Huisdake, I got this error from neutron server master17:54
Huibranch17:54
sdakeHui did you clean up your local environment and delete your database container data container?17:55
sdakewe dont yet implement database migratiosn for update17:55
HuiI did clean-containers17:55
*** vilobhmm11 has left #kolla17:55
Huisdake,do I need to remove /etc/kolla on the deploy node?17:55
sdakenope shouldn't need to unless yoru using ceph17:55
sdakethen the answer is YES17:56
HuiI mean /etc/kolla/neutron-server, I do not use ceph17:56
*** kproskurin has joined #kolla17:56
*** ashishjain has quit IRC17:59
sdakehui shoouldn't need to do so18:00
sdakethe problem you have is your database is not migrating18:01
sdakei may be experirinecing that exception and not seeing it18:01
sdakehow do yo duplicate?18:01
rhalliseysdake, I think my subscription repos are just messed up18:02
rhalliseysince rdo isn't working ..18:02
sdakemake sure to use virtial repo pool id18:02
sdakenot physical18:02
sdakevirtual are unlimited pool subscriptions physical has imit18:02
sdakeyou don't want a physical in the build18:02
sdakethat should probably go in the build docs18:02
Huisdake, there is no existing database. I just install kolla from scratch today and neutron fails18:03
sdakehui are you doihng dev or production deployo/eval?18:04
openstackgerritMerged openstack/kolla: Fix RHEL builds  https://review.openstack.org/24255318:04
rhalliseysdake, uh how do I do virtual18:04
rhalliseyonly see physical pool18:04
sdakeask the bossses for one18:04
sdakei asked my boss britthouser in this case ;-)18:05
sdakeits just a separate pool id18:05
sdakeneed to hunt one down18:05
Huisdake, I have a dev testbed of 4 nodes to build kolla everyday18:05
sdakehui nice!18:05
openstackgerritMerged openstack/kolla: Make RHEL build properly  https://review.openstack.org/24255418:06
Huiif a build success, I upgrade the images in a larger testbed18:06
jpeelerrhallisey: mine has virtual if you need to use it again18:06
sdakenice canary deploooyment -that would be a huge killer feture18:06
Huisdake, now as you pointed out, I may consider pin to stable/liberty for the stable reason18:06
rhalliseyjpeeler, how do I get to it18:07
rhalliseyI'm using yours18:07
sdakeya ounds more like your doing eval use csae hui18:07
sdakerather then dev18:07
jpeeleri always have to relook it up because i use it like once a year...18:07
Huisdake, yes18:07
sdakehui what is your max scale size your gunning for?18:09
Hui5 compute nodes + 2 network nodes + 1 controller as of now18:10
sdakehui 'd recommend trying out scale teting of ha if that is in your future :)18:11
Huisdake, of course18:11
Huisdake, is there any official doc about kolla HA?18:11
sdakejust the spec but its dated18:11
sdakemost fo the good info is floating about in devs brains18:11
*** achanda has quit IRC18:12
Huiok18:12
sdakewe need an architecture.rst document18:12
*** unicell has joined #kolla18:12
britthouserrhallisey: if you are on baremetal, the virtual pools won't show up18:13
britthouserbut if you are in a VM, you should see both virtual and physical pools.18:13
britthousermake sure the pick the virtual pool.18:13
rhalliseyis it possible to attach to a virtual pool on baremetal18:16
rhalliseynot sure if it was accpeted..18:16
rhalliseypool is restricted to virtual guests18:17
britthouserI think subscription manager is smart enough not to let you do that. =)18:18
rhallisey:'(18:18
britthouserOr else your paycheck would be very very small. =P18:18
rhalliseyya18:19
*** cemmason has quit IRC18:22
sdakebritthouser see this http://paste.fedoraproject.org/287756/4468345118:29
sdakehttp://paste.fedoraproject.org/287757/8345621418:30
britthouserYeah...that is what keystone container looks liek today18:30
sdakenm aparently i didn't build keystone correctly18:30
sdakei had added user in but didn't build it propery18:31
britthouserOk.18:31
britthouserI guess what we're expecting to see is that the inital HTTPd and the logging processes owned by apache?18:31
sdakeya although i'm not sure that is viable18:31
sdakeit may work if we change permissiosn on /dev/log or whatever it s called18:32
britthouser*seems* like we're mostly doing the right thing here alrady.18:32
*** absubram has joined #kolla18:32
britthouserbut I'm not sure what the goal is.  100%  or where its just flagrantly wrong.18:33
*** achanda has joined #kolla18:34
*** achanda has quit IRC18:35
sdakebritthouser here is what I get with user at the end18:41
sdakehttp://paste.fedoraproject.org/287765/1446835218:41
sdakedid you ge tthat ls jammed in there?18:41
severionhello everyone. have a question regarding the kolla heat template for testing kolla. stack is in the creation process right now, but i noticed the images in the "stack view" details are broken. they seem to work with other stack templates i have running (like magnum). is this a kolla thing or do i need to reach out to the heat team about this?18:43
severionhaven't really ever seen this before...so new to me18:43
sdakeseverion just a heads up, the heat template is currently totally and completely broken18:43
severionlol18:44
sdakeseverion someone needs to moidfy it to match ansible vs compose where it is at now18:44
severionwell, man...18:44
sdakei'd recommend bare metal personally18:44
sdakebut alternatively vagrant works18:44
severioncrap...my test environment is all OS at the moment18:44
britthousernot yet sdake....getting pulled into other things this afternoon.18:44
severioni tried vagrant...but ran into issues and couldn't find answers on that one either.18:44
sdakeyou can use os - just treat the vm like bare metal18:44
sdakeroger18:44
sdakeseverion I highly doubt you can use vagrant on top of openstack18:46
severionno18:46
sdakeand if you do it would be painfully slow sine it would be virt in virt18:46
severionthe vagrant option i was building in virtualbox18:46
severionon mac18:46
sdakeroger18:46
severionand yes...it was slow anyway :D18:46
sdakeya that is tessted and works18:46
severionvery18:46
sdakepbourke and mandre are points of contact there18:46
dmsimardsdake: you can with an openstack provider18:46
sdakei'd recommend just spinning up a 16 gig vm and treating it like bare metal18:47
dmsimardsdake: say you use openstack instead of virtualbox18:47
severionso my issues...hmmm...let me look back18:47
severionnever got past this: "unable to connect, check login credentials (login_user, and login_password, which can be defined in ~/.my.cnf),18:47
severionunder creating keystone18:47
sdakeand following the quickstart guide there18:47
sdakewhich version ar eyou using?18:47
sdakei just deployed master 1 hour ago18:47
severiongrabbed github a couple of days ago.18:47
severiondid that fix some issues?18:48
severionmeaning, your resent push to master...18:48
sdakewe have 4-6 patches hitting the repo every day18:48
sdakei'd recommend a fresh pull18:48
severionoye18:48
severionok18:48
severioni'll try again18:48
*** achanda has joined #kolla18:48
severionthanks for the tip. i'll try again...follow instructions exactly...and then report back.18:49
sdakeseverion if your doing an eval, checkout stable/liberty18:50
severionif that doesn't work, i'll do the OS bare metal attempt. will probably run faster that way anyway.18:50
sdakeit is more stable then master ;-)18:50
sdakeclone the repo then git checkout stable/liberty and use that18:50
severionah, good call. is magnum included or does that need to be baked in after turn-up?18:50
sdakewe have not implmeneted ansible playbooks for magnum - it is in the review queue atm as WIP18:51
severionthe inception since containers is wild...container kolla, running caas magnum! haha18:51
sdakei have reviewed it - code looks ready for mergef imo18:51
severionas is magnum in general anyway...understandable entirely18:51
severionlast time i checked (maybe last week) kubernetes started working, mesos had additional requirements and swarm...well, that was just plain broken from what i could tell.18:52
*** absubram has quit IRC18:53
Huisdake, I found why my neutron fails. The bootstrap node fails due to empty /etc/neutron/neutron.conf in the container19:05
*** gfidente has quit IRC19:05
sdakeidea why that file is empty?19:05
Hui    volumes: "{{ node_config_directory }}/glance-api/:{{ container_config_directory }}/:ro",  container_config_directory  /var/lib/kolla/config_files/neutron.conf19:06
Huido you know which line of code copy  /var/lib/kolla/config_files/neutron.conf to /etc/neutron/ in the boostrap container?19:07
sdakeset_configs.py does the job19:07
sdakeit is based upon config.json19:07
sdakeit reads config.json and does its thing19:07
sdakeare you running ansible as sudo?19:07
sdake(you should be imo:)19:07
rhalliseyHui, wasn't this fixed in sam's patch?19:08
sdakeoh right19:08
Huino19:08
sdakehui how old is your repo?19:08
Huiupdated to the latest HEAD19:08
sdakeare you running as sudo?19:09
sdakeI suspect you hae a permissions problem on one of the files sansible is using19:09
HuiI am running as a root19:09
sdakeout of disk space?19:10
sdakeso I am running as root, but I didn't clean up my confi dirs and whatnot19:10
*** mbound has joined #kolla19:10
sdakeso its possible master is broken19:10
dmsimardHey so I have a kind of a stupid question, how do you see Kolla different than OSAD ? From an untrained eye they overlap a lot. Using very similar means to get things running (ansible, containers). Perhaps the main difference I see is that they do not provide support for binary installs.19:13
sdakedms moment19:13
* britthouser gets some popcorn19:13
Huisdake, I saw the neutron-server.json file. It should copy the file in the container19:14
rhalliseydmsimard, they use container like vms19:14
rhalliseyand they don't containerize everything19:14
rhalliseyit's a completely different model19:14
rhalliseywe use the micro service approach19:14
dmsimardSorry for the (somewhat on purpose) trick question :) I just want to see where you guys are coming from and ensure there's no wheel re-invention going on19:15
sdakehang tight19:15
sdakedmsimard https://review.openstack.org/#/c/206789/19:16
rhalliseydmsimard, this is a completely different methodology. It's a fair question that we had to pass to get into the big tent19:16
sdakedmsimard see my comment on patch set 519:17
sdakebritthouser enjoy the salt-free popcorn ;-)19:17
sdakethe tc had a similar question - how is it different19:17
sdakethat comment answers that quesstion19:18
sdakethere are a slew more differences19:18
sdakebut those are the big ones19:18
sdakeand to be fair, kolla started first on the concept ;-)19:18
sdakeso if anyone is wheel reinventing its osad ;)19:18
rhalliseyhell ya19:18
dmsimardsdake: I'm happy with your explanation in the patchset, you should keep that link handy for people who ask that :)19:21
sdakeits more fun to go see all the drama in my inbox re that topic ;)19:22
*** achanda has quit IRC19:24
*** achanda has joined #kolla19:29
*** openstackstatus has joined #kolla19:34
*** ChanServ sets mode: +v openstackstatus19:34
-openstackstatus- NOTICE: Gerrit will be offline at 20:00-20:15 UTC today (starting 20 minutes from now) for scheduled project rename maintenance19:39
openstackgerritMichal Rostecki proposed openstack/kolla: Add Ansible support for Magnum  https://review.openstack.org/23622319:47
dmsimardoh yeah support for magnum in kolla, we've gone full circle, now let's go full triple-o inception19:48
* dmsimard runs away19:49
*** cloudnau_ has joined #kolla19:50
nihiliferno, before that we have to go in deeper inception19:50
nihiliferKolla which runs LXD instances on which Magnum runs containers19:51
nihiliferthat's the way to go19:51
*** cloudnautique has quit IRC19:53
*** tzn has quit IRC19:53
*** tzn has joined #kolla19:59
-openstackstatus- NOTICE: Gerrit is offline until 20:15 UTC today for scheduled project rename maintenance20:00
*** ChanServ changes topic to "Gerrit is offline until 20:15 UTC today for scheduled project rename maintenance"20:00
*** tzn has quit IRC20:04
*** tzn has joined #kolla20:06
*** tzn has quit IRC20:11
*** tzn has joined #kolla20:11
*** tzn has quit IRC20:12
*** tzn has joined #kolla20:12
*** achanda has quit IRC20:15
*** nihilifer has quit IRC20:23
*** nihilifer has joined #kolla20:25
*** ChanServ changes topic to "Kolla IRC meetings on Wednesday 16:30 UTC (Day light savings doesn't apply) - Agenda @ https://wiki.openstack.org/wiki/Meetings/Kolla - IRC channel is *LOGGED* @ http://eavesdrop.openstack.org/irclogs/%23kolla/"20:35
*** dims is now known as dimsum__20:37
openstackgerritMerged openstack/kolla: Switch to RDO untested master repository  https://review.openstack.org/24257920:58
*** openstackgerrit has quit IRC21:01
*** openstackgerrit has joined #kolla21:01
sdakebritthouser ive got glance rolling once i post patche you can bse keystone off hat21:04
*** absubram has joined #kolla21:13
*** kproskurin has quit IRC21:25
*** kproskurin has joined #kolla21:25
britthouserThanks sdake!21:37
britthouserI needed someone to blaze that trail for me. =)21:37
SamYaplesdake: keystone is probably going to be a speical case21:38
SamYaplerecommend all things other than keystone first21:39
SamYapleand acutally its going to be tricky for all of them...21:39
SamYaplelike nova nad neutron need special sudo files21:39
sdakeyup21:39
SamYapletheyll still have some commands they need to run as root21:39
sdakethe infra is down so i can't post my patches21:39
sdakei hve to run to pick u pdaugther bbiaf21:39
SamYapleim going back to sleep21:40
SamYapledmsimard: OSAD isn't a thing! It's OSA now. Maybe they'll have another rename WHO CAN KNOW THESE THINGS??!!21:41
dmsimardSamYaple: it used to be called oasd before big tent21:48
dmsimardI guess it kind of stuck21:48
dmsimard:)21:48
*** jasonsb has joined #kolla21:57
openstackgerritMerged openstack/kolla: Finish implementation of spice  https://review.openstack.org/24232021:57
SamYapledmsimard: i know i supported it for over a year21:58
SamYapledmsimard: many name changes21:58
dmsimard:)21:59
dmsimardLots of good things in there, if there's one thing I was surprised about is how rackspace managed the Swift rings21:59
dmsimardcool and clever22:00
SamYapledmsimard: having to declare every disk on every node in a yaml file is not clever in my book22:01
SamYaplepain in the ass, sure.22:01
SamYapleand it was.22:01
*** blahRus has quit IRC22:02
dmsimardSamYaple: huh ? Last I checked didn't they iterate over disks with ansible, generate a config file based on that (csv?) and then actually parse that file and build the ring files with a python script ?22:02
dmsimardmemory is fuzzy and friday laziness is kicking in22:02
SamYapledmsimard: they do... but the operator had to first declare those disks in a big blob of yaml22:03
sdakevs a label, sounds painful22:04
sdakei hope pbourke  adds labeing for swift22:04
sdakeit would be a sweet feature22:04
SamYapledmsimard: https://github.com/openstack/openstack-ansible/blob/master/doc/source/install-guide/configure-swift-devices.rst22:05
*** jasonsb has quit IRC22:05
SamYaplepretty awful in my opinion (and my experince....)22:05
SamYapleand dmsimard they also require setting up the ring settings in a yaml file22:07
SamYapleall of them22:07
SamYaplehttps://github.com/openstack/openstack-ansible/blob/master/doc/source/install-guide/configure-swift-config.rst22:08
dmsimardoh man from my perspective that was all automated22:08
SamYaplenone of it is22:08
dmsimardI didn't read the fine print :p22:08
SamYaplelike none of it22:08
SamYapleit was always a pain22:08
*** achanda has joined #kolla22:08
SamYapledmsimard: have you seen my ceph implementation for Kolla?22:09
sdakeglance works22:10
sdakeyay22:10
sdakegoing to eat for a bit - bbiaf22:10
*** daneyon has quit IRC22:10
dmsimardSamYaple: I haven't dug into how each project has been implemented (yet) but I am a core for puppet-ceph fwiw so I can have a look eventually :)22:11
SamYapleoh youll like it i think22:11
SamYaplecompletely automated22:11
SamYapleoperator sets a marker in the form of a disk label on the disks they want to bootstrap22:12
SamYaplethats it22:12
SamYapleautomated cache tier setup22:12
SamYapleerasure pool22:12
dmsimardwell to be fair setting a simple cluster automatically isn't so hard, it's about managing the crush map where it gets tricky (e.g, cache pool, erasure coding and their settings)22:12
SamYaple^^22:12
SamYaplesee above22:12
dmsimardoh i'll have to look then :)22:12
sdakeya ew got al that automated with tags22:12
SamYaple:D22:12
sdakerather labels22:12
sdakeit rocks22:12
sdakeit would be better if I could share a disk22:12
* sdake flogs SamYaple a bit get to work!22:13
SamYapleyea its got a few limitations dmsimard, im not saying its perfect22:13
SamYaplebut it does do erasure and automated cache tiers22:13
SamYaplei deploy exclusively with erasure and cahce tiers22:13
dmsimarddoes it spin one container for each osd or what ?22:13
SamYapleyup22:13
sdakethe ceph code in kolla is relaly the gem of koll aimo22:13
sdakethat is something nobody else could possibly hope to hae ;-)22:14
SamYapleits also basically stand alone22:14
sdakeanyway my mortal form grows weak22:14
dmsimardno kidding, i'm still so jetlagged22:15
SamYapleme too i should be sleep22:16
SamYapleanyway, i would love to have some additional ceph verification and code support dmsimard22:16
SamYaplethis has been basically 100% code from me22:17
SamYaple(its from an older project of mine)22:17
dmsimardSamYaple: historically there was enovance's puppet-ceph module https://github.com/redhat-cip/puppet-ceph and we started a new module from scratch just to ensure we wouldn't use exported resources - just that was kind of challenging22:18
dmsimardI'll look at the stuff in kolla when I have a chance :)22:19
SamYapleyea but ceph in docker is.... harder :)22:19
SamYaplei was the first that I know of to get seperate osds to work with ipv4 in docker containers22:20
SamYapletheres some interprocess communication that requires the processes to be in the same pid namespace to not break all the things22:20
dmsimardOooh wait, didn't you work with Sebastien Han for that22:24
dmsimardI remember asking you guys "Why?" on Twitter or something :p22:25
SamYapleyea thats leseb22:27
SamYaplei had a project going in parallel22:27
*** achanda has quit IRC22:27
SamYaplewe do mix company and share ideas, but ive always built it targeting openstack while he builds stuff thats meant to be general ceph22:28
SamYapleto be honest though, i really hate teh ceph-dockerstuff22:28
SamYaple(the other ceph in docker containers project)22:28
SamYapleits pretty bad in my opinion22:28
*** alisonh has quit IRC22:32
rhalliseyI hate it too22:38
*** kproskurin has quit IRC22:38
rhalliseytheir container solution wasn't the best22:38
*** alisonh has joined #kolla22:50
*** achanda has joined #kolla23:08
*** achanda has quit IRC23:08
*** Hui has quit IRC23:24
*** achanda has joined #kolla23:29
kfox1111how far out is 1.1?23:32
rhalliseykfox1111, I think steak id building23:32
rhalliseyif*23:33
rhalliseyis*23:33
rhallisey:/23:33
kfox1111steak?23:33
rhalliseysdake23:33
kfox1111ah.23:33
rhalliseyor stake23:34
rhalliseyI'm just hungry atm23:34
kfox1111:)23:34
kfox1111for ha, why was keepalived choosen over pacemaker?23:37
*** achanda has quit IRC23:37
*** dimsum__ has quit IRC23:52

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!