Wednesday, 2015-09-30

mandrewoohoo00:00
*** openstackgerrit has quit IRC00:01
*** openstackgerrit has joined #kolla00:01
*** openstackstatus has quit IRC00:02
*** openstack has joined #kolla00:05
*** banix has quit IRC00:06
*** devlaps has quit IRC00:09
*** tpot has quit IRC00:17
*** sdake_ has joined #kolla00:38
*** sdake has quit IRC00:43
*** sdake_ has quit IRC00:53
*** jtriley has joined #kolla01:10
*** erkules_ has joined #kolla01:11
*** erkules has quit IRC01:14
*** jtriley has quit IRC01:17
*** unicell has quit IRC01:35
*** Kennnan2 is now known as Kennan01:37
*** marun has quit IRC01:52
*** tpot has joined #kolla01:55
*** sdake has joined #kolla01:58
*** tpot has quit IRC02:00
*** loth has quit IRC02:28
*** tpot has joined #kolla02:41
*** achanda has joined #kolla02:43
*** unicell has joined #kolla02:56
*** dims_ has quit IRC02:57
SamYaplewooo03:13
SamYaplerc1 sdake03:13
SamYapleRC1!03:13
SamYaplesdake: i have pushed the ubuntu rc1 images to docker registry03:16
sdakenice03:33
sdakethanks for that sam03:33
*** sdake has quit IRC03:38
*** gfidente has quit IRC03:39
SamYapleonly too 86 minutes btw03:40
SamYaplei pushed it in parralaler 1003:40
SamYaplewow. parralel03:40
*** sdake has joined #kolla03:41
*** sdake has quit IRC03:46
*** sdake has joined #kolla03:49
*** dims has joined #kolla03:57
*** dims has quit IRC04:03
*** diga has joined #kolla04:04
*** alisonh has quit IRC04:04
*** alisonh has joined #kolla04:14
*** sdake has quit IRC04:28
SamYaplehttp://paste.openstack.org/show/474778/05:00
SamYaplefor those interested, those instructions above are tested ^^ and working with the images i pushed to the docker hub05:00
*** CBR09 has joined #kolla05:12
CBR09Hello all05:13
CBR09which version ubuntu images on docker hub?05:14
CBR09kilo or liberty?05:14
CBR09and rsyslog container on ubuntu doesn't work05:16
CBR09https://bugs.launchpad.net/kolla/+bug/150116705:16
openstackLaunchpad bug 1501167 in kolla "Rsyslog doesn't work on ubuntu" [Undecided,New]05:16
SamYapleCBR09: right now the only images pushed and working to docker hub would be the liberty-rc105:17
SamYapleCBR09: looks like an AUFS issue. not sure about that. it works for overlayfs and btrfs (ive personally tested)05:17
SamYapleCBR09: can you update teh bug with your kernel and docker versions?05:17
CBR09my version docker 1.9 and kernel 3.19, both are ok?05:19
SamYapleI havent tested 1.9 but i think it would be fine yes05:19
SamYaple3.19 should work for ubuntu and AUFS05:20
SamYapleCBR09: I am kicking a AUFS deploy now05:20
SamYapleshould be about ~10 min05:21
CBR09yes, but rsyslog doesn't work :(05:21
CBR09yes, thanks you:)05:22
SamYapleCBR09: ive asked for exact command output in the bug, please update when you can :)05:24
*** CBR09_ has joined #kolla05:29
CBR09_sorry Sam:D05:29
CBR09_I'm using my phone05:29
*** CBR09 has quit IRC05:29
CBR09_wait me for 15minutes05:30
CBR09_:D05:30
CBR09_Ah Sam, could you give me your skype nick?. My IP public workplace banned, not access IRC:(05:33
SamYaplei don't have a skype thing CBR09_05:34
SamYapleCBR09_: I do have an irc proxy node...05:34
*** inc0 has joined #kolla05:34
SamYapleCBR09_: ok full new environment rekick with AUFS and kernel 3.19, no problems with rsyslog05:35
inc0good moringing05:37
inc0morning even05:37
inc0yeah, I was testing everything on aufs + 3.1905:37
inc0so if you have different conf and you experience problems, do tell plz05:37
*** CBR09_ has quit IRC05:42
inc0SamYaple, rsyslog was causing problems?05:48
SamYaplenot on AUFS for me05:48
SamYaplelooking into the error it looks like a wierd kernel bug that people "fix" by changing versions05:48
SamYaplehavent seen a reason why yet05:48
inc0SamYaple, I was meant to ask you, what's biggest deployment you've made using ansible? does ansible scale?05:52
SamYapleansible _can_ scale yes, but you have to configure it as such05:52
SamYaplethe biggest factor for larger deploys would probably be --forks05:53
SamYapleit defaults to 8, but if you have 800 nodes it will take 100 times longer05:53
inc0based on that it's a single ansible node05:54
inc0it would be cool to make this scallable, so you have ansible on node 1 which ssh to 10 different ansible hosts from which each run actual playbooks on N forks05:54
SamYapleinc0: you can configure ansible in pull mode06:01
SamYapleso each host runs pull06:01
SamYapleand you can adjust the forks06:02
inc0so puppet-like stuff06:02
SamYaple100 forks is fine06:02
SamYapleand the 100 times longer bit isnt completely accurate, it was just to give oy uan idea06:02
inc0yeah I get that06:03
SamYaplealso things like fact caching _can_ be helpful, though i tend to only use them if you have many many nodes. at the scale of 10-20 i would never consider it06:03
*** itsuugo has joined #kolla06:04
inc0SamYaple, ansible question: what's best approach to situation in which I have 2 types of nodes in cluser and different interface names in each?06:16
inc0if I use group_vars in inventory, that will work?06:16
inc0in which I override globals.yml06:16
SamYapleinc0: i have been thinks about this alot06:17
SamYaplelong term I think for that we would have to have overrides pernode, but you can set them in the inventory file06:17
inc0well, per node would be hard, but since I have these grouped, inv and group_vars there should work06:17
SamYapleagreed that would be the more propery way to do it06:18
SamYapleesspecially at scale wher eyour inventory arguably _should_ be in a database06:18
SamYaplebut thats ansible stuff outside the scope of kolla06:18
inc0sure06:18
inc0I'm deploying this right now so it's my personal inventory06:19
SamYaplebut yea variables attached to groups would work fine06:19
inc0also, ceph caching stuff we were talking about yesterday06:20
inc0I spoke about it with our ceph team06:20
SamYapleoh boy06:20
inc0and well...if you have ssd cache on top of hardware cache...06:20
inc0you don't really need journal right?06:20
SamYapleim not sure i follow? my setup is as follows:06:20
SamYaple4 spinning disk osds in 'slow-tier'06:21
SamYaple2 ssh in 'fast-tier'06:21
SamYaplethe journal is colocated on the disks06:21
inc0how did you do tiering?06:21
SamYaplethe fast-tier is used to cache the slow-tier06:21
SamYaplehttp://docs.ceph.com/docs/master/rados/operations/cache-tiering/06:21
inc0bache?06:21
SamYapleits pretty straight forward06:21
SamYapleno offical ceph06:21
inc0ah, so in ceph06:21
inc0ok06:22
SamYapleit also means i can do erasure coding for the underlying tier and still support rbd06:22
SamYaplecurrently earaure coding doesnt support rbd, but the cachetier makes the possible!06:22
inc0noob question: whats erasure coding?06:22
SamYaplefancy cloudy raid06:23
SamYapleinstead of 3 copies of the data you can get better reliability for only 1.4 copies06:23
SamYaple(numbers depend on erasure profile)06:23
inc0wow06:24
SamYaplebasically save a whole bunch of space _and_ make the data more resiliant06:24
inc0I'll read about it, sounds like fun06:24
inc0(damn I should become operator)06:24
SamYapleearasure coding was around before RAID but it wasn't used due to its overhead which is nothing nowadays06:24
SamYaplenah the only reason i get to play with tihs stuff is because i do it in my free time06:25
inc0I play too much games :S06:25
*** sdake has joined #kolla06:25
inc0morning sdake06:25
sdakeyo06:26
SamYaplehey sdake06:26
sdakeyo06:26
SamYapletested a full deploy from docker hub with rc106:26
SamYapleits awesome06:26
SamYaplevery quick06:26
inc0brb, food06:26
*** itsuugo has quit IRC06:28
*** sdake has quit IRC06:32
SamYapleha! inc0 https://01.org/intel%C2%AE-storage-acceleration-library-open-source-version/06:38
SamYapleintel only erasure coding profile06:38
*** erkules_ is now known as erkules06:55
*** erkules has quit IRC06:55
*** erkules has joined #kolla06:55
*** tpot has quit IRC07:05
inc0cool, I'll ask around about that07:07
inc0sounds like fun07:07
inc0SamYaple, for ceph I'm to prepare paritions or disks?07:08
inc0it's partitions right?07:09
*** Kennan has quit IRC07:23
*** Haomeng has quit IRC07:23
*** Haomeng has joined #kolla07:23
*** Kennan has joined #kolla07:25
*** sdake has joined #kolla07:25
*** shardy has joined #kolla07:29
SamYapleinc0: it uses the entire disk07:31
SamYaplebut you wipe the partition and apply a special label to the first partition to bootstrap it07:31
*** itsuugo has joined #kolla07:34
inc0aaa ok07:48
*** achanda has quit IRC07:48
*** mat425 has joined #kolla07:49
SamYapleparted /dev/$DISK -s -- mklabel gpt mkpart KOLLA_CEPH_OSD_BOOTSTRAP 1 -107:50
SamYaplewhere $DISK is something like 'sdb'07:50
SamYapleinc0: ^07:50
*** gfidente has joined #kolla07:59
*** itsuugo has quit IRC08:08
*** athomas has joined #kolla08:16
SamYapleinc0: raw text, orugh for ceph08:20
SamYaplehttp://paste.openstack.org/show/474788/08:20
SamYapleoops, this one http://paste.openstack.org/show/474789/08:21
inc0yeah, I'm trying to write ansible to do it;)08:21
inc0on 10 nodes with 22 disks each08:21
SamYapleand you can, but that wont ever be in Kolla due to its destructive nature08:21
SamYaplei expect people will automate this step08:22
SamYaplebut i dont want to be in the habbit of wiping disks08:22
SamYapleinc0: http://paste.fedoraproject.org/273040/14436013/08:23
inc0no no no, it's side script08:23
SamYaple{{ disk }} should be {{ item }}08:23
*** tfukushima has joined #kolla08:28
*** shardy_ has joined #kolla08:49
*** shardy has quit IRC08:51
*** shardy_ has quit IRC08:55
*** shardy has joined #kolla08:55
Kennanhi SamYaple: I tried late kolla code, and find09:02
Kennanironic-pxe:The command '/bin/sh -c echo 'source not yet available for ubuntu'     && /bin/false' returnon-zero code: 109:02
Kennandid you try if ironic-pxe related image can be built from source ?09:02
SamYapleKennan: yes it is not available on ubuntu *yet*09:03
SamYaplei have some code i was about ot push up09:03
SamYaplebut the ansible bits dont work if i remember..09:03
KennanOK. ironic-pxe is newly added ? OK09:04
SamYapleyes past few days09:04
*** mat425 has quit IRC09:17
inc0guys, when do we open mitaka branch?09:24
SamYapleinc0: i believe sdake said he was opening up master after tagging rc109:25
SamYaplebut he may not have yet09:26
SamYapleyes he did09:26
SamYaplewe have stable/liberty now09:26
SamYapleso master is effectively mitaka09:26
inc0cool, I'm thinking about modifying your ceph so you can specify journal disk09:28
inc0but we'll see if that's required09:28
SamYapleyea it does need to expand, but i want to be careful of how _much_ we expand it09:29
SamYaplethis is Openstack, not ceph deploy yo uknow?09:29
SamYaplei mean deploying ceph like this is almost its own project09:29
inc0well I guess09:31
inc0but if we presume to deploy ceph, which we do09:31
inc0we kinda introduced it into our scope09:31
SamYaplewhen this was initially purposed it was purposed as a 'basic ceph deploy' with the ability to use an external ceph cluster09:32
SamYaplein case it wasn't clear, seperate journal device is cool with me, but i want to avoid complication and it adds a good bit of complication to do seperate journals09:32
*** dims has joined #kolla09:32
inc0but to be honest, deploying ceph like that would be quite cool stuff itself09:33
inc0and I know it's a large topic09:33
SamYaplewhich we could break out that into its own project that kolla consumes09:33
SamYaplethen yea we can add all that craziness in09:33
inc0sure, that would work too09:33
SamYaplebut im weary of maintaining that within kolla09:33
inc0wanna be PTL Sam?09:34
SamYapleim not trying to be dicouraging, just keep that in mind please09:34
SamYaplewell it wouldnt be an openstack project09:34
inc0yeah, I know09:34
inc0storage is whole new space of problems09:34
SamYaplesince it doesn't have anything to do with openstack09:34
inc0well, that depends09:34
*** dims has quit IRC09:39
*** dims has joined #kolla09:42
SamYapledepends on what?09:46
SamYaplei mean they are related, but if we break out ceph _completely_ then i dont know...09:47
*** itsuugo has joined #kolla09:56
inc0politics10:04
inc0quick question, delegate host raises authentication error for me10:04
inc0doesn't it take info from inv?10:04
SamYapleyea it wont work for AIO and doesn't respect ssh_user according ot pbourke_10:05
SamYapleidk10:05
SamYaplebasically it initiates an ssh session to the host it delegates to10:05
SamYapleit doesnt respect ansible_connection=local or ansible_ssh_user=blah apparently10:05
inc0it's multinode10:06
SamYapleok. then its probably the ssh_user thing10:06
inc0so I need to make paswordless?10:06
SamYapleprobably10:06
*** mat425 has joined #kolla10:08
*** tfukushima has quit IRC10:11
*** pbourke_ has quit IRC10:34
*** pbourke_ has joined #kolla10:35
*** shardy_ has joined #kolla10:39
*** shardy has quit IRC10:41
*** shardy_ has quit IRC10:45
*** shardy has joined #kolla10:45
mat425kolla installation on vagrant takes incredibly long time... :|10:49
SamYaplesorry mat425 i haven't tried the vagrant stuff yet10:50
SamYaplepbourke_ harmw_ mandre can you comment on speed of vagrant?10:50
pbourke_a deploy of core services takes maybe about 3 mins10:51
SamYaplethat can't be10:51
SamYaplethats way faster than even baremetal10:51
SamYaplei can do 6min in AIO on very fast baremetal10:51
pbourke_maybe im being optimistic10:51
SamYaplewith images prestaged10:51
pbourke_i'll time a run now10:51
mat425just the build-kolla command is running from 1.5h. but even the vagrant up cmd last about 1h10:52
SamYaplemat425: something wrong... gotta be10:52
SamYaplewhats your environment?10:52
mat425vcpu issue? the param is not set in vagrantfile10:53
mat425i leave vmemory as is (4GB)10:53
mat425(single node)10:53
mat425xubuntu 14.04 64bit10:53
SamYapleare you out of memory? and swapping to the disk?10:54
mat425sometimes it used swap10:55
mat425not for the whole process10:56
inc0damn these 2 ifaces are real pain11:01
SamYaplewhich ones?11:02
pbourke_7 mins :)11:03
inc0fact that I don't have homogenous setup11:05
SamYaplehey pbourke_11:09
SamYaplepbourke_: https://github.com/openstack/kolla/blob/master/docker/swift/swift-object-base/config-swift-object.sh11:09
SamYapleis that still used...11:09
pbourke_no11:09
SamYapleok removing11:09
pbourke_there's some crud in swift that needs to go11:09
SamYaplewell like that whole file i dont see where it gets called at all11:10
pbourke_i dont think object-base is needed after we moved to ansible11:10
mat425pbourke_ 7 min to deploy kolla on vagrant?11:10
pbourke_mat425: yes, but no every service, just the ones needed for nova11:11
pbourke_*not11:11
mat425oh jeez...11:11
pbourke_yours in longer?11:11
pbourke_*is11:11
mat425yes still running11:11
pbourke_try boosting memory11:12
pbourke_I would give minimum of 8GB in an AIO11:12
mat425already reach the cap of my laptop :'11:12
mat425i have 8 GB11:12
mat425and 4GB for vagrant11:12
pbourke_yeah its going to struggle I think11:13
mat425and inside the vm still has 2GB free11:13
mat425okok, i'm gonna try on a bigger one11:14
*** itsuugo has quit IRC11:21
openstackgerritSam Yaple proposed openstack/kolla: Common start.sh  https://review.openstack.org/22937111:29
openstackgerritMick Thomspon proposed openstack/kolla: Add missing core library to murano  https://review.openstack.org/22937211:29
SamYaplewowzers a 700 line reduction in codebase size for removal of duplicate start.sh11:33
inc0SamYaple, failed: [10.237.154.17] => (item=(11, {'device': '/dev/sdn', 'fs_uuid': u''})) => {"changed": true, "cmd": ["docker", "wait", "bootstrap_osd_11"], "delta": "0:00:00.034045", "end": "2015-09-30 07:42:36.637229", "failed": true, "failed_when_result": true, "item": [11, {"device": "/dev/sdn", "fs_uuid": ""}], "rc": 0, "start": "2015-09-30 07:42:36.603184", "stdout_lines": ["1"], "warnings": []}11:44
inc0output of "Waiting for bootstrap containers to exit"11:44
inc0can you make any sense out of it?11:44
SamYapleinc0: the bootstrap container should still exist11:46
SamYapledocker logs bootstrap_osd_1111:46
SamYapledocker wait returns the exist code of the server11:46
SamYaplecontainer*11:46
SamYapleso the container bombed out, but why it bombed out you will ahve to find out from the container11:47
inc0Error connecting to cluster: PermissionError it's not too helpful tbh :S11:47
SamYapleyea thats just the output of the ceph client11:47
SamYaplethats not much i can help with other than guess11:47
SamYapledid _all_ containers go away?11:48
inc0all bootstrap ones11:48
SamYapleyou shoudl fix your networking :P11:48
inc0 librados: client.admin authentication error (1) Operation not permitted that was real problem, but still not helpful11:51
SamYapleah so youve probably rekicked a few times11:51
inc0seems like keys are broken, which might be the case cause I changed place of mons11:51
inc0yeah I did11:51
SamYapleand you didnt removed /etc/kolla/ceph-mon11:51
SamYapleso you had old keys11:51
inc0makes sense, let me fix that11:51
*** jruano has joined #kolla11:52
inc0shouldn't ansible detect that it's supposed to change?11:52
SamYapleno you had old cluster keys11:52
SamYaplenot like it didnt copy them11:52
SamYapleyou generated them in an old cluster11:52
*** dwalsh has joined #kolla12:17
*** dwalsh_ has joined #kolla12:17
*** sdake has quit IRC12:22
*** sdake has joined #kolla12:23
openstackgerritSam Yaple proposed openstack/kolla: Common start.sh  https://review.openstack.org/22937112:25
*** mat425 has quit IRC12:27
SamYapleinc0: also watch out, ansible will populate the /etc/fstab on your host box12:28
inc0these are dedicated machines for this purpose so I don't care12:29
inc0right now I'm still battling12:29
inc0aaaand delegate-f***ing-host again12:29
inc0this thing is broken12:30
*** diga has quit IRC12:31
*** jruano has quit IRC12:31
*** rhallisey has joined #kolla12:32
*** jtriley has joined #kolla12:33
sdakesamyaple ping q re your patch12:34
sdakedoes the CMD in base always run if no CMD is specifiedin a child container?12:34
*** banix has joined #kolla12:37
*** jtriley has quit IRC12:37
*** Kennan has quit IRC12:38
*** Kennan has joined #kolla12:39
SamYaplesdake: correct, but im testing12:40
SamYapleoops +w was added12:40
SamYaplesdake: ill make sure it doesn't merge before you verify centos12:41
SamYapleinc0: yea so basically just make sure you have ssh keys for root pushed around12:42
SamYapleinc0: thats what our docs say. liberty requires root12:42
SamYaplethings gonna break without it12:42
inc0LM will12:42
inc0I know that12:42
inc0anyway, I'm already past that12:43
SamYaplestill borked?12:43
*** achanda has joined #kolla12:44
inc0yeah, now ceph_mon doesn't want to start, also we need to fix error handling there https://github.com/openstack/kolla/blob/master/ansible/roles/ceph/tasks/distribute_keyrings.yml#L512:44
inc0no json object could be decoded is not very helpful12:44
inc0but I'm past that as well, now debugging ceph_mon not starting12:45
*** itsuugo has joined #kolla12:46
*** jtriley has joined #kolla12:47
openstackgerritMartin André proposed openstack/kolla: Fix wrong option name in kolla-build.conf  https://review.openstack.org/22940612:53
inc0http://paste.openstack.org/show/474830/ SamYaple halp!12:55
*** achanda has quit IRC13:00
inc0ceph_mon doesn't start13:07
inc0ceph_mon_data is ok13:08
*** jtriley has quit IRC13:12
inc0I've added this clobber opt and rebuilt container, might be worth checking out what have happened13:13
*** dwalsh_ has quit IRC13:18
*** dwalsh has quit IRC13:18
SamYapleinc0: stahp13:18
SamYapleinc0: what are yo udoing13:18
SamYaplecleanup your environment13:19
SamYapleyou did not remove previous keys13:19
*** dwalsh has joined #kolla13:19
*** banix has quit IRC13:23
SamYaplemandre: well also need to cherry-pick that to the liberty branch13:23
SamYaplesdake: we need a proceedure around tracking backport things, how do other projects do this?13:24
SamYapleive seen a few ways13:24
openstackgerritMerged openstack/kolla: Fix wrong option name in kolla-build.conf  https://review.openstack.org/22940613:26
*** itsuugo has quit IRC13:28
mandreSamYaple, you're right, thanks for the reminder13:30
*** banix has joined #kolla13:30
inc0SamYaple, where are these keys?13:31
inc0 /etc/kolla/ceph-mon on ansible host?13:31
SamYapleinc0: /etc/kolla/ceph-mon on all the monitor nodes13:34
inc0ah13:34
SamYapleansible ceph-mon -m file -a 'state=absent' -i /path/to/inventory13:34
SamYaplethat should work....13:34
inc0I have just one mon node right now13:35
*** dwalsh_ has joined #kolla13:35
SamYapleok. just fyi without 3 nodes bad things all around13:36
SamYaplei mean ceph can do aio but you have to hack the crush map to do it right and its a mess13:36
inc0without 3 mons or 3 nodes?13:37
inc0I mean, I have 1 mon and shitload of osd13:37
inc0but ok, I'll add 2 more mons13:37
SamYapleinc0: yea 3 mons and many osds on several different hosts13:39
SamYaple(50 on one host will still violate the crush map placements stuffs)13:39
*** jtriley has joined #kolla13:40
*** dims has quit IRC13:49
*** dims has joined #kolla13:49
*** banix has quit IRC13:56
*** itsuugo has joined #kolla14:08
*** shardy_ has joined #kolla14:13
*** jtriley has quit IRC14:14
*** shardy has quit IRC14:14
*** shardy_ has quit IRC14:18
*** shardy has joined #kolla14:19
*** athomas has quit IRC14:22
*** jruano has joined #kolla14:27
*** athomas has joined #kolla14:27
pbourke_SamYaple: do you have a neutron setup atm?14:33
*** jtriley has joined #kolla14:35
sdakepbourke_ is there a neutron problem?14:36
pbourke_sdake: maybe14:36
sdaketell me more pls14:37
pbourke_can anyone do an 'ip netns exec' on the network host14:37
sdakei donthave a deployment activec  and have a presentation in 15 mins14:37
sdakebut after I can look14:37
pbourke_e.g. ip netns exec qdhcp-blah ip a14:37
sdakewhat is neutron doing or not doing propery?14:37
pbourke_doing so gives me: seting the network namespace "qdhcp-46f7fce8-65a5-4123-859c-26682b811b3e" failed: Invalid argument14:38
pbourke_now, this may not be a problem - it works from within the neutron containers themselves14:38
pbourke_but if you restart the container it then gives the same error as above14:38
pbourke_again, maybe benign, but it doesn't seem right14:39
pbourke_Im having trouble getting an instance to get an ip via dhcp, which lead me down this path14:39
sdakeip address assignment has been wroking pretty solidly for me14:42
sdakei use centos binary tho14:42
sdakefor the most part14:42
sdakeoh if we only had functionaltesting :)14:42
*** shardy_ has joined #kolla14:42
pbourke_yeah14:43
sdakepbourke_ do you notice a regression lately in neutron or has it always been this way?14:43
pbourke_it probably does work for most people, otherwise there's be more shouting14:43
pbourke_the network namespace error worries me slightly as I remember seeing it way back when we were beginning to containerise neutron14:43
pbourke_and assumed it was sorted14:43
*** inc0 has quit IRC14:44
*** shardy has quit IRC14:44
SamYaplepbourke_: yes14:45
SamYaplepbourke_: no14:45
SamYaplepbourke_: you cannot do that from the host14:45
pbourke_SamYaple: ok so that's expected14:46
pbourke_SamYaple: what about restarting the container14:46
SamYapleyes, but we should add a docker volume to prevent those netns from showing up on the host like that14:46
SamYaplepbourke_: yea you can restart teh container14:46
sdakeSamYaple I suggested that about a month ago but you said it wouldnt work so I never added it14:46
SamYaplepbourke_: fyi i justa added a comment about that https://review.openstack.org/#/c/229371/2/docker/neutron/neutron-base/ip_wrapper.py14:47
SamYaplesdake: i don't recall that... maybe i did try this before and it broke?14:47
SamYapleor maybe i miscommunicated?14:47
sdakenot sure14:47
sdakei was pretty clear about addinga  data container for neutron and you said itwouldl not function properly14:47
sdakefor the netns files14:48
*** shardy_ has quit IRC14:48
SamYapleohhhh i may have confused what yo uwere asking14:48
SamYaplei thought you were saying make it accessible from host14:48
SamYapleand no, that wouldnt make it accessible form host14:48
SamYaplejust hide it from host14:48
pbourke_can I get a summary of why they're not accesible from the host?14:48
sdakenah just to make one to hide it from the host14:48
SamYapleyes sorry. my fault14:48
pbourke_or why they can't be14:48
SamYaplepbourke_: lol yea....14:49
SamYapleits dockers fault14:49
*** shardy has joined #kolla14:49
SamYaplethat ip_wrapper.py i linked you too works!14:49
SamYapleon docker 1.7.0-dev2.....14:49
SamYapleit was removed as a regressing in -dev314:49
SamYaplesdake was talking with someone to get it added back in.... sdake?14:49
*** banix has joined #kolla14:49
sdakeneed a bug link to communicate what needs to be done14:49
SamYapleits not a bug14:50
sdakeneed some type of reference14:50
sdakerather then dev3 added a regession14:50
sdakei thought i had it in my inbox but i can't find it14:50
SamYaplelet me get the email i sent to you last time14:50
SamYaplethis conversation seems familiar14:50
sdakeright we have had it several times :)14:50
sdakeand each time i think we end up at not being able to find the email with the references14:51
sdakeyou did sned it once i remember erading it14:51
SamYapleactually i think i dont look14:51
sdakebut cant find it14:51
sdakeoutlook kind of sucks for searching14:51
sdakeit had somethign to do with docker machine14:52
pbourke_id like to understand it better14:52
SamYaplefloker was bbroken14:52
SamYaplei think14:52
SamYaplepbourke_: i will explain in a minute14:52
sdakeit could be that14:52
SamYaplegoing to get this information out14:52
pbourke_ok thanks14:52
sdakeSamYaple email to steven.dake@gmail.com14:52
sdakeso i can search for it14:52
sdakeput something like "docker regression" in the subject or something14:53
pbourke_if anyone gets a few mins, could you please try this scenario: http://paste.openstack.org/show/474711/ (ignore the nova flavor its way too small)14:53
sdakeit will take a lot of conversation to get it fixed up14:53
sdakeif at all14:53
SamYaplesdake: https://github.com/openstack/kolla/commit/62e02fdbf4a4f714d008943c4e5c8512a93807ed14:57
SamYaplehttps://github.com/docker/docker/commit/ff770d33cdee5ab4dc0bdeb06dcfea49aa72ab8e14:57
sdakesamyaple can you amil that to me14:57
sdakeso i can find it later14:57
SamYapleyes which email14:57
sdakei have meetings all morning14:57
sdakesteven.dake@gmail.com plz14:57
SamYaplehey guys, milestone14:59
sdakecan someone look at this bug https://bugs.launchpad.net/kolla/+bug/149869414:59
openstackLaunchpad bug 1498694 in kolla "neutro-server broken in kolla" [Critical,Triaged]14:59
SamYapleoutside confirmation ceph works14:59
SamYaplesdake: i think thats delorean only15:00
SamYapleok pbourke_ you around for a story?15:02
pbourke_yup15:02
SamYaplestory time then15:02
SamYaplewhen you create a network namespace $(ip netns add namepsace_name) it creates a mountpoint at /run/netns/namespace_name15:03
SamYapleit uses the for proc for that namespace15:03
SamYaplenow when you create a container it creates a _new_ mountnamespace15:03
SamYapleit will inherit all host mounts if you bind it in15:03
SamYapleso if you bind in /run/netns with an already create mount namespace both host and container work15:04
SamYaplebut if you create another one on the host, it wont propogate into the container15:04
SamYapleif you create one in the container it wont ever propogate up15:04
SamYaple(mounts are inherited)15:04
SamYaplewell if you create your mount namespace when you create teh container in a certain way the host /run/netns _can_ propogate into the container on the fly15:05
SamYapleso new netns on the host can be accessed in the container15:05
SamYapledocker doesnt create it this special way, and they dont allow that to be configurable15:05
pbourke_why dont they propagate15:05
SamYaplepbourke_: http://www.ibm.com/developerworks/library/l-mount-namespaces/index.html15:06
SamYaplegreat article on practicle usage of private mount namepsace15:06
SamYapleshared mount is what we want, private mount is what we have15:06
pbourke_which is why we have fat container for agents right15:07
SamYapleyes15:07
SamYaple(ps i build custom binaries to make this work for me ;)15:07
pbourke_why are we binding in /run then15:08
*** sdake has quit IRC15:08
SamYapleshared socket between containers i think15:08
SamYapledont really recall15:09
pbourke_right15:09
SamYapleeither way we can put /run/netns in a volume for the agents container and make them invilible to the host15:09
pbourke_ok think that makes sense thanks!15:09
pbourke_will read the article also15:09
SamYapleyes it should make it more clear15:09
pbourke_have you had any problems with instances not being able to get an ip via dhcp15:10
*** sdake has joined #kolla15:10
SamYaplenope15:10
SamYapleive been kicking it all day with the rc1 images i pushed to docker hub15:10
SamYaplei did not push centos images15:10
SamYaple(i cant test them on ubuntu so i didnt push)15:10
*** Hui has joined #kolla15:11
SamYaples/win 715:14
*** dims_ has joined #kolla15:16
openstackgerritSam Yaple proposed openstack/kolla: Common start.sh  https://review.openstack.org/22937115:17
*** itsuugo has quit IRC15:17
*** JohnN has joined #kolla15:19
*** dims has quit IRC15:20
sdakeSamYaple can you push the centos images anyway15:21
sdakeand I'll test them15:21
sdakeif they are bust, it wont be much harm15:21
sdakeas we will push final images for release15:22
SamYaplesdake: sure but they arent build atm15:22
SamYaplei purges15:22
HuiHi, Sdake, I have created a blueprint about consul. If you have time, could you help to review?15:22
SamYaplewill build15:22
sdakehui I did review the patch15:23
HuiThanks.15:23
sdakehui I didnt' provide a review inline but looked it over15:23
SamYaplesdake: i dont see a review15:23
sdakeI have trouble understanding the value of consul for kolla compared to its relative complexity level which is high15:23
sdakelike I said I didn't provide a review but looked itover15:23
SamYapleoh i see15:23
SamYaplesorry15:23
sdakeno sweat15:24
sdakei think our messages wer emid-air :)15:24
SamYapleso happy right now everyone15:24
SamYapleCBR09 just reported a successful deployment with ceph15:24
sdakehui I read the spec about 5-10 times and I just don't see a compelling value to it15:24
sdakesamyaple what is cbr0915:25
SamYaplea person in hte channel (someitmes)15:25
sdakeoh i see15:25
HuiSdake, the basic idea is for dynamic service discovery15:25
SamYaplepbourke_: https://bugs.launchpad.net/kolla/+bug/1500858 i responded here. lets continue the discussion in the upgrade blueprint i think?15:26
openstackLaunchpad bug 1500858 in kolla "Config changes do not cause a container restart when using COPY_ALWAYS" [Undecided,New]15:26
sdakesamyaple https://bugs.launchpad.net/kolla/+bug/150116715:27
openstackLaunchpad bug 1501167 in kolla "Rsyslog doesn't work on ubuntu" [Undecided,New]15:27
SamYapleclosing now15:27
SamYapleit was apparmor15:27
SamYapleupdating it literally right now15:27
sdakehui yes I just don't understand why kolla needs dynamic service discovery compared to the high degree of complexity15:27
pbourke_SamYaple: sure, thanks for that15:27
sdakeessentially it adds something that doesn't offer a lot of value with a high degree of complexity15:27
sdakethe idea of kolla is to keep things simple so the development effort can scale well15:28
sdakehui for example, the HA system in kolla is highly complex in my opinion, but offers huge value15:28
sdakeso I accept it :)15:28
Huisdake, do you mean the HA blueprint under spec?15:29
sdakethe gain is worth the pain15:29
sdakeno i mean the ha implementation in general15:29
sdakehui we dont use specs in kolla typically15:29
sdakeonly for contentious items15:29
sdakethat way we can have a common viewpoint15:29
sdakewe typically rollcall vote specs15:29
Huisdake, where can I find the HA doc in kolla?15:29
sdakethe only one is in the specs directory15:30
sdakehui I realize you did alot of work with the consul dev, and maybe down the road it would be appropriate, but atm it seems like overkill for a simple problem15:30
sdakeand I hope you aren't put off by lack of reception around the idea, but a whoel bunch of code was written without submitting a blueprint of communicating the idea was to be done15:31
sdakeso the development could be guided15:31
Huisdake, ok15:31
sdakehui clearly you are very sharp and understand kolla, so I like that :)15:32
SamYaplejust to clarify something here. I didn't see what it added that we dont already do in one way or another. We already hot reload haproxy, while its not dynamic discovery it is still dynamicly updating the environment when running the playbook15:32
sdakeits just that the complexity of consul I dont get15:32
SamYapleif you can make it add more value or explain how it already add more value i think youll find it more well recieved15:32
sdakeagree with SamYaple15:32
sdakehui besides the complexity vs gain issue, the workflow is not ideal15:33
Huisdake, could you give some guide on the workflow?15:34
sdakeI want workflows that are essentially single commands that gte a huge bunch of output15:34
sdakewhat I mean is in the spec you wrote, to get an operational deployment several different commands must be run15:34
sdakeatm, we hae one command that must be run to get a deployment15:34
Huisdake, I use ansible tags to explain what you will expect at each step15:35
sdakeyes, but i dont want the operator to have to understand how ansible tags work15:35
Huibut essetionally, you only need one ansible command to deploy15:35
sdakei want thme to run a command "deploy" and out the other end comes a deployment15:36
sdakei see, that was possbily a misunderstanding on my part15:36
HuiI add a variable enable_consul, so that you can choose whether or not use this feature15:36
sdakethe fundametal issue though is the complexity vs the gain15:36
Huiok15:36
sdakeperhaps I don't understand the gain well enough15:36
sdakeand as sam sai,d if that were better communicated itwould help15:37
sdakeperhaps a mailinglist thread would help in this case15:37
SamYapleyea Hui i do see some advatanges here. please do pursue this so we can make an informed decision here15:38
Huisdake, shall we use the launchpad as the mailinglist?15:38
sdakelaunchpad works too, we can discuss in the blueprint15:38
sdaketypically what happens is I put blueprints into the discussion phase15:38
sdakeand then discussion hapepns in launchpad15:39
Huiok, sounds great15:39
sdakewe have been super busy with liberty-3 and rc1, so wehavne't been on top of all the new blueprints or folowing that process15:39
sdakebasically we have been locked down for about 6 weeks preparing for the release15:39
Hui: )15:40
sdakehui it almost looks liek you had that code written based around the kolla single node deployment using compose15:41
sdakeis that accurate?15:41
Huisdake, no. The current implementation requres multi-node deployment15:41
Huiusing ansible15:41
sdakeok thanks15:42
sdakei saw that, it just looked like it was baed upon older kolla work15:42
Huithe code is based on the master branch15:42
sdakeyes I saw that15:42
Huilike Sam removed the config-external.sh15:42
sdakewhat I mean is it looks like it was started some time ago and then rebased15:42
sdakeand ryan as well :)15:42
sdakehey folks if you triage a bug go ahead and set the priority and assign it to thenext rc15:43
openstackgerritSam Yaple proposed openstack/kolla: Common start.sh  https://review.openstack.org/22937115:43
sdakepbourke_ SamYaple  you guys seem to help with the traiging so keep that in mind pls :)15:43
sdakeand set to "triaged"15:43
sdakeif your not sure what the priority shouldbe just make a best gues15:44
sdakei always error on the side of making the bug more critical even if its not confirmed15:44
SamYapleyea ive been cleaning up15:45
sdakeonce a bug enters the confirmed state the priority shoudl have a second pass15:45
SamYaplebut i want to get any major changes done first15:45
sdakeSamYaple sorry don't understand what you mean?15:45
SamYaplesorry wrong window15:46
sdakethat would explain it :)15:46
pbourke_jpeeler: but submitting reviews for master is the usual case right?15:51
jpeelerpbourke_: the policy is you must check master and submit first there if applicable i believe15:52
jpeelerhonestly i forgot how the stable policy works as far as marking for backports, etc15:52
pbourke_I think I dont understand the current branching policy15:52
pbourke_is there a wiki page or something15:52
jpeelershould talk about it in the meeting i guess15:52
pbourke_ah yeah that will do15:53
sdakewhat is the issue pbourke15:53
pbourke_sdake: don't understand why reviews are being proposed for stable/liberty instead of master15:53
sdakethey should be proposed for master first15:53
jpeelerthat review doesn't apply to master15:54
sdakethen once in master, git cherrypick to stable/liberty15:54
pbourke_https://review.openstack.org/#/c/229429/15:54
sdakei think thre ws a master version of that15:54
jpeeleryeah that one should have been cherry picked with -x15:55
jpeelersdake: there was15:55
sdakeright15:55
jpeelerhttps://wiki.openstack.org/wiki/StableBranch15:55
pbourke_ok cool15:55
pbourke_but I dont think 'git review' should target stable/liberty by default15:56
jpeelerthat page even goes so far as to say that cherry pick with -x is needed to avoid a gerrit bug15:56
jpeelerpbourke_: it should in the liberty branch15:56
sdakeas we are not a clinet/server project we will have slightly different policies towards backports15:56
sdakewhich we can define on our own15:56
jpeelerstill, it's a good practice15:56
sdakethat backport policy is targeted around 90% of the projects, which are client/server project15:56
pbourke_jpeeler: aaaaaah15:58
pbourke_jpeeler: sorry I get it now15:58
sdakepbourke ya stable/liberty should have the default gerrit set as it is in the depends-on patch15:58
jpeelerok cool15:58
pbourke_thbanks15:58
*** inc0 has joined #kolla15:59
inc0wassup my dear kolla'gues?15:59
sdakelol15:59
sdakekolla-gues15:59
*** bmace has quit IRC16:00
*** blahRus has joined #kolla16:00
inc0SamYaple, I16:00
inc0I've given up on ceph16:00
inc0I'll try again in few days maybe16:01
inc0but that was because of mess I made16:02
*** dims_ has quit IRC16:06
openstackgerritHui Kang proposed openstack/kolla: Add consul containers for dynamic configuration and discovery.  https://review.openstack.org/22871916:06
SamYapleinc0: i understand16:10
SamYaplei have some cleanup aliases if you are interested16:10
inc0not right now, I'll be off for 4 days - famili matters16:10
inc0but hold on to them please16:10
inc0this ceph is now my idee fixe, I will bring it to the end...his or mine16:11
*** bmace has joined #kolla16:13
*** jtriley has quit IRC16:14
SamYaplelol16:14
inc0family**16:14
sdaketeam meeting in 3 mins16:27
*** banix has quit IRC16:29
sdakeirc meeting in #openstack-meeting-416:31
sdakepbourke_ meeting time if your available16:32
pbourke_sdake: thanks for the reminder!16:32
*** dims_ has joined #kolla16:33
*** unicell has quit IRC16:39
*** banix has joined #kolla16:44
*** devlaps has joined #kolla16:50
*** achanda has joined #kolla16:58
*** achanda has quit IRC17:01
*** dwalsh_ has quit IRC17:01
*** jtriley has joined #kolla17:03
*** harmw_ is now known as harmw17:09
harmwSamYaple: already did :)17:09
*** jtriley has quit IRC17:09
inc0thanks harmw :)17:09
harmwwelcome inc0, I guess :>17:09
SamYapleharmw: awesome!17:09
SamYapleexciting times17:09
inc0so now I got +1 from all of cores;) cool! thank you guys! I really appreciate that17:09
SamYapleharmw: can you help me bring designate to ansible?17:09
SamYapleor can you do it :P17:10
harmwand excuse me for not being as active as I was some months ago17:10
harmwansibalising designate is about the one biggest thing on my list :p17:10
SamYaplelol ok17:10
SamYapleill trust you17:10
SamYapleso sdake to be clear on something17:10
SamYaplewe are not backport something like designate?17:11
SamYaplethat seems fairly nonintrusive to backport17:11
harmwdesignate is quote selfsupporting, indeed17:11
harmwno overly wicked dependencies or something17:11
sdakeSamYaple I am not certain17:12
SamYaplei just mean if it doesnt change the container then the ansible code should be pretty backport flexible i think17:12
sdakemy gut tells me if we don't we will end up lagging for liberty17:12
*** unicell has joined #kolla17:12
sdakemyg ut also tells me it is dangerous to backport a bunch of code17:12
SamYaplebut the way our roles work its super isolated17:12
inc0well, this is pretty portable17:12
sdakeright17:12
SamYapleso enable_designate: false would be the default obviously17:13
sdakelike I said, we will end with different policies thn the typical client/server project17:13
SamYaplebut if it doesnt affect other hosts....17:13
inc0so if someone have liberty stabe and wants designate, this should be pretty easy to do anyway17:13
sdakethe one probelm is the /etc directory17:13
SamYaplejust dont -2 it inc017:13
SamYaplesdake: agree. we need a good script for that17:13
SamYaplebasically i want a script to run and generate/update /etc/kolla17:14
SamYapleso touch new files if needed, change globals.yml if they change17:14
SamYaplethat kinda thing17:14
SamYaplepreserve settings of course17:14
sdakeya its a shame we weren't able to write that already17:14
inc0I won't just let's not rush17:14
SamYaplelol i was kidding inc0 dont let it get to your head17:14
inc0and I'd rather keep enable_designate: no as default17:14
inc0SamYaple, you did throw -2s freely:P17:15
SamYaplelies! look at my stats17:15
SamYaple2.5% of my reviews were -217:16
sdakethe /etc directory is almost like a database migration17:16
inc0you have more -2s than few ptls I know17:16
SamYapleand 4 of those were for my patches!17:16
sdakebecause we use /etc almost as a database17:16
inc0during all of their coreship (is this a word?)17:16
SamYaplesdake: yea. working on that17:16
sdakeand its obvious database migration is not ideal during a backport17:16
SamYapleyou need to look at stackalytics more. ireally dont17:16
SamYaplesdake: well only if you enable designate17:17
SamYapleif you dont no migration needed17:17
SamYapleif we are calling it that17:17
sdakelets do ourselves a favor and not call it migration17:17
SamYapleyes17:17
SamYapleplease17:17
sdakelets make up a new word if need be :)17:17
SamYaplekollafication17:17
inc0etcigration17:17
sdakeetcgration17:17
sdakewinning17:18
inc0SamYaple, only thing I'm concerned about is that this is our first release and you don't get second "first impression"17:19
inc0so please let's not rush things17:19
SamYaplenon-default things don't impact first impressions17:19
openstackgerritJeff Peeler proposed openstack/kolla: WIP - tidy up syntax in all.yml  https://review.openstack.org/22953517:20
SamYaplebut im not looking for change17:20
SamYaplewe just have a way of being very consistent accross all branches17:20
SamYapleand we should use that advantage17:20
inc0well, anyway, I'm seeing double, good sigh I should probably go off17:20
sdakethis really only matters for liberty/mitaka, after that I think we will be right on top of the big tent in terms of delivery of ansible + containers17:21
inc0have a good weekend guys, I'll be off for few days17:21
sdakeso it may not be worth the risk for liberty/mitaka17:21
sdaketake care inc017:21
SamYaplewell lets not talk about things before they happen17:21
SamYaplenight inc017:21
*** inc0 has quit IRC17:21
SamYaplebut i am fairly confident there is no risk involved considering you don't even need to update your inventory file17:21
SamYapleever comment out the swift or ceph hosts in an inventory file? try it its actually much cleaner17:22
SamYapleso a backport will be very clean. but well discuss later for sure!17:22
jpeelerSamYaple: thoughts? https://review.openstack.org/#/c/229535/1/ansible/group_vars/all.yml,cm17:23
*** jmccarthy has quit IRC17:23
SamYaplejpeeler: youre making dicts, its fine with me17:23
jpeelerif it's not clearer though, it's not worth doing17:24
SamYaplei see the advantage for organization17:24
SamYapleim not complaining :)17:24
jpeelerok i'll go ahead and get it fixed up then17:25
SamYaplethat is a breaking change though just fyi17:25
SamYaplethat'll be harder to backport17:25
SamYaplethe globals.yml will ahve to address things differently and then we expose having ot use dicts in globals.yml17:26
jpeeleroh, well maybe now's not the right time then17:26
SamYaplenot saying no, but it is pretty late in the game to change this17:26
SamYapleother cores weigh in?17:26
SamYaplei guess the real question comes down to this:17:28
SamYapleso we want a 'flat' globals.yml without knowlege of yaml and dicts17:28
SamYapleor a structured one17:28
jpeeleri don't really want to go doing search/replace if we decide it's not worth the structure. but if you think it's worth doing, i'll go ahead and do it17:31
SamYaplejpeeler: im neutral. i like the change, but my question above stands.17:32
SamYaplecheck with other cores17:32
jpeelersdake: do you have any opinion?17:32
jpeelerthis was just something i came across while learning about ansible (which clearly is still not a lot)17:33
sdakejpeelr sorry was doing something else let me read scrollback17:34
SamYapleyea this is just dicts implemented in yaml17:34
jpeelerso it's not even an ansible thing, just yaml17:34
SamYaplejpeeler: did you know you can address dicts in ansible as {{ dict_name.key }} or {{ dict_name['key'] }}17:35
SamYaplejpeeler: correct17:35
*** shakamunyi has quit IRC17:35
sdakeok read scrollback17:35
sdakelet me read patch17:35
jpeelerSamYaple: amazingly i did, but keep the tips of the day coming heh17:35
SamYaple:)17:35
sdakei like it17:35
SamYaplesdake: it will require a non-flat globals.yml17:36
SamYapleim neutral with that change. just fyi17:36
sdakeSamYaple caan you expand on what that means17:36
SamYapleright now you do:17:36
SamYapleapi_interface: 'eth0'17:36
SamYaplethis would require -17:36
*** banix has quit IRC17:36
SamYaplenetowrk:17:36
SamYaple  api_interface: 'eth0'17:36
SamYaplein globals.yml17:37
sdakei understand17:37
sdakeI like that, it groups things together17:37
SamYaplefair enough17:37
SamYaplei like the structure too17:37
SamYaplebetter now than later as well in my opinion17:37
sdakethis is a PNR17:37
SamYapleso i can go either way, but if we do it, we need to do it now17:37
*** achanda has joined #kolla17:37
SamYaplePNR?17:37
sdakepoint of no return17:37
SamYapleyea17:38
sdakelets make sure everyone is on board jpeeler17:38
sdakeplease send an email to the mailing list17:38
sdakeand explain the change17:38
sdakeand see if anyone has any objections17:38
sdakebeyond the obvious "why did you wait until the lsat possible minute" :)17:38
jpeeleri wish i had the change up during the meeting17:38
SamYapleyes17:39
jpeeleri waited because i thought this sort of thing could go in at the start of M17:39
jpeelerbut forgot about backports and such17:39
sdakeit basically makes the config more object oriented17:39
SamYapleyes17:39
sdakeSamYaple from the context of ansible is there any downside?17:40
SamYaplenah. we already use a dict for the auth strings17:40
SamYapleworks fine17:40
sdakein other words does it limit flexiblikty?17:40
SamYapleno but its a bit more confusing to override a variable17:40
SamYaplebut anyone knowing they can do that should know how to do that17:40
sdakehwo do you do an override?17:41
sdakejust specify x:17:41
sdake     var: "xyz" ?17:41
SamYapleyes17:41
sdakejppeeler I'd like ithis in rc2 (2nd) if possible17:41
sdakeunless someone in the cr team has a problem with it17:42
jpeelerso send out an email or go ahead and complete the review assuming people will be fine?17:42
sdakeyes17:42
SamYaplejpeeler: both17:42
sdakeunless you have something better to do :)17:42
SamYapleyea pbourke_ you around?17:42
sdakeit is posible it could b ewasted effort17:42
sdakethe only folks I think may find objection are oracle17:43
SamYaplehey guys. without maxing out my ceph cluster at all. I got 5,000 iops for 4k writes.... FROM INSIDE THE INSTANCE17:43
SamYaplei did that fio test on 8 instances at once17:43
SamYapleall doing 5,000 iops17:43
sdakethat gear can do 50k iosp I think17:43
SamYaplei think the qemu driver is the bottlenexk17:44
SamYaplesdake: yes17:44
sdakeyup memcpy17:44
SamYapleno its virtio-scsi driver17:44
SamYapleway better17:44
SamYaplebut still not perfect17:44
SamYapleeither way 900MB/s for seq large blocks17:44
SamYapleand 5,000 random 4k17:44
SamYapleiops17:44
sdakeis the caching for ceph different to set up then using the disks as raw disks?17:46
sdakeor does kolla figure it out17:46
SamYaplewhat do you mean?17:46
sdakeSamYaple ^^17:46
sdakei mean you have ceph which has 750s for caching17:46
sdakehow doy ou set that up ?17:46
sdakedoes it require custom deploy steps?17:46
SamYapleah. so you take your backing disks and ssds (what you want for cache) and bootstrap them the same way17:47
SamYapleparted /dev/nvme0n1 -s -- mklabel gpt mkpart KOLLA_CEPH_OSD_BOOTSTRAP 1 -117:47
SamYapledo that for all disks17:47
SamYaplethen you have to do ceph things to setup the tiering17:47
SamYaplehttp://docs.ceph.com/docs/master/rados/operations/cache-tiering/17:47
sdakeis it possible to make automatic?17:47
SamYaplei wont no17:47
sdakelike in mitaka17:47
SamYapleif you want to break out ceph into its own project, then yes17:48
SamYaplebut it could easily balloon out paste all of the openstack code17:48
SamYapleceph is complicated and storage is sensitive17:48
sdakeis it something that can be documented instead of automatic?17:48
SamYapleof course17:48
SamYaplebut in this case the ceph offical docs are far more useful17:49
sdakei do not want to break out ceph separately17:49
SamYapleright, so we need to be alot less flexible with the ceph 'automation'17:49
SamYapleim happy just getting the osds in the cluster17:49
SamYaplethe maintianing performance is the operator all the way17:49
SamYaplesdake: I would be ok with a scripts/tools directory script/ansible that could do some of that automation17:51
SamYaplebut again this could easily balloon out way past the openstack stuff17:51
sdakei was thinking something pretty inflexible, like specify the cache devices, specify the regular devices17:52
sdakeand have kolla figure it out17:52
sdakelike how you have the labels now17:52
sdakebut with a second label for the cache devices17:52
sdakebut I dont know the first thing about ceph or if that is feasible17:52
SamYapleyes i thought about that17:55
SamYapleits possible but ceph is complex17:55
SamYapleand im worried about a cluster with 200TB of data and breaking it....17:55
*** jtriley has joined #kolla17:57
sdakewell try to keep an open mind :)17:59
sdakepossibly an optional feature for the future17:59
SamYapleyea no worries but its complex18:00
sdakeI think it would be pertty sweet18:00
SamYaplebecause you have to redo the crush map18:00
sdakeI dont even know what that means ;)18:00
SamYaplei know and thats my point18:00
SamYapleits do able. ill look into it18:00
sdakecool18:00
sdakewould be a hot featur I think :)18:01
*** shakamunyi has joined #kolla18:01
*** banix has joined #kolla18:01
sdakesomething not flexible just "ya I want lazy-mans caching"18:01
SamYapleno substitue for a good ceph admin18:01
SamYaplemore probe to breakage18:01
sdakeif someone wants highly tuned storage they are oging  to have ceph cats on staff and probably not run it in a container18:01
*** loth has joined #kolla18:01
sdakeif they are a 100 person org, they are going to turn on poor mans storage setup18:01
SamYapleyou say that but let me tell you running in a container actually _SOLVES_ some issues18:01
sdakerather not poor man lazy man18:02
SamYapleit gets rid of a nasty hack for looking up the osd id18:02
SamYapledouble mounting18:02
* SamYaple shakes a bit18:02
sdakehey guys, inc0's vote was unanimous18:05
sdakeinc0 welcome to the CR team!18:05
sdakeoh I guess inc0 is not around18:05
sdakewell anway maybe he will read the logs18:05
Huicould anyone point to me the HA design in Kolla?18:07
sdakeI dont think there is any modern documentation on the HA design18:07
sdakeour docs are really lagging hui18:07
Huiso to be HA, the current approach is to deploy multiple containers for a service. Is my understanding correct?18:08
SamYapleyes18:09
HuiThanks Sam18:09
sdakesamyaple I'm thinking for our demo at summit we should demo writing a file with the caching of ceph18:09
sdaketo show how blindingly fast it is18:09
SamYaplesdake: as long as we can ssh to my gear :P18:09
sdakeis it possible to access your cluster from the internets?18:09
SamYaplethe whole world even!18:10
HuiIf, say a container fails, there is no way to recover it now. Right?18:10
SamYapleI even have dns and icmp tunnels18:10
SamYapleHui: the container will auto restart which *may* fix it18:10
SamYapledepends on failure18:10
Huiok18:10
sdakeya i think ssh to your gear is the way to go, as long as we can setup a dns nat or something like that18:10
sdakeyou saw my slidedeck on tthe demo18:11
sdakethat is easy to explain to folks18:11
SamYapleyes looks good18:11
sdakeand it is understandable18:11
SamYapleso fyi we are doing cachemode=writeback for ceph18:11
sdakeeven though nobody si really going to run a netgear wrt router :)18:11
sdakeright18:11
SamYaplethis is safe for ceph (and only ceph) due to using librbd and not kernel pages18:12
SamYapleusing that with anything else would be dangerous and live-migration could move stale-data18:12
SamYapleceph fixed this in 2014 with librbd and qemu patches18:12
SamYaplein case anyone askes you :)18:12
sdakewriteback is generall ysfe unless there is power loss18:13
SamYapleyea but its not actually safe with live-migration see above18:13
sdakein the 90s when I worked on raid stuff, we used writeback but we had battery backups on our raid cards18:13
sdakeright18:13
SamYapleonly with ceph can you actually issue a cache invalidate call18:13
SamYaplei still need to put in the dvr yes/no option in the configs for koll18:16
SamYaplea18:16
*** openstackgerrit has quit IRC18:16
SamYaplei do dvr here a bit when testing18:16
*** openstackgerrit has joined #kolla18:16
SamYapleit doesnt really matter for such a small test lab, but still18:16
SamYaplemore people would try it if it were a yes/no option18:17
openstackgerritSam Yaple proposed openstack/kolla: Common start.sh  https://review.openstack.org/22937118:22
SamYapleugh sdake ansible in kolla_ansible container is foobar18:42
SamYaplei guess some breakage upstream18:42
SamYaplelocking down to a tag now18:42
SamYaplesdake: so there are more deps we can move to the openstack-common container18:43
SamYaplethat should decrease size and improve build times18:44
SamYapleholy crap i need to sleep18:44
*** dims_ has quit IRC18:46
*** openstackgerrit has quit IRC18:46
*** openstackgerrit has joined #kolla18:46
*** dims_ has joined #kolla18:46
sdakesamyaple groan18:46
SamYaplesdake: its cool. tested latest tag, all good18:47
sdakeok cool18:47
sdakewhat was the probllem then?18:47
SamYaplerc1 image upstream for ubuntu works18:47
SamYaplebut the centos kolla_ansible container gonna be foobared18:47
SamYapleno idea18:47
SamYapleits stil la proble18:47
sdakei probalbybusted it when i was wirting code after being up for 30 hous18:48
* sdake sighs18:48
sdakei'll look into it today18:48
SamYaplesdake: no its upstream _ANSIBLE_18:49
sdakeoh i see18:49
sdakei was pretty sur eI teted that :)18:49
sdakewhich is why I was surprised when you said it18:49
sdakebut now i get it18:49
SamYapleit literally just broke for me18:49
SamYapleim guessing latest commit or two18:49
sdakethe build breaks ?18:50
SamYapleno build works, container breaks18:50
sdakeok18:50
SamYapleso no doing any commands in container18:50
SamYapleanyway ih ave patch testing now18:50
*** achanda has quit IRC18:50
SamYapleworking thus far18:50
*** achanda has joined #kolla18:53
openstackgerritSam Yaple proposed openstack/kolla: Common start.sh  https://review.openstack.org/22937118:58
openstackgerritSam Yaple proposed openstack/kolla: Lock down ansible version  https://review.openstack.org/22957918:58
*** achanda has quit IRC19:02
*** jtriley_ has joined #kolla19:03
*** jtriley has quit IRC19:06
*** inc0 has joined #kolla19:08
inc0just come back to say thanks! I'm really excited;)19:09
SamYaplego for the gold!19:09
sdakeinc0 grats bro!19:09
inc0now the hard work starts:)19:10
sdakeinc0 if you want a bit of coaching and have 5 minutes i can giveoyu the quick how to do the thing19:10
sdakedo you have 5 minutes?19:10
inc0sure19:10
sdakeok so basically one person comes alnog and +2s a review19:10
sdakeif your the first person to a review, and you like the patch, go ahead and hit +219:11
sdakebut do not change the workflow19:11
sdakeif your the second person to a a review and it already has a +2, set +2 and workflow to +119:11
sdakedo not ever approve a patch with only one review19:11
sdakeno matter what the circiumstance19:11
sdakeif you are part of the author chain you may not self +2 a patch19:11
sdakeeven if your only a coauthor19:12
sdakeit requires an independent party to approve the patch19:12
openstackgerritSam Yaple proposed openstack/kolla: Common start.sh  https://review.openstack.org/22937119:12
sdakethe one exception is if two core reviewers have already reviewed the patch19:12
sdakeand its a rebase19:12
sdakeand gerrit retains the other 2 +2s19:12
SamYaplesdake: https://review.openstack.org/#/c/229371/19:12
sdakeyou may set +2/+a19:12
SamYapleplase test centos19:12
SamYapleubutntu 100%19:12
sdakesam sure i'll pull it down now and reubild19:13
sdakehang tight19:13
sdakeit takes about 45 mins19:13
SamYapleno prob im going to sleep19:13
SamYaplerepatch if needed19:13
sdakeenjoy19:13
sdakeinc0 any questions?19:13
inc0sdake, so far no, I've seen a lot of reviews so I think I get how it works;)19:14
SamYapleinc0: a good patch to flex your +2+1 :) https://review.openstack.org/#/c/229579/19:14
inc0I'll ask around if not19:14
sdakeya the only tricky one is the rebase19:14
sdakesamyaple that is waiting on the gate19:14
SamYapleim asleeps now19:15
sdakeword enjoy19:15
inc0SamYaple, quick question...alpha?19:15
inc0we pin to alpha?19:15
SamYapleinc0: its the only 2.0 tag avaialbe19:15
sdakethat is all thee is19:15
SamYaplealpha1 and alpha219:15
SamYaplewe need 2.019:16
inc0for shade licence? I get it19:16
SamYaplei actually comment on this in the commit message19:16
SamYapleno nothing to do with shade19:16
sdakeinc0 one other thing19:17
sdakei need to bring this up with al lthe crs19:17
sdakebut i'd like people not to approve changes until the non-voting gates complete successfully19:17
sdakeits annoying we can't just vote and move on19:18
sdakebut that is the world we live in :)19:18
inc0we need to think of good way of stable and reliable voting gates19:18
sdakei know how to do it19:19
sdakeit involves getting infra to mirror the repos19:19
sdakegoing to tackle in mitaka19:19
inc0I made first +2 ;) but not +w19:19
sdakenice19:19
sdakeinc0 its complictaed by the fact that the ptl for infra changed19:20
inc0new one will be less sympathetic?19:20
inc0I can ask intel's CI guys to spare few machines for us19:20
inc0if that's lack of hardware problem19:21
*** jtriley_ has quit IRC19:32
sdakeits lack of people to implement it problem in the current infra19:32
sdakeinc0 ^^19:32
sdakenew ptl not sure if sympathetic or not, but will have hands full with very busy project19:33
sdakeso hard to get ear of ptl for this effort19:33
*** jtriley has joined #kolla19:33
inc0I see19:33
sdakeif you could find someone to magically implement it in the infra team, that would be fantastic :)19:34
sdakeotherwise I'm going to hve to beg around19:34
sdakethat is more important then having a functional gate imo19:34
inc0or outside of infra, just accessible19:34
inc0and dedicated for us19:34
sdakevoting gate is critical19:34
sdakei'd like it to be in the infra19:34
sdakeexternal ci can't vote19:34
inc0no, I mean mirrors19:34
sdakeoh right19:35
sdakeya doesn't matter where it is19:35
sdakebut in infra woudl be better19:35
sdakeit isn't that hard to mirror rpm repos19:35
sdakeand deb repos19:35
inc0yeah, and github is pretty stable19:35
sdakegithub doesn't do the mirroring19:35
sdakeyou have ot setup a http server19:35
sdakeand run a cron job to mirror various rpm repos19:36
sdakethe cron job runs a mirroring script19:36
inc0well, that's something we'll talk about in Tokyo I guess19:36
sdakeand we have to modify the build.py to add a specific set of mirrors19:36
inc0I'll ask around if we can spare few machines from our community servers19:36
sdakethe work is easy just need the mirrors running19:36
sdakedoesn't even have ot be bare metal19:37
inc0we have 3rd party CI in intel, maybe we could place mirrors there19:37
sdakecould be virt19:37
inc0anyway, I'm going off, cya and thanks again:)19:37
sdakeenjoy :)19:37
*** inc0 has quit IRC19:37
openstackgerritJeff Peeler proposed openstack/kolla: Group variables by dictionary in all.yml  https://review.openstack.org/22953519:40
sdakejpeeler you sure cranked that out fast :)19:41
jpeeleryou call that fast?19:41
jpeeleri still need to test it19:41
sdakelooks groansihly long to review19:41
sdakeset workflow so folks know you aren't done testing it19:41
jpeeleri did19:42
sdakecool19:42
jpeelerthat was painful, so i really hope it goes in19:42
sdakeit will be harder to review then it was to develop19:43
sdakeyou have the easy part of it :)19:43
sdakejpeeler please make a blueprint19:44
jpeeleri really hope nobody reviews it for two hours19:44
sdakethat is how long it will take i suspect19:44
sdakei will review it for however long it takes19:44
sdakejpeeler hwere does kolla_internal_address come from?19:48
sdakewhy not kolla.internal_address?19:48
jpeelerthat's from globals19:48
sdakei would expect globals to not have a flat structure?19:49
jpeeleri didn't touch that file19:49
sdakei think the intent is to touch that file19:49
jpeeleri'll need to test to see what that looks like19:51
jpeelerreally didn't mean to spend all day on this19:52
jpeelerbut i do think it's a good change...19:52
jpeeleractually i guess you're right19:53
*** JohnN has quit IRC20:00
openstackgerritJeff Peeler proposed openstack/kolla: Group variables by dictionary in all.yml  https://review.openstack.org/22953520:10
*** shardy has quit IRC20:14
*** banix has quit IRC20:18
*** thumpba has joined #kolla20:25
*** tpot has joined #kolla20:52
*** itsuugo has joined #kolla20:53
*** thumpba has quit IRC21:01
*** thumpba has joined #kolla21:03
*** thumpba has quit IRC21:08
*** thumpba has joined #kolla21:09
*** jtriley has quit IRC21:20
*** dwalsh has quit IRC21:34
*** thumpba_ has joined #kolla21:40
*** thumpba has quit IRC21:43
*** thumpba_ has quit IRC21:46
*** thumpba has joined #kolla21:46
*** banix has joined #kolla21:55
*** sdake has quit IRC21:55
*** sdake has joined #kolla21:58
*** thumpba has quit IRC22:05
*** jruano has quit IRC22:10
*** diogogmt has joined #kolla22:16
*** jtriley has joined #kolla22:25
*** devlaps has quit IRC22:30
*** dims__ has joined #kolla22:30
*** dims_ has quit IRC22:33
*** jtriley has quit IRC22:55
*** alisonh has quit IRC23:03
*** alisonh has joined #kolla23:03
*** jtriley has joined #kolla23:04
*** jruano has joined #kolla23:08
*** jtriley has quit IRC23:25
*** diogogmt has quit IRC23:36
*** blahRus has quit IRC23:47
*** diogogmt has joined #kolla23:48

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!