Tuesday, 2015-10-13

*** achanda has joined #kolla00:11
*** achanda has quit IRC00:13
*** exploreshaifali has quit IRC00:16
*** cloudnautique has quit IRC00:17
*** sdake_ has quit IRC00:18
*** juggler_ has quit IRC00:19
*** sdake has joined #kolla00:33
*** tobe has joined #kolla00:49
*** tpot has quit IRC01:03
*** tpot has joined #kolla01:09
openstackgerritMerged openstack/kolla: Update Vagrant directories in .gitignore  https://review.openstack.org/23356701:10
*** erkules_ has joined #kolla01:11
*** erkules has quit IRC01:14
*** achanda has joined #kolla01:31
*** jasonsb_ has quit IRC01:47
bigjoolsIs Kolla suitable for deploying a heavily custom Openstack with many custom dependencies? It seems like the image layering is too inflexible for that but maybe I am missing something?01:51
openstackgerritMerged openstack/kolla: Ignore the 'Disk Flags:' line in parted  https://review.openstack.org/23359201:58
*** alisonh has quit IRC02:01
*** unicell1 has quit IRC02:02
rhalliseybigjools, if you layer properly I think it should be fine02:05
rhalliseyactually even if you don't it shouldn't really matter02:05
*** vinkman has quit IRC02:07
*** rhallisey has quit IRC02:09
*** vinkman has joined #kolla02:14
*** alisonh has joined #kolla02:19
asalkeldanyone else's kolla-build failing with:02:19
asalkeld    if (image['parent'] is not None and02:19
asalkeldKeyError: 'parent'02:19
asalkeldthe image has "parent_name" but not parent02:20
asalkeldhacked around it, but odd that i am getting that ...02:23
*** zhiwei has joined #kolla02:28
*** dims_ has quit IRC02:28
*** sdake has quit IRC02:29
*** zhiwei has left #kolla02:29
*** klint has joined #kolla02:40
*** dims__ has joined #kolla02:41
*** CBR09 has joined #kolla02:53
mandreasalkeld, that's odd02:55
asalkeldyeah, clean vagrant setup02:55
mandreimage['parent'] is normally set in find_parents()02:55
mandrewhat's the value of image['parent_name']?02:56
asalkeldexample: 'parent_name': 'operator.local:4000/lokolla/centos-binary-ceilometer-base:latest',02:58
*** dims__ has quit IRC02:59
asalkeldno worries, let me re-create the vagrant operator vm and try again02:59
*** dayou has joined #kolla02:59
*** rajalokan has joined #kolla02:59
*** tpot_ has joined #kolla03:00
*** tpot has quit IRC03:01
mandrewell, that bugs me, I'd like to understand why it's failing03:01
mandreyou should normally have an image with 'fullname' being the same, that's what how we find a matching parent03:04
asalkeldmandre: it's ok. emacs make a weird file in my kolla git tree (unsaved change) and that cause an install error03:15
asalkeld(i was just adding some notes to the vagrant rst file)03:15
bigjoolsI am trying to make Kolla build on a 12.04.5 Ubuntu base and when it gets to openstack-base I see this in the log: "INFO:__main__:openstack-base:Step 0 : FROM kollaglue/ubuntu-source-base:latest"03:20
bigjoolsI used --base-tag 12.04.503:20
bigjoolsis this a bug or did I screw up?03:20
*** rajalokan has left #kolla03:26
*** cloudnautique has joined #kolla03:27
*** tfukushima has joined #kolla03:27
kjellybigjools: Based on the information you provided, it's not bug. But ubuntu 12.04 is a little old. Maybe try it without base-tag03:30
bigjoolskjelly: no, I need 12.0403:30
bigjoolsfor $reasons03:31
bigjoolswhat dictates the tag it uses for that image?03:31
*** jasonsb_ has joined #kolla03:31
kjellykjelly: well, it seems that most developer develop in latest tag.03:32
kjellybigjools:well, it seems that most developer develop in latest tag.03:32
bigjoolskjelly: ok - there's no way to override?03:32
*** sdake has joined #kolla03:33
kjellybigjools: you can use "--base-tag 12.04.5". But it maybe doesn't work.03:33
bigjoolskjelly: yes, that's what I am using already. that's why I thought it was a bug :)03:33
*** sdake has quit IRC03:37
SamYaplemorning03:38
*** sdake has joined #kolla03:38
sdakeevening folks03:39
sdakeasalkeld i do not run into that problem03:39
sdakebigjools could you go into more detail03:39
SamYaplesdake: got a successful gate deploy03:39
sdakeSamYaple nice03:40
bigjoolssdake: sure. I am trying to get a custom git repo building on 12.04. When it starts to build the base images it ignores the --base-tag I put on the command line and just uses latest.03:41
bigjools"docker images" shows ubuntu at 12.04 but the others are 'latest'03:41
sdakecustom git repo contains what?03:41
SamYaplebigjools: you aren't going to be able to build these images on 12.0403:41
SamYaplebigjools: but also --base-tag speciffcaly only refers to base03:42
bigjoolssdake: well I am starting with keystone03:42
SamYapleso that wounds like its working03:42
bigjoolsoh ok - can I override the others?03:42
SamYaple--tag03:42
SamYaplebasically --base-tag is just used to pick the base image (the only image kolla doesnt build)03:42
SamYaple--tag is used to tag images kolla does build03:43
bigjoolsaha ok let me try that, thanks03:43
sdakeSamYaple got my 1.2 tb intel 750 today03:44
SamYaplei read03:44
SamYaplei also have 1.2tb of intel 75003:44
SamYaplebut with 3x the iops!03:44
sdakei havae 3 400gb 750s as well ;-)03:44
SamYaplei would be very intersted in the performance numbers of your 800$ switch03:45
SamYaplemy guess is the backplane can only do 1-2GB a second03:45
sdakeit claims 10gb03:46
sdakei'll let you knw owhen i get it03:46
sdakepeople in the q&a claim 6-8gb03:46
sdakewhich is GEFN compared to 5k for a cisco switch03:46
SamYapleno thats not what im saying03:46
SamYaplea switch with 24 10gb ports may only be able to switch 160gb a second03:47
sdakei'll let you know what my ceph numbers are as soon as i get my gear installed03:49
sdakemy 2 400gb are in transit atm03:49
sdakeas wel las tehe switch - arrives tomorrow  ithink03:50
bigjoolsSo this isn't doing what I expected, it's downloading 14.04 packages. Is that hard-coded somewhere?03:50
SamYaplebigjools: yes all over the place03:51
*** zhiwei has joined #kolla03:51
SamYaplethis wont work with 12.04 like i said above03:51
bigjoolsdamn :(03:51
bigjoolsok03:51
SamYapleim fairly certain they dont even make liberty packages for 12.0403:51
SamYapleill check03:51
bigjoolswell like I said above, I'm using custom repos03:52
SamYaplenope03:52
SamYapleonly icehouse03:52
*** zhiwei has left #kolla03:52
SamYaplesure but there are expectations about the versions of packages03:52
sdakeSamYaple i think what bigjools is  getting at his his dependencies need to be older then theya re03:53
sdakewe do not have a dependency pinning solutiion on koll atm03:53
SamYaplewhat im saying is that wont happen03:53
sdakethat said, I think we need one03:53
SamYaplewe _need_ newever versions03:53
SamYapleubuntu is pinned03:53
sdakei.e. specify a requirements.txt and pin eah install to a custom requirements.txt03:54
openstackgerritKuo-tung Kao proposed openstack/kolla: update doc for root is needed when deploying  https://review.openstack.org/23238103:54
SamYaplewhat are you talking about sdake03:54
bigjoolsI was using include_header to set up pins and repos etc, but the stuff I can't change is using 14.04, obviously03:54
*** tpot_ has quit IRC03:54
sdakeif oslo.log is 1.1.0 and 1.2.0 is broken, yo uwill want to use 1.1.0  = in other words custom pin to oslo.log 1.1.003:54
SamYaplei dont htink bigjools is talking about that at all03:54
sdakepossible i dont he enough information03:55
bigjoolsThat part works OK I think03:55
bigjoolsit's the fact that when it installs the other -base images it uses 14.0403:55
bigjoolswell, all of them in fact03:55
sdakeand the problem wit htht is what/03:55
bigjoolsI have to use 12.04, I don't get a choice here because of other reasons03:56
SamYapleyea i dont think this will work for you bigjools03:56
bigjoolsif it's not possibe then fair enough03:56
SamYaplethere are lots of small changes throughout03:57
bigjoolshow are you going to deal with this when newer Ubuntu releases are made?03:57
SamYapleadditionally its expected you have certain version of packages (qemu, rabbitmq, etc)03:57
SamYaplejust upgrade it?03:57
SamYapleits in a container so itll be fine03:57
bigjoolsI mean installing from scratch03:57
SamYaplelike baremetal?03:57
SamYapleim confused03:58
SamYapleall of this is in the container so that doesnt really affect anything03:58
sdakebigjools probably ahs ssome requirements driven by his business not to upgrade03:58
bigjoolsI mean when the next LTS comes out and I want to install from scratch with Kolla, will it still use 14.04 or does it just grab the latest LTS? Or something else?03:58
SamYaplekolla will always target the latest LTS03:58
sdakekolla atm doesnst dod bare metal provisioning03:59
bigjoolsrighto03:59
bigjoolsok that's cleared things up very well, thanks both03:59
SamYaplethe packages may overlap in the case of 14.04->16.04 but probably not for 3 years03:59
*** dims__ has joined #kolla03:59
SamYaplebigjools: just to be clear, some of these reqs are imposed by openstack, mainly the libvirt and qemu versions04:00
SamYaplecanonical provides the updated repos and kolla consumes them04:00
bigjoolsyeah I know, we have workarounds for that :)04:00
SamYapleok just making sure04:00
SamYapletruthfully you _could_ make this work if you wanted to04:00
sdakesoudns ike metapod ;-)04:00
bigjools*cough* :)04:00
SamYapleit would be a simple fork of kolla with a few tweaks04:00
bigjoolsright04:01
bigjoolsI'll have to see if we want to go that route, I dislike forking but ...04:01
sdakeits not a fork if you rebase against master periodically04:02
sdakethis is how the upstream openstack orkernel.org work04:02
sdakeor most open source software04:02
sdakeredhat takes kernel.org, forks tmeporarily (temporary being 7 years)04:03
sdakebut eventuall yrebases against master when new version of rhel comes out04:03
sdakecentos operates same way04:03
*** dims__ has quit IRC04:06
bigjoolsyeah, tech-debt is my bugbear04:10
*** bmace__ has quit IRC04:11
*** bmace__ has joined #kolla04:12
nihilifero/04:12
sdakesamyaple is it possible to use ceph with 1 disk?04:13
sdakewith 3 control nodes04:13
SamYapleyea04:13
SamYaplejust bootstrap one disk. dont change anything else04:14
SamYapleyoull have to change number of copies or modify crush map04:14
*** tpot has joined #kolla04:14
sdakecool can you hand hold me throug hthat, in about 20 mins afteri finish eating?04:15
SamYapleits in the docs04:15
*** jasonsb_ has quit IRC04:19
*** tpot has quit IRC04:23
*** tpot has joined #kolla04:25
*** sdake has quit IRC04:30
*** sdake has joined #kolla04:31
SamYaplesdake: hey04:44
SamYaplegnocchi builds just fine for ubuntu04:44
SamYaplei dont want to remove that04:44
sdakeoh wasnt aware04:45
sdakei thought it was broken all around04:45
SamYapleyea i put the work into fixing that like 2 months ago04:45
SamYapleits been building fine since then04:45
sdakecan you abandon the change then please04:45
sdakesince you havve it open04:45
sdakedont abandon others changes unless asked btw :)04:46
SamYaplethe remove gnocchi change?04:46
sdakeya04:46
SamYaplealright. and i dont abandon peoples changes :)04:46
sdakei expire peoples changes after a couple a months04:46
sdakebut i usually give warning :)04:46
SamYaplewel lthat i do04:46
*** tfukushima has quit IRC04:48
*** inc0 has joined #kolla04:49
tpotyay - my first successful tools/kolla-ansible deploy04:49
SamYapleawesome tpot04:50
inc0good early morning04:50
SamYaplemorning inc004:50
sdakenice job tpot04:50
sdakewhat are your first impressions04:50
SamYaplesdake: https://review.openstack.org/#/c/233879/04:52
SamYaplethis is breaking builds now04:52
sdakesounds like it needs a revert04:52
SamYaplethats what that patch is04:53
inc0sam I've hit +a04:53
SamYaplethanks inc004:54
SamYapleill kick the backport immediately04:54
openstackgerritMerged openstack/kolla: Remove obsolete backport for oslo.log  https://review.openstack.org/23387904:55
tpotfirst impressions are it's a bit fiddly and lots of linux admin knowledge required to troubleshoot04:56
*** tpot has quit IRC04:56
openstackgerritSam Yaple proposed openstack/kolla: Remove obsolete backport for oslo.log  https://review.openstack.org/23393804:56
openstackgerritMichal Rostecki proposed openstack/kolla: [WIP] Separate exclusion list for source and binary types  https://review.openstack.org/23393904:59
SamYapletpot thats probably a fair assessment04:59
SamYapleMichal Rostecki are you in IRC?04:59
nihiliferyes, I am04:59
SamYapleah hi nihilifer! didnt realize that was you04:59
inc0so much MichaƂs05:00
inc0so many of them05:00
inc0;)05:00
SamYapleso sdake wiegh in here, but didnt we say we were going to do a 'main, infra, aux' list?05:00
nihiliferinc0: where is second Michal from Intel? ;)05:00
SamYapleif so i dont think we need a seperate exlusion list05:00
inc0he's doing docker now...docker itself05:00
*** tpot has joined #kolla05:01
inc0SamYaple, elaborate please? what's main, infra, aux?05:01
SamYaplehttps://blueprints.launchpad.net/kolla/+spec/restructure-template05:01
nihiliferSamYaple: didn't realize that. well, I just picked up this bug05:01
SamYapleno its cool05:01
SamYapleand that may be what we do05:01
SamYapletrying to make sure we arent duplicating efforts05:02
SamYapledont want to waste your time05:02
tpotfirst impressions - a bit fiddly but that's probably my unfamiliarity with ansible and the project itself05:02
inc0ah, so "contrib" topic?05:02
SamYapleyea inc005:02
SamYapletpot: i would agree with your assesment05:02
inc0I think we have session about that05:02
inc0something I want to move in this topic is also 3rd party05:03
tpotHey I'm going to be at the summit in Tokyo and would love to say hi to you all if you are there also05:03
inc0so ease of adding new stuff by consuming parties05:03
inc0not to core code, but as plugins05:04
SamYapletpot: I will be there, so will alot of us05:04
sdaketpot we have 8 design sessions so we will be there :)05:04
inc0tpot, feel free to join us at any of kolla sessions05:04
tpotI'll definitely be there05:04
SamYaplejust not the non-kolla sessions05:04
SamYaplestay away from us in those05:04
inc0no, why? there are few ops summit sessions that sounds fun05:04
tpotvery impressive to run pstree and see everything running under docker05:05
inc0I don't want to be seen too close to devs05:05
SamYapletpot: not.. everything05:05
SamYapletpot: libvirt qemu instances will not05:05
SamYaplesame with ceph-osd05:05
SamYaplefor various reasons05:05
tpotbah - details (-:05:05
SamYaple:)05:05
inc0SamYaple, what do you mean? docker is wrapping this stuff05:05
sdakedocker-engine 1.8.3 and kolla don't work together05:05
tpotit's interesting to see how much the docker abstraction needs to be broken, and where, for it all to work05:05
inc0ofc it meddles with base host05:06
inc0same as ovs05:06
SamYapleinc0: qemu and ceph-osd are not under the docker parents05:06
SamYaplethey are under pid 1 as parent05:06
inc0ah that's what you mean05:06
inc0yeah, pstree05:06
SamYaplehtop for me!05:06
inc0I need to drink caffeine.05:06
inc07am here, brain is slow, send help05:07
SamYaplevbel you around?05:08
openstackgerritMerged openstack/kolla: Fix up loc with change to devenv  https://review.openstack.org/23340405:11
SamYaplepbourke: you around?05:14
sdakeya docker-engine 1.8.3 is busted05:15
SamYapleive been using 1.8.3 for a while05:15
SamYapleno issues05:15
sdakewierd05:15
sdakeit prints out something about m yregistry is old05:15
sdakeand this causes ansible to implode05:15
SamYapleansible only interacts with docker-py05:16
SamYaplei would start by looking there05:16
sdakewhich interacts with docker05:16
sdakei am running same version of docker-py05:16
*** vinkman has quit IRC05:16
SamYapleis that version of docker-py new enough to interact with 1.8.305:16
sdakeunknown05:16
SamYaplei would start by looking at docker-py05:17
sdakey series is supposed to be api lockdown05:17
SamYapleand yet 1.4.0 to 1.4.1 had an api bump if i remember correctly05:17
SamYaplefor pid host support05:17
sdakeanyway docker .1.81 works well enough05:17
*** tfukushima has joined #kolla05:20
*** tfukushima has quit IRC05:20
sdaketpot how many nodes did you deploy - and did you deploy with a registry?05:22
tpotjust a single node spare PC05:22
*** tfukushima has joined #kolla05:22
tpotall images built locally05:23
sdakewhat did yo uthink of speed of deployment05:25
tpoti think i've got the wrong config for the (required) devicemapper backend05:26
tpotso creating a container is pretty slow05:26
sdaketo build, best performance happens with btrfs05:27
tpotit may work faster if I use a dedicated partition for containers05:27
sdaketo deploy best perfomrnace uses overlayfs05:27
tpotyeah i think the default dm backend uses multiple layers of loopback and is really slow05:27
tpotdoesn't overlayfs not work with systemd?05:27
tpotthere was a launchpad bug report about the systemd RPM not installing onto a container with overlayfs and I ran into that early o n05:28
tpotanyway, i thought things were reasonably fast given my old hardware and slow network in .au05:29
SamYapletpot: it doesnt work for builds at all05:29
SamYapleoverlayfs is great for _running_ containers05:29
tpotgotcha - good tip05:29
*** tfukushima has quit IRC05:32
openstackgerritMerged openstack/kolla: Remove obsolete backport for oslo.log  https://review.openstack.org/23393805:36
sdakeyes you can't build on overaly05:37
sdakebut you can deploy to overlayfs05:37
*** unicell has joined #kolla05:38
*** tobe has quit IRC05:39
*** daneyon has joined #kolla05:40
openstackgerritMichal Rostecki proposed openstack/kolla: Push images to registry by default in Vagrant  https://review.openstack.org/23394605:40
*** daneyon has quit IRC05:41
sdakesamyaple what are the cinder pool names?05:42
*** tfukushima has joined #kolla05:43
sdakevolumes and backups?05:44
openstackgerritMichal Rostecki proposed openstack/kolla: Update Vagrant docs  https://review.openstack.org/23337105:47
openstackgerritMichal Rostecki proposed openstack/kolla: Update Vagrant docs  https://review.openstack.org/23337105:50
*** tpot has quit IRC05:52
SamYaplesdake: cinder is volumes05:58
SamYaplecinder-backup is backups05:59
sdakeValueError: No JSON object could be decoded06:01
sdakefrom TASK: [ceph | Fetching Ceph keyrings] *****************************************06:02
sdakeany ideas how to rectify?06:02
SamYapleclean your environment06:02
sdakei killed all containers rebooted, parted labels rebooted06:02
sdakeand removed the entry from etc/fstab06:02
SamYapleumount /var/lib/ceph/osd/*; rm -rf /var/lib/ceph/osd/*; ceph_wipe; sed -ir '/ceph/d' /etc/fstab; rm -rf /etc/kolla/!(config|globals.yml|passwords.yml) /usr/share/kolla;06:03
SamYapledo something like that then06:03
*** dims__ has joined #kolla06:03
SamYapleyou didnt remove /etc/kolla/ceph-*06:03
sdakeis it not possible to label a parition, only a disk?06:04
SamYaplewell you are labeling a partition not the disk06:05
SamYaplebut the general notion yes, it uses the whole dis06:05
SamYapledisk*06:05
sdakeso you can hve a bootstrap with 3 partitions on one disk?06:05
SamYapleits going to use the whole disk and youll end up with one OSD, but sure i guess06:06
SamYapleyou can label 3 partitions as bootstrap.06:06
SamYapleit wont do anything sepcial06:06
openstackgerritMichal Rostecki proposed openstack/kolla: Install tox in Vagrant  https://review.openstack.org/23396006:06
sdakei want to add poor mans cache auto-setup via labeling in the future06:06
sdakejust seing if its posislbe06:07
SamYapleyea we can probably do that06:07
SamYaplebut scope creep on ceph end is a bad thing06:07
SamYapleitll grow larger than kolla in a hurry06:07
sdakecould you expand on that point06:08
SamYapleceph is big and complex. to do automated tihngs with ceph requires lots of code06:08
SamYaplecache we could probably pull off reasonably06:08
SamYaplebut ceph can to a million and one things and i dont want that stuff in kolla06:09
sdakeya i thnk th elabel thing is a solid solution to avoid witch doctor skills with ceph06:09
sdakei have to eval is caching is even worthwhile06:09
*** dims__ has quit IRC06:09
SamYapleyea i agreed to the cache thing last week06:10
SamYaplei just want to set expectations06:10
sdakeright i recall06:10
sdakei dont even know if it will be worthwhile06:11
SamYaplethe cache thing will06:11
sdakeit looked from your prelim tests it was06:11
sdakebut i didn't see the baseline with no cache06:11
SamYapleno cache _and_ no ssd journal was ~100MB/s, 1000 iops total cluster06:11
SamYaple8 disks i think06:12
sdakeis a cache ssd required per compute node?06:14
sdakeor only a limited amount required?06:14
SamYapleyou dont have to have the cache on any particular nodes06:15
SamYapleits all over the network06:15
sdakewhere do you think it tops out?06:15
sdakejust depends on size?06:15
SamYaplewhat do you ean?06:15
SamYaplemean*06:15
sdakehow much disk space of cache to regular disk space is necessary for improved throughput06:16
SamYaplelust like any cache it depends on how you use it06:16
SamYapleif you are really write heavy, alot06:16
SamYapleif not just a little cache is fine06:16
sdakedoes ceph run over a storage network separate from the rest of the control network?06:19
sdakeor can it be made to do so?06:19
sdakecontrol/managemnet network06:20
SamYaplethere is no control/management network06:20
SamYaplejust api_interface06:20
SamYaplewe also have a storage_interface06:20
SamYapleit defaults to api_interface06:20
*** openstack has joined #kolla06:29
sdakewhich disk benchmark tool didyo uuse to mesaure iops?06:31
SamYaplefio06:31
openstackgerritSam Yaple proposed openstack/kolla: Bump ansible to 1.9.2  https://review.openstack.org/23398306:33
openstackgerritSam Yaple proposed openstack/kolla: Fix issue with su and bad kernels  https://review.openstack.org/23398406:33
SamYaplesdake: those patches are requirements for the gate06:33
SamYapleplease review if you want them in liberty06:33
*** daneyon has joined #kolla06:34
sdake[sdake@minime-01 ~]$ docker exec ceph_mon ceph -s06:36
sdake    cluster 5fba2fbc-551d-11e5-a8ce-01ef4c5cf93c06:36
sdake     health HEALTH_WARN 128 pgs incomplete; 128 pgs stuck inactive; 128 pgs stuck unclean06:36
sdake     monmap e1: 3 mons at {minime-01=192.168.1.101:6789/0,minime-02=192.168.1.102:6789/0,minime-03=192.168.1.103:6789/0}, election epoch 4, quorum 0,1,2 minime-01,minime-02,minime-0306:36
SamYaplepastebin!06:36
sdakenova hypervisor-list shows no hypervisors06:37
* sdake sighs06:37
sdakeagent-list works06:37
SamYapledid anyone ever implement ceph for centos?06:37
SamYapleif not thats probably why its broken06:37
* sdake groans06:37
sdakewhat is involved, just installing packages?06:38
SamYaplei believe so06:38
SamYaplebut i cant test it so i dont know06:38
sdake2015-10-13 06:37:59.417 1 ERROR nova.compute.manager [req-433d9427-bf27-441b-9973-e0089e0b8906 - - - - -] Error updating resources for node minime-03: rbd python libraries not found06:39
SamYapleyea youll need ceph-common package for nova and cinder06:39
*** daneyon has quit IRC06:39
sdakedoes rbd python librar ycome from ceph-common?06:39
SamYapleceph-common includes a dep for that library yea06:39
sdakewhat about glance?06:40
sdakealso needs ceph-common?06:40
SamYapleglance needs python-rbd and python-rados packages only06:40
SamYaple(packages not pip installed)06:40
sdakejust nova-compute needs ceph-common?06:40
SamYapleyoull need ceph-common package for nova and cinder06:40
SamYapleyou also probably need to a a ceph repo for ceph-hammer in the base06:41
sdakein nova-base and cinder-base or some other container?06:41
SamYaplei dont know what centos runs, but hammer is what we are targetting06:41
sdakei'll sort that out06:41
sdakewhere did yo uput em for ubuntu06:41
sdakethe bsae packages or somethign else?06:41
SamYaplei dont recall. im sure yo ucan grep for it06:42
SamYaplei dont know what the centos package names are, but ubuntu package names are ceph-common and python-rbd06:42
sdakewhat about qemu-utisl?06:42
sdakeutils06:42
SamYapleyea youll probably need to do something with that. depending on the package version06:42
SamYapleidk though06:43
SamYaplei had to file upstream bugs to sort it out with ubuntu06:43
sdakeany other distro specific things?06:45
SamYapleunknown06:45
*** achanda has quit IRC06:46
*** achanda has joined #kolla06:48
sdakeis 0.80 hammer?06:49
SamYapleno06:50
SamYaplethats like firefly i think06:50
SamYaple0.94 is hammer i believe06:50
openstackgerritSam Yaple proposed openstack/kolla: In gate functional testing  https://review.openstack.org/23399307:02
SamYapletada07:02
*** dims__ has joined #kolla07:05
*** jmccarthy has quit IRC07:07
*** akwasnie has joined #kolla07:07
*** jmccarthy has joined #kolla07:07
*** dims__ has quit IRC07:10
*** exploreshaifali has joined #kolla07:11
*** achanda has quit IRC07:20
*** exploreshaifali has quit IRC07:20
SamYaplequestion about delorean07:23
SamYapleis that still enabled in the liberty branch sdake ?07:23
SamYapleif so how is that considered liberty?07:24
sdakei am workign on getting an answer on that07:24
sdakei thought i got one a couple days ago but it was incorrect repo location07:24
sdakeand the cat who told me the answer hs been ooo07:25
*** Travis__ has joined #kolla07:26
*** daneyon has joined #kolla07:29
*** daneyon has quit IRC07:34
*** Travis__ has left #kolla07:34
openstackgerritSteven Dake proposed openstack/kolla: Make Ceph work for RPM based distributions  https://review.openstack.org/23402007:58
*** dims__ has joined #kolla08:06
*** jmccarthy has quit IRC08:07
sdakewell SamYaple  fantastic job on ceph08:07
*** jmccarthy has joined #kolla08:07
sdakeit works liek a cham with my centos port08:07
SamYaplehuzzah08:08
*** dims__ has quit IRC08:11
sdakegood thing smoene tested rpm prior to our final releae ;)08:11
SamYaplei test ubuntu lal the time!08:12
SamYapleget with the program!08:12
sdakejust got hardware today08:12
SamYaplealways somthing with you isnt it...08:13
*** athomas has quit IRC08:14
sdakecan't test with invible electrons :)08:15
SamYaplenot with that attitude08:15
SamYaplehttps://github.com/docker/docker/commit/668a1758bd7bb1007fe5a44194dc3ae3cba3934c08:15
SamYaplesdake: ^^ thats probably what you hit08:15
SamYapleits got a daemon flag08:15
*** athomas has joined #kolla08:15
jmccarthySamYaple: not with that attitude - LOL !08:16
SamYapleI am very clever. Praise me08:16
*** gfidente has joined #kolla08:17
sdakepossible08:17
sdakebut the default is to work with v1 registries from wht i can tell08:17
SamYapleno08:18
SamYapleread the commit message08:18
sdake1am going to bed08:18
sdakenight all08:18
jmccarthyNite !08:19
SamYaplenight08:21
*** daneyon has joined #kolla08:23
*** zhiwei has joined #kolla08:24
*** mbound has joined #kolla08:26
*** sdake has quit IRC08:27
*** daneyon has quit IRC08:28
*** openstack has joined #kolla08:48
jmccarthySamYaple: Hmm is what is done for swift currently risky by default ? i.e. should there be iptables/firewalld/etc rules documented about it ? I saw your earlier comments about pulling tokens08:48
*** cemason has joined #kolla08:49
*** pbourke has quit IRC08:56
*** pbourke has joined #kolla08:57
SamYaplejmccarthy: yea documentation is best here09:01
SamYaplebasically if you can talk to memcache you have full access to al lcontents09:01
SamYapleso iptables/netowrk segregation either one09:01
SamYapleyou need that stuff09:01
*** exploreshaifali has joined #kolla09:03
SamYaplewelp sdake i was mistaken. i was running docker 1.8.209:04
SamYapledocker 1.8.3 is b0rked09:05
*** dims__ has joined #kolla09:08
*** dayou has quit IRC09:09
CBR09SamYaple: hi, you around?09:12
*** dims__ has quit IRC09:13
*** daneyon has joined #kolla09:17
SamYaplemaybe...09:20
SamYapleproblem CBR09?09:21
*** daneyon has quit IRC09:22
CBR09ah, some problem, but I found it09:25
CBR09same as https://bugs.launchpad.net/mos/+bug/139016809:25
openstackLaunchpad bug 1390168 in Mirantis OpenStack 6.1.x "nova-conductor failed to start while galera cluster is building after reboot" [High,Invalid] - Assigned to MOS Nova (mos-nova)09:25
CBR09nova-conductor can not connect galera09:27
SamYapleCBR09: sounds like mariadb isnt running09:28
CBR09problem when mariadb isn't running, nova-conductor trying establish connection09:31
SamYaplesure but thats to be expected09:31
SamYaplethings break when they cant reach the database09:31
CBR09and when mariadb is running, nova-conductor stop reconnecting mariadb09:31
CBR09just restart nova-conductor09:32
CBR09it's fix09:32
CBR09or change database.max_retries=-109:34
CBR09always reconnect09:35
*** openstackstatus has joined #kolla09:37
*** ChanServ sets mode: +v openstackstatus09:37
-openstackstatus- NOTICE: gerrit is undergoing an emergency restart to investigate load issues09:40
*** ChanServ changes topic to "gerrit is undergoing an emergency restart to investigate load issues"09:40
kjellySamYaple: Should I document "/etc/" in next patch ?  this is ansible problem, not kolla. https://review.openstack.org/#/c/232381/4/doc/ansible-deployment.rst09:41
CBR09should we put max_retries=-1 in nova.confg ?09:42
CBR09default it's 1009:42
*** zhiwei has quit IRC09:44
SamYapleCBR09: sounds like a great canidate for us to put in there for sure09:47
*** dims__ has joined #kolla09:49
CBR09Ok, I'm file a bug and fix it09:49
*** zhiwei has joined #kolla09:55
*** zhiwei has quit IRC10:00
*** openstackgerrit has quit IRC10:01
*** openstackgerrit has joined #kolla10:01
*** tfukushima has quit IRC10:04
*** diga has quit IRC10:11
*** daneyon has joined #kolla10:11
*** daneyon has quit IRC10:16
openstackgerritMerged openstack/kolla: Push images to registry by default in Vagrant  https://review.openstack.org/23394610:16
*** dims__ is now known as dims10:18
*** cemason has quit IRC10:20
*** shardy_ has joined #kolla10:20
*** shardy has quit IRC10:22
*** cemason has joined #kolla10:22
*** shardy_ has quit IRC10:25
*** shardy has joined #kolla10:26
openstackgerritPaul Bourke proposed openstack/kolla: Also copy over Swift builder files for rings  https://review.openstack.org/23411710:37
SamYaplepbourke: you around10:37
pbourkeSamYaple: yup10:37
SamYaplecool10:37
SamYaplecouple of this10:37
SamYapledocker 1.8.3 is b0rked with ansible10:37
SamYaplespecifically the ansible docker.py module10:38
SamYaplethey changed api returns without updating the versions10:38
SamYaplehuzzah10:38
SamYaplegot a patch up at ansible to fix it10:38
SamYaplehttps://github.com/ansible/ansible-modules-core/pull/225810:38
SamYapleso put some heat on it if you can and stay away from 1.8.3 if yo ucant10:39
SamYaplesecond thing is the uppercontainers patch10:39
SamYaplelets talk about that10:39
pbourkeare you familiar with the requirements repo?10:41
SamYaplevery10:41
SamYaplebut the problem i have is with upper contstraints10:41
pbourkeis my patch using it in an incorrect way?10:41
SamYapleif we used it right now all logging would be b0rked10:41
SamYaplenope upper constraints is10:41
pbourkeany idea what other projects do in this situation10:42
SamYapleignore it?10:42
pbourkeactually I think there's an exclusion mechanism?10:42
SamYaplethere is, but im not sure we want to use the upper constraints is my point10:43
SamYaplebecause we are building images in a point-in-time we should be alright not using them10:43
SamYaplejust tlaking here of course10:43
pbourkefor me its a step to make builds more repeatable and reliable10:44
SamYapleyea im not opposed to it (i left a comment to this affect) ive just got some reservations10:45
pbourkeyeah im just trying to explain my goal10:45
SamYaplewe have no expectation that liberty will build a year down the road10:45
SamYapledistro packages will break that10:45
SamYaplethe goal is repeatable reliable _deploys_ in my opinion10:45
SamYapleand we do that with images10:45
SamYapleso will this help or hinder that is what im getting out10:46
SamYaples/out/at/10:46
pbourkethis will help, let me say why10:46
SamYaplecool10:46
pbourkeyou're correct in that deploys should be the goal rather than builds. its quite likely the image will build with a newer version of dependency x10:46
pbourkebut it's also likely a runtime error can result from the newer dep10:46
pbourkewhich is exactly what I ran into yesterday10:47
pbourke(not on master but no reason it doesn't happen on master from time to time)10:47
pbourkethere's also a corporate tinge to this requirement in that legal/security types like to have a full list of versions that goes into a release10:48
SamYaplewell thats the thing10:48
SamYapleupper-constraints can change10:49
SamYaplein a release i mean10:49
SamYapleits not a hard cap for that release10:49
pbourkeright, but not if you check out a ref of it10:49
SamYaplewhich we should because it only changes to fix a bug10:49
pbourkeI didnt add that in this patch though as kolla always goes from master10:49
SamYaplechouldnt*10:49
SamYaplethe requirements project doesnt do tags10:49
SamYapleits master (or head of a branch) always10:50
SamYaplethats _always_ the right thing to pull and it does change from time to time10:50
pbourkeagain you can just specify a ref, or mirror the repo and only sync it when you need (which is what we do in oracle)10:50
pbourkeits up to the deployer, the thing is the option is there10:51
SamYapleactually its up to the builder in that case10:51
SamYapledeployer this problem doesnt affect them10:51
pbourkeright sorry that's what I meant10:51
pbourkeanyway, I do share the concern that this may break stuff, you called out syslog but there may be others. In which case maybe we should do it image by image10:52
SamYapleyea again I think this is the right move, but if we merge it right now it will break logging again (a patch we just reverted because its fixed upstream)10:52
SamYaplei just want to consider the implications of doing this10:52
*** inc0 has quit IRC10:53
openstackgerritSam Yaple proposed openstack/kolla: In gate functional testing  https://review.openstack.org/23399310:53
openstackgerritSam Yaple proposed openstack/kolla: Pin docker to 1.8.2 in gate  https://review.openstack.org/23412510:53
SamYaplefyi pbourke i updated https://review.openstack.org/#/c/233983/10:54
SamYapleits the issue with docker 1.8.2 and not the patch10:54
pbourkeSamYaple: got it, the fact that a deploy is a test in itself somehow escaped me!10:56
SamYaplehow do you pin packages in centos?10:56
SamYaplei need to pin docker in the gate10:56
openstackgerritPaul Bourke proposed openstack/kolla: Remove some outdated info from swift-guide  https://review.openstack.org/23412710:59
*** inc0 has joined #kolla11:01
*** daneyon has joined #kolla11:05
openstackgerritPaul Bourke proposed openstack/kolla: Add small reminder to use latest vagrant  https://review.openstack.org/23414711:07
*** daneyon has quit IRC11:10
*** cemason has quit IRC11:10
*** cemason has joined #kolla11:11
*** ChanServ changes topic to "Kolla IRC meetings on Wednesday - Agenda @ https://wiki.openstack.org/wiki/Meetings/Kolla - IRC channel is *LOGGED* @ http://eavesdrop.openstack.org/irclogs/%23kolla/"11:15
-openstackstatus- NOTICE: Gerrit has been restarted and is responding to normal load again.11:15
*** diogogmt has quit IRC11:16
*** cemason has quit IRC11:29
*** exploreshaifali has quit IRC11:32
*** rhallisey has joined #kolla11:35
openstackgerritVu Nguyen Duy (CBR09) proposed openstack/kolla: Add max_retries to nova.conf  https://review.openstack.org/23416011:36
openstackgerritVu Nguyen Duy (CBR09) proposed openstack/kolla: Add max_retries to nova.conf  https://review.openstack.org/23416011:36
*** dims has quit IRC11:47
*** dims has joined #kolla11:48
*** daneyon has joined #kolla11:59
*** daneyon has quit IRC12:04
*** CBR09 has quit IRC12:04
*** inc0 has quit IRC12:06
*** inc0 has joined #kolla12:07
*** achanda has joined #kolla12:32
*** achanda has quit IRC12:37
openstackgerritSam Yaple proposed openstack/kolla: In gate functional testing  https://review.openstack.org/23399312:45
openstackgerritSam Yaple proposed openstack/kolla: Bump ansible to 1.9.2  https://review.openstack.org/23398312:45
openstackgerritSam Yaple proposed openstack/kolla: Pin docker to 1.8.2 in gate  https://review.openstack.org/23412512:45
openstackgerritSam Yaple proposed openstack/kolla: Fix issue with su and bad kernels  https://review.openstack.org/23398412:45
openstackgerritMichal Rostecki proposed openstack/kolla: Update Vagrant docs  https://review.openstack.org/23337112:45
*** nihilifer has quit IRC12:53
*** daneyon has joined #kolla12:53
*** nihilifer has joined #kolla12:54
*** nihilifer has quit IRC12:55
*** inc0 has quit IRC12:57
*** nihilifer has joined #kolla12:57
*** daneyon has quit IRC12:58
*** jruano has joined #kolla13:01
*** mbound has quit IRC13:06
*** mbound has joined #kolla13:10
*** mbound has quit IRC13:11
*** achanda has joined #kolla13:11
*** mbound has joined #kolla13:11
*** klint has quit IRC13:17
*** achanda has quit IRC13:20
*** cemason has joined #kolla13:40
*** dwalsh has joined #kolla13:43
*** cemason has quit IRC13:45
*** jtriley has joined #kolla13:47
*** daneyon has joined #kolla13:48
*** cemason has joined #kolla13:48
SamYaplepbourke: do you know how to pin packages in centos?13:52
pbourkeSamYaple: not without googling unfortunately13:53
*** daneyon has quit IRC13:53
SamYapleok13:53
SamYaplegood news13:53
SamYaplegate worked!13:53
SamYaplehttps://jenkins03.openstack.org/job/gate-kolla-build-images-ubuntu-source/108/console13:53
SamYaple^^ successful in gate deploy with proper patches13:53
openstackgerritSam Yaple proposed openstack/kolla: In gate functional testing  https://review.openstack.org/23399313:58
openstackgerritSam Yaple proposed openstack/kolla: Pin docker to 1.8.2 in gate  https://review.openstack.org/23412513:58
SamYaplepbourke: figured it out13:58
SamYaplewere good for the gate at lesat13:58
*** jruano has quit IRC14:01
pbourkenice job14:04
*** jmccarthy has quit IRC14:08
*** jmccarthy has joined #kolla14:08
openstackgerritPaul Bourke proposed openstack/kolla: Add build profiles to build.py  https://review.openstack.org/23422914:09
*** dwalsh has quit IRC14:20
*** cemason has quit IRC14:20
openstackgerritJeff Peeler proposed openstack/kolla: Bump ansible to 1.9.2  https://review.openstack.org/23398314:29
*** achanda has joined #kolla14:30
*** diogogmt has joined #kolla14:34
*** dwalsh has joined #kolla14:37
*** achanda has quit IRC14:40
*** daneyon has joined #kolla14:42
*** daneyon has quit IRC14:47
*** vinkman has joined #kolla14:50
*** achanda has joined #kolla14:59
*** achanda has quit IRC15:04
*** dwalsh has quit IRC15:07
SamYaplejpeeler: you messed with my patch and all my otherones are outdated now :/15:10
jpeeleryeah i shouldn't have done it15:10
SamYaplenot a problem15:10
SamYapleill repush them all15:10
openstackgerritMerged openstack/kolla: Add small reminder to use latest vagrant  https://review.openstack.org/23414715:13
openstackgerritSam Yaple proposed openstack/kolla: In gate functional testing  https://review.openstack.org/23399315:14
openstackgerritSam Yaple proposed openstack/kolla: Bump ansible to 1.9.2  https://review.openstack.org/23398315:14
openstackgerritSam Yaple proposed openstack/kolla: Pin docker to 1.8.2 in gate  https://review.openstack.org/23412515:14
openstackgerritSam Yaple proposed openstack/kolla: Fix issue with su and bad kernels  https://review.openstack.org/23398415:14
*** akwasnie has quit IRC15:19
*** dwalsh has joined #kolla15:20
*** blahRus has joined #kolla15:21
*** CBR09 has joined #kolla15:26
CBR09hi pbourke, about max_retries, problem same as https://bugs.launchpad.net/mos/+bug/139016815:30
openstackLaunchpad bug 1390168 in Mirantis OpenStack 6.1.x "nova-conductor failed to start while galera cluster is building after reboot" [High,Invalid] - Assigned to MOS Nova (mos-nova)15:30
CBR09hi SamYaple, what does you mean?, sorry for my not really understand15:31
CBR09I can implement it for project-wide15:31
*** cloudnautique has quit IRC15:35
*** daneyon has joined #kolla15:36
*** juggler_ has joined #kolla15:36
*** bkosick has joined #kolla15:36
*** jruano has joined #kolla15:38
*** juggler_ has quit IRC15:40
*** juggler_ has joined #kolla15:40
*** daneyon has quit IRC15:41
*** juggler_ has quit IRC15:43
*** juggler_ has joined #kolla15:44
*** mbound has quit IRC15:44
*** juggler_ has quit IRC15:45
*** daneyon has joined #kolla15:47
*** daneyon_ has joined #kolla15:49
*** daneyon has quit IRC15:53
*** shardy_ has joined #kolla15:55
*** shardy has quit IRC15:56
*** jasonsb has joined #kolla15:58
*** bkosick has quit IRC15:59
*** shardy_ has quit IRC16:00
*** shardy has joined #kolla16:01
*** juggler_ has joined #kolla16:03
*** mbound has joined #kolla16:05
*** juggler_ has quit IRC16:07
*** cloudnautique has joined #kolla16:08
*** diga has joined #kolla16:09
*** juggler has joined #kolla16:11
*** jpeeler has quit IRC16:13
*** jpeeler has joined #kolla16:14
*** jpeeler has quit IRC16:16
*** jpeeler has joined #kolla16:16
*** unicell has joined #kolla16:21
*** mbound has quit IRC16:22
*** unicell1 has quit IRC16:23
*** juggler has left #kolla16:28
*** unicell has quit IRC16:28
*** diga has quit IRC16:34
*** dwalsh has quit IRC16:39
*** CBR09 has quit IRC16:55
*** unicell has joined #kolla17:02
*** achanda has joined #kolla17:14
*** sdake has joined #kolla17:25
*** sdake has quit IRC17:31
*** achanda has quit IRC17:31
*** sdake has joined #kolla17:36
*** achanda has joined #kolla17:48
*** sdake_ has joined #kolla17:52
*** sdake has quit IRC17:52
*** athomas has quit IRC18:06
*** jmccarthy has quit IRC18:07
*** shardy_ has joined #kolla18:12
*** shardy has quit IRC18:13
*** shardy_ has quit IRC18:18
*** shardy has joined #kolla18:18
*** dwalsh has joined #kolla18:20
sdake_samyaple ping me when yu get in plz18:22
*** achanda has quit IRC18:25
*** cloudnau_ has joined #kolla18:26
*** achanda has joined #kolla18:27
*** cloudnautique has quit IRC18:27
*** smeisner has quit IRC18:55
*** daneyon has joined #kolla18:56
*** daneyon_ has quit IRC19:00
*** daneyon has quit IRC19:00
*** smeisner has joined #kolla19:23
*** cloudnau_ has quit IRC19:23
*** cemason has joined #kolla19:29
*** cloudnautique has joined #kolla19:32
*** cloudnautique has quit IRC19:41
*** smeisner has quit IRC19:47
*** cloudnautique has joined #kolla19:48
*** sdake has joined #kolla19:52
*** sdake_ has quit IRC19:52
*** bkosick has joined #kolla20:03
*** bmace__ has quit IRC20:27
*** shardy_ has joined #kolla20:28
*** jtriley has quit IRC20:29
*** shardy has quit IRC20:29
*** shardy_ has quit IRC20:33
*** shardy has joined #kolla20:34
*** bkosick has quit IRC20:34
*** jmccarthy has joined #kolla20:37
*** jmccarthy has quit IRC20:39
*** exploreshaifali has joined #kolla20:55
*** tpot has joined #kolla21:05
*** cemason has quit IRC21:08
*** rhallisey has quit IRC21:09
*** cemason has joined #kolla21:12
*** gfidente has quit IRC21:15
*** bmace__ has joined #kolla21:44
*** daneyon has joined #kolla21:58
*** exploreshaifali has quit IRC21:59
*** jasonsb has quit IRC22:08
*** jasonsb has joined #kolla22:09
*** shardy_ has joined #kolla22:15
*** shardy has quit IRC22:17
*** cemason has quit IRC22:17
*** cemason has joined #kolla22:18
*** shardy_ has quit IRC22:21
*** shardy has joined #kolla22:22
*** tpot has quit IRC22:30
*** tpot has joined #kolla22:32
*** sdake has quit IRC22:41
*** daneyon has quit IRC22:41
*** tpot has quit IRC22:44
*** tpot has joined #kolla22:52
*** rhallisey has joined #kolla23:03
*** jruano has quit IRC23:07
*** dims has quit IRC23:41
*** dims has joined #kolla23:42
*** dwalsh has quit IRC23:42
*** tpot has quit IRC23:44
openstackgerritMerged openstack/kolla: Remove some outdated info from swift-guide  https://review.openstack.org/23412723:52
*** achanda has quit IRC23:53
openstackgerritMerged openstack/kolla: Install tox in Vagrant  https://review.openstack.org/23396023:54

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!