Wednesday, 2015-08-19

*** achanda_ has joined #kolla00:02
*** achanda has quit IRC00:05
*** achanda_ has quit IRC00:07
*** jasonsb has quit IRC00:10
*** jasonsb has joined #kolla00:11
*** banix has quit IRC00:19
*** rhallisey has quit IRC00:43
*** jasonsb has quit IRC00:43
*** jasonsb has joined #kolla00:44
*** dims has joined #kolla00:46
*** dims_ has quit IRC00:47
*** jasonsb has quit IRC00:48
*** dims has quit IRC00:56
*** dims has joined #kolla00:56
*** banix has joined #kolla01:11
*** weiyu has joined #kolla01:14
*** SourabhP has quit IRC01:17
*** SourabhP has joined #kolla01:17
*** erkules_ has joined #kolla01:23
*** erkules has quit IRC01:26
*** weiyu has quit IRC01:27
*** weiyu has joined #kolla01:30
*** banix has quit IRC01:32
*** Haomeng has joined #kolla01:46
*** britthouser has joined #kolla01:48
*** britthou_ has joined #kolla01:50
*** britthouser has quit IRC01:53
*** loth has quit IRC01:58
*** diogogmt has joined #kolla02:11
*** jasonsb has joined #kolla02:28
*** achanda has joined #kolla02:32
*** unicell has quit IRC02:33
*** Windir has quit IRC02:49
SamYaplepbourke: how do you mean? each hostvars?02:50
SamYaplelike a loop02:50
openstackgerritSam Yaple proposed stackforge/kolla: Remove start.yml abstraction for kolla-ansible  https://review.openstack.org/21360002:53
*** Windir has joined #kolla02:53
*** dims has quit IRC02:55
SamYaplesdake: ping02:56
*** weiyu has quit IRC03:18
openstackgerritSam Yaple proposed stackforge/kolla: Move defaults.yml to group_vars/all.yml  https://review.openstack.org/21396103:19
openstackgerritSam Yaple proposed stackforge/kolla: Move lots of global.yml options  https://review.openstack.org/21420003:19
*** weiyu has joined #kolla03:21
openstackgerritMerged stackforge/kolla: Cinder bootstrap container is not initializing the DB  https://review.openstack.org/21384503:22
*** diogogmt has quit IRC03:42
*** shardy_ has joined #kolla03:47
*** unicell has joined #kolla03:48
*** shardy has quit IRC03:49
*** diga has joined #kolla03:49
*** shardy_ has quit IRC03:52
*** shardy has joined #kolla03:53
openstackgerritSteven Dake proposed stackforge/kolla: Remove data containers and adjust compose files  https://review.openstack.org/21283004:00
openstackgerritSteven Dake proposed stackforge/kolla: Use one data container for all data volumes  https://review.openstack.org/21280404:00
sdakesamyaple shoot04:00
SamYapleglobals.yml and comments04:03
SamYaplejust so were clear, those comments are essentially docs04:03
SamYaplethat means they will probably get out of date04:03
SamYapleAdditionally I dont want to maintain docs in two locations for where options are in both globals.yml and all.yml04:04
sdakerebasing ftl04:06
sdakeagree they are docs04:08
sdakehopefully they dont need to change, except to improve04:08
SamYaplei agree, but lets please take that approach04:08
sdakei'm going to try to make sphinx publish our defaults04:08
sdakeas in the docs will come right out of etc/kolla/globals.yml04:09
sdakeother projects do it with ini files, should be possible with yaml04:09
*** weiyu has quit IRC04:17
*** diga has quit IRC04:26
sdakecoolsvap|away priority #1 is gnocchi templating so we can finish the job on the dockerfile-templates04:27
sdakethats the last container set needed04:27
sdakeyour signed up for it, if you want to pass it along please do so ;)04:27
sdakeok bedtime04:27
*** sdake has quit IRC04:29
*** diga has joined #kolla04:35
*** coolsvap|away is now known as coolsvap04:46
*** weiyu has joined #kolla04:53
*** coolsvap is now known as coolsvap|away05:08
*** weiyu has quit IRC05:16
openstackgerritSam Yaple proposed stackforge/kolla: Add Dockerfile template for kolla-ansible  https://review.openstack.org/21339405:20
*** coolsvap|away is now known as coolsvap05:23
openstackgerritSwapnil Kulkarni (coolsvap) proposed stackforge/kolla: Add Dockerfile template for Gnocchi  https://review.openstack.org/21445205:24
coolsvapSamYaple, can you have a quick look at http://www.slideshare.net/SwapnilKulkarni9/project-kolla-bof-at-containercon05:25
SamYaplecoolsvap: mission statement is outdated05:27
coolsvapI picked it up from the etherpad05:27
SamYapleeither way its outdated05:28
coolsvapcan you point me to the correct one? the review?05:29
SamYaplei dont really agree with the structure of that kolla architecture page but meh05:29
SamYaplehttps://review.openstack.org/#/c/209343/1/gerrit/projects.yaml05:29
SamYaplemultinode and single node are both done and have been for a while05:30
openstackgerritSam Yaple proposed stackforge/kolla: Remove barbican  https://review.openstack.org/21445305:31
coolsvapyes, but i we are still testing it, so just added it as in progress05:32
SamYaplei only mention it since you said single node was done05:32
SamYaplethere is no difference between the two05:32
coolsvaphmm, makes sense will update it a bit and i forgot to add the vagrant part05:33
openstackgerritSam Yaple proposed stackforge/kolla: Add Dockerfile template for nova  https://review.openstack.org/21347805:42
*** weiyu has joined #kolla06:01
*** coolsvap is now known as coolsvap|away06:10
*** britthou_ has quit IRC06:21
*** britthouser has joined #kolla06:22
openstackgerritSam Yaple proposed stackforge/kolla: Remove mysql-server from base  https://review.openstack.org/21447106:28
*** achanda has quit IRC06:34
*** SourabhP has joined #kolla06:37
*** SourabhP_ has joined #kolla06:40
*** SourabhP has quit IRC06:42
*** SourabhP_ is now known as SourabhP06:42
*** dwalsh has joined #kolla07:06
*** inc0 has joined #kolla07:07
inc0good morning07:08
inc0yay, docker 1.807:09
SamYaplehey inc0 you still working on logging?07:09
SamYaplei see the blueprints still in my name07:09
inc0we were to coop there, you've said you have ideas07:09
SamYapleyea im just swamped07:10
SamYaplebasically we just need an rsyslog container I believe07:10
inc0yeah, that's not a problem, do you want me to take over alltogether?07:10
SamYapleyea sure. i thought that was the plan when we talked about coop, you leading and all07:11
inc0misunderstaning then07:11
SamYapleno problem07:11
inc0anyway, I'll take over then07:12
SamYapleim hammering the ubunt ustuff out right now07:13
inc0ok07:14
inc0SamYaple, do we want to have syslog on each container? It seems wrong. How else do we want to solve this?07:18
inc0mount volume and log to volume, then read from volume in rsyslog container?07:19
SamYapleyea07:20
SamYaplethats what we had talked about I believe07:20
SamYaplebut no07:20
SamYapleI think we should have each service log directly to a remote syslog07:20
SamYapleor have the /dev/ log device shared07:20
SamYaplelots of options07:20
inc0yeah, let's pick one07:21
SamYaplesharing the dev device is probably the best choice07:21
SamYaplesince it would appear as a log device and things would write straight to it07:22
inc0ok, so we'll need to add this line to every ansible role right?07:22
*** dolpher has joined #kolla07:23
inc0anyway, I'll think this through07:23
inc0will start from rsyslog tho07:23
*** weiyu has quit IRC07:24
SamYapleyea we need an rsyslog container first07:24
SamYaplethen sharing the dev will require some thought07:24
*** weiyu_ has joined #kolla07:26
inc0brb07:29
*** SourabhP has quit IRC07:33
*** gfidente has joined #kolla07:35
*** dolpher1 has joined #kolla07:45
*** dolpher has quit IRC07:47
*** dwalsh has quit IRC08:05
*** jmccarthy has joined #kolla08:16
*** inc0 has quit IRC08:23
*** akwasnie has joined #kolla08:23
*** inc0 has joined #kolla08:43
*** achanda has joined #kolla08:48
pbourkeSamYaple: hey. not sure if you saw some of the conversation from earlier, I was basically trying to figure out how we can maintain host specific vars and so far it looks like hostvars would be the place to access that08:55
pbourkethe specific host vars I need are things like the list of devices on that host that should be added to the swift ring, the weights for each, etc.08:56
SamYaplepbourke: hmmm08:59
SamYaplei dont think thats such a good way t odo it08:59
pbourkeSamYaple: so far it's been a bit tricky but it may well be my lack of experience with ansible08:59
SamYaplethats the way OSAD does it, its... bad08:59
pbourkeid really welcome suggestions08:59
SamYaplewel lthe way i did ceph was dynamically08:59
pbourkeIm not familiar with ceph at all, did it have a similar situation?09:00
SamYapleyea im looking for the code09:00
SamYaplebasically iy has disks that need ot be mounted09:00
SamYapleit*09:00
SamYapleyea so i collect 'osds' as a variable09:01
SamYapleit has all initialized and unintiazed volumes09:01
pbourkeI was thinking Id still like to use the data vol for an 'out of the box' kind of setup09:02
SamYapleI wrote this module to do that09:02
SamYaplehttps://github.com/SamYaple/yaodu/blob/master/ansible/library/ceph_osd_list09:02
pbourkejust to help with demos/testing etc09:02
SamYaplei have no opinion on that09:02
SamYapleSwift won't ever be default so it doesnt have to be simple to setup09:02
SamYaplelike one click deploy simple i mena09:02
pbourkehmm09:04
SamYaplehere is what I dont want to see, manually specified disks for each thing in swift09:05
SamYaplethat might work with 5 nodes with 4 disks each, but try 100 nodes with 20 disks09:05
SamYapleits not a good plan09:05
pbourkethe thing is, do we need to allow for ops to specify that09:07
SamYapleyea this is a ops project. ops rules here09:07
SamYaplethat doesnt mean it can't be dynamic09:07
pbourkeyeah we need both09:08
SamYaplewhat do you mean both?09:08
SamYaplethe manually specified is not a good thing at all09:08
pbourkeso the values would be done dynamically but the option to override is there09:09
pbourkesomething like {{ hostvars[host][weight] | default(1) }}09:09
SamYapleIf the disks are looked up dynamically im not sure how the override could ever happen09:09
*** jmccarthy has quit IRC09:09
*** jmccarthy has joined #kolla09:09
pbourkethe disks are discovered dynamically but the values for each should be overridable09:10
pbourkeweight, zone, etc09:10
SamYapleLets take a step back for a second. Why are we trying to solve swift?09:10
SamYapleWhy not make the operator generate the rings and we just distribute?09:10
pbourkethat did cross my mind09:10
SamYapleThen this issue goes away and the operator can rely on official documentation09:10
SamYapleI can assure you we are OK with that too09:10
pbourkesdake kind of said he'd like it to work out of the box09:11
SamYapleIm sure he does09:11
SamYapleone click and all09:11
SamYaplebut thats not how this stuff works09:11
SamYapleits swift09:11
SamYapleits complicated and requires raw disks09:11
SamYapleit wont work 'out of the box'09:11
pbourkeyeah. maybe for the first iteration at least we dont have to solve this09:11
pbourkeI'll look again and see what I come up with over the next few hours09:12
pbourkeit sucks because you can't test without rings and they're such a manual process09:13
SamYaplebut its not on us ot fix that09:13
SamYaplethats a swift thing09:13
pbourkeagree09:13
SamYaplewhat if we do 'fix it' and swift goes a different direction? now we have this wierd non-standard thing09:13
*** jmccarthy has quit IRC09:17
*** achanda has quit IRC09:17
SamYapleyou know this problem goes away when you back swift with ceph :P09:18
*** dolpher1 has quit IRC09:35
*** jmccarthy has joined #kolla09:36
openstackgerritMerged stackforge/kolla: Use one data container for all data volumes  https://review.openstack.org/21280409:52
*** tfukushima has joined #kolla09:55
*** apmelton has quit IRC10:06
*** apmelton has joined #kolla10:09
pbourkeSamYaple: wrt https://review.openstack.org/#/c/214471/110:15
pbourkeI have a feeling some of the source images need the server package to build10:15
*** tfukushima has quit IRC10:15
pbourkemight be wrong10:15
*** tfukushima has joined #kolla10:16
SamYapleon the ubuntu side ther need python-mysqldb10:17
SamYapleon the centos side... idk?10:17
SamYaplebut either way this is bad since the mariadb repo isnt added until the mariadb container10:18
SamYapleim rewritting the ubuntu stuff so it actually works10:18
SamYaplepbourke: i put it as wip10:19
pbourkekk10:19
openstackgerritMerged stackforge/kolla: Move defaults.yml to group_vars/all.yml  https://review.openstack.org/21396110:25
openstackgerritMerged stackforge/kolla: Move lots of global.yml options  https://review.openstack.org/21420010:31
openstackgerritSam Yaple proposed stackforge/kolla: Remove defaults.yml from docs  https://review.openstack.org/21414710:35
openstackgerritSam Yaple proposed stackforge/kolla: Remove barbican  https://review.openstack.org/21445310:43
inc0SamYaple, remove barbican...lol10:53
SamYapleim a man of few words10:54
inc0+1 from me on that10:54
inc0anyone uses barbican anyway?10:54
inc0although10:56
*** dims has joined #kolla10:56
inc0after a thought...that looks like perfect contrib container for me10:56
SamYapleit doesnt work10:57
SamYapleit needs to be split into 4 containers10:57
SamYapleit never worked10:57
inc0whatever, code is there10:57
inc0don't throw it away10:57
inc0its contrib, it's just...there10:57
inc0use it on your own risk10:57
inc0maybe someone actually would like to fix that10:57
SamYapleBut i guess thats my point, if it never worked then what code is there is nothing like it should be10:58
inc0then let it go to contrib to die10:58
SamYapledod you know if you ^C and then hit enter in a docker contaienr it wil lsay 'Eh?'10:58
inc0oO11:00
*** dims_ has joined #kolla11:03
openstackgerritSam Yaple proposed stackforge/kolla: Fix small typo in task name  https://review.openstack.org/21455911:05
*** dims has quit IRC11:07
*** britthouser has quit IRC11:10
SamYaplepbourke: would you mind reviewing https://review.openstack.org/#/c/211442/11:15
*** weiyu_ has quit IRC11:17
SamYaplethanks man11:18
SamYapleAlso if you wouldn't mind changing your vote on https://review.openstack.org/#/c/213480/ since I do have that patch in place for docs11:19
openstackgerritRyan Hallisey proposed stackforge/kolla: Libvirt is moving around the wrong config file  https://review.openstack.org/21424111:24
*** dims has joined #kolla11:29
*** dims_ has quit IRC11:31
*** dims_ has joined #kolla11:31
pbourkeSamYaple: sure, but I dont see a DocImpact flag there?11:34
*** dims has quit IRC11:34
SamYaplesdake never set it up11:35
SamYaplethe patch is already +2+111:35
SamYaplejpeeler: ping11:40
*** tfukushima has quit IRC11:42
*** diogogmt has joined #kolla11:46
*** diga has quit IRC11:55
*** dwalsh has joined #kolla12:01
*** jmccarthy has quit IRC12:07
*** jmccarthy1 has joined #kolla12:07
*** jpeeler has quit IRC12:07
openstackgerritSam Yaple proposed stackforge/kolla: Remove Ansible abstraction for Neutron  https://review.openstack.org/21144212:17
*** diogogmt has quit IRC12:20
*** rhallisey has joined #kolla12:21
*** britthouser has joined #kolla12:28
openstackgerritMichal Jastrzebski (inc0) proposed stackforge/kolla: rsyslog container  https://review.openstack.org/21458912:33
inc0soo...now let's talk - what do with logs once they land there12:34
pbourkeinc0: it seems there's additional work before that discussion takes place?12:40
pbourkeinc0: as right now they're not landing anyway12:40
pbourkes/anyway/anywhere12:40
*** tfukushima has joined #kolla12:43
*** tfukushima has quit IRC12:48
inc0well, in containers /var/log/messages12:49
openstackgerritPaul Bourke proposed stackforge/kolla: Add ansible role for memcached  https://review.openstack.org/21459312:50
inc0anyway yeah, it's wip12:50
inc0we need to discuss what to do with logs and how to push logs there in the first place12:51
inc0also how to setup rest of logging stack (logstash, kibana...)12:51
pbourkeinc0: ah yeah. I would think most would just need the correct config settings to log to the remote rsyslogd12:51
inc0in every container12:51
inc0and ansible roles to point to correct rsyslog12:51
inc0this container is supposed to be this "remote rsyslog";)12:52
openstackgerritPaul Bourke proposed stackforge/kolla: Add ansible role for memcached  https://review.openstack.org/21459312:54
*** erkules_ is now known as erkules13:01
*** erkules has joined #kolla13:01
*** banix has joined #kolla13:14
*** dims_ has quit IRC13:29
*** tfukushima has joined #kolla13:29
*** dims has joined #kolla13:29
*** diogogmt has joined #kolla13:43
*** jpeeler has joined #kolla13:45
*** diogogmt has quit IRC13:47
jpeelerSamYaple: hey13:48
*** jtriley has joined #kolla13:48
*** weiyu has joined #kolla13:51
openstackgerritMerged stackforge/kolla: Libvirt is moving around the wrong config file  https://review.openstack.org/21424113:56
*** sdake has joined #kolla13:56
*** sdake_ has joined #kolla13:58
openstackgerritSam Yaple proposed stackforge/kolla: Fix up base, RabbitMQ, and MariaDB for Ubuntu  https://review.openstack.org/21462513:59
sdake_oh glorious 9 hours of sleep13:59
sdake_morning folks13:59
SamYaplesdake_: cool! now get to work13:59
* SamYaple cracks the whip14:00
sdake_samyaple have you tried rebuilding all your images from sratch14:00
sdake_i think it doesn't work if you do14:00
SamYaplesdake_: yea i mentioned neutron broke14:00
sdake_it being kolla14:00
SamYapleonly on the new images14:00
sdake_ya i fixed that neutron bug14:00
sdake_i am getting nms in the "ERROR" state14:00
sdake_on nva boot14:01
SamYapleive been hammering on ubuntu so i havent tried booting recently14:01
sdake_ok14:01
sdake_i'll see if i can sort it out today14:01
sdake_see if its a regressoin in our code or upstream rdo14:01
*** sdake has quit IRC14:01
openstackgerritSam Yaple proposed stackforge/kolla: Fix group name in vagrant docs  https://review.openstack.org/21395014:02
openstackgerritSam Yaple proposed stackforge/kolla: Update support to control  https://review.openstack.org/21348014:03
akwasniehi14:03
SamYaplehey akwasnie14:03
sdake_hey akwasnie14:03
akwasniesdake: https://bugs.launchpad.net/kolla/+bug/1485742 in this bug you want to replace all /opt/kolla occurences with /var/lib/kolla so that files like config-external will be copied to /var/lib/kolla ?14:04
openstackLaunchpad bug 1485742 in kolla "global search and replace /opt/kolla with /var/lib/kolla" [Critical,Confirmed] - Assigned to Alicja Kwasniewska (alicja-kwasniewska)14:04
sdake_akwasnie right14:05
akwasnieI can do it if u don't mind14:05
SamYaplesdake_: no14:05
SamYaplethose are configs14:05
SamYaplethose go in /etc14:05
SamYaplethe templates can go to /var/lib/kolla14:05
sdake_the tricky part is wha tto do about kolla-common.sh14:06
*** weiyu has quit IRC14:06
sdake_kolla-ocmmon.sh and config-external.sh should probably go in /usr/share/kolla14:06
SamYapleoh you mean the container stuff?14:06
SamYapleor the ansible dirs?14:06
sdake_welll basically we shouldn't be using /opt14:06
SamYapleagreed14:07
sdake_so we might as well get it right now ;)14:07
SamYaplebut what yare you talking about14:07
sdake_there is a bug i filed to search and replace /opt/kolla with /var/lib/kolla14:07
*** inc0 has quit IRC14:08
sdake_but i really link shell scripts should end in /usr/share/kolla14:08
sdake_think14:08
SamYapleOk, I have the Ansible directory changes (where it lays down configs) handled under the update-configs blueprint14:08
SamYaplethis bug is not related to what im talking about14:08
SamYapleI have no strong opinion on this14:08
*** bmace has joined #kolla14:09
sdake_i just think we should be following lsb14:09
sdake_otherwise people will think our implementation is "out of place"14:09
SamYaplewell that means /etc for configs14:09
sdake_the generated configs?14:09
sdake_for each node?14:09
SamYapleyup14:09
SamYaplethose arent required to be generated for one (this was in the spec even if we dont have al lthe code in place)14:10
sdake_those are more like persistent data i think14:10
sdake_because they are generated14:10
SamYapleread above14:10
SamYapleand for COPY_ALWAYS those configs are designed to change14:10
sdake_ok let me wake up a bit14:10
sdake_and then we can discuss14:10
SamYapleFor those of you who care about Ubuntu, weigh in now https://review.openstack.org/#/c/214625/14:12
sdake_here is the ops meetup etherpad14:13
sdake_https://etherpad.openstack.org/p/PAO-ops-meetup14:13
*** sdake_ is now known as sdake14:13
*** jtriley has quit IRC14:13
sdakehttps://etherpad.openstack.org/p/PAO-ops-containers-for-deployment14:13
sdakesamyaple that is worth a read14:15
sdakeabove14:15
sdakerequirements in there14:15
sdakeit's what the cool kids are doing14:15
sdakelol14:15
*** thumpba has joined #kolla14:15
SamYaple'data outside the container'14:18
SamYapletold you ops would want that14:18
sdakewell duh14:18
SamYapleHa you say that now and yet I have logs of you fighting me!14:18
sdakei'm pretty sure i prposed that in the ansible-multi spe ;)14:18
sdakenot fighting grilling so icould write that spec14:19
sdakeiirc14:19
SamYapleafter some prodding....14:19
sdakequestion14:19
SamYaplenot how I recall it but I also dont care so proceed14:19
sdakewhat nova containers require privilege14:19
SamYaplethe ones that have "privileged: True"14:19
sdakeyes14:19
SamYaple:)14:19
sdakeyou mean look at the code14:19
sdakeso tired14:20
SamYaplertfm14:20
sdakei am pretty sure nova-compute doesn't require privileged container14:21
SamYapleyou can say that, but youd be wrong!14:21
SamYapleloading modules requires CAPS14:21
SamYaplewe can isolate said CAPs, but I would focus on dropping to a user before I do that14:22
SamYaplein M I will be pushing al lthe containers to drop privileges and run as unpriveleged in the containers14:22
SamYapleand hopefullly Isolating caps14:22
SamYapleI spent about a month on yaodu doing just that14:22
SamYaplesdake: https://review.openstack.org/#/c/211442/ please merge that in before I have t orebase AGAIN14:23
SamYapleI have had t orebase 5 patched :(14:23
*** absubram has joined #kolla14:23
SamYaplepbourke: let me know if you need any ansible helps with memcached/swift. I can provide guidance should you need it14:27
pbourkeSamYaple: thanks. I think I'm going to submit a WIP in the next hour or two just without the ring generation14:27
pbourkeget feedback at least on what's there14:27
sdakeya rebasing sucks14:27
SamYapleOk guys I am heading out for a bit. be back for the team meeting14:29
SamYapleIf anyone has an desire to influence Ubuntu, please look at https://review.openstack.org/#/c/214625/14:29
SamYapleIf not I am going t odo things my way!14:29
sdakewhat is missing today which would get you on the container path?14:31
sdakeTime14:31
sdakekernel bugs14:31
sdakeunprivilaged containers14:31
sdakeless industry hype, more best practices14:31
sdakewe will have  to work on the unprivileged c ontainers in mitaka14:32
sdakeidentify minimum set of capacibiltiies to get the job done14:33
sdakei still dont think nova-compute  rquires privileged but my system doesn't run so  Ican verify !14:33
openstackgerritMerged stackforge/kolla: Remove Ansible abstraction for Neutron  https://review.openstack.org/21144214:34
*** jtriley has joined #kolla14:39
*** sdake_ has joined #kolla14:41
sdake_damn out of disk space14:43
sdake_i bet that is why nova isn'tworking14:43
*** sdake has quit IRC14:44
sdake_samyaple can you confirm docker-py 1.1.0 wont work for us?14:49
sdake_for building14:49
sdake_that is what is in the global-requirements.txt file14:49
*** diga has joined #kolla14:49
SamYapledocker 1.1.0 wont work14:51
sdake_ok14:51
SamYaple1.2.0 is minimum14:51
sdake_which featuere is needed (so I can put in commit log)14:51
sdake_so i can get the global requirements updated14:51
SamYaplesdake_: nova-compute _does_ require privilieged because it needs to load modules14:51
sdake_so we can get a requirements sync job14:52
*** blahRus has joined #kolla14:53
openstackgerritSteven Dake proposed stackforge/kolla: The requirements.txt file isn't correct  https://review.openstack.org/21339814:53
sdake_samyaple do you know which api feature is needed from docker 1.2.0?14:53
SamYaplesdake_: oh sorry14:53
SamYaplei see you say for building14:53
SamYaplei think were fine for building at 1.1.014:53
SamYaplei think14:53
SamYapledunno14:54
*** diogogmt has joined #kolla14:54
sdake_might as well try to get the job updated14:54
*** dwalsh has quit IRC14:54
SamYaplemight need 1.2.0 to support docker 1.7.0 (which we dont need)14:54
sdake_rather get the requirement updated14:54
sdake_docker.1.7.0 is needed to bindmount /dev iiuc14:54
SamYapleyea but the reason that supposedly worked was taken out before 1.7.014:55
SamYapleBut im not pushing it14:55
SamYaplewere going to be requiring the newest docker soon enough I think14:55
sdake_ok so back on topic :) I want to update global rquirements to docker 1.2.014:55
sdake_what is my argument14:55
sdake_(for the commit log)14:55
sdake_if you dont know that is a fine answer, i'll make something up14:56
sdake_just looking for anything to help along the review14:56
*** pbourke has quit IRC14:57
*** pbourke has joined #kolla14:57
SamYaplewait globals has an upperlimit on the docker-py?14:58
sdake_no a lower limit14:58
sdake_upper limits have been removed14:58
SamYapleoh thats fine leave that alone14:58
openstackgerritMerged stackforge/kolla: Add oslo_versionedobjects dependency for neutron  https://review.openstack.org/21436214:59
sdake_so what will happen when i set up the global sync job14:59
sdake_is our requirements.txt will be changed by a script to docker-py > 1.1.014:59
SamYaplei know14:59
SamYaplethats fine for building14:59
sdake_ok i'll change our requirements.txt to 11.0 then15:00
openstackgerritSteven Dake proposed stackforge/kolla: The requirements.txt file isn't correct  https://review.openstack.org/21339815:00
SamYaplewont the update job do that for us?15:00
SamYapleproposalbot15:01
sdake_once i have it in place15:01
sdake_its not enabled atm15:01
*** dims has quit IRC15:01
sdake_only so many hours in the day15:01
sdake_i also dont know how to enable it15:01
*** dims has joined #kolla15:01
SamYaplehar har har15:01
openstackgerritSwapnil Kulkarni (coolsvap) proposed stackforge/kolla: Add Dockerfile template for Gnocchi  https://review.openstack.org/21445215:02
sdake_i can figure itout15:02
sdake_coolsvap|away thanks bro :)15:02
openstackgerritSteven Dake proposed stackforge/kolla: The requirements.txt file isn't correct  https://review.openstack.org/21339815:03
pbourkehas anyone seen issues with mariadb since the data container change?15:03
pbourkesdake_: ^15:03
SamYaplepbourke: define issues15:03
SamYaplethere are some tricky bits surrounding that15:04
sdake_pbourke you must rebuild all your images15:04
SamYaplepermissions wise15:04
pbourkehttp://paste.openstack.org/show/421959/15:04
pbourkeyeah permissions15:04
sdake_sorry about the rebuid requirement15:04
pbourkeno worries15:04
sdake_its in the ocmmit log15:04
SamYaplepbourke: yea we rm -rf the /var/lib/mysql folder15:04
SamYapleor at least teh contents15:04
sdake_no docker mounts /var/lib/mysql as root15:04
SamYapleanyway a rebuild and youll be fine15:04
SamYaplesdake_: not quite15:05
*** dims_ has joined #kolla15:05
SamYaplebut end result is the same15:05
sdake_pbourke the code in ansible is incompatible with the old images15:05
sdake_it sucks but  it is what it is15:05
sdake_i'd like to stop making these breaking changes15:05
sdake_and i thiink we are almost there, once we get /run in data containers15:05
SamYaplethats unlikely to happen...15:05
sdake_samyaple you may be right15:06
SamYaplerun in a data container im not so keen on, but I can come around on the subject15:06
*** dims has quit IRC15:06
SamYaplemaybe15:06
SamYaplewell discuss not in a position to atm15:06
sdake_samyaple i'm not even sure if it would work15:06
sdake_its more like a research project for someone15:07
SamYapleit would work for most things15:07
sdake_probably not for libvirt15:07
SamYaplemy issue is the /run default socket kinda blocks other apps from starting15:07
sdake_i have a bug or blueprint up for it15:07
sdake_which default socket15:07
SamYaplelike you could have two different libvirt or openvswitch things going with different sockets, but the same kernel15:07
SamYaplethat would be ba15:07
sdake_ya libvirt needs to share /run host os because it communicates with systemd15:08
sdake_there is no way around that from what i can tell15:08
SamYapleThere are15:08
SamYaplebut i dont know that this is worth it15:08
sdake_you mean bindmount the specific run stuff needed by libvirt15:08
SamYapleno, but that would work too15:09
sdake_rather then the whole directory15:09
* sdake_ wtb someone to fix the keepalive pid bug15:09
SamYaplewhat bug?15:09
sdake_https://bugs.launchpad.net/kolla/+bug/148507915:14
openstackLaunchpad bug 1485079 in kolla "if /run/keepalived.pid file exists, keepalive wont start" [Critical,Confirmed]15:14
SamYapledo you not properly stop your keepalived container?15:15
SamYapleif not then youre leaving your vip aroun15:15
SamYapled*15:15
SamYapleThis should be a check to cleanup though15:15
SamYaplestill you shoudl stop the container rather than forcibly removing it15:16
sdake_ok well if the container crashes for example15:16
sdake_same outcome15:16
SamYapleyea but thats not why you hit that bug ;)15:16
sdake_keepalived is busted, when you createa pid file yoursupposed to run fcntl to lock it15:16
sdake_then if the process dies he lock si removed15:17
sdake_if the file is llocked then you dont start, not if it exists15:17
SamYaplethat doesnt work when you sigkill the process15:17
SamYapleand thats what you are doing when you remove the container like that15:17
sdake_sigkilll doesn't remove the lock on the file?15:17
SamYaplenot that I am aware15:18
*** coolsvap|away is now known as coolsvap15:18
SamYaplethis is why all the daemons have a check/cleanup15:18
sdake_how shoudl i cleanup between runs15:18
sdake_if not container-cleanup -fv/15:18
sdake_do we have a script for that?15:19
SamYapleyea i dont use that cleanup script cause it bad15:19
SamYaplebut I can fix it up for you15:19
sdake_something to get me going would be fantastic ;)15:19
SamYaplestop the container, then remove it15:19
*** dims_ has quit IRC15:19
*** dims has joined #kolla15:19
coolsvapsdake_, hi15:19
coolsvapplease have a look at http://www.slideshare.net/SwapnilKulkarni9/project-kolla-bof-at-containercon15:19
sdake_sometimes i  forget to rm the /run/keepalived pid file15:19
sdake_and then all kinds of bad things happen15:19
SamYaplewell if yo ustop the cotnainer, the file gets removed properly15:19
SamYapleI run this beast:15:20
SamYapleecho kolla_ansible log_data heat_{api{,_cfn},engine} mariadb{,_data} rabbitmq{,_data} keystone glance_{api,registry,data} nova_{data,scheduler,novncproxy,consoleauth,conductor,api,compute,libvirt} neutron_{server,agents,linuxbridge_agent,openvswitch_agent} openvswitch_{vswitchd,db} haproxy keepalived | xargs -P100 -n1 sh -c 'docker stop $1; docker rm -v -f $1' sh15:20
SamYaplethats one line btw15:20
sdake_samyaple nice can you put that in clenaup-containers -fv ;)15:20
sdake_rathre the clenaup-containers script15:20
SamYaplecan you file a bug and assign it to me?15:20
sdake_of course15:21
SamYapleI will do any bugs you assign to me15:21
*** coolsvap is now known as coolsvap|away15:22
*** daneyon has joined #kolla15:22
SamYaplehey sdake_ did you email your docker contact? we still dont have that namespace propgation in master15:22
pbourkestill getting permission issues on rabbitmq15:23
sdake_i talked to him15:23
sdake_and he was keen to help us out15:24
pbourkeI've rebuilt and 'docker rm' ed everything15:24
SamYaplepbourke: I have tested rabbitmq as working15:24
sdake_pbourke ugh15:24
SamYapleis it folder permissions?15:24
pbourkeSamYaple: how did you know it was rabbit :)15:24
SamYaple15:24 < pbourke> still getting permission issues on rabbitmq15:25
sdake_i wish i was young and could remember everything everyone says :)15:25
SamYaplei dont remember anything15:25
SamYapleI have logs and grep15:25
sdake_pbourke try a build with --no-cache15:25
sdake_pbourke areyou building -t binary?15:25
SamYapleI grep across all my IRC channels when I have an issue ;)15:25
pbourkeI tend not to use the build.py as it builds everything15:26
pbourkei'll try no-cache15:26
sdake_-t binary works15:26
*** daneyon_ has quit IRC15:26
SamYapleI built against the templates fyi15:26
sdake_not sure about -t source, tbh i didn't test oracle linux with -t source15:26
sdake_maybe i missed something with -t source15:26
SamYaplerabbitmq source and binary are teh same15:27
sdake_cool so the templates are actually working?15:27
SamYapleits all ive been using15:27
sdake_have you been able to nova boot?15:27
SamYaplehavent tested recent15:27
SamYaplea couple days ago15:27
SamYaplefyi two main templates left15:27
sdake_yesterday on master i was unable to nova boot, but i may had an out of disk space problem15:28
SamYaplehttps://review.openstack.org/#/c/213394/15:28
SamYaplehttps://review.openstack.org/#/c/213478/15:28
sdake_reviewing15:28
jpeelershould there be a when check for enable_haproxy here? https://github.com/stackforge/kolla/blob/master/ansible/roles/mariadb/tasks/register.yml#L1215:28
sdake_git clone ansible.git so evil ;)15:28
SamYaplejpeeler: that is used as a check of sorts15:29
SamYapleto ensure the database is active15:29
SamYaplethat user is not given any privleges at all15:29
SamYaplewe can change the username to health_check probably15:30
sdake_gate is maddening - need --retry asap!!15:30
SamYaplebtw guys. read https://github.com/docker/docker/blob/c4d45b6a29a91f2fb5d7a51ac36572f2a9b295c6/docs/extend/plugins_volume.md15:31
*** inc0 has joined #kolla15:35
*** britthouser has quit IRC15:35
SamYaplewhat time is meeting today?15:36
inc0in 30 mins Sam15:36
SamYaplethought so15:37
SamYaplewe still ahvent updated the time to 163015:37
sdake_i am sending out an announcement after this meeting for an a adjust in the scheduling15:37
SamYapleslacker15:38
sdake_i'll talk more about it at our meeting today15:38
inc0ah...damn I hoped it wont move before I move;)15:38
sdake_i want to remove the 3pm pst meeting too15:38
SamYapleyea that seems like a must15:38
SamYapleunfortunately15:38
sdake_not remove,but move to 9:3015:38
SamYapleGMT man15:38
SamYapleUTC15:38
sdake_ya exepcted apac contribs, and mandre was one, but he moved to france15:38
sdake_everyone that contribs is in europe/us15:39
inc0cool, I won't miss half of meetings15:39
inc0guys...about logging15:40
inc0are you comfortable with having logs on volume?15:40
SamYapleinc0: we talked a bit about this already. Where they spit out to a volume (which can be bind-mounted to the host)15:41
SamYapleso the logs are accessible from the host or a single container15:41
SamYaplesetup in a way we can forward them to a logstash type server too15:42
SamYaplebut thats not a requirement initially15:42
inc0we will need them to be accessible from other containers as well15:42
inc0either way, volume works15:42
inc0I'll do this in separate, ansiblish patch15:42
SamYapleyea I dont know that I agree with teh other containers part15:42
SamYaplethe way we talked about the containers themseleves wouldn't even have the log since its spit to a syslog container15:43
*** dwalsh has joined #kolla15:43
SamYapleright?15:43
inc0yeah, I meant containers that can parse files15:43
SamYapleoh yea well that would be done via forwarding15:43
inc0although if we'll push it to logstash, that would be redundant15:43
sdake_all logs go to data container, logstash mounts data container and pushes to one place15:44
inc0so we do data container instead of volume?15:45
SamYaplesdake_: no thats not going to work15:45
sdake_lets discuss in meeting so there is a meeting log heh? :)15:45
SamYaplethe logs go from the application to an rsyslog container15:45
inc0yeah, and rsyslog pushes it to logstash container15:45
inc0and elasticsearch container uses this logstash stuff15:46
sdake_rsyslog container is good for me - as long as it ends up with all the files as well :)15:46
inc0we'll need to make it work ofc15:46
sdake_lets discuss in meeting15:46
inc0kk15:46
sdake_thanks to whoever updated the meeting agenda15:49
* sdake_ been slacking on that15:49
*** dims has quit IRC15:49
*** sdake_ is now known as sdake15:49
*** dims has joined #kolla15:49
SamYapleslacker15:51
*** dwalsh has quit IRC15:52
inc0hmm there is logstash on dockerhub15:52
inc0I'll give it a try, and with a bit of luck we will get this stuff for free15:53
*** dims_ has joined #kolla15:53
inc0and just configure ansible/rsyslog to use it15:53
sdakei'd prefer any dependencies be in our repo (logstash)15:54
SamYapleinc0: well logstash wont be included by default15:54
SamYaplesdake: it depends on if its an official repo15:54
sdakemeeting in 5 minutes - lets discuss then :)15:54
*** dims has quit IRC15:54
openstackgerritPaul Bourke proposed stackforge/kolla: Fix missing line break in memcached template  https://review.openstack.org/21468315:54
SamYapleif logstash is pumping out an official container, I would like to use that15:54
inc0yeah, let's not create brave new wheel15:55
SamYaplebut only if its offical15:55
jmccarthy1sdake: Hmm I'm bit concerned about swift bugs #1477993 and #1477990 .. I've noticed there is more involved in #1477993 than I first thought .. it's just under two weeks left in L3 is it ? What's up for discussion this meeting ?15:55
openstackbug 1477993 in kolla "kolla bp/swift-container: Fix up supporting swift container functionality" [Critical,Triaged] https://launchpad.net/bugs/1477993 - Assigned to James McCarthy (jmccarthy)15:55
openstackbug 1477990 in kolla "kolla bp/swift-container: Method for bind mounts with swift-data" [Critical,Triaged] https://launchpad.net/bugs/1477990 - Assigned to James McCarthy (jmccarthy)15:55
sdakethe problem is distro15:55
sdakeif the distro used to build is ubuntu and its running on centos i don't have high confidence it will work ;)15:56
sdakemeeting in 4 minutees, we can discuss all these things jmccarthy115:56
jmccarthy1Okie15:56
SamYaplesdake: if its a self contained app that talks over the network, it will work15:56
sdakewould like in official log15:56
SamYaplebut in this case it sint15:56
SamYapleanyway we dont have to worry about that now15:56
pbourkesdake: did you have a bug logged for oraclelinux template builds?15:57
*** jmccarthy1 has quit IRC15:57
sdakepbourke no15:57
sdakecould you file one15:57
pbourkesdake: kk will file now15:57
pbourkehave patch ready15:57
*** jmccarthy has joined #kolla15:57
sdakenice15:57
pbourkethis is just for binary, will test source next15:58
openstackgerritPaul Bourke proposed stackforge/kolla: Fix binary builds for oraclelinux templates  https://review.openstack.org/21468715:58
openstackgerritMerged stackforge/kolla: Add Dockerfile template for kolla-ansible  https://review.openstack.org/21339416:01
openstackgerritMerged stackforge/kolla: Remove barbican  https://review.openstack.org/21445316:01
sdakemeeting time16:02
*** stvnoyes has joined #kolla16:04
*** daneyon_ has joined #kolla16:05
*** daneyon has quit IRC16:08
*** jtriley has quit IRC16:17
*** jtriley has joined #kolla16:20
*** unicell has quit IRC16:23
*** thumpba has quit IRC16:32
*** dims_ has quit IRC16:39
*** dims has joined #kolla16:39
openstackgerritRyan Hallisey proposed stackforge/kolla: Ceph container  https://review.openstack.org/21470616:43
*** SourabhP has joined #kolla16:48
*** erkules_ has joined #kolla16:50
*** SourabhP_ has joined #kolla16:50
*** SourabhP has quit IRC16:52
*** SourabhP_ is now known as SourabhP16:52
*** erkules has quit IRC16:53
*** erkules has joined #kolla16:55
*** unicell has joined #kolla16:55
*** erkules_ has quit IRC16:56
* rhallisey gets lunch17:00
harmwo/17:00
inc0https://etherpad.openstack.org/p/kolla-logging-stack I'll also put link to bp17:00
harmwgreat inc017:00
inc0tomorrow I'll write something there17:00
*** britthouser has joined #kolla17:00
inc0hopefully17:00
sdakeinc0 sounds fantastic - appreciate the help here :)17:01
inc0imho this is critical17:01
inc0for people to appreciate kolla17:01
inc0I mean, ops will love this17:02
*** diga has quit IRC17:02
harmwdef. inc017:02
inc0I'm out for tonight, please, if you have any ideas feel free to put these to etherpad17:02
inc0I'll do the same tomorrow17:03
inc0have a good day guys, cyas17:04
*** unicell has quit IRC17:04
harmw\o/17:04
*** inc0 has quit IRC17:04
*** unicell has joined #kolla17:04
sdakepbourke rebuilding now with master and more disk sspace, i'll let you know if i can get  a deploy17:11
pbourkesdake: I think I was missing building the new data container17:11
pbourkesdake: building now and I reckon it will work17:11
sdakeoh that would be a problem ;)17:11
pbourkesdake: strange though that it tried to start at all17:11
pbourkelike all came up just the services that used data containers went into a restarting state17:12
sdakethat would explaint he rabbitmq permissions problem ;)17:12
pbourkeok better run! btw i was chatting to SamYaple and he doesn't think we should solve the generation of swift rings in ansible17:13
pbourkemaybe we could, but it probably shouldn't hold back this blueprint17:13
sdakei htink we need to solve it17:14
sdakebut it can be in rc117:14
sdakesuggest making a new blueprint for swift-ring-generation and putting in  rc117:14
pbourkei need to take a better look at what ops are currently doing to generate these rings17:14
sdakebecause without it, swift is unable17:14
sdakeunusable17:14
pbourkeyeah :(17:15
harmwguys, anyone knows a decent openstack-running cloudprovider?17:15
sdakerax is good17:15
sdakelittle pricey17:15
harmwto expensive17:15
harmwtoo, that is17:15
sdakebut has best deployment that is public17:15
harmwI was thinking of cloudvps17:16
sdakehavne't used them17:16
harmwwhich happens to be NL-based17:16
harmwbut they don't utilize much of the openstack eco17:16
jmccarthyGotta jet also, catch you guys next time !17:16
harmwo/17:16
*** sdake_ has joined #kolla17:17
*** SourabhP has quit IRC17:17
*** jmccarthy has quit IRC17:17
harmwdigitalocean is using some homecooked piece, right17:18
harmwhpcloud is also expensive17:19
harmwhm, dreamhost, ofcourse...17:20
*** sdake has quit IRC17:20
sdake_i only have experience with rackspace17:26
sdake_and it is expensive :)17:26
sdake_but otherwise rock solid17:26
*** britthouser has quit IRC17:27
*** britthouser has joined #kolla17:28
*** loth has joined #kolla17:35
*** sdake has joined #kolla17:37
*** sdake_ has quit IRC17:40
*** achanda has joined #kolla17:42
*** sdake has quit IRC17:47
*** sdake has joined #kolla17:51
*** SourabhP has joined #kolla17:52
*** achanda_ has joined #kolla17:54
*** dwalsh has joined #kolla17:54
*** achanda has quit IRC17:55
*** dwalsh has quit IRC18:02
harmwever been to FOSDEM sdake ?18:07
sdakeno harmw18:13
harmwok18:14
*** tfukushima has quit IRC18:33
*** banix has quit IRC18:33
*** tfukushima has joined #kolla18:42
*** jasonsb has quit IRC18:43
*** jasonsb has joined #kolla18:44
*** tfukushima has quit IRC18:46
*** cfarquhar has joined #kolla18:47
*** cfarquhar has quit IRC18:47
*** cfarquhar has joined #kolla18:47
*** jasonsb has quit IRC18:48
*** banix has joined #kolla18:53
*** akwasnie has quit IRC19:09
*** jasonsb has joined #kolla19:25
*** gfidente has quit IRC19:34
*** achanda_ has quit IRC19:37
*** shardy_ has joined #kolla19:41
*** shardy has quit IRC19:42
*** shardy_ has quit IRC19:46
*** shardy has joined #kolla19:47
*** achanda has joined #kolla19:51
vincent_vdkyou should definitely come to FOSDEM20:29
*** achanda has quit IRC20:47
*** achanda has joined #kolla20:52
*** achanda has quit IRC21:06
openstackgerritimain proposed stackforge/kolla: Add an environment variable to allow args to be set on startup.  https://review.openstack.org/21479921:15
*** shardy has quit IRC21:31
*** rhallisey has quit IRC21:40
*** jtriley has quit IRC21:44
*** unicell has quit IRC21:45
openstackgerritWilliam Frischemeier proposed stackforge/kolla: Load ip_vs module for keepalived  https://review.openstack.org/21481121:45
*** unicell has joined #kolla21:45
*** absubram has quit IRC22:03
*** unicell has quit IRC22:12
*** unicell has joined #kolla22:12
*** loth1 has joined #kolla22:46
*** loth has quit IRC22:48
*** jasonsb has quit IRC22:55
*** britthouser has quit IRC22:56
*** stvnoyes has quit IRC23:02
*** banix has quit IRC23:02
*** diogogmt has quit IRC23:03
*** tfukushima has joined #kolla23:03
*** banix has joined #kolla23:31
*** achanda has joined #kolla23:33
*** dims_ has joined #kolla23:33
*** dims has quit IRC23:36
*** dims_ has quit IRC23:39
*** dims has joined #kolla23:39
*** dims has quit IRC23:44
*** loth1 is now known as loth23:46
sdakesamyaple if you appear23:47
sdakesamyaple I can't install from master23:47
sdakeglance fails with some wierd ansible spam trace23:48
*** blahRus has quit IRC23:52
*** Haomeng|2 has joined #kolla23:55
*** Haomeng has quit IRC23:59
*** jasonsb has joined #kolla23:59

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!