Friday, 2018-01-12

klindgren_no it doesn't00:00
SamYapleare you saying they are not both active at the same time so it doesnt?00:00
klindgren_you top the existing rpc service, you start the new container.  As long as hostname is set in the config file its fine00:00
klindgren_stop*00:00
SamYapleok concern relieved there00:00
SamYaplei think, with the rename stuff, this could work for kolla-ansible without as much work as i initially stated, yes00:01
klindgren_the rpc flow is test rpc connectivity, stop _active container - start _new container if start fails.... remove _new container, start _active00:01
SamYaplei thought, at first, you wanted multiple active old and new on the same host, but that workflow seems good00:01
klindgren_no - only api containers are active/active for some period.00:02
SamYaplewell that cant work because kolla-ansible uses host networking00:02
SamYapleyoud have two apis on the same ip:port00:02
klindgren_some period == amount of time to test new api node, and time to drain connections to existing running api container.00:02
klindgren_it works because we use a static set of ip's that we map between api containers00:03
klindgren_in a different network00:03
SamYaplei mean for kolla-ansible00:03
klindgren_which the local haproxy is configured to use.00:03
SamYapleoh. hmm. youre saying there will be another layer of haproxy in the mix here? i thought you meant doing the draining at the main haproxy container00:04
klindgren_so you just configure the container services to listing on 0.0.0.0 so when you map in a new ip, the service just works on that ip automatically00:04
klindgren_So - the way we currently do it is every api node is fronted by a local haproxy00:05
klindgren_so we run all the api servers on the same set of servers.00:05
SamYapleone of kollas design principles is "dont touch the host", so swapping in new ips wont work. the other is a limiting access so it doesnt use 0.0.0.0 (anywhere that can be avoided)00:05
SamYaplethe haproxy per api host is a tougher sell in my opinion00:06
SamYaplebut its doable00:06
klindgren_why wont new ip's work?00:06
klindgren_its jsut another docker network00:06
klindgren_that you create?00:06
SamYaplewe dont use docker networks00:06
SamYaplehost-networking only00:06
klindgren_so you are exposing ports in docker?00:07
SamYapleno00:08
SamYaplethe container sees allthe host interfaces00:08
SamYapledocker is unaware of anything related to kolla-networking00:08
klindgren_you mean docker-networking?00:11
klindgren_sorry - I guess I jsut dont see how using docker networks breaks the tenant of "dont touch the host"?00:12
SamYapleklindgren_: no thats a different thing00:13
SamYaplei said that because i thought you meant add ips to the host interfaces00:13
SamYaplekolla-ansible doesnt use docker networking for two reasons00:13
SamYaplefirst, back inthe day when kolla came out, docker networking was dirt slow00:13
SamYaplesecond, there is a bug with iscsi and network namespaces and qemu that breaks iscsi containers00:14
SamYapleerrr nova-compute containers00:14
SamYapleanyway, thepoint is that kolla-ansible doesnt use container networking, it uses host networking00:14
klindgren_so the way that I use docker networks, its just as fast as host networking and the docker binary is not inline for any network traffic. 2.) so your not willing to do a much better container rollout strategy because of a single edge case in a single service?  which isn't even an API service, so it could use host-networking anyway?00:17
SamYapleklindgren_: i left kolla-ansible almost 2 years ago. im not blocking anything00:18
SamYapleim just letting you know what you canexpect00:18
klindgren_kk00:19
*** jtriley has quit IRC00:19
kolla-slack<inc0> I think it's good idea but as Sam said it would have to be done on haproxy00:25
SamYapledont listen to him, he wont be PTL much longer00:26
kolla-slack<inc0> Ha ha ha...00:26
kolla-slack<inc0>  /slowlaugh00:26
SamYapleinc0: using container networking in kolla-ansible instead of host-networking... possible?00:26
kolla-slack<inc0> Uh not now00:26
inc0I can't think of compelling reason to do it00:27
SamYapleinc0: klindgren_ was saying thats how this method works00:28
SamYaplei feel most of what klindgren_ wants to do can be done.... with the exception of running multiple active api hosts00:28
*** jascott1 has quit IRC00:28
inc0well...multiple active api hosts on same host00:28
*** jascott1 has joined #openstack-kolla00:29
inc0and same port00:29
SamYapleyep00:29
inc0you could modify default port and deal with it on haproxy00:29
inc0I think it would be easier to handle than rewiring our networking00:29
inc0not to mention that docker networking itself is well...uh.00:29
inc0maybe just for API nodes?00:29
SamYaplebut at that point, without docker networking, local haproxy isnt needed since we can do that on the other haproxy00:29
*** yangyapeng has quit IRC00:30
inc0well you can do this on regular haproxy too assuming different port00:30
SamYapleweve been talking about it for year, maybe klindgren_ can finally implemetn something00:30
inc0you'll manage backends dynamically00:30
*** yangyapeng has joined #openstack-kolla00:30
inc0well on that note, using some networking underlay like calico or weave (but ansible driven, not k8s) might also be something worth looking at00:31
klindgren_its possible to not do active/active api's - thats jsut how I was able to do it00:31
klindgren_I jsut want some agreement on the general idea.00:31
inc0well active/active apis on same node is not possible today00:31
klindgren_since I already have code to bring - I was hoping I could get started on a spec and start trying to get some stuff landed.00:32
SamYaplei dont think youll get opposition on the end goal, just the imeplmentation like using local haproxy and docker networking00:32
klindgren_its possbile if not using host networking.00:32
inc0totally, but let's disect idea and split it into smaller chunks of functionality00:32
inc0and fwiw I think APIs could be exempt from our host-network-only policy00:33
klindgren_so I would push for running local haproxy, because if people use real lb's with dsr for loadblaancing - you have no way without local haproxy of maitnaenancing a node00:33
klindgren_been there - done that.00:33
inc0it *has* to be there for stuff like neutron agents00:33
inc0but other than that...it was just easy to handle and that was written in time when docker network was less than ideal00:33
*** jascott1 has quit IRC00:33
inc0so ad "check if API is alive" we had this sanity test tasks00:34
inc0that did pretty much what you explained00:34
*** yangyapeng has quit IRC00:34
inc0https://github.com/openstack/kolla-ansible/blob/d474987ad9722c63bf6bf964ca80d810b89a313e/ansible/roles/glance/tasks/check.yml00:35
inc0but it wasn't ever finished00:35
SamYaplesanity checks told me everything i needed to know about you inc000:36
*** jtriley has joined #openstack-kolla00:36
inc0don't blame me to use words that I'm familiar with...intimately00:37
*** k_mouza has quit IRC00:43
*** tovin07_ has joined #openstack-kolla00:45
inc0anyway, I'm off for 2 weeks00:49
inc0see you all aroind!00:50
inc0I should be reachable by mail00:50
*** inc0 has quit IRC00:53
*** pengdake has joined #openstack-kolla00:56
*** zhaochao has joined #openstack-kolla00:56
*** threestrands has quit IRC00:58
*** k_mouza has joined #openstack-kolla00:58
*** itlinux has joined #openstack-kolla01:01
*** ljjjustin has joined #openstack-kolla01:02
*** threestrands has joined #openstack-kolla01:02
*** k_mouza has quit IRC01:02
*** threestrands has quit IRC01:03
*** threestrands has joined #openstack-kolla01:04
*** mdnadeem has joined #openstack-kolla01:10
*** threestrands has quit IRC01:11
sbezverkkfox1111: ping01:18
*** yangyapeng has joined #openstack-kolla01:21
*** numans has quit IRC01:24
*** threestrands has joined #openstack-kolla01:24
*** threestrands has quit IRC01:24
*** threestrands has joined #openstack-kolla01:24
*** numans has joined #openstack-kolla01:25
*** threestrands has quit IRC01:25
*** threestrands has joined #openstack-kolla01:26
*** numans has quit IRC01:29
*** jascott1 has joined #openstack-kolla01:30
*** caowei has joined #openstack-kolla01:30
*** numans has joined #openstack-kolla01:35
*** jtriley has quit IRC01:38
*** duonghq has joined #openstack-kolla01:43
*** zhurong has joined #openstack-kolla01:45
*** dave-mccowan has joined #openstack-kolla01:46
duonghqmorning guys01:46
kiennt26duonghq: morning :)01:48
*** tone_z has joined #openstack-kolla01:49
duonghqmorning kiennt2601:50
*** threestrands has quit IRC01:54
*** mm6021 has quit IRC01:56
*** threestrands has joined #openstack-kolla01:56
*** hnl has joined #openstack-kolla02:16
*** gkadam has joined #openstack-kolla02:18
*** jianghuaw_ has quit IRC02:29
*** jianghuaw has joined #openstack-kolla02:29
*** unicell has quit IRC02:35
openstackgerritMerged openstack/kolla-ansible stable/ocata: Config my_ip to api_interface_address in cinder.conf  https://review.openstack.org/51360602:41
*** gkadam has quit IRC02:42
*** chmarkus has quit IRC02:58
*** chmarkus has joined #openstack-kolla02:59
*** numans has quit IRC03:05
*** numans has joined #openstack-kolla03:07
*** itlinux has quit IRC03:19
*** ljjjustin has quit IRC03:24
*** dave-mccowan has quit IRC03:33
*** larsks has quit IRC03:56
*** larsks has joined #openstack-kolla03:56
*** zhurong has quit IRC04:03
*** unicell has joined #openstack-kolla04:07
*** goldyfruit has quit IRC04:13
*** itlinux has joined #openstack-kolla04:32
*** caowei has quit IRC04:43
*** absubram has joined #openstack-kolla05:23
*** absubram has quit IRC05:23
*** Herbert_Guan has joined #openstack-kolla05:32
*** caowei has joined #openstack-kolla05:33
*** gkadam has joined #openstack-kolla05:34
*** zhurong has joined #openstack-kolla05:42
*** aqsa has joined #openstack-kolla05:45
aqsaHi all while im trying to deploy multinode i find that the images which got built are 6.0.0 and when i try to pull the images they fail with the error 404 saying cannot find image <image_name>5.0.205:47
aqsaany help or any idea why im using a stable pike branch05:47
Jeffrey4laqsa, configure the "openstack_release" variable to the proper tag name in your env.05:48
aqsawhy? Note: Im using a stable/pike branch **05:48
Jeffrey4laqsa,  where images are you using?05:48
aqsaook in the globals.yml right..05:48
aqsaJeffrey4l: i just used this  `kolla-build --base centos --type source --registry 192.168.1.61:4000 --push `05:49
aqsaim using `--base ubuntu ` sorry05:50
Jeffrey4laqsa, where the images come from?05:50
aqsaas per my understanding we first build images right  using `kolla-build --base centos --type source --registry 192.168.1.61:4000 --push`05:51
aqsai have not used kolla tar ball05:52
Jeffrey4laqsa, you need check what's the image tag really is.05:52
aqsaim presuming they come from docker hub right05:52
aqsaJeffrey4l: so how do i check it ..05:52
Jeffrey4lno, you built it.05:52
Jeffrey4lshow me the "docker images" on you node ran "kolla-build"05:53
aqsahttps://hastebin.com/junayizazu.vbs I deleted all the images and i just built nova images and attached is the output05:53
Jeffrey4laqsa, you can the TAG is 6.0.0, so check the "openstack_release" in /etc/kolla/globals.yml file to "6.0.0", then try to deploy again.05:54
aqsayes so now i have to set the #openstack_release: "" option to 6.0.0 in the globals.yml05:56
aqsaif im not wrong to avoid that error05:56
Jeffrey4laqsa, technically, it is not. before we can not guess what's the image tag name on the kolla-ansible side.05:58
aqsaJeffrey4l: now how do i do it ??06:13
aqsaI mean how can i fix this ??06:13
*** hnl has quit IRC06:20
*** hnl has joined #openstack-kolla06:21
Jeffrey4laqsa, you can the TAG is 6.0.0, so check the "openstack_release" in /etc/kolla/globals.yml file to "6.0.0", then try to deploy again.06:26
*** absubram has joined #openstack-kolla06:26
*** harlowja has quit IRC06:35
*** hnl has quit IRC06:42
*** gkadam has quit IRC06:45
*** hnl has joined #openstack-kolla06:46
*** jbadiapa has joined #openstack-kolla06:56
*** hnl has quit IRC06:57
*** daidv has quit IRC06:57
*** duonghq has quit IRC06:57
*** daidv has joined #openstack-kolla06:58
*** duonghq has joined #openstack-kolla06:58
*** hnl has joined #openstack-kolla06:58
openstackgerritDuong Ha-Quang proposed openstack/kolla-ansible master: Check enable_cinder before deploy iscsi and tgtd  https://review.openstack.org/53309906:58
*** jascott1 has quit IRC06:58
*** jascott1 has joined #openstack-kolla06:58
*** hieulq has quit IRC06:58
*** kiennt26 has quit IRC06:58
*** tovin07_ has quit IRC06:58
*** hieulq has joined #openstack-kolla06:59
*** tovin07_ has joined #openstack-kolla06:59
*** kiennt26 has joined #openstack-kolla06:59
*** harlowja has joined #openstack-kolla07:10
*** harlowja has quit IRC07:10
*** feldmann has joined #openstack-kolla07:14
*** itlinux has quit IRC07:20
*** Angeh has joined #openstack-kolla07:24
*** pcaruana has joined #openstack-kolla07:29
openstackgerritMerged openstack/kolla-ansible stable/pike: Fix dhcp_agent configuration for vmware DVS  https://review.openstack.org/53169107:31
*** cah_link has joined #openstack-kolla07:32
*** egonzalez has joined #openstack-kolla07:44
*** rajivk has joined #openstack-kolla07:45
*** rajivk has left #openstack-kolla07:45
openstackgerritXinliang Liu proposed openstack/kolla master: Consume RDO packaged mariadb version  https://review.openstack.org/46863207:47
*** Mr_Broken has joined #openstack-kolla07:48
*** unicell has quit IRC07:51
openstackgerritZhaokun Fu proposed openstack/kolla-ansible master: FIX inject password error  https://review.openstack.org/53194907:55
egonzalezaqsa, if have image tags 6.0.0, means you builded queens (master branch) images instead of stable/pike07:56
*** threestrands has quit IRC07:56
egonzalezfrom stable/pike would be 5.0.*07:56
egonzalezaqsa, if have internet access from all hosts and not have any custom change in images, not build them. Use dockerhub images07:56
*** cah_link has quit IRC08:05
*** gkadam has joined #openstack-kolla08:10
*** zhangfei has joined #openstack-kolla08:13
duonghqJeffrey4l, ping08:13
duonghqJeffrey4l, about this patch: https://review.openstack.org/#/c/533099/1 I left ironic untouch, just fix cinder-related logic08:14
openstackgerritDuong Ha-Quang proposed openstack/kolla-ansible master: Check enable_cinder before deploy iscsi and tgtd  https://review.openstack.org/53309908:15
*** wojdec has joined #openstack-kolla08:15
*** shardy has joined #openstack-kolla08:24
*** unicell has joined #openstack-kolla08:24
Jeffrey4legonzalez, duonghq could you review this patch https://review.openstack.org/508407 thx.08:28
*** wojdec1 has joined #openstack-kolla08:30
*** wojdec has quit IRC08:31
openstackgerritMerged openstack/kolla-ansible master: Ensure docker running before execute the destroy action  https://review.openstack.org/53168808:32
*** knsahm has joined #openstack-kolla08:33
*** wojdec has joined #openstack-kolla08:33
*** wojdec1 has quit IRC08:36
*** lvdombrkr has joined #openstack-kolla08:44
*** k_mouza has joined #openstack-kolla09:10
egonzalezare there free tickets for PTG to contributors?09:11
*** lvdombrkr89 has joined #openstack-kolla09:13
*** lvdombrkr has quit IRC09:15
*** wojdec has quit IRC09:18
*** wojdec has joined #openstack-kolla09:19
*** zhurong has quit IRC09:20
jianghuawegonzalez: I have the same question:-) At least it should be free for cores (like you) who contributed a lot.09:21
hrwtone_z: elo09:21
hrwmorning09:21
egonzalezjianghuaw, if request travel support program, maybe can only request just the ticket09:21
jianghuawegonzalez, don't know if the support program also include the ticket.09:23
hrwegonzalez: good point. no mails about it compared to previous events09:23
egonzalezjianghuaw, includes the ticket, iirc last time i got rejected they sent me a free code ticket09:23
*** robbbe has joined #openstack-kolla09:24
*** feldmann has quit IRC09:24
jianghuawok. The ticket costs much less than a full travel support. If allow to request just the ticket, it can cover more people.09:25
hrwI had discount for Barcelona, Boston and Sydney summits. Did not used any as I did not went there09:25
jianghuaw:-)09:26
hrwjianghuaw: I do not have a luck with CfP09:26
jianghuawhrw: CfP?09:26
hrwCall for Papers09:27
jianghuawah. got it:-)09:27
hrwjianghuaw: was co-speaker for Barcelona and Sydney talks. Both were not accepted.09:27
jianghuawhrw, wish you'll have a good luck for the next summit.09:28
hrw;d09:28
* hrw off09:28
egonzalezhrw, talk title should be something like "Stop using OpenStack and move to k8s" in order to get accepted09:28
*** dougsz has joined #openstack-kolla09:28
kolla-slack<aslancimbom100> That's actually funny :slightly_smiling_face:09:29
jianghuawegonzalez, lol09:29
*** aqsa has quit IRC09:29
*** feldmann has joined #openstack-kolla09:30
*** Angeh has quit IRC09:33
jianghuawegonzalez, the variable of xenapi_facts_file defined at: https://review.openstack.org/#/c/531640/7/etc/kolla/globals.yml@41509:33
jianghuawwill be used by both nova and neutron.09:33
jianghuawhttps://review.openstack.org/#/c/525999/16/ansible/roles/neutron/tasks/config.yml@15109:33
jianghuawhttps://review.openstack.org/#/c/531640/7/ansible/roles/nova/tasks/bootstrap_xenapi.yml@809:34
*** jascott1 has quit IRC09:35
*** jascott1 has joined #openstack-kolla09:35
egonzalezjianghuaw, better put it in group_vars (shared for various roles) but not that generic as globals.yml09:36
*** duonghq has quit IRC09:36
jianghuawegonzalez, Agreed. Surely will rework it accordingly. thanks.09:37
*** jascott1 has quit IRC09:39
openstackgerritJianghua Wang proposed openstack/kolla-ansible master: XenAPI: Support compute node bootstrap  https://review.openstack.org/53164009:44
openstackgerritJianghua Wang proposed openstack/kolla-ansible master: XenAPI: Support neutron configuration  https://review.openstack.org/52599909:44
egonzalezPTG confirmed o/09:45
jianghuawegonzalez, confirmed for a free ticket?09:46
egonzalezpaying :(09:46
jianghuaw:(09:46
jianghuawegonzalez, ^reworked the 1st patch and rebased the 2nd one.09:47
*** k_mouza_ has joined #openstack-kolla09:48
*** k_mouza has quit IRC09:48
*** wojdec1 has joined #openstack-kolla09:50
jianghuawberendt, I see you've +2 on the first one. Thanks very much:-)09:51
*** wojdec has quit IRC09:52
jianghuawberendt, What's the benefit for the additional ml2_conf_xenapi.ini?09:55
jianghuawhttps://review.openstack.org/#/c/525999/17/ansible/roles/neutron/tasks/config.yml@15809:55
berendtjianghuaw: yes, i think it makes no sense, you can use a separate folder for a xen host09:56
hrwegonzalez: k8s is now that mandatory word?09:56
egonzalezhrw, been in last 3 summits09:57
berendtthis way you can inject ml2 parameters for xen hosts and non xen hosts without mixing them in a global ml2 config09:57
*** gfidente has joined #openstack-kolla09:57
*** gfidente has quit IRC09:57
*** gfidente has joined #openstack-kolla09:57
hrwegonzalez: shit. that's why my talk for devconf.cz got rejected... too old buzzwords used09:57
jianghuawberendt, got your point. Thanks.09:57
hrwI had containers, docker, openstack in title but no k8s09:57
berendthrw: lol09:58
berendtjianghuaw: but i thinks an additional custom ml2_conf_xen.ini is not required, changed to +209:59
jianghuawberendt, see it. Thanks very much.10:00
hrwberendt: "Kolla - deploying OpenStack using containers"10:01
berendtsounds very unspectacular10:02
berendtbut I'm really bad at writing good titles and abstracts10:03
*** kevinz has quit IRC10:05
*** numans has quit IRC10:05
hrwso do I10:06
*** pengdake has quit IRC10:06
*** numans has joined #openstack-kolla10:07
jianghuawberendt, will you attend the Dublin PTG?10:07
*** wojdec has joined #openstack-kolla10:07
*** kiennt26 has quit IRC10:07
*** wojdec1 has quit IRC10:09
*** daidv has quit IRC10:10
*** tovin07_ has quit IRC10:11
*** nathharp has joined #openstack-kolla10:12
*** pbourke has quit IRC10:16
*** pbourke has joined #openstack-kolla10:18
*** sambetts|afk is now known as sambetts10:23
*** ktibi has joined #openstack-kolla10:23
ktibiHi kolla :)10:24
hrwhi ktibi10:24
ktibiany news about kubernetes in kolla ?10:25
ktibiI guess not for Q release.10:25
sean-k-mooneyktibi: k8s in kolla or kolla-kubernetes?10:37
*** jascott1 has joined #openstack-kolla10:38
sean-k-mooneyktibi: i think queens should be the first production release fo kolla-kubernets at least that was a cycle goal10:38
*** caowei has quit IRC10:38
ktibisean-k-mooney, kolla-kubernetes yes10:38
ktibiho great :)10:39
sean-k-mooneyktibi: as i said i dont know if that goal was achived but that was the intent at least10:40
sean-k-mooneykolla-ansible had a couple of cycles head start so its more feature complete and stable but the k8s port has been comming along nicely too10:41
*** serlex has joined #openstack-kolla10:49
*** shardy has quit IRC10:50
*** tone_z has quit IRC10:51
*** shardy has joined #openstack-kolla10:52
*** pabbi has joined #openstack-kolla10:54
sean-k-mooneyegonzalez: pbourke o/11:04
sean-k-mooneyegonzalez: pbourke QQ if you are using images form the docker hub and you want to do an minor upgrade to you just do a pull then upgrade since the tag (pike) is the same11:06
egonzalezsean-k-mooney, deploy or reconfigure11:06
egonzalezupgrade will do db syncs which are not needed11:06
*** k_mouza_ has quit IRC11:10
*** k_mouza has joined #openstack-kolla11:12
sean-k-mooneyegonzalez: oh ok so is upgrade only needed for major version updates then?11:16
*** zhangfei has quit IRC11:22
egonzalezsean-k-mooney, yep11:23
*** pabbi has quit IRC11:30
*** nathharp has quit IRC11:34
*** zhangfei has joined #openstack-kolla11:39
*** lpetrut has joined #openstack-kolla11:47
*** nathharp has joined #openstack-kolla11:59
chmarkusis it possible to limit the 'kolla-ansible deploy' command for an existing deployment to a specific service? Let's say I want to redeploy a removed barbican container in an existing installation without having ansible going/skipping through all the existing services?12:08
sean-k-mooneychmarkus: you can use --tags for that12:09
sean-k-mooneyit will still check if all the other roles should run but since the tag wont be in the list it will skip them exept for the tags you set so --tags barbican12:10
*** semoac has quit IRC12:11
chmarkussean-k-mooney, beautiful! That did the trick - thanks a bunch!12:14
chmarkusI didn't find such tags in the playbook roles' *.yml files, is this some implicit ansible magic?12:15
hrwsean-k-mooney: thanks for hint12:15
sean-k-mooneychmarkus: no its done at the top level playbook here https://github.com/openstack/kolla-ansible/blob/master/ansible/site.yml#L75 when the role is imported12:17
*** Zophar has quit IRC12:17
sean-k-mooneya couple of things like precheck or gather fact are not tagged or set to tags:always as then are assumed to have run always before the other roles but all roles that actully change things on the system are tagged12:18
chmarkussean-k-mooney, I see. Thanks for the link and the explanation - that is very helpful :)12:19
sean-k-mooneychmarkus: if we didnt compose the playbooks this way your other option would have been to create your own playbook and just import just the role you want to run12:20
sean-k-mooneykolla-ansible deploy just translates to ansible-playbook -e action=deploy -e@/etc/kolla/globals.yml ansible/site.yml12:21
sean-k-mooneywhen you add -i or --tags to kolla ansible deploy they just get added on to the ansible-playbook command12:22
*** Herbert_Guan has quit IRC12:28
*** janki has joined #openstack-kolla12:41
*** nathharp has quit IRC12:50
*** nathharp has joined #openstack-kolla12:51
*** dave-mccowan has joined #openstack-kolla12:54
*** goldyfruit has joined #openstack-kolla13:03
*** goldyfruit has quit IRC13:05
ktibiI can see prometheus image in kolla, anyone works on the integration of prometheus in kolla-ansible ?13:08
*** absubram has quit IRC13:11
egonzalezktibi, WIP https://review.openstack.org/#/q/project:openstack/kolla-ansible+prometheus13:13
*** goldyfruit has joined #openstack-kolla13:14
*** goldyfruit has quit IRC13:19
*** numans has quit IRC13:20
*** numans has joined #openstack-kolla13:20
*** yangyapeng has quit IRC13:32
*** feldmann has quit IRC13:49
*** david-lyle has quit IRC13:49
*** egonzalez has quit IRC13:49
*** spsurya has quit IRC13:53
*** yangyapeng has joined #openstack-kolla14:02
*** kbaegis has joined #openstack-kolla14:07
kbaegisMorning gents14:08
kbaegisI had a question14:08
kbaegisI'm using ceph-ext in my environment and I was wondering if there was a convenient method to get logging and mairadb mapped to ceph rbds via kolla14:09
kbaegissomething like appending -v /mnt/logs/:/var/log to each of the services14:09
sean-k-mooneykbaegis: i just deployed the same congiuration and have been thinking about the same problem14:10
sean-k-mooneythe short answer is yes you can change the volumes14:10
sean-k-mooneythe mariadb volumns are defined here https://github.com/openstack/kolla-ansible/blob/master/ansible/roles/mariadb/defaults/main.yml#L10-L1414:11
sean-k-mooneyin your globals.yml on in your inventory you can override mariadb_services:{mariadb:{volumes:[]}}}14:12
sean-k-mooneythe syntax is wrong but you get the idea14:12
sean-k-mooneyto make this work with external ceph you either need to use a cephfs fileshare per db or have systemd script automount the rbd volumn on system boot before docker starts14:14
sean-k-mooneykbaegis: this conficuration is not offically supported by kolla-ansible however and i have not tested it personally but that is how i was considering moving the the centralised logs to be stored in ceph14:15
kbaegissean-k-mooney: Yeah, it would certainly be easier14:20
*** goldyfruit has joined #openstack-kolla14:21
kbaegisfstab runs before docker, correct?14:21
sean-k-mooneykbaegis: im sure people would be interested in a way to do this automatically in kolla-ansible in the future so if you do go this route and automate it it would be a nice addtion14:22
sean-k-mooneyyes so if you go with cephfs then you can use that to auto mount it14:22
sean-k-mooney* fstab14:22
sean-k-mooneyyou cant use rbd volumes directly from fstab as far as i know as it only deals with filesystems not block devices hence systemd script with before docker.target set14:23
kbaegisWell- let's see.14:25
kbaegisYou got your ceph rbd module, then it populates dev via the mapper file14:25
kbaegisYeah, you might very well be right. You'd need an explicit init dependency for docker on ceph14:26
sean-k-mooneyi have never had much luck using the rbdmapper service so i just write my own14:26
kbaegisBut the rbd shouldn't care about the contents of fstab. It's just a module and a generic nod14:26
kbaegisAhh14:26
kbaegisYeah, I'm just doing a rbd map operation14:27
kbaegisDoes cephfs even have ckfs yet?14:27
sean-k-mooneyam not sure. i deployed it at home and had a 10% perfromance hit vs maping an rbd volume so i jsut stuck with rbd14:28
sean-k-mooneyi have not tried it on real server hardware14:28
kbaegisOuch. Yeah, and I thought cephfs still had jumbo 'experimental' label because there was a risk of losing your metadata contents and not being able to recover14:29
sean-k-mooneycephfs is easier to use though if you can make use of the added utility of having a shared network filesystem14:29
kbaegismy entire ceph setup is wonky. I'm using a partition rather than an entire device. I know that's not best practice, but it's working fine and I don't know the specific reason14:30
sean-k-mooneykbaegis: i think luminous allows active active metadata services now so not sure if that has been resolved14:30
kbaegisUnless I/O gets pegged by the system, in which case it's a cluster and can stand to move out/down a couple of nodes anyway14:30
sean-k-mooneykbaegis: ya same.  although i deployed a new one yesterday which is nolonger doing that14:31
kbaegisWhat's the penalty to using a partition? I only have one disk currently per blade14:31
*** knsahm has quit IRC14:31
kbaegisOne of my more junior guys wiped the OS installs on accident because he was unaware that ceph would gobble up the entire drive by default14:32
kbaegisHence ceph-external14:32
*** numans has quit IRC14:32
sean-k-mooneymy old developer cloud use 1 ssd for os + osd 2 journal  + cache osd data + 1 hdd for osd data per node14:32
sean-k-mooneyit worked well for the last 14 ish months14:32
sean-k-mooneyhowere i realsed because i was useing earsure encoded pools with a cache teir with replica count 3 i was writing all data 8.8 times total14:33
sean-k-mooney3 cache journals + 3 caches + 1.4 x for ec pool journal +1.4x for ec pool backing hdds14:34
kbaegisYeah, erasure kinda sucks imo. Gobbles up too much cpu, and I have shared storage/compute14:34
sean-k-mooneyso every 1BG writen was 7.4 to the ssds14:34
kbaegisIt's super handy though because you can control replicas on a per-pool basis, so you just configure cinder with different drivers pointing to discrete pools14:35
kbaegisNot that xor is too expensive.14:35
sean-k-mooneythe down side of haveing the ssd have the journsal + cache +os was when that wright heavy workload stated to casue them to fail it took the entire node with it14:36
kbaegisDid you have cascading failures, or was a node loss not such a big deal?14:36
*** janki has quit IRC14:36
sean-k-mooneywell i was doing a convered openstack deployment so the node was both an openstack controler, ceph mon node, openstack networking node and a ceph data node so not fun14:37
sean-k-mooneyi manganged to catch it before the drive completely failed and a simple dd of the drive to a spare ssd was all that was needed to fix it14:38
sean-k-mooneybuy  ya on erasuer pools i like to use them for nova vms and cinder volumes/backups and then put glance on an ssd-primary replicated pool14:39
sean-k-mooneyssd-primarly meaning 1 replica on ssd +2 on hdd14:39
kbaegisYeah, I'd be doing that if I had a dedicated ceph cluster14:39
kbaegisAs it stands, it's way cheaper to do a combined compute/storage pool14:40
sean-k-mooneywell i combine them too14:40
kbaegisThere's a cpu toll and latency added when using erasure isn't there?14:40
sean-k-mooneymy newest deployment has 3 dedicated servers for openstack contol/network group and they are my ceph mons too. then all the computes are mixed ceph osd + compute node servers14:41
kolla-slack<aslancimbom100> what are you using to monitor them?14:42
*** athomas has quit IRC14:42
sean-k-mooneykbaegis: well one i work at intel so lets just say cpu capasity is not really a problem the servers im useing have 2 10 or 14 core cpus on the older systesm and i have a couple of 2X 22 core systems for compute nodes14:42
*** athomas has joined #openstack-kolla14:43
sean-k-mooneyaslancimbom100 currently myself unfortunetly. i am hoping ot use collectd + aodh+ gnocchi in the futre but also looking at collectd+influx+grapana14:45
kolla-slack<aslancimbom100> i see, I had ceph-dash but no alerting configured14:47
kolla-slack<aslancimbom100> Haven't tried ceph mgr yet14:47
kbaegisOh, hey that's cool. So there's a bridge from irc into slack?14:48
sean-k-mooneykbaegis: yep14:48
glycerineUnder ceph luminous, I get a warning about "application not enabled for pools" Would I be correct in setting my pools (images, volumes, backups, vms) to 'rgw'14:49
sean-k-mooneyi think it was set up to help people working on kolla-kubernetes easilly talk to the k8s dev team that work only on slack14:49
sean-k-mooneyglycerine: no i dont think that would be correct14:49
glycerineI have these options  'cephfs', 'rbd', 'rgw', or freeform14:50
kbaegisWell, slack is just about as secure as IRC so :)14:50
sean-k-mooneykbaegis: it was not about security just ease of use. that is what k8s project chose to use14:51
sean-k-mooneyglycerine: i would say rbd would be the better choice but maybe openstack would also make sense14:53
sean-k-mooneyglycerine: http://ceph.com/community/new-luminous-pool-tags/14:54
glycerinethanks I'll read up on it14:55
kbaegis<<Just learning about luminous14:55
sean-k-mooneyvolumes,backups and vms are all block devices so should be rbd and image is an object store style usecase so maybe rgw but rdb would also work.14:56
kbaegisBlueStore sounds neat. I'm wondering what it actually does :)14:57
sean-k-mooneytoday the application has no effect really so you could also use a custom openstack applicaiton name14:57
sean-k-mooneykbaegis: bluestore replaces the files system used by the osd with a custume minimal filesystem based on a leveldb data base and write ahead journal14:58
sean-k-mooneykbaegis: it get ride of some of the issue around makeing sure data is commited to disk transactionally by removing the intermediate filesystem layer and allowing ceph to manage the disk directly14:59
kbaegisrather than deploying under xfs?14:59
kbaegisgotcha14:59
kbaegisNeat :D14:59
sean-k-mooneythat means more perfomance and better data integrty vs xfs/btrfs and it enable some new featres such as partail overries on ec pool that could not be done safely before15:00
kbaegisSounds awesome. It also makes it much more difficult I would imagine to troubleshoot15:01
kbaegisI can't use xfs/btrfs utlities to mount and repair15:01
kbaegisBut that's worth doing15:01
*** spsurya has joined #openstack-kolla15:01
sean-k-mooneywell true but i dont think you could do that before safely15:01
kbaegisYeah. And it's built for node failure anyways.15:02
kbaegisThanks for the primer.15:02
kbaegisAnyone here have some handy scripts they're willing to share to roll off old docker image versions?15:03
sean-k-mooneykbaegis: well i am not a ceph expert but just learning the bits i need to deploy a cloud for my team at work based on kolla SamYaple and kfox1111 know a lot more about ceph then i do15:03
kbaegisI've read the rados whitepaper too15:05
kbaegisJust didn't know about their new custom db/fs15:05
sean-k-mooneykbaegis: SamYaple had some interesting articles on ceph on https://yaple.net but it seams to be down at the moment15:07
*** jtriley has joined #openstack-kolla15:16
kbaegisLooks like it's on archive.org still15:16
sean-k-mooneysam did the origial kolla ceph support so for a time his blog was the only documentation for external journals or cache teirs for kolla deployed ceph.15:17
sean-k-mooneyhe did some interesting experimets with bcache that i use at home but for work im using a different setup now15:18
*** semoac has joined #openstack-kolla15:20
*** zhangfei has quit IRC15:26
openstackgerritTudosoiu Marian proposed openstack/kolla-ansible master: Rework mariadb recovery tasks  https://review.openstack.org/53112215:31
*** lvdombrkr89 has quit IRC15:35
*** david-lyle has joined #openstack-kolla15:38
*** robbbe has quit IRC15:39
openstackgerritMark Goddard proposed openstack/kolla-ansible master: Use bool filter with enable_ironic_pxe_uefi  https://review.openstack.org/53321415:52
*** jschlueter has joined #openstack-kolla15:58
jschluetermandre: in reviewing recent changes I see sensu-base has a run command creating sensu.repo in it15:58
*** nathharp has quit IRC16:00
jschluetermandre: I see it was injected by tripleo-common16:01
*** nathharp has joined #openstack-kolla16:04
openstackgerritPradeep Kilambi proposed openstack/kolla master: Drop ceilometer client package requirement  https://review.openstack.org/53322416:07
*** mchlumsky has joined #openstack-kolla16:09
*** itlinux has joined #openstack-kolla16:12
*** mchlumsky has quit IRC16:14
jschluetermandre: ^^16:14
*** zhaochao has quit IRC16:14
*** mchlumsky has joined #openstack-kolla16:16
*** itlinux has quit IRC16:20
sdakemorning peeps16:25
*** kbaegis1 has joined #openstack-kolla16:30
*** kbaegis has quit IRC16:30
*** jistr has quit IRC16:37
*** adrianreza has quit IRC16:37
*** berendt has quit IRC16:37
*** cloudnull has quit IRC16:37
*** ansiwen has quit IRC16:37
*** dosaboy has quit IRC16:37
*** dosaboy has joined #openstack-kolla16:37
*** cloudnull has joined #openstack-kolla16:37
*** adrianreza has joined #openstack-kolla16:38
*** berendt has joined #openstack-kolla16:38
*** chmarkus has quit IRC16:39
*** itlinux has joined #openstack-kolla16:39
*** jistr has joined #openstack-kolla16:40
*** serlex has quit IRC16:40
*** ansiwen has joined #openstack-kolla16:40
*** chmarkus has joined #openstack-kolla16:42
*** Mr_Broken has quit IRC16:43
-openstackstatus- NOTICE: Zuul has been restarted and lost queue information; changes in progress will need to be rechecked.16:45
*** wojdec has left #openstack-kolla16:47
*** mdnadeem has quit IRC16:48
sean-k-mooneyjust as a sainity check since our merge_configs fuction calls template here https://github.com/openstack/kolla-ansible/blob/master/ansible/action_plugins/merge_configs.py#L94 i shoululd be able to add jinja vairbale suppsutiontion {{my_var}} to a config over file in /etc/kolla/config/... and set the var in my inventory correct?16:53
*** nathharp has quit IRC16:55
*** absubram has joined #openstack-kolla17:00
*** pcaruana has quit IRC17:03
*** ntpttr_laptop has joined #openstack-kolla17:06
sean-k-mooneysdake: pbourke kfox1111 ^ any idea if that will work? ill be testing it in a few minuts but trying to avoid having to create multiple nova.conf overrides17:09
*** dciabrin_ has joined #openstack-kolla17:09
*** dciabrin has quit IRC17:11
SamYaplesean-k-mooney: kbaegis1 yea sorry, blogs down at the moment. anythingi an help with?17:12
sean-k-mooneySamYaple: no i was chating to people about ceph and mention you had done the original ceph support in kolla and your blog.17:13
sean-k-mooneySamYaple: unless you know the answer to can i use jinja in template overrides17:13
sean-k-mooneySamYaple: you did the original merge_configs plugin too right17:14
*** jascott1 has quit IRC17:15
*** jascott1 has joined #openstack-kolla17:15
*** jascott1 has quit IRC17:16
*** shardy has quit IRC17:28
*** gfidente is now known as gfidente|afk17:32
sdakesean-k-mooney SamYaple wrote nearly half the original code base17:33
sdakeso yes17:33
*** k_mouza has quit IRC17:34
*** absubram has quit IRC17:34
sean-k-mooneysdake: haha thats true. im almost ready to test it so ill just run the reconfigure and find out17:38
sdakesean-k-mooney sounds good17:38
SamYaplesean-k-mooney: yea sorry, template overrides were after my time(mostly)17:39
sean-k-mooneySamYaple: im refering to the merge config not template overrides but i just hit enter on the reconfigure so ill know soo enought17:40
*** lpetrut has quit IRC17:40
SamYapleoh no, you cant put jinja templating into the merge config17:40
SamYaplethough that would be pretty easy to extend support for17:40
klindgren_So as mentioned yesterday here is our current ansible role which does the flow chart stuff I talked about17:41
klindgren_https://github.com/krislindgren/openstack-deploy17:41
sean-k-mooneySamYaple: damb i was hoping this would render the template https://github.com/openstack/kolla-ansible/blob/master/ansible/action_plugins/merge_configs.py#L9417:41
klindgren_I tireid to remove thee godaddy-ism's like we have an inventory system and dynamic inventory script that gives a bunch of the infromation over to ansible.  The goal being that though shalt never have passwords anywhere but an encrypted vault.  It also means that some information comes from that topology information.  Which allows us to do things like, not having to login to memcache servers, to get facts, to use as a variable so17:44
klindgren_mewhere else.  We jsut pull the relevant information from out topology information.17:44
klindgren_SamYaple, inc0, kfox111117:45
kolla-slack<inc0> All passwords are stored in passwords.yml17:47
kolla-slack<inc0> So if you put them to vault under these keys you should be fine17:47
sean-k-mooneySamYaple: actully i think it worked the way i wanted it to and templated it out correctly17:49
klindgren_Thats fine - but the way we store password is not currently comptable with kolla.  tl;dr is that we have our own set of topology information/refrences and we translate those into whatever tool we are using.  IE we use the same base information for ansible/puppet/pythonscripts, (insert tool here) with render's for whatever we need to put out.  We got tired of being slaves of our tools.  So we went with a tool (ansible/puppet/salt17:53
klindgren_) agnostic format, and when we need something in a specific format for that tool we simply build a renderer for that tool.17:53
sean-k-mooneySamYaple: sdake alittle bit sanitzed but http://paste.openstack.org/show/643955/ works17:54
*** dougsz has quit IRC17:55
sdake@sean-k-mooney you shouldn't have to set virt type as the defaults are merged with your updates17:55
sean-k-mooneydefault should be DEFAULT though to match up with the template17:56
*** ktibi has quit IRC17:56
sean-k-mooneysdake: on ya i know it just force of habit17:56
sean-k-mooneythe fact that you can in fact use jinja in your template overrides make doing tweeking like this much simpler as i can have templated template overrides17:58
*** gkadam has quit IRC17:58
sean-k-mooneysorry templated config overrides17:58
SamYapleklindgren_: there is all kinds of dynamic ansible inventory and vars stuff, surely you could write something to hook into kolla without much issue17:59
*** robbbe has joined #openstack-kolla17:59
klindgren_We can and we will.  I was just saying that if something is missing in the above thing from me, its probably because it comes from the dynamic inventory stuff/vars and I missed it in my attempt to try to make the module with less Godaddy-ism's18:01
sean-k-mooneyklindgren_ i cant rembere who did it but i know someone wrote a custom dynamic inventory plugin that pull all the config varible form txt records in there dns system so others have hooked into kolla sucessfully in the past without needing to change the code18:01
klindgren_to be clear, my comments were solely related to: https://github.com/krislindgren/openstack-deploy18:02
*** david-lyle has quit IRC18:12
*** k_mouza has joined #openstack-kolla18:14
*** kbaegis has joined #openstack-kolla18:15
*** kbaegis1 has quit IRC18:17
*** k_mouza has quit IRC18:19
*** athomas has quit IRC18:20
*** spsurya has quit IRC18:23
*** ntpttr_laptop has quit IRC18:23
*** ntpttr_laptop has joined #openstack-kolla18:29
*** jtriley has quit IRC18:33
*** yee379 has joined #openstack-kolla18:36
*** yee37932 has quit IRC18:38
*** semoac has quit IRC18:51
*** ntpttr_laptop has quit IRC19:03
*** iclon has joined #openstack-kolla19:04
*** sambetts is now known as sambetts|afk19:09
*** ntpttr_laptop has joined #openstack-kolla19:19
*** bmace has quit IRC19:19
*** bmace has joined #openstack-kolla19:20
*** kbaegis has left #openstack-kolla19:25
*** ntpttr_laptop has quit IRC19:27
*** harlowja has joined #openstack-kolla19:27
*** david-lyle has joined #openstack-kolla19:33
*** sbezverk has quit IRC20:03
*** harlowja has quit IRC20:04
*** sbezverk has joined #openstack-kolla20:04
*** iclon has quit IRC20:11
*** goldyfruit has quit IRC20:14
*** robbbe has quit IRC20:15
*** robbbe has joined #openstack-kolla20:16
*** robbbe has quit IRC20:16
*** robbbe has joined #openstack-kolla20:18
*** robbbe has quit IRC20:18
*** robbbe has joined #openstack-kolla20:21
*** strgout has joined #openstack-kolla20:21
*** robbbe has quit IRC20:21
strgouthowdy all, i'm using the images from docker. I've noticed that ovsdb won't start because the unix socket.. is a directory. Removing the directory and restarting fixed the problem but how do i commit this change. More of a docker question i guess.20:23
strgouti don't have a local registry20:23
*** robbbe has joined #openstack-kolla20:24
*** robbbe has quit IRC20:24
*** robbbe has joined #openstack-kolla20:27
*** robbbe has quit IRC20:28
*** robbbe has joined #openstack-kolla20:30
*** robbbe has quit IRC20:31
*** robbbe has joined #openstack-kolla20:31
*** robbbe has quit IRC20:32
*** robbbe has joined #openstack-kolla20:33
*** openstackstatus has quit IRC20:46
*** openstack has joined #openstack-kolla20:50
*** ChanServ sets mode: +o openstack20:50
sean-k-mooneyhow do you start kolla ansible container again after running kolla-ansible stop.20:52
sean-k-mooneydeploy and upgrade dont seam to work20:52
*** jascott1 has joined #openstack-kolla20:52
*** lpetrut has joined #openstack-kolla20:53
sean-k-mooneymariadb_recovery also fails with "kolla_toolbox container is not running." after starting the dbs20:56
sean-k-mooneywell this explains it... https://bugs.launchpad.net/kolla-ansible/+bug/171401520:59
openstackLaunchpad bug 1714015 in kolla-ansible "containers not started during deploy if are stopped" [High,In progress] - Assigned to Eduardo Gonzalez (egonzalez90)20:59
*** itlinux has quit IRC21:02
*** Jeffrey4l has quit IRC21:03
*** Jeffrey4l has joined #openstack-kolla21:11
*** gfidente|afk has quit IRC21:17
*** jascott1 has quit IRC21:22
*** Jeffrey4l has quit IRC21:24
*** harlowja has joined #openstack-kolla21:26
*** goldyfruit has joined #openstack-kolla21:27
*** itlinux has joined #openstack-kolla21:32
*** jascott1 has joined #openstack-kolla21:34
*** Jeffrey4l has joined #openstack-kolla21:37
*** lpetrut has quit IRC21:39
openstackgerritPierre Blanc proposed openstack/kolla-ansible master: Homogenize the topics conf variable on templates  https://review.openstack.org/53331021:43
*** unicell1 has joined #openstack-kolla21:50
*** strgout has quit IRC21:59
*** itlinux has quit IRC22:10
*** Jeffrey4l has quit IRC22:11
*** Jeffrey4l has joined #openstack-kolla22:14
*** mchlumsky has quit IRC22:38
*** dave-mccowan has quit IRC22:39
*** robbbe has quit IRC22:59
*** robbbe has joined #openstack-kolla22:59
*** robbbe has quit IRC22:59
*** robbbe has joined #openstack-kolla23:00
*** robbbe has quit IRC23:00
*** robbbe has joined #openstack-kolla23:01
*** harlowja has quit IRC23:01
*** robbbe has quit IRC23:01
*** vskilet has joined #openstack-kolla23:04
*** vskilet has quit IRC23:06
*** owalsh has quit IRC23:12
*** owalsh has joined #openstack-kolla23:12
*** itlinux has joined #openstack-kolla23:22
*** itlinux has quit IRC23:39
*** itlinux has joined #openstack-kolla23:57
jascott1portdirect around?23:58
kolla-slackAction: portdirect hides23:58
portdirectsup dude?23:59

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!