Sunday, 2015-10-11

*** dims_ has joined #kolla		00:03
*** dimsum__ has quit IRC		00:06
*** dimsum__ has joined #kolla		00:06
*** jmccarthy1 has quit IRC		00:07
*** jmccarthy has joined #kolla		00:07
*** dims_ has quit IRC		00:10
jmccarthy	Anyone run into an issue where trying to get a console to an instance always times out, but the 'click to view full screen' ones works ?	00:12
*** achanda has joined #kolla		00:28
*** achanda has quit IRC		00:35
*** masterbound has quit IRC		00:46
*** dimsum__ has quit IRC		00:57
openstackgerrit	Merged openstack/kolla: Fix path of synced_folder in Vagrantfile https://review.openstack.org/233374	00:59
*** erkules_ has joined #kolla		01:14
*** erkules has quit IRC		01:17
*** achanda has joined #kolla		01:30
*** achanda has quit IRC		01:34
*** mbound has joined #kolla		01:46
*** sdake has quit IRC		01:51
*** mbound has quit IRC		01:52
*** sdake has joined #kolla		01:54
*** dimsum__ has joined #kolla		01:57
*** dimsum__ has quit IRC		02:03
*** bmace__ has quit IRC		02:06
*** bmace__ has joined #kolla		02:07
*** sdake_ has joined #kolla		02:12
*** klint has joined #kolla		02:39
*** sdake_ has quit IRC		02:44
*** dimsum__ has joined #kolla		03:00
*** dimsum__ has quit IRC		03:04
SamYaple	morning	03:20
SamYaple	jmccarthy: https://bugs.launchpad.net/nova/+bug/989337	03:21
openstack	Launchpad bug 989337 in OpenStack Compute (nova) "multiple nova-consoleauth instances cause issues with novncproxy" [Undecided,Fix released] - Assigned to Vish Ishaya (vishvananda)	03:21
SamYaple	been meaning to fix that	03:21
SamYaple	pretty simple but i keep forgetting to bring it up in a meeting	03:21
sdake	samyaple about	03:33
sdake	have a critical bug	03:33
sdake	can you take a look at the logs	03:33
sdake	i am tagging rc2 as is now	03:33
sdake	https://bugs.launchpad.net/kolla/+bug/1504883	03:35
openstack	Launchpad bug 1504883 in kolla "VirtualInterfaceCreateException: Virtual Interface creation failed" [Critical,Triaged]	03:35
SamYaple	so whats the issue?	03:36
SamYaple	is there a stack trace	03:36
*** achanda has joined #kolla		03:38
sdake	yes stack trace	03:40
sdake	the vms dont start up is the issue	03:40
sdake	there are a slew of backtraces in the minime-01 log	03:40
sdake	that is all the log files on that node	03:40
sdake	the first 24 start, i kill them with stack delete	03:40
sdake	the second 24 dont start, only 15-20 start	03:40
sdake	also python-heatclient 0.8.0 is broken with kolla	03:41
sdake	i'm not sure if its kolla or heat	03:41
sdake	heatclinet 0.6.0 works though	03:41
sdake	i ordered 2 750 400gb and 1 750 1.2 tb for my workstations/minime amchines	03:45
sdake	and 3 4tb red drives for backend storage	03:45
sdake	hopefully all that gear will work with ceph properly	03:46
sdake	i am putting the 400gb as cache tiers on minime, and the 4tb red as storage tier	03:46
sdake	and 1.2 tb as my main linux drive in my bigiron mahcine	03:47
SamYaple	eh its not like im running out of TBs over here	03:49
SamYaple	hey whats up with heatclient	03:49
SamYaple	"its broken" means nothing	03:49
sdake	says something about a x thing not found	03:49
sdake	sec, i'll paste	03:49
sdake	heat client explosion http://ur1.ca/nzden -> http://paste.fedoraproject.org/277850/35557144	03:52
sdake	0.8.0 bust 0.6.0 works	03:53
sdake	so looks like we have aout 4 or 5 ritical bugs that absolutely must be fixed between now and the 15th	03:53
* sdake groans		03:54
sdake	atleast stable/liberty is in pretty solid shape	03:54
SamYaple	you should probably open a but with heat	03:54
sdake	yes i am	03:54
sdake	i suspect its our config that is busted	03:54
SamYaple	might be. i dont know anything about heat	03:56
sdake	heat is the new hotness	03:57
SamYaple	not so sure about that	03:59
*** dimsum__ has joined #kolla		04:02
sdake	new laptop faster	04:02
* sdake likes		04:02
sdake	kolla has 194 stars	04:03
* sdake yayas		04:03
openstackgerrit	Steven Dake proposed openstack/kolla: Fix up loc with change to devenv https://review.openstack.org/233404	04:06
*** dimsum__ has quit IRC		04:06
*** bmace__ has quit IRC		04:10
*** bmace__ has joined #kolla		04:11
openstackgerrit	Steven Dake proposed openstack/kolla: Fix up loc with change to devenv https://review.openstack.org/233404	04:11
openstackgerrit	Sam Yaple proposed openstack/kolla: DO NOT MERGE - Gate things https://review.openstack.org/231881	04:12
openstackgerrit	Sam Yaple proposed openstack/kolla: DO NOT MERGE - Gate things https://review.openstack.org/231881	04:19
*** achanda has quit IRC		04:23
*** sdake has quit IRC		04:34
*** achanda has joined #kolla		04:54
openstackgerrit	Sam Yaple proposed openstack/kolla: DO NOT MERGE - Gate things https://review.openstack.org/231881	04:57
openstackgerrit	Sam Yaple proposed openstack/kolla: DO NOT MERGE - Gate things https://review.openstack.org/231881	05:11
openstackgerrit	Sam Yaple proposed openstack/kolla: DO NOT MERGE - Gate things https://review.openstack.org/231881	05:18
*** sdake has joined #kolla		05:30
sdake	yo	05:31
sdake	SamYaple what do you make of this	05:42
sdake	Oct 11 02:55:51 minime-01 nova-scheduler: Seems service is down. Last heartbeat was 2015-10-11 02:54:44. Elapsed time is 67.338841	05:42
SamYaple	sdake: im not sure the context	05:49
sdake	the failure to allocate vms as mentioned erarlier	05:49
sdake	when spawning 24 vms via heat hten killing then wpawning 24 again	05:50
sdake	that seems t o imply amqp is on the rails	05:50
SamYaple	i blame your date time!	05:51
SamYaple	did yo usetup ntp?	05:52
sdake	no	05:55
sdake	i guess i'll do that	05:56
sdake	hang tight	05:56
*** CBR09 has joined #kolla		05:59
sdake	boy looking at wall clock time to determine healthchecking is a bunch of fail	06:17
sdake	bunch o fail	06:17
*** dimsum__ has joined #kolla		06:28
*** dimsum__ has quit IRC		06:33
SamYaple	sdake: what do you mean?	06:47
SamYaple	the messages are timestamped	06:48
SamYaple	its not looking at when it got it, its looking at when it was timestampped until now	06:48
SamYaple	its a great way to do it	06:48
SamYaple	that way it can deal with retransmits	06:48
sdake	SamYaple check this out xhttp://ur1.ca/nzemf -> http://paste.fedoraproject.org/277883/45462921	06:51
sdake	3 control nodes	06:52
sdake	only 1 of them is actually returning the image	06:52
sdake	the other 2 error out	06:52
sdake	nd in the server say teh image not preent	06:52
SamYaple	thats because only one actually has the image	06:52
SamYaple	do you know how glance works in this regard?	06:52
SamYaple	you have the file backend	06:52
SamYaple	it has no syncing mechanism	06:52
SamYaple	you need ceph or nfs	06:53
sdake	so peple do glance with nfs?	06:55
SamYaple	sorta. they do an nfs share for /var/lib/glance	06:55
SamYaple	mostly its ceph	06:55
SamYaple	or swift	06:55
SamYaple	maybe cinder, but that has its own SPOF	06:55
sdake	i find it hard to believe ther eis no database storage mechanism for the bakend - images are small	06:56
sdake	without ceph ha is unusable with kolla ssince you can't launch vms	06:57
SamYaple	the default limit size for images is 1TB in glance. ive had to raise it before :)	06:57
SamYaple	nah that can be fixed with config options sdake	06:57
SamYaple	if you choose to use a file backend	06:57
SamYaple	this isnt a kolla issue, this is a non-ha backend issue sdake	06:58
SamYaple	dont freak out	06:58
sdake	ok well can e get those config options in as default then	06:58
*** Kennan2 has joined #kolla		06:58
sdake	what do the config options do?	06:58
SamYaple	no it doesnt make it ha....	06:58
SamYaple	the files still only exist in one spot	06:58
SamYaple	its not going to sync them, thats on you	06:58
*** Kennan has quit IRC		06:59
sdake	roger, we can document that	06:59
sdake	that is better then nova not booting because it didnt' find the right glance server :)	06:59
sdake	syncing would havebeen the first thing i ever implmented in glance	06:59
sdake	i am surprised it isn't built in	06:59
SamYaple	im pretty sure RAX built a tool for this at one point	07:00
SamYaple	but then they wised up and said "use an HA backend like Glance was designed to use"	07:00
SamYaple	yes https://github.com/rcbops/glance-image-sync	07:01
SamYaple	dont use it btw	07:01
sdake	is http writeable?	07:02
SamYaple	wut	07:02
sdake	Various repository types are supported including normal file systems, Object Storage, RADOS block devices, HTTP, and Amazon S3. Note that some repositories will only support read-only usage.	07:02
sdake	so how do we get access to just one registry service for the particular image	07:04
sdake	rather then round robning to machines that don't have the image	07:04
SamYaple	no thats the solution	07:04
SamYaple	roundrobining	07:04
SamYaple	thats the designed solution...	07:04
SamYaple	sorry im just a bit shocked youve never dealt with this. this is like glance 101	07:05
sdake	how do we make kolla work without ceph where most of the ystem is ha but glance?	07:05
SamYaple	its glance it wont be ha	07:05
SamYaple	it never has been without an HA backend	07:05
sdake	roger	07:06
sdake	so the answer is use ceph?	07:06
SamYaple	if you need an HA backend, use an HA backend	07:06
SamYaple	ceph works	07:06
SamYaple	S3 can be configured	07:06
sdake	what can we get working out of the box	07:06
SamYaple	if it requires an HA backend, nothing	07:07
sdake	you must understand how unappealing not being able to use openstack without ceph is	07:07
sdake	can you just run one galnce service	07:07
SamYaple	you must understand that this is openstack period	07:07
SamYaple	you can	07:07
SamYaple	i mean this isnt news	07:07
SamYaple	this is exactly how its always been	07:08
SamYaple	use an ha backend	07:08
sdake	i think what makes sense is to change storage to "general_storage" and "image_storage"	07:09
sdake	and hae image_storage only hae 1 host, general storage have all the hosts	07:09
SamYaple	what do you mean?	07:09
sdake	that solves the problem well enough	07:09
sdake	we hve a [storage] section	07:09
SamYaple	no thats awful	07:09
sdake	but we have two types of storage	07:09
SamYaple	besides we dont put glance on storage	07:09
SamYaple	no we dont	07:09
SamYaple	we have containers	07:09
SamYaple	all the "storage" hosts have physical disks	07:10
sdake	where is glance put?	07:10
SamYaple	storage == physical disks	07:10
SamYaple	glance stores its images in the containers like oyu wanted	07:10
SamYaple	but it can be configured with ceph or swift or s3	07:10
sdake	on control nodes or storage nodes?	07:10
SamYaple	whereever glance-api runs	07:10
SamYaple	glance has nothing to do with storage nodes	07:10
sdake	do you kno wwhwere it runs?	07:10
SamYaple	it depends on your inventory configuration	07:11
SamYaple	what is the problem here again?	07:11
sdake	what is the out of the box default	07:11
SamYaple	check the inventory file....	07:11
sdake	the problem is an out of the box multinode with multiple control nodes does not work	07:11
sdake	without serious hacking on the inventory file	07:11
sdake	and someone magically knowing glance doesn't work ha	07:11
sdake	without a ha backend	07:12
SamYaple	i wouldnt call that magically	07:12
SamYaple	more like common sense and how its always always been	07:12
sdake	most of our users are not rocket scientistss when it comes to openstack ;)	07:12
sdake	most openstack is not ha for that matter	07:13
SamYaple	most openstack is RESTful and as such is HA	07:13
sdake	its ok i know how to take care of it	07:13
sdake	i'll resolve it - easy to fix	07:13
SamYaple	ok dont change the group names though	07:13
sdake	we need a new group name	07:13
sdake	one for glance	07:13
SamYaple	you cant backport that.	07:14
SamYaple	inventory is API	07:14
SamYaple	you dont have to use those inventory files iwth ansible, they are exaples	07:14
sdake	well then now is the time to backport it because what we have now is doa	07:14
SamYaple	-2	07:14
SamYaple	you are freaking out about something yet again	07:14
SamYaple	glance is not HA without an HA backend	07:15
sdake	every time someone comes into the channel andasks how to get glance ogin ha, i'm pointing them at you	07:15
sdake	enjoy	07:15
SamYaple	you freak out alot when you dont understand something	07:15
sdake	i dont understand the issue with having an additional gropu name	07:15
SamYaple	its an api change....	07:15
sdake	i fully understandthe situation	07:15
SamYaple	you do not	07:15
SamYaple	its a freaking config change whats the issue	07:15
sdake	we have not released liberty yet	07:16
SamYaple	you cant make glance sync images	07:16
sdake	release is the 15th	07:16
sdake	along with the rest of openstack	07:16
SamYaple	im -2 on changing group names rather that the config	07:16
SamYaple	15th or not	07:16
sdake	i dont understand the last sentence coud you reprhase	07:17
SamYaple	change nova.conf not the gropu names in the inventory. as ive been saying	07:17
sdake	ok tell me more plz	07:17
SamYaple	thats the entirety of my sentence	07:17
sdake	if tht is viable i'm good with that, pleae point me at docs	07:17
sdake	or tell me the config option	07:18
sdake	fwiw i brought this up several months ago and you said it wasn't a problem	07:19
SamYaple	because of being able to fix it with a config change....	07:19
sdake	but i hadn't confirmed it until just now	07:19
sdake	cool lets get er done then :)	07:19
SamYaple	youre making it a problem like you did with all the other non issues because you didnt get the situation	07:20
sdake	the config option is what?	07:20
sdake	see sam, i lsiten whenyu explain	07:20
SamYaple	you just dont remember	07:21
sdake	i do not have esp powers unfortunately	07:21
sdake	ya crs, it will happen to you too :)	07:21
SamYaple	i did say the config option was configurate like 30 minutes ago dude	07:21
openstackgerrit	Sam Yaple proposed openstack/kolla: Please stop freaking out its a small change https://review.openstack.org/233429	07:23
SamYaple	try that	07:23
SamYaple	left a special message for oyu in the commit	07:23
SamYaple	;)	07:23
sdake	thanks i'll give it a spin	07:27
sdake	sam i only have kolla's best interests in mind ;)	07:28
SamYaple	thats not in question, but you do tend to freak out about things when i tell you not too	07:28
SamYaple	this is the 4th time I believe	07:28
sdake	you haven't seen a gade 9 freakout yet bro ;)	07:30
SamYaple	you probably shouldnt be ok with that	07:30
SamYaple	its quite irratating to me	07:30
*** dimsum__ has joined #kolla		07:30
*** jmccarthy has quit IRC		07:31
sdake	from marvell movie	07:31
sdake	clearly lost on you :(	07:31
SamYaple	i stand by my statements	07:32
*** dimsum__ has quit IRC		07:35
sdake	SamYaple have you deployed murano	07:35
SamYaple	yup	07:35
SamYaple	api works, not horizon integration	07:35
sdake	need to sort out the horizon plugins in mitaka	07:37
sdake	i read an email from ttx, made me rethink liberal backport policy	07:38
SamYaple	btw making this default change to the confs seriously affects performance when launching a bunhc of instances	07:38
SamYaple	just fyi	07:38
sdake	cool let me eval	07:39
sdake	i'd like poeple to just roll with ceph	07:39
sdake	is it possible to make it ceph conditional?	07:39
SamYaple	yea im working on it. also the hosts thing is deprecated in liberty so im updating that	07:39
sdake	then our recommended deploy (ceph) won have any impact	07:40
SamYaple	file a bug for this would you please	07:40
sdake	sure	07:40
sdake	shll I assign to you?	07:40
SamYaple	sure	07:40
sdake	assuming this fixes openstack locing up on my boxes, kolla wil be in good shape :)	07:41
sdake	locking	07:41
sdake	SamYaple does kolla behave correctly when there are multiple dhcp metadata and l3 agents running on control nodes?	07:48
SamYaple	define behave correctly	07:49
sdake	networkign will work correctly?	07:50
SamYaple	sure?	07:50
SamYaple	im not sure i follow	07:50
SamYaple	neutron is SPOF in legacy mode	07:50
SamYaple	and only one dhcp server will be assigned by default	07:51
sdake	do you know anything about l3 ha?	07:51
SamYaple	yup	07:51
sdake	is that the dvr thing?	07:52
SamYaple	nope	07:52
*** achanda has quit IRC		07:54
openstackgerrit	Sam Yaple proposed openstack/kolla: Balance by source for horizon and nova-consoleauth https://review.openstack.org/233430	07:55
sdake	fresh reboot, only 9 vms started	07:56
sdake	with patch applied	07:57
*** achanda has joined #kolla		07:57
SamYaple	yea thats the deprecated option probably	07:57
SamYaple	have a new patch just wating on bug	07:57
sdake	roger filign now	07:57
sdake	https://bugs.launchpad.net/kolla/+bug/1504902	08:02
openstack	Launchpad bug 1504902 in kolla "nova doesn't always find images in glance in HA mode" [Critical,Confirmed] - Assigned to Sam Yaple (s8m)	08:02
openstackgerrit	Sam Yaple proposed openstack/kolla: Glance round robin for default file backend https://review.openstack.org/233429	08:02
*** inc0 has joined #kolla		08:13
*** achanda has quit IRC		08:13
inc0	good morning	08:13
openstackgerrit	Steven Dake proposed openstack/kolla: Recommend enabling NTPD in the documenation https://review.openstack.org/233432	08:18
sdake	testing	08:20
sdake	hey inc0	08:20
inc0	2 weeks till Tokyo	08:23
inc0	do you know how much sessions we'll have after all?	08:23
sdake	8	08:23
sdake	3 fishbowl 5 regular	08:23
sdake	40 mins in length	08:23
inc0	cool, so most of sessions we wanted to do will land	08:24
sdake	yup	08:24
inc0	will we have dev meeting as well?	08:24
sdake	yes half day contrib meetup	08:24
sdake	2-5pm	08:24
sdake	then sakibombers!!	08:24
inc0	cool	08:24
inc0	I have flight back home at 1am	08:24
sdake	nice best time to get the booze rolling then :)	08:25
inc0	yup	08:25
inc0	just to keep mind clean enough to be able to show papers at border control	08:25
inc0	but I'm polish, I'm not rich enough to get enough alcohol for my slavic blood;)	08:26
SamYaple	inc0: 1am friday morning?	08:27
inc0	Saturday	08:28
SamYaple	cool	08:28
inc0	technically it Saturday already	08:28
SamYaple	yea my flights 1230 am i think	08:28
inc0	cool	08:28
*** vinkman has quit IRC		08:28
sdake	2015-10-11 08:28:20.100 1 ERROR nova.compute.manager [instance: 2413c892-5650-4482-9a17-277076cd23f4] ImageNotFound: Image c463e50c-7478-4236-866c-f1df143fa6cb could not be found.	08:35
sdake	\| c463e50c-7478-4236-866c-f1df143fa6cb \| cirros \|	08:35
sdake	SamYaple looks like needs more love	08:36
openstackgerrit	Michal Jastrzebski (inc0) proposed openstack/kolla: Add ceph and ironic to index https://review.openstack.org/233434	08:38
*** mbound has joined #kolla		08:40
*** jmccarthy has joined #kolla		08:41
jmccarthy	SamYaple, you about ?	08:42
SamYaple	im here jmccarthy	08:42
jmccarthy	Your here all times ;)	08:42
SamYaple	me? never	08:43
openstackgerrit	Steven Dake proposed openstack/kolla: Recommend enabling NTPD in the documenation https://review.openstack.org/233432	08:44
jmccarthy	Quick question re: 'Balance by source for horizon and nova-consoleauth' patch, can you elaborate on why specify hash-type consistent ?	08:44
jmccarthy	Actually I have another question, but one at a time :)	08:44
jmccarthy	SamYaple: I mean is that a preference, or it's really also needed ?	08:45
SamYaple	jmccarthy: http://blog.haproxy.com/2015/05/06/haproxys-load-balancing-algorithm-for-static-content-delivery-with-varnish/	08:45
SamYaple	needed? no. helpful though	08:46
jmccarthy	Ok grand, I'll read up a bit more, cool	08:46
jmccarthy	SamYaple: Question 2 :) - Ok so with the getting a console to an instance, I find the embedded one always fails, while the 'click to view fullscreen' or whatever that says, always works ?	08:47
SamYaple	that patch fixes it	08:47
jmccarthy	The embedded one too though ?	08:47
jmccarthy	I have this behaviour even with only one consoleauth running	08:48
SamYaple	this is for horizon and consoleatuh	08:49
SamYaple	give it a try	08:49
jmccarthy	I'm unclear if or what they do different - don't they both use the websocket on 6080 to the vip ?	08:49
SamYaple	yea but thats not breaking	08:50
SamYaple	the auth is breaking	08:50
SamYaple	fyi havent gotten around to testing that yet, just threw it up realy quick in the midst of phones and tickets	08:51
jmccarthy	It's cool, just thought you might have some ideas on it, yes I'll give it a whirl today	08:55
CBR09	morning all	08:56
inc0	hey CBR09	08:56
CBR09	is ha currently work well?	08:57
CBR09	to do HA with two controller node, beside edit [control] group and change kolla_internal_address to VIP, what else?	08:59
inc0	3 nodes	08:59
inc0	with 2 nodes you might have problem with galera	08:59
jmccarthy	SamYaple: I'll try now actually	09:02
CBR09	inc0: for testing purpose, are two nodes ok?	09:05
inc0	CBR09, 3 nodes are because of quorum election and such	09:06
inc0	it has to be odd number	09:06
inc0	(but don't quote me on that, I'm not 100% sure how galera internals work)	09:07
CBR09	ok, I see, minimal require 3 nodes	09:09
inc0	yeah, its algorythmical requirement rather than performance	09:10
sdake	samyaple can you add https://github.com/openstack/nova/blob/master/nova/image/glance.py#L224-L226	09:10
sdake	samyaple as is, there is zero retries meaning the first failure is the last failurle :)	09:10
CBR09	inc0: yes, two nodes can work, but always have threats	09:11
SamYaple	sdake: thats retry per host. it walks the hosts	09:12
inc0	CBR09, quite possibly it won't break right away, it might cause data loss or cluster failure on split brain, but that's not a concern in testing env	09:12
inc0	on prod, I strongly reccomend 3 nodes at least	09:13
SamYaple	it doesnt have to be an odd number	09:13
SamYaple	you just gain no benefit from an even number	09:13
SamYaple	4 is no better than 3	09:13
inc0	so one node will be read-only right?	09:13
SamYaple	no	09:13
SamYaple	no difference, its just no added benfit for having an even number than an odd	09:14
SamYaple	minimum of 3	09:14
SamYaple	past that its fine	09:14
sdake	samyaple this code sets retries = 1 https://github.com/openstack/nova/blob/master/nova/image/glance.py#L226	09:14
SamYaple	just keep >50% up at one time	09:14
sdake	that creates 1 cient	09:14
sdake	and only parses api_servers one time	09:14
SamYaple	https://github.com/openstack/nova/blob/master/nova/image/glance.py#L206	09:16
SamYaple	yea i think youre right	09:16
SamYaple	i dont thinks its always been that way, but hold on	09:16
openstackgerrit	Sam Yaple proposed openstack/kolla: Glance round robin for default file backend https://review.openstack.org/233429	09:17
SamYaple	try that	09:17
sdake	will do one moment	09:17
sdake	samyaple num_retries	09:19
openstackgerrit	Sam Yaple proposed openstack/kolla: DO NOT MERGE - Gate things https://review.openstack.org/231881	09:19
SamYaple	ugh so picky	09:19
sdake	tell my wife about it	09:20
openstackgerrit	Sam Yaple proposed openstack/kolla: Glance round robin for default file backend https://review.openstack.org/233429	09:20
SamYaple	im sure she know	09:20
jmccarthy	SamYaple: Hmm ok with those changes, I still only get console 50% of the time	09:23
sdake	definately going to make glance loading pokey	09:23
SamYaple	sdake: that is how it works without an ha backend	09:25
SamYaple	its how its worked since essex	09:25
sdake	right i see	09:25
SamYaple	jmccarthy: ok give me a chance to test it	09:25
SamYaple	im really really against memcached for security reasons	09:25
SamYaple	its halariously dangerous and no one seems to care	09:25
sdake	i think memcached s in the tree ofor some reason	09:26
jmccarthy	SamYaple: no probs, yea it's there for swift proxy_server	09:26
SamYaple	they implemented it for swift	09:26
sdake	it should be removed if its a ssecurity problem	09:26
SamYaple	you dont need it for swift either	09:26
SamYaple	no sdake it juts shouldnt be on by default	09:26
SamYaple	i use it personally because i know the risks and how to secure it	09:26
SamYaple	but kolla cant ensure that its secured	09:26
sdake	well wfm, i'm more inclined to have no exposure :)	09:26
SamYaple	memcached needs to be in tree, but it needs to be have big bold HERE IS WHY YOU SHOULDN'T USE THIS letters around it	09:27
SamYaple	in the docs at least	09:27
sdake	or more importantly herei is how you use it correctly :)	09:27
SamYaple	well thats a complicated subject	09:27
jmccarthy	SamYaple: you dont need it for swift either - you mean by having similar config changes in haproxy ? I thought there was something else that had it as well at one point	09:27
SamYaple	jmccarthy: its the same reason with swift, shared auth tokens	09:28
SamYaple	if you always go to the same proxy (with balance source) youll be fine	09:28
SamYaple	i didnt say anything at the time because i havent had a chance to audit all the security	09:28
* sdake wtb faster deploys!!		09:28
sdake	15 minutes 50 times a day gets old, real old	09:28
SamYaple	i heard OSA is faster sdake	09:29
sdake	huh?	09:29
sdake	then kolla?	09:29
jmccarthy	SamYaple: I hear you, I've just not tried that, and if that node drops out, will it re-do tokens etc with new one ok ? I suppose so	09:29
SamYaple	i guess it just depends on who yo utalk to	09:29
sdake	from what I know kolla is 3-4x as fat as everyone elses deploy project ;)	09:29
SamYaple	the people who have used and and the people hearing the propoganda :P	09:29
sdake	fast	09:29
sdake	not fat :)	09:29
SamYaple	jmccarthy: yea it will just reauth. you just dont want it reauthing every request	09:30
sdake	if you include the image bulding in kolla from china, ya kolla is slower	09:30
sdake	becaue it goes through the gfw	09:30
jmccarthy	fat vs fast - unfortunate typo ;)	09:30
sdake	and that is slow	09:30
SamYaple	well i take that back jmccarthy, the application has to reauth. just an api call returns 401	09:30
jmccarthy	SamYaple: I'm ok as long as it sorts it self out without too much grief ;)	09:31
SamYaple	sdake i have a `kolla_reset` alias that wipes my host	09:31
SamYaple	you should use it	09:31
sdake	cool hook me up	09:31
sdake	steven.dake@gmail.com plz ;0	09:31
jmccarthy	SamYaple: Ok let me know if that change helps with console for you, it didn't seem to help me but not sure why offhand	09:32
SamYaple	http://paste.fedoraproject.org/277897/14445559/	09:32
sdake	i do a manual cleanup with reboots which takes forever	09:32
SamYaple	sdake: youll want to adjust or remove ceph_wipe and fix_networking	09:32
SamYaple	it also wipes the images from the system so keep that in mind	09:32
*** dimsum__ has joined #kolla		09:32
SamYaple	eh youre smart you can figure it out	09:33
SamYaple	thats my aliases you make them work for oyu	09:33
sdake	my ceph hardwre arrives monday	09:33
sdake	then i'll be harrassing you for setup help :)	09:33
SamYaple	fyi, using ceph populates /etc/fstab and that must be cleaned	09:34
SamYaple	reboots will hang otherwise	09:34
SamYaple	sed -ir '/ceph/d' /etc/fstab	09:35
inc0	I'll give a try this aio ceph guide	09:35
inc0	on vm	09:35
SamYaple	ya good job CBR09	09:35
SamYaple	ill try to update it when i have a moment to clarify a few points	09:35
sdake	ya cbr09 wrote some docs, and as a side benfit he knows how kolla works now :)	09:35
inc0	writing docs is cool for ramp up	09:36
inc0	same as writing tests	09:36
SamYaple	yea me and CBR09 spent a good bit of time on chat while he was getting that up and running	09:36
sdake	someone complained today that we have too much ocumentation in a bug tracker ;-)	09:36
CBR09	yea: )	09:37
CBR09	I'll try ceph on aio and multi-node	09:37
CBR09	both work	09:37
SamYaple	apart from me, you are the only one to confirm that!	09:37
CBR09	and thank Sam again, for help me : )	09:37
sdake	SamYaple can you put the backport tag in that glance bug plz	09:38
*** dimsum__ has quit IRC		09:38
SamYaple	whats it worth to you?	09:38
CBR09	I'll try HA projects with Kolla, tomorrow	09:39
CBR09	hope it work : )	09:39
SamYaple	i want to bump ansible requirements up to 1.9.2	09:40
SamYaple	that way we can remove that docker_api_version thing	09:40
SamYaple	the gate is not liking that	09:40
SamYaple	its kinda holding up the deploy	09:40
sdake	gate is a priority do what you need there	09:41
SamYaple	can we bump ansible reqs to 1.9.2 in liberty?	09:41
SamYaple	i think we said 1.8.4 before	09:41
sdake	ya to hit centos7	09:41
sdake	but i think eveyrone will be roling to 2.0 soon enough	09:41
SamYaple	ha	09:41
SamYaple	doubtful	09:41
SamYaple	its a mess	09:41
SamYaple	i dont want to touch 2.0	09:42
sdake	i was heistent to bump the gate before becauses 2.0 was unreleased	09:42
SamYaple	no bump to 1.9.2 not 2.0	09:42
sdake	yes	09:42
sdake	i understand	09:42
SamYaple	yea 2.0 isnt going to work for anyones playbooks out of the gate	09:42
sdake	the reason i didn't want to bump to 1.9.2 is becuse 2.0 ws unannounced	09:42
SamYaple	its been a mess everytime ive tried it	09:42
sdake	leaving us without a way to eploy kolla without a pip install	09:42
SamYaple	and i really want to use blocks	09:43
sdake	butpeople will be pip intalling it by default	09:43
sdake	samyaple somehow you get this in nova.conf	09:43
sdake	[glance]	09:43
sdake	api_servers = 192.168.1.101:9292,192.168.1.102:9292,192.168.1.103:9292num_retrie	09:43
sdake	s = 3	09:43
SamYaple	ugh jinja	09:43
SamYaple	hold on	09:44
SamYaple	you know oyu dont have t orekick to test	09:44
SamYaple	just purge the nova_copute containers	09:44
SamYaple	or all nova if yo uwant to be safe	09:44
openstackgerrit	Sam Yaple proposed openstack/kolla: DO NOT MERGE - Gate things https://review.openstack.org/231881	09:44
openstackgerrit	Sam Yaple proposed openstack/kolla: Glance round robin for default file backend https://review.openstack.org/233429	09:45
inc0	aaand ceph aio failed	09:46
SamYaple	gotta have ssh	09:47
inc0	<putting on debugging hat>	09:47
CBR09	ssh auth	09:47
inc0	I know...I removed delegate thing;)	09:47
*** Kennan has joined #kolla		09:47
inc0	failed on bootstrap containers	09:47
*** Kennan2 has quit IRC		09:48
inc0	failed: [localhost] => (item=(0, {u'device': u'Flags', u'fs_uuid': u''})) => {"changed": true, "cmd": ["docker", "wait", "bootstrap_osd_0"], "delta": "0:00:00.010079", "end": "2015-10-11 11:45:51.239149", "failed": true, "failed_when_result": true, "item": [0, {"device": "Flags", "fs_uuid": ""}], "rc": 0, "start": "2015-10-11 11:45:51.229070", "stdout_lines": ["2"], "warnings": []}	09:48
inc0	fs_uuid shouldn't be "" should it?	09:48
SamYaple	for bootstrap it should	09:50
SamYaple	whats the error	09:50
inc0	http://paste.openstack.org/show/475975/ pay attention do container id...it just knew...	09:51
SamYaple	im confused, whats going on?	09:52
inc0	id is "bad"	09:52
SamYaple	why?	09:53
inc0	full id bad16a088cfd	09:53
inc0	it just happend this way	09:53
inc0	anyway, that was poor joke from my part	09:53
SamYaple	i agree	09:53
SamYaple	but iget it know	09:53
SamYaple	i aint laughing, but i get it	09:54
inc0	but you're not laughing	09:55
inc0	anyway, back to the issue at hand	09:55
inc0	my ceph is broken and I have no clue why	09:55
sdake	it was so bad it was funny	09:57
sdake	;-)	09:57
CBR09	inc0: you parted these disk ?	10:00
inc0	yup	10:00
inc0	http://paste.openstack.org/show/475976/	10:01
CBR09	give me full log when running playbook	10:02
inc0	it's pretty inconclusive...	10:05
CBR09	looks not find these disks for osd	10:07
SamYaple	inc0: need the playbook logs and the logs from the container that bombed out (bootstrap container)	10:09
CBR09	yea, agree with Sam, playbook logs is useful	10:10
inc0	http://paste.openstack.org/show/475977/ plays	10:14
inc0	(before that it's all green)	10:14
SamYaple	u'device': u'Flags',	10:16
SamYaple	dats your problem	10:16
SamYaple	can you post the output of `parted -l`	10:16
CBR09	not find disk	10:16
CBR09	: )	10:16
inc0	http://paste.openstack.org/show/475979/ parted -l	10:17
SamYaple	what version of parted is that	10:18
SamYaple	"Disk Flags:" is the issue btw	10:18
inc0	parted (GNU parted) 3.2 <- parted -v	10:19
inc0	what's wrong with them?	10:19
SamYaple	just the way i screen scrape it	10:19
SamYaple	hold on ill write a patch youll need to try	10:19
SamYaple	can you pop a bug	10:20
inc0	bug I'll pop then	10:20
openstackgerrit	Sam Yaple proposed openstack/kolla: Ignore the 'Disk Flags:' line in parted https://review.openstack.org/233441	10:22
SamYaple	inc0: try that	10:22
openstackgerrit	Sam Yaple proposed openstack/kolla: DO NOT MERGE - Gate things https://review.openstack.org/231881	10:25
inc0	it failed later	10:25
inc0	but passed this point	10:25
SamYaple	LOGS DAMMIT	10:26
SamYaple	:)	10:26
inc0	hold on, let me file bug you wanted	10:26
SamYaple	work harder!	10:26
SamYaple	faster!	10:26
SamYaple	better!	10:26
SamYaple	stronger?	10:26
inc0	http://paste.openstack.org/show/475980/	10:27
openstackgerrit	Sam Yaple proposed openstack/kolla: Ignore the 'Disk Flags:' line in parted https://review.openstack.org/233441	10:28
SamYaple	no so sure. did you cleanup the environment properly before retrying?	10:29
inc0	hold on, I will	10:29
CBR09	should be remove /etc/kolla/ceph-* : D	10:30
SamYaple	umount /var/lib/ceph/osd/; rm -rf /var/lib/ceph/osd/ /etc/kolla/ceph-*; sed -ir '/ceph/d' /etc/fstab;	10:30
SamYaple	inc0: ^^ thats like the bare minimum	10:31
inc0	removing etc and containers didn't help	10:31
inc0	still the same	10:31
SamYaple	i dont believe you are cleaning it up properly this quickly!	10:32
inc0	it's one command	10:32
inc0	...	10:32
CBR09	I always remove /etc/kolla/ceph-* and add new disk	10:32
SamYaple	inc0: see above	10:32
SamYaple	its more than one command	10:32
SamYaple	4 at minumum	10:32
SamYaple	umount /var/lib/ceph/osd/; rm -rf /var/lib/ceph/osd/ /etc/kolla/ceph-*; sed -ir '/ceph/d' /etc/fstab; parted /dev/sdb -s -- mklabel gpt	10:33
SamYaple	try that inc0	10:33
CBR09	Ah Sam, sometime I run into error:	10:34
CBR09	http://paste.openstack.org/show/475981/	10:34
inc0	umount: /var/lib/ceph/osd/82575176-3a45-4165-b739-c68a784a770f: not mounted	10:34
CBR09	due to remaining /etc/kolla/ceph-*	10:34
CBR09	is it error ?	10:34
inc0	we need proper clean scripts, I'll file a bug for that as well	10:34
inc0	while I'm at it	10:34
SamYaple	inc0: i have alieases	10:35
SamYaple	http://paste.fedoraproject.org/277897/14445559/	10:35
SamYaple	`kolla_reset` and its all cleaned	10:35
inc0	https://bugs.launchpad.net/kolla/+bug/1504921	10:36
openstack	Launchpad bug 1504921 in kolla "ceph clean scripts" [High,Triaged]	10:36
SamYaple	really not a bug dude	10:36
SamYaple	and you cant really have a clean script like that	10:37
SamYaple	i dont want to go removing and unmounting things...	10:37
inc0	yeah, wishlist is better	10:37
SamYaple	well thats just the proper thing for it	10:37
inc0	at least a doc or set of commands	10:37
SamYaple	since its not a bug, not to lessen the value	10:37
inc0	hmm....it seems it worked now	10:38
SamYaple	ha!	10:38
SamYaple	i knew you didnt clean it	10:38
SamYaple	mwhahahaa	10:38
inc0	https://bugs.launchpad.net/kolla/+bug/1504920	10:39
openstack	Launchpad bug 1504920 in kolla "ceph fails on aio deploy" [High,In progress] - Assigned to Sam Yaple (s8m)	10:39
inc0	by the way, for your patch	10:39
inc0	SamYaple ... it requires pretty complicated thing to clean that up:P	10:39
inc0	if it's at least in docs	10:39
inc0	I'm ok with it, but it should be somewhere	10:39
SamYaple	im perfectly fine with it being documents	10:40
SamYaple	or a script	10:40
SamYaple	just want to be carful with disks	10:40
openstackgerrit	Sam Yaple proposed openstack/kolla: Ignore the 'Disk Flags:' line in parted https://review.openstack.org/233441	10:41
inc0	yeah, that's my point	10:42
inc0	well...meh	10:43
inc0	no osds	10:44
inc0	I've just noticed, it skipped the ODSs	10:44
SamYaple	you didnt bootstrap the osd maaaaan	10:44
inc0	aaaaa	10:44
inc0	yeah	10:44
SamYaple	parted /dev/sdb -s -- mklabel gpt mkpart KOLLA_CEPH_OSD_BOOTSTRAP 1 -1	10:44
inc0	yup	10:45
inc0	aaand we get last error	10:45
inc0	failed: [localhost] => (item={u'device': u'/dev/sdb', u'fs_uuid': u'6efd1972-340a-4f5d-86fb-855bde978e16'}) => {"failed": true, "item": {"device": "/dev/sdb", "fs_uuid": "6efd1972-340a-4f5d-86fb-855bde978e16"}}	10:46
inc0	msg: Error mounting /var/lib/ceph/osd/6efd1972-340a-4f5d-86fb-855bde978e16: mount: can't find UUID=6efd1972-340a-4f5d-86fb-855bde978e16	10:46
SamYaple	do you have xfs installed on your host?	10:46
inc0	now I have	10:47
SamYaple	yea we should document that	10:48
SamYaple	parted and xfs required	10:48
SamYaple	on host	10:48
inc0	still, didn't help	10:48
inc0	apt-get install xfsprogs <- that one?	10:49
SamYaple	well what it is doing in that exact moment is trying to mount the filesystem	10:49
SamYaple	figure that ouy	10:49
SamYaple	mount -t xfs /dev/sdb1 /mnt	10:49
SamYaple	somethings not working	10:49
SamYaple	arrrrgggggg	10:50
SamYaple	I have it a bad kernel in the gate	10:50
SamYaple	all is lost sdake	10:50
SamYaple	i cannot into gating	10:50
SamYaple	we have to do changes to all the bootstrap scripts :(	10:50
inc0	mounting worked:/	10:50
SamYaple	ok, well just poke around there	10:51
SamYaple	thats literally all its trying to do	10:51
inc0	yeha I have it	10:51
inc0	it's trying to mount different uuid	10:51
inc0	where does it take it from?	10:51
SamYaple	i forget	10:51
SamYaple	`blkid /dev/sdb*`	10:52
SamYaple	it scrapes that	10:52
inc0	is it possible that inside container it has different fs_uuid?	10:55
SamYaple	i doubt it	10:56
SamYaple	but it could be a kernel caching thing	10:57
SamYaple	whats the output from `blkid /dev/sdb*`	10:58
inc0	that's the thing, it's different	10:59
SamYaple	well ansible runs that on the host not hte container	10:59
SamYaple	so its not a container thing	10:59
inc0	ansible shows: b0a25c1d-35ab-4731-a6ec-021b6e73f8a1 while normal blkid is "a61e6cbf-5b5b-42e5-96ec-3d05ca78bcdf"	10:59
SamYaple	reset and run it again	11:00
*** sdake has quit IRC		11:00
SamYaple	if ansible tries to mount a61e6cbf its gotta be a kernel caching thing	11:00
inc0	ansible tries b0a...	11:00
SamYaple	always b0a	11:01
SamYaple	are you sure you reset	11:01
inc0	I'm rebooting now	11:01
inc0	maybe that's because its b0a, and ansible scripts are in python?	11:02
inc0	<< badum tss >>	11:02
SamYaple	i dont follow	11:02
inc0	boa is different kind of snake	11:02
SamYaple	im kidding im being mean	11:02
inc0	well, I did explain it for everyone who were ashamed to say they don't follow	11:03
inc0	something is very, very wrong	11:05
inc0	now boot can't start the disk	11:05
SamYaple	did oyu clean up /etc/fstab?	11:05
inc0	same uuid as ansible shown before	11:05
SamYaple	umount /var/lib/ceph/osd/; rm -rf /var/lib/ceph/osd/ /etc/kolla/ceph-*; sed -ir '/ceph/d' /etc/fstab; parted /dev/sdb -s -- mklabel gpt	11:06
SamYaple	make sure to always run that	11:06
SamYaple	i mean when you are tearing down an environment	11:06
inc0	yay read-only file system	11:07
inc0	it seems my machine is busted	11:07
SamYaple	i tolded you	11:07
inc0	any way to change that? recovery mode can't	11:08
SamYaple	mount it readwrite	11:09
SamYaple	lurn2linux man	11:09
SamYaple	mount -o remount,rw /	11:09
inc0	I'm just a dev	11:11
SamYaple	so you admit it. operators rule, devs drool	11:12
inc0	ops have egos big enough that it's often good to appeal to one	11:13
SamYaple	esspecially then you cant do the thing you need to ;)	11:13
inc0	exactly, its called effective manipulation	11:14
SamYaple	sure thing. thats why we are here. to do all the things you dont know how to	11:14
inc0	so we can focus on moving things forward;)	11:15
SamYaple	yea would hate to be dragged down with making it work	11:15
*** Kennan2 has joined #kolla		11:16
*** Kennan has quit IRC		11:16
inc0	damn, now my / is read-only	11:17
inc0	it's working tho after remount, but where to make it mount as rw by default?	11:18
SamYaple	/etc/fstab	11:18
SamYaple	comeon man	11:18
inc0	yeah, I though / is not in fstab	11:19
SamYaple	it is	11:19
SamYaple	dont worry	11:19
inc0	duh my fstab is busted	11:20
inc0	but I'll sort this out	11:20
inc0	also I know why my fstab got busted..	11:23
SamYaple	ansible writes to /etc/fstab for persistent mounts you know	11:23
inc0	my lvm group has "ceph" in it so your clean scripts removed stuff	11:23
SamYaple	yea that would do it	11:24
inc0	still, same error as before	11:24
inc0	different uuid set up by ansible than correct one	11:25
SamYaple	yes but what is the uuid	11:25
inc0	hold on	11:27
inc0	I have no idea where it takes this id from	11:29
SamYaple	what ar ethe uuids	11:29
inc0	SamYaple, do parted -l on your machne plz	11:30
SamYaple	answer my question first	11:30
SamYaple	im trying ot confirm something and youre making it difficult	11:30
inc0	/dev/sdb1: UUID="8bdc5651-5367-4380-98c1-21585fdce5b7" TYPE="xfs" PARTLABEL="KOLLA_CEPH_DATA" PARTUUID="e25d30ff-871c-45ac-9888-6c407a659ee1" <- blkid by hand	11:31
inc0	failed: [localhost] => (item={u'device': u'/dev/sdb', u'fs_uuid': u'6d7ab8fa-5293-465b-a0cf-8f1a3650955b'}) => {"failed": true, "item": {"device": "/dev/sdb", "fs_uuid": "6d7ab8fa-5293-465b-a0cf-8f1a3650955b"}}	11:31
inc0	msg: Error mounting /var/lib/ceph/osd/6d7ab8fa-5293-465b-a0cf-8f1a3650955b: mount: can't find UUID=6d7ab8fa-5293-465b-a0cf-8f1a3650955b	11:31
inc0	this is output from ansible	11:31
inc0	there is no such id in ls -l /dev/disk/by-uuid	11:32
inc0	well..small disclaimer	11:32
SamYaple	seems like the stale uuid stuff	11:32
inc0	this is ubu 15.04	11:32
inc0	hold on, I'll try to clean and rebuild and we'll see which uuid will pop up	11:33
inc0	in error: 0b79bdab-7a9d-4095-9772-8b7bdacaecd5	11:34
inc0	I haven't seen this uuid anywhere before, so it doesn't seem to be cached	11:34
*** dimsum__ has joined #kolla		11:35
*** dimsum__ has quit IRC		11:41
openstackgerrit	Sam Yaple proposed openstack/kolla: DO NOT MERGE - Gate things https://review.openstack.org/231881	11:42
inc0	I think I have it	11:52
*** CBR09 has quit IRC		11:52
SamYaple	whatd you find	11:54
inc0	'/dev/sdb: PTUUID="94c69718-9a84-4348-beef-83018c7d307d" PTTYPE="gpt"\n/dev/sdb1: UUID="cd7e020d-acd7-41f0-a532-aed06f31ff75" TYPE="xfs" PARTLABEL="KOLLA_CEPH_DATA" PARTUUID="be3d754b-556b-4bdc-a47e-e29ea51c8fb8"\n/dev/sdb2: PARTLABEL="KOLLA_CEPH_JOURNAL" PARTUUID="812a9fb3-9814-425e-848a-52c0afec68ee"\n'	11:59
inc0	this is output from blkid	11:59
inc0	so it has /dev/sdb with PTUUID	11:59
SamYaple	ok yea thats simple	11:59
SamYaple	hold on ill fix it	11:59
inc0	I'll look into libs that does that stuff instead of scrapping outputs	12:00
inc0	just change blkid /dev/sdb* to /dev/sdb1	12:01
inc0	you have part number in "line" var	12:01
SamYaple	there are no libs for this	12:03
SamYaple	and i refuse to use pyparted as its awful	12:03
SamYaple	i believe i made a note about that	12:03
inc0	http://xzased.github.io/reparted/ this doesn't look that bad at first glance	12:05
SamYaple	what do you think we are doing here?	12:06
inc0	all we need is to find uuid	12:06
SamYaple	that wont help yo uwith blkid	12:06
*** akscram has quit IRC		12:06
inc0	anyway, I'll look into it later	12:06
SamYaple	please do i hate screen scraping	12:07
inc0	let's get this working first	12:07
SamYaple	but additional libraries are no good either	12:07
inc0	no argument there	12:07
inc0	still better than screen scraping	12:07
inc0	and with ansible container it's not terrible either	12:07
inc0	would need to run this inside ansible container th	12:07
inc0	o	12:07
SamYaple	yea but then you need to keep /dev:/dev bound into EVERY ansible container	12:07
SamYaple	its awful	12:07
*** akscram has joined #kolla		12:08
inc0	maybe it's just changing commands to something more...scrapable	12:08
inc0	because parted -l is cool for reading	12:08
SamYaple	:)	12:08
SamYaple	good luck	12:09
SamYaple	this is the most distiled version i could come up with	12:09
SamYaple	off and on working on it for 6 months	12:09
SamYaple	wihtout libraries its a mess	12:09
SamYaple	also i want to point out i asked oyu to run `blkid /dev/sdb*` like right off the bat	12:10
SamYaple	and oyu didnt	12:10
SamYaple	that would have identifed the issue immediately	12:11
inc0	I did...just after ansible fail	12:11
SamYaple	would have show it still	12:11
inc0	it didnt show it it	12:11
SamYaple	it shows PTUUID	12:11
inc0	I know, there was no PTUUID when I was calling it	12:12
inc0	for some reason	12:12
SamYaple	you know it doesnt relaly matter :) ill get it fixed right up	12:12
SamYaple	hehe youll love this one	12:13
inc0	scraping with grep in it? -.-	12:13
SamYaple	ew no	12:13
openstackgerrit	Sam Yaple proposed openstack/kolla: Ignore the 'Disk Flags:' line in parted https://review.openstack.org/233441	12:14
SamYaple	better!	12:14
inc0	....	12:14
*** diogogmt has quit IRC		12:16
inc0	well that's a nope Sam;)	12:16
inc0	but that might be me	12:18
*** diogogmt has joined #kolla		12:18
SamYaple	yea thats how its going to go unless it just doesnt work for some reason	12:20
SamYaple	i want a library more than anything, but there isn't one yet	12:20
inc0	it's not me, now bootstrap is busted	12:20
inc0	let me fix this one	12:20
SamYaple	i need more than that	12:20
*** dimsum__ has joined #kolla		12:24
openstackgerrit	Sam Yaple proposed openstack/kolla: DO NOT MERGE - Gate things https://review.openstack.org/231881	12:31
inc0	debugging ansible modules is pain.	12:33
SamYaple	you could just give me logs and i could fix it	12:39
SamYaple	or ask how to properly do debugging	12:39
SamYaple	bad dev!	12:39
larsks	SamYaple: Why are you scraping output from parted instead of just taking JSON output from lsblk?	12:41
SamYaple	larsks: o/	12:42
larsks	:)	12:42
SamYaple	looking at the PARTITION_NAME	12:42
*** diogogmt has quit IRC		12:42
openstackgerrit	Michal Jastrzebski (inc0) proposed openstack/kolla: Ignore the 'Disk Flags:' line in parted https://review.openstack.org/233441	12:43
larsks	Hmmm, and nothing else gives you that? That does seem annoying.	12:43
inc0	just do not, and I mean DO NOT -1 it because of lambda	12:43
SamYaple	why did you patch over my stuff inc0	12:44
inc0	because it works this way	12:44
SamYaple	not when im actively working on the patch...	12:44
inc0	it broke on OSD bootstrap	12:44
inc0	well, feel free to disregard my patch	12:44
SamYaple	yea larsks its a bit annoying. ive got a note in module. i want a better solution!	12:45
SamYaple	you really like to make things complicated inc0	12:45
larsks	SamYaple: My system has /dev/disk/by-partlabel, which seems to be exactly what you want.	12:47
SamYaple	larsks: alas i cannot rely on that	12:47
SamYaple	not all systems do	12:48
larsks	Too recent?	12:48
SamYaple	yea I believe so	12:48
SamYaple	there are nasty hackish things in ceph to work around this issue as well	12:48
larsks	What's the minimal supported kernel for kolla?	12:48
SamYaple	hold on ill find some comments	12:48
SamYaple	larsks: there is no minum	12:48
SamYaple	minimum*	12:49
SamYaple	whatever	12:49
SamYaple	larsks: https://github.com/ceph/ceph/blob/master/src/ceph-disk#L72	12:50
SamYaple	that talks a bit about it	12:50
SamYaple	anyway 3.19 kernel ubuntu 14.04 no partlabel	12:50
SamYaple	its in 15.04 and the newest debian though	12:50
SamYaple	but cant rely on it so cant use it :(	12:50
inc0	it's alive!	12:51
larsks	Bummer,	12:51
inc0	my ceph is working at last!	12:51
SamYaple	w00t inc0 bout time	12:52
inc0	http://paste.openstack.org/show/475983/ SamYaple please confirm it's working	12:53
SamYaple	larsks: im all ears for better solutions!	12:53
inc0	tell me this is how working ceph look like	12:53
SamYaple	inc0: yea that looks good enough	12:53
SamYaple	its not actually working working, but mostly	12:53
inc0	well, no replicas and such	12:53
inc0	but its aio	12:53
inc0	so SamYaple feel free to disregard this one-like-that-fixes-it-but-has-lambda-in-it-so-its-evil	12:56
SamYaple	dude its really bad	12:56
inc0	man, let's face it, whole this module is not so good	12:57
inc0	and let's look at how to fix it	12:58
inc0	just...not now	12:58
openstackgerrit	Merged openstack/kolla: Glance round robin for default file backend https://review.openstack.org/233429	12:58
inc0	my hatred for parted -l had reached todays limit	12:58
*** klint has quit IRC		12:59
SamYaple	indeedso since you never sent me any logs i dont know why you are saying this doesnt work for you without that horrible lambda filter	12:59
SamYaple	since its doing the same thing	12:59
SamYaple	only without extra steps	12:59
inc0	it didn't work without my lambda filter, because find_disk in bootstrap_osd task didn't return anything	13:00
inc0	because parted created PTUUID not UUID	13:00
SamYaple	yea its not going to return anything	13:00
SamYaple	you should have let the playbooks play through	13:00
SamYaple	since this is all tested	13:00
inc0	it failed for me	13:01
SamYaple	with what logs was my question	13:01
SamYaple	and the lambda and filter is still overkill and bad	13:01
inc0	index error while bootstraping	13:01
inc0	lambda filter AND removal of whitespace	13:01
inc0	before UUID	13:01
inc0	so PTUUID and sich will also be splitted	13:02
SamYaple	it shouldnt be split	13:02
SamYaple	thats the problem	13:02
SamYaple	your code isnt working like you think	13:02
SamYaple	it should be the partition not the disk	13:02
inc0	my code only shows partition which is named by partition_name	13:02
SamYaple	thats why i grabbed " UUID"	13:02
SamYaple	so does the priop patch	13:02
SamYaple	prior*	13:03
SamYaple	your code is breaking on my system since it finds PARTUUID	13:04
inc0	sorry, its PARTUUID="85d07d55-82c3-450f-acd9-e5c15ee59125" after parted /dev/sdb -s -- mklabel gpt mkpart KOLLA_CEPH_OSD_BOOTSTRAP 1 -1	13:04
SamYaple	yea you DONT want that	13:04
SamYaple	ugh now i have to make some change to it to repush it up	13:05
SamYaple	can you tell me why the other code didnt work?	13:05
SamYaple	did you even try it?	13:05
inc0	ofc I did	13:05
SamYaple	so what was wrong with patchset 4	13:05
SamYaple	because patchset 5 does _NOT_ do what its suppose to	13:05
inc0	index error on line 74	13:06
inc0	since nothing there had uuid	13:06
inc0	and I just deployed ceph again with my patchset	13:06
SamYaple	oh thats a simple fix	13:06
SamYaple	without all that lambda nonsense	13:06
openstackgerrit	Sam Yaple proposed openstack/kolla: Ignore the 'Disk Flags:' line in parted https://review.openstack.org/233441	13:06
SamYaple	if that doesnt break my system ill let you know	13:07
larsks	SamYaple: udev appears to give you partition names, even in trusty. There are udev bindings for Python, as well, so it's pretty simple.	13:08
larsks	E.g., pyudev.Device.from_name(ctx, 'block', 'dm-0')['ID_PART_ENTRY_NAME']	13:09
SamYaple	larsks: that would work best	13:09
SamYaple	whats the overhead for that library	13:10
SamYaple	oh its tiny	13:10
SamYaple	yay	13:10
SamYaple	time to find out if i can loop through all available devices	13:10
SamYaple	fantastic larsks thats much better	13:11
inc0	SamYaple, http://paste.openstack.org/show/475984/	13:13
inc0	on your patchset:P	13:13
inc0	but cool, let's pursue this udev thing	13:14
SamYaple	inc0: yea im working it up right now, but it is a dep	13:14
SamYaple	that would mean the kolla_ansible container....	13:14
SamYaple	its a much better solution though	13:14
SamYaple	i really hate screen scraping	13:15
inc0	I'm going off for now	13:15
inc0	cya, thanks for help	13:15
SamYaple	help me out with the udev thing later inc0	13:15
SamYaple	ill need some reviews on it	13:16
SamYaple	problably tommorow	13:16
inc0	sure	13:16
inc0	if you submit anything today I can do it first thing tomorrow	13:16
SamYaple	im going to bed	13:17
SamYaple	so probably not	13:17
inc0	go to bed	13:17
inc0	I'm going to spend rest of sunday far away from anything work-related	13:17
inc0	cya tomorrow then	13:17
SamYaple	cool	13:17
SamYaple	just finished building the images for ceph	13:18
SamYaple	gonna test real quick	13:18
SamYaple	alright i see the problem with why your patch broke for me inc0	13:20
SamYaple	and this latest patch won't work	13:20
SamYaple	this one does....	13:20
openstackgerrit	Sam Yaple proposed openstack/kolla: Ignore the 'Disk Flags:' line in parted https://review.openstack.org/233441	13:20
inc0	yeah it should	13:21
SamYaple	horrible i know but this fixes it for you for the time being and doesnt break it for me	13:21
SamYaple	udev will fix it but then we have to bind /dev in for kolla_ansbile :(	13:21
SamYaple	oh wait	13:21
SamYaple	udev in containers == bad	13:21
SamYaple	ugh ill think about it later	13:21
SamYaple	see ya	13:21
SamYaple	thanks larsks	13:21
inc0	cya, it's working	13:22
inc0	thanks SamYaple	13:23
*** inc0 has quit IRC		13:25
openstackgerrit	Sam Yaple proposed openstack/kolla: DO NOT MERGE - Gate things https://review.openstack.org/231881	13:26
*** jainman has joined #kolla		13:36
jainman	Bug 1500245] Re: Consistently Failing - Deploy OpenStack all in one node using Ansible - Want to update doc for closure of issue	13:37
openstack	bug 1500245 in kolla "Consistently Failing - Deploy OpenStack all in one node using Ansible" [Undecided,Triaged] https://launchpad.net/bugs/1500245	13:37
jainman	Should I fork the master branch for changes - please advise	13:38
*** dimsum__ has quit IRC		13:44
*** dimsum__ has joined #kolla		13:50
*** dimsum__ has quit IRC		13:56
*** dimsum__ has joined #kolla		13:56
larsks	jainman: you would typically create a new branch for whatever changes you're making. It doesn't really matter where you do your work locally because all changes get submitted through gerrit reviews.	14:01
nihilifer	jainman: please look at http://docs.openstack.org/infra/manual/developers.html	14:04
nihilifer	or watch https://www.youtube.com/watch?v=DX9NYMZgj80	14:06
nihilifer	generally, we are making changes via gerrit on review.openstack.org	14:06
nihilifer	github is only a mirror, so please don't make pull requests on github	14:06
jainman	Ok, Let me go through it, at work I create a branch and checkin followed by pull request	14:09
*** jmccarthy has quit IRC		14:10
*** kysse has joined #kolla		14:50
kysse	hello guys. So what is your opinion? is kolla production ready and what's the status of high availability? (and what is meant to be ha)	14:51
*** jmccarthy has joined #kolla		14:53
*** dims_ has joined #kolla		15:04
*** dimsum__ has quit IRC		15:06
*** jmccarthy1 has joined #kolla		15:08
*** jmccarthy has quit IRC		15:08
*** dims_ has quit IRC		15:40
*** jmccarthy1 has quit IRC		16:10
*** dimsum__ has joined #kolla		16:10
*** jmccarthy has joined #kolla		16:10
*** dimsum__ has quit IRC		16:37
*** jmccarthy has quit IRC		17:11
*** dimsum__ has joined #kolla		17:37
*** dimsum__ has quit IRC		17:42
*** vinkman has joined #kolla		17:57
*** jmccarthy has joined #kolla		18:15
jmccarthy	SamYaple: You about ?	18:17
*** dimsum__ has joined #kolla		18:39
*** dimsum__ has quit IRC		18:45
*** sdake has joined #kolla		19:23
sdake	morning	19:26
*** achanda has joined #kolla		19:26
*** sdake_ has joined #kolla		19:34
*** sdake has quit IRC		19:37
*** dimsum__ has joined #kolla		19:41
*** dimsum__ has quit IRC		19:47
*** dimsum__ has joined #kolla		20:04
*** jainman has quit IRC		20:23
*** Slower has joined #kolla		20:29
*** sdake has joined #kolla		20:38
*** sdake_ has quit IRC		20:39
*** achanda has quit IRC		20:48
*** jmccarthy has quit IRC		20:57
*** achanda has joined #kolla		21:11
*** sdake has quit IRC		21:15
*** sdake has joined #kolla		21:26
*** achanda has quit IRC		21:28
*** alisonh has joined #kolla		21:34
*** achanda has joined #kolla		22:28
*** achanda has quit IRC		22:34
*** dimsum__ has quit IRC		22:44
*** asalkeld has quit IRC		22:46
*** achanda has joined #kolla		22:53
*** asalkeld has joined #kolla		23:06
*** dimsum__ has joined #kolla		23:09
*** dimsum__ has quit IRC		23:23
*** mbound has quit IRC		23:34
*** achanda has quit IRC		23:38

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!