Friday, 2014-02-14

derekh	lifeless: so before you redeploy a new testenv, shall we	00:00
derekh	for p in $(neutron port-list \| grep te_testenv \| awk '{print $2}') ; do neutron port-delete $p ; done	00:00
lifeless	derekh: we need to make this less manual - can you add that to the TODO list on the etherpad and perhaps file a bug/trello card?	00:02
derekh	lifeless: yup, will do,	00:02
SpamapS	lifeless: You know...	00:03
SpamapS	lifeless: there's this thing that orchestrates APIs..	00:03
derekh	lifeless: and running command	00:03
lifeless	derekh: I'm on it	00:03
derekh	lifeless: k	00:03
* SpamapS runs off to fetch the rugrats		00:03
lifeless	derekh: spurious use of grep ftw btw	00:03
lifeless	SpamapS: yeah, and when you get involved in this stuff you can throw stones ;)	00:03
lifeless	SpamapS: (please do!)	00:04
lifeless	derekh: btw:	00:05
lifeless	neutron port-list \| awk '/te_testenv/ {print $2}' \| xargs -n1 neutron port-delete	00:05
lifeless	derekh: more compact	00:05
derekh	lifeless: thanks	00:05
lifeless	deploying	00:11
lifeless	grabbing food while it churns	00:12
*** e0ne has joined #tripleo		00:14
ccrouch	rwsu: did i get confused here: https://review.openstack.org/#/c/72524/3/scripts/devtest_variables.sh ?	00:15
lifeless	ccrouch: I think so :)	00:18
*** e0ne has quit IRC		00:19
ccrouch	doh :-(	00:23
*** cadenzajon has quit IRC		00:24
ccrouch	lifeless: but given he took out the sed of overcloud.yaml https://review.openstack.org/#/c/72524/2/scripts/devtest_overcloud.sh	00:24
ccrouch	how will the heat template know to use rabbit on fedora?	00:25
openstackgerrit	A change was merged to openstack/tripleo-image-elements: Seed specific Fedora network configuration https://review.openstack.org/71998	00:29
ccrouch	lifeless: the exports aren't deleted, they are just moved down. So MESSAGING_BACKEND can be inserted	00:30
openstackgerrit	A change was merged to openstack/os-refresh-config: Move migration to run after post-configure https://review.openstack.org/71977	00:31
*** ramishra has joined #tripleo		00:32
*** ramishra has quit IRC		00:37
openstackgerrit	Derek Higgins proposed a change to openstack/tripleo-incubator: Restore generation of ssh keypair https://review.openstack.org/73458	00:39
lifeless	ccrouch: heat shouldn't need to know. if the parameters are really different then you need to either a) expose them both or b) have make create an appropriately specialised heat template	00:39
derekh	lifeless: ^^ did you mean to remove that here https://review.openstack.org/#/c/73089/7/scripts/install-dependencies	00:39
lifeless	derekh: I did	00:40
lifeless	derekh: its testenv specific. look in devtest_testenv.sh	00:40
lifeless	ok \| 3a54ea68-4921-4a33-9b19-dc189a2279d7 \| testenv \| CREATE_COMPLETE \| 2014-02-14T00:11:22Z \|	00:41
lifeless	so, we should be good to ci something now, in principle	00:42
derekh	lifeless: ahh ok, so maybe I'm not understanding something, shouldn't we have a ~/.ssh/id_rsa and place that into authorized_keys for people running devtest locally? so the combination of local-config/stackuser works	00:43
lifeless	derekh: + cat ~/.ssh/id_rsa_virt_power.pub >> ~/.ssh/authorized_keys	00:43
lifeless	derekh: no, we inject the ssh key to use via the testenv ssh-key key.	00:43
*** CaptTofu has quit IRC		00:43
derekh	lifeless: yes I see that but we also need a non virt_power keypair (I'm talking about non ci runs)	00:44
derekh	lifeless: will try a local fresh run and see what happen and abandon if all is ok	00:44
lifeless	derekh: huh why?	00:44
derekh	lifeless: so authorized_keys is place in ~stack/.ssh/authorized_keys on images , people will have to start using ssh stack@192.0.2.2 -i ~/.ssh/id_rsa_virt_power to ssh in as stack	00:46
derekh	anyways will check and report back	00:46
lifeless	derekh: i think we have crossed wires. OTP for a minute	00:48
*** ccrouch has quit IRC		00:55
lifeless	derekh: back	00:55
lifeless	so looking now...	00:55
lifeless	dprince: hey	00:59
lifeless	dprince: so I keep poking - can you remove your -1 on https://review.openstack.org/#/c/71706/. Or ask a followup question.	00:59
*** edmund has quit IRC		01:00
jog0	lifeless: ConnectionFailed: Connection to neutron failed: Maximum attempts reached	01:01
jog0	in n-apu	01:01
jog0	n-api	01:01
openstackgerrit	lifeless proposed a change to openstack/tripleo-incubator: Make the overcloud-user config identify itself https://review.openstack.org/73462	01:02
openstackgerrit	lifeless proposed a change to openstack/tripleo-incubator: Permit specifying ovs bridge mappings too. https://review.openstack.org/71706	01:02
openstackgerrit	lifeless proposed a change to openstack/tripleo-incubator: Permit setting a /etc/hosts entry for the API. https://review.openstack.org/71727	01:02
lifeless	jog0: ah so yes, neutron	01:02
derekh	hmm, I can't ssh to any of the testenvs	01:02
*** ccrouch has joined #tripleo		01:02
lifeless	derekh: wups	01:02
*** tzumainn has quit IRC		01:02
lifeless	derekh: I can't ping the first few I've tried	01:02
lifeless	derekh: but maybe ...	01:03
derekh	an none of them have registered with neutron on the overcloud	01:03
lifeless	nuts	01:03
lifeless	I was just about to look	01:03
lifeless	console time I guess	01:03
lifeless	derekh: /opt/stack/diskimage-builder/bin/disk-image-create -a amd64 -o testenv-worker stackuser ubuntu testenv-worker dhcp-all-interfaces hosts -p git -p build-essential -p python-dev -p libssl-dev -p python-pip	01:03
lifeless	derekh: is how I built the image	01:03
jog0	lifeless: neutron api doesn't have any stacktraces :/	01:04
jog0	lifeless: and its getting traffic	01:05
lifeless	jog0: yeah, great eh :)	01:05
derekh	lifeless: When I last did it I did ./diskimage-builder/bin/disk-image-create -a amd64 -o testenv-worker ubuntu testenv-worker dhcp-all-interfaces hosts	01:05
lifeless	derekh: right, I just bake in the packages	01:05
lifeless	derekh: since it doesn't have networking	01:05
lifeless	derekh: erm, no, confused robert. I think I was working around unlanded bugs or something	01:06
jog0	lifeless: unfortunately I have to go AFK in a few minutes (dinner time)	01:06
lifeless	jog0: ciao	01:06
lifeless	derekh: I'll look at a console in a minute (or you can - the creds are all in the nova bm db ...	01:06
*** ccrouch has quit IRC		01:06
derekh	lifeless: will look, have never accessed console before	01:07
lifeless	derekh: https://review.openstack.org/#/c/71727/ should be trivially +A able - it had 2x+2 before I rebased it to the bottom of the stack	01:07
openstackgerrit	lifeless proposed a change to openstack/tripleo-incubator: Stop trying to copy out SSH keys from notcompute. https://review.openstack.org/73465	01:09
openstackgerrit	lifeless proposed a change to openstack/tripleo-incubator: Make the overcloud-user config identify itself https://review.openstack.org/73462	01:09
openstackgerrit	lifeless proposed a change to openstack/tripleo-incubator: Permit specifying ovs bridge mappings too. https://review.openstack.org/71706	01:09
lifeless	derekh: ok so https://review.openstack.org/73465 is what I think is needed - I missed that before	01:10
*** ccrouch has joined #tripleo		01:10
*** ccrouch has quit IRC		01:10
*** ccrouch has joined #tripleo		01:10
*** epim has quit IRC		01:10
*** taps has quit IRC		01:13
*** e0ne has joined #tripleo		01:13
lifeless	derekh: did you try pinging /all/ the hosts ?	01:14
derekh	lifeless: I tried ssh to about 5 or 6 of them	01:15
*** ccrouch has quit IRC		01:15
lifeless	I betok, in the console on 10.10.16.148	01:15
derekh	lifeless: +2'd both	01:16
lifeless	derekh: the users own key should let them ssh in	01:16
lifeless	derekh: we certainly don't want to be copying their keys around	01:16
lifeless	148 is meant to be a3edd6e1-dad4-4924-b1b6-49955ec41cda \| testenv-testenv7-bd2uzs6jl22n	01:16
lifeless	but the virtual console shows	01:17
lifeless	Ubuntu 12.10 compute-test tty1	01:17
lifeless		01:17
lifeless	compute-test login: [ 255.895626] block nbd15: Unexpected reply (ffff8817d249dc	01:17
lifeless	odd	01:17
*** e0ne has quit IRC		01:18
lifeless	oh nup.got it to reset. weird	01:18
derekh	lifeless: talking about the patch to generate id_rsa ? agreed, we shouldn't be copying it around but we should generate it if it doesn't exist	01:18
lifeless	stack:stack fails	01:19
lifeless	prompt is ubuntu	01:19
lifeless	I'm thinking metadata server hookup fail	01:19
lifeless	derekh: mmm, I see. I'm not sure I agree :)	01:19
lifeless	resetting the box, seeing what happens	01:20
lifeless	derekh: we probably need a waitcondition on the testenv heat template to avoid silent failures like this	01:20
derekh	lifeless: that is what we allways did, it needs to be generated so that the public key finds its way into authporized_keys for the stack user	01:20
derekh	lifeless: agreed	01:20
derekh	lifeless: will add to trello	01:20
lifeless	k	01:21
lifeless	restarted n-d-a just in case we have that lease problem too	01:21
*** morazi has joined #tripleo		01:24
lifeless	Cloud-init v. 0.7.3 running 'init-local' at Thu, 13 Feb 2014 21:05:21 +0000. Up	01:26
lifeless	22.83 seconds.	01:26
lifeless	cloud-init-nonet[24.33]: waiting 10 seconds for network device	01:26
lifeless	cloud-init-nonet[34.37]: waiting 120 seconds for network device	01:26
lifeless	which is a good start	01:26
lifeless	got a kernel and all	01:26
rwsu	lifeless: question about https://review.openstack.org/#/c/72524/3/scripts/devtest_variables.sh	01:26
lifeless	rwsu: shoot	01:26
openstackgerrit	lifeless proposed a change to openstack/tripleo-image-elements: Look for local.json in preference to config.json. https://review.openstack.org/73401	01:27
rwsu	lifeless: can I go back and use the $MESSAGING_BACKEND variable like I have it in patch 2?	01:27
lifeless	rwsu: ah so when I said 'just put in DIB_COMMON_ELEMENTS' I meant within that file only	01:29
lifeless	rwsu: since that file is seedrc specific	01:29
lifeless	rwsu: I was wrong in https://review.openstack.org/#/c/72524/2/scripts/devtest_overcloud.sh	01:29
rwsu	ok, put it in DIB_COMMON_ELEMENTS for seed and undercloud, but for overcloud I'll have to use $MESSAGING_BACKEND	01:30
lifeless	derekh: also 2014-02-13 21:09:40,240 - util.py[WARNING]: Failed: growpart /dev/sda 3 might be relevant	01:30
rwsu	lifeless: is that ok?	01:30
derekh	lifeless: so on my clean run of devtest, devtest can't ssh to the seed "Permission denied (publickey,gssapi-keyex,gssapi-with-mic)." keeps looping	01:31
derekh	lifeless: http://paste.openstack.org/show/65240/	01:31
lifeless	SpamapS: so for seed and undercloud, I suggest putting it in https://review.openstack.org/#/c/73032/	01:31
derekh	lifeless: which is why I think we need that patch	01:31
lifeless	rwsu: bah, SEED_DIB_EXTRA_ARGS and UNDERCLOUD_DIB_EXTRA_ARGS	01:31
lifeless	rwsu: for overcloud we need a new feature to let you add elements to just the control plane	01:31
lifeless	rwsu: which I'd make a separate patch	01:31
derekh	lifeless: yup, growpart looks like the problem	01:32
lifeless	rwsu: so start with 73032, add a patch on that to finesse overcloud images, and then your actual thing on top	01:32
lifeless	derekh: you don't have an ssh key ?	01:32
lifeless	derekh: hangon, I'm super confused here.	01:32
lifeless	derekh: too much context switching. Lets start with ssh.	01:33
*** ramishra has joined #tripleo		01:33
derekh	lifeless: I removed it, the point is we specificaly used to generate it if it wasn;t there and place the public key into to hosts authorized keys for exactly this reason	01:33
derekh	because that authorized_keys file is what ends up getting placed into authorized_keys for both root and stack in the image	01:34
lifeless	derekh: this isn't making sense to me.	01:35
lifeless	derekh: we have two situations we run in	01:35
lifeless	a) people, b) ci.	01:35
lifeless	which one are you simulating ?	01:35
rwsu	lifeless: ok I can do that, but I find using *_EXTRA_ARGS a bit confusing, isn't the message broker required?	01:35
lifeless	rwsu: it is,so you should be appending it to the args, no ?	01:36
derekh	lifeless: a) people , specifically so that this will work https://github.com/openstack/diskimage-builder/blob/master/elements/local-config/install.d/62-ssh-key#L9	01:36
lifeless	derekh: so people have keys already. I don't think deleting the key is a good test.	01:37
lifeless	derekh: or we can halt and tell them to get their key on their, but adding a new key without warning is pretty odd.	01:37
derekh	lifeless: even if they have a key, we need to place the public part of the pair into authorized_keys on the host	01:37
*** ramishra has quit IRC		01:37
lifeless	derekh: why?	01:38
lifeless	(I'm not trying to be difficult, there are multiple things we use ssh for, I want to be unambiguous)	01:38
derekh	lifeless: because that authorized_keys file is what get placed onto the image of the seed and ends up allowing us to ssh in as root	01:38
lifeless	ok, so my concern there is that adding an entry to authorized_keys lets other people ssh into their machine.	01:39
lifeless	derekh: the failure is dib - dprince's dhcp-all-interfaces landed and broke	01:39
*** nosnos has joined #tripleo		01:39
lifeless	yup	01:40
derekh	lifeless: yes it does, but they would have to have access to the private key that is already on the host and its what we always did and removing that code broke devtest	01:40
lifeless	derekh: context switch for a secnod please	01:42
derekh	lifeless: k	01:42
openstackgerrit	lifeless proposed a change to openstack/diskimage-builder: Revert "Support adding DHCP interfaces one at a time. " https://review.openstack.org/73472	01:43
openstackgerrit	lifeless proposed a change to openstack/diskimage-builder: Revert "Add Fedora DHCP interfaces via udev rules." https://review.openstack.org/73473	01:43
openstackgerrit	lifeless proposed a change to openstack/diskimage-builder: Better apt-sources docs. https://review.openstack.org/73035	01:43
lifeless	derekh: these ^ need to land to unbreak dhcpa-ll-interfaces	01:43
lifeless	the work isn't baked yet	01:43
lifeless	https://review.openstack.org/73472 and https://review.openstack.org/73473 specifically	01:43
rwsu	lifeless: I don't understand why in boot-seed-vm, we wouldn't put $MESSAGING_BACKEND in ALWAYS_ELEMENT but in SEED_DIB_EXTRA_ARGS, I guess i'm missing something	01:44
*** peoplemerge has quit IRC		01:45
lifeless	rwsu: I'm sorry, I can't focus on this right now - its super late for derekh and CI is down.	01:45
lifeless	rwsu: do what works and looks good to you.	01:45
lifeless	rwsu: the key issue is that we can't put it on the hypervisors	01:45
rwsu	ok, np	01:45
lifeless	thats the proble,m in the current patch,a nd sorry for steering you badly before	01:46
*** wusWWW has quit IRC		01:46
derekh	lifeless: looking, as a matter of interest any idea what the problem was on ubuntu? its hardly causing the growparts problem is it	01:47
lifeless	derekh: growparts isn't a problem	01:47
lifeless	df -h	01:47
lifeless	Filesystem Size Used Avail Use% Mounted on	01:47
lifeless	/dev/sda3 469G 1.7G 448G 1% /	01:47
lifeless	I think its coplaining because it has already grown.	01:48
lifeless	its odd that /mnt isn't there	01:48
lifeless	I have NFI what was up with that, but the fundamental problem is that dhcp-all-interfaces is broken	01:48
lifeless	derekh: I don't think we should go down the rabbit hole - revert to known good, give dan time to get it fixed.	01:49
derekh	lifeless: ok, fair enough if your sure its what the problem is	01:49
lifeless	derekh: totally sure - I watched the boot on the console fail	01:50
lifeless	derekh: logged in on console, no network, eth2 down	01:50
lifeless	thats the problem d-a-i is meant to solve	01:50
derekh	lifeless: +A'd both	01:51
lifeless	thanks	01:51
derekh	brb	01:52
lifeless	derekh: ok	01:52
lifeless	derekh: ssh. so I see your point.	01:52
lifeless	pragmatic side - if we put generation back, will it work? we'll have default ssh into a node work, which we need to run init-keystone	01:52
lifeless	we can file a bug about the abuse of the users authorized_keys files	01:53
lifeless	derekh: so ok, lets put it back, but can we put some prose around it ?	01:53
lifeless	wow, what you want to see aon a server image build	01:55
lifeless	'regenerating fonts cache'	01:55
StevenK	Haha	01:55
openstackgerrit	A change was merged to openstack/diskimage-builder: Revert "Support adding DHCP interfaces one at a time. " https://review.openstack.org/73472	01:56
openstackgerrit	A change was merged to openstack/diskimage-builder: Revert "Add Fedora DHCP interfaces via udev rules." https://review.openstack.org/73473	01:56
derekh	lifeless: I know it doesn't work without it, I'll try now with it back to verify	01:57
openstackgerrit	A change was merged to openstack/tripleo-incubator: Permit setting a /etc/hosts entry for the API. https://review.openstack.org/71727	01:57
openstackgerrit	Richard Su proposed a change to openstack/tripleo-heat-templates: Add username parameter for rabbit https://review.openstack.org/71430	01:57
lifeless	ah heat stack-delete, one machine at a time.	02:01
lifeless	StevenK: seen the remove i386 from foreign arch on server install thread ?	02:02
lifeless	I was considering whether we had that enabled just the other day.	02:02
StevenK	lifeless: I've seen it	02:02
StevenK	I'm currently ambilivant about it on my own servers	02:03
openstackgerrit	Derek Higgins proposed a change to openstack/tripleo-incubator: Restore generation of ssh keypair https://review.openstack.org/73458	02:03
StevenK	Read as: I have a local mirror, so apt-get update is already fast	02:03
derekh	lifeless: added a note to find an alternative, is that what you meant by "add prose" ?	02:04
lifeless	testenv deploy running	02:04
StevenK	lifeless: However, it probably makes sense for us to kill it	02:04
derekh	ok, time I went to bed	02:05
lifeless	derekh: will tweak and push	02:05
lifeless	gnight	02:05
derekh	fingers crossed all comes up this time :-)	02:05
lifeless	derekh: thanks for your help!	02:06
derekh	lifeless: np	02:06
StevenK	lifeless: chroot'd into an amd64 ubuntu image created about 80 minutes ago:	02:06
*** derekh has quit IRC		02:06
StevenK	root@undermined:/# dpkg --print-foreign-architectures	02:06
StevenK	root@undermined:/#	02:06
lifeless	StevenK: ah cool	02:06
StevenK	lifeless: But I agree with you. It makes no sense for cloud images.	02:07
* StevenK tries to teach his muscle memory to use pkill rather than skill		02:07
openstackgerrit	lifeless proposed a change to openstack/tripleo-incubator: Restore generation of ssh keypair https://review.openstack.org/73458	02:12
*** e0ne has joined #tripleo		02:13
*** dprince has quit IRC		02:15
*** dprince has joined #tripleo		02:17
*** e0ne has quit IRC		02:18
lifeless	yay te coming online	02:19
*** CaptTofu has joined #tripleo		02:20
*** ramishra has joined #tripleo		02:30
*** dprince has quit IRC		02:32
lifeless	nuts, after all that we still had 2 te machines fail to deploy	02:33
lifeless	stabbity stab stba	02:33
lifeless	31 test environments; little low :(	02:39
*** ccrouch has joined #tripleo		02:52
openstackgerrit	A change was merged to openstack/tripleo-image-elements: Parameterize geard broker and test env public interface https://review.openstack.org/72305	02:57
lifeless	SpamapS: Ithink some of the fragility is more bad BMC interactions	02:59
lifeless	SpamapS: very slow crotchety embedded machines	02:59
*** newell has quit IRC		03:01
*** e0ne has joined #tripleo		03:13
*** whayutin_ has quit IRC		03:13
lifeless	SpamapS: also remember to restart n-d-a between each deploy :(	03:14
*** vkdrao has joined #tripleo		03:17
*** e0ne has quit IRC		03:18
*** cwolferh has quit IRC		03:19
*** jcooley_ has joined #tripleo		03:26
*** cwolferh has joined #tripleo		03:30
StevenK	Hm	03:39
*** ramishra has quit IRC		03:39
StevenK	Changes to toci writing the datafile isn't escaped	03:39
openstackgerrit	Steve Kowalik proposed a change to openstack-infra/tripleo-ci: Add more escaping the JSON writing https://review.openstack.org/73491	03:44
openstackgerrit	Steve Kowalik proposed a change to openstack-infra/tripleo-ci: Add cpu-{node,mem,disk} to the testenv JSON https://review.openstack.org/73492	03:46
*** ramishra has joined #tripleo		03:47
StevenK	lifeless, SpamapS: 73491 is likely quite important, since it may lead to invalid JSON or bash getting confused	03:48
*** vkdrao has quit IRC		03:51
*** edmund has joined #tripleo		03:59
*** ramishra has quit IRC		04:12
*** ramishra has joined #tripleo		04:12
*** e0ne has joined #tripleo		04:13
*** CaptTofu has quit IRC		04:16
*** e0ne has quit IRC		04:18
*** chrisfoster has quit IRC		04:23
*** UtahDave has joined #tripleo		04:23
*** chrisfoster has joined #tripleo		04:24
*** jcooley_ has quit IRC		04:24
*** saju_m has joined #tripleo		04:39
openstackgerrit	Steve Kowalik proposed a change to openstack-infra/tripleo-ci: Add more escaping when writing the JSON datafile https://review.openstack.org/73491	04:42
openstackgerrit	Steve Kowalik proposed a change to openstack-infra/tripleo-ci: Add cpu-{node,mem,disk} to the testenv JSON https://review.openstack.org/73492	04:42
StevenK	SpamapS: ^ Fixed, sorry	04:42
lifeless	StevenK: and thats why we used python :)	04:45
lifeless	nuts undercloudcloud fail	04:45
lifeless	9 failed testenvs	04:45
lifeless	SpamapS: are yo around ?	04:46
lifeless	SpamapS: I'm betting keystone fail	04:46
StevenK	lifeless: Could it be due to 73491 ?	04:46
lifeless	StevenK: no	04:46
StevenK	Aw	04:46
lifeless	StevenK: \| c2a3c40e-ced6-4695-91be-8ca8524d9c5b \| testenv-testenv3-xrgjgoycffwa \| ERROR \| None \| NOSTATE \| \|	04:46
lifeless	wrong layer	04:46
StevenK	Right	04:46
lifeless	hmm, 87K tokens, thats not in OMG territory yet.	04:48
* lifeless truncates anyhow		04:48
*** epim has joined #tripleo		04:55
SpamapS	lifeless: eh?	04:58
SpamapS	lifeless: we should be keeping tokens under control now that the purge thing actually works	04:58
SpamapS	lifeless: 87k is 1.32 million less than we had back when it was zomg fail ;)	04:58
lifeless	SpamapS: yeah, but I had 9 failed instances :(	05:03
lifeless	SpamapS: I'm flailing around for a cause, since all 10 of thsoe machines deploy ed ok before	05:05
lifeless	SpamapS: (and then totally failed to be useful	05:05
SpamapS	lifeless: are they failing after having pxe booted once into deploy ramdisk?	05:07
SpamapS	lifeless: I saw that a few times but couldn't keep my attention on ILO enough to figure out why.	05:08
lifeless	dunno, I just kicked it all back to start	05:08
lifeless	9 in a row exceeded my per-machine credularity	05:08
SpamapS	Yeah I'm wondering if it is a tftp fail or something silly like that.	05:08
*** ramishra has quit IRC		05:11
*** ramishra has joined #tripleo		05:12
*** jcooley_ has joined #tripleo		05:13
*** e0ne has joined #tripleo		05:13
lifeless	one spawned	05:17
lifeless	9 pending	05:17
lifeless	3.6G in /tftpboot	05:18
lifeless	but only kernels and ramdisks go there	05:18
*** e0ne has quit IRC		05:18
*** akuznetsov has joined #tripleo		05:24
lifeless	no requests in the deploy helper	05:24
lifeless	machine seems stuck on boot-from-hard-drive	05:31
*** epim_ has joined #tripleo		05:53
*** epim has quit IRC		05:54
*** epim_ is now known as epim		05:54
*** saju_m has quit IRC		05:58
*** al has quit IRC		06:01
*** rpodolyaka1 has joined #tripleo		06:02
lifeless	ahha	06:03
lifeless	Stderr: 'Error in open session response message : insufficient resources for session\n\r\nError: Unable to establish	06:03
lifeless	IPMI v2 / RMCP+ session\r\nUnable to get Chassis Power Status\r\n'	06:03
lifeless	SpamapS: ^ le failure	06:03
lifeless	Active Session Count : 59	06:06
* lifeless sees another night of fixing ahead		06:08
lifeless	devananda: ^ this may interest you	06:08
*** sballe has quit IRC		06:11
*** e0ne has joined #tripleo		06:13
*** rcarrillocruz1 has joined #tripleo		06:15
*** CaptTofu has joined #tripleo		06:17
*** akuznetsov has quit IRC		06:17
*** rcarrillocruz has quit IRC		06:17
lifeless	these machines have 4 slots	06:18
lifeless	one for the screen sessions	06:18
*** e0ne has quit IRC		06:18
lifeless	one for periodic checks	06:18
lifeless	one for deploy actions	06:18
lifeless	one for sol - we're at capacity	06:18
*** vkozhukalov has joined #tripleo		06:21
*** CaptTofu has quit IRC		06:21
*** jcooley_ has quit IRC		06:23
*** akuznetsov has joined #tripleo		06:26
openstackgerrit	Richard Su proposed a change to openstack/tripleo-incubator: Messaging backend selection https://review.openstack.org/72524	06:27
*** mrunge has joined #tripleo		06:28
*** rcarrillocruz has joined #tripleo		06:33
*** tzumainn has joined #tripleo		06:33
*** rcarrillocruz1 has quit IRC		06:34
*** lazy_prince has joined #tripleo		07:01
lifeless	24310 ? S 0:00 /usr/bin/expect -f /usr/lib/conman/exec/ipmitool.exp 10.10.16.25	07:02
lifeless	ok, kill -9'd em all	07:03
lifeless	trying a deploy	07:04
lifeless	I wonder, is it possible that these machines share BMCs or something?	07:04
lifeless	that would explain 'deploy N at once' => overload BMC ?	07:04
lifeless	killed the screen session with ilos too	07:08
*** akuznetsov has quit IRC		07:09
*** akuznetsov has joined #tripleo		07:09
*** rcarrillocruz1 has joined #tripleo		07:14
*** rcarrillocruz has quit IRC		07:17
*** sballe has joined #tripleo		07:24
*** saju_m has joined #tripleo		07:25
*** tserong has quit IRC		07:29
*** vkozhukalov has quit IRC		07:33
*** jprovazn has joined #tripleo		07:41
openstackgerrit	A change was merged to openstack/tuskar: Rename Openstack to OpenStack https://review.openstack.org/73212	07:46
*** rpodolyaka1 has quit IRC		07:50
openstackgerrit	A change was merged to openstack/tuskar: Fix delete cascade for attribute and counts https://review.openstack.org/72464	07:50
*** epim has quit IRC		07:51
jprovazn	SpamapS, Hi, you still around?	07:52
*** jistr has joined #tripleo		08:06
*** dko has joined #tripleo		08:09
*** CaptTofu has joined #tripleo		08:17
*** shardy_afk is now known as shardy		08:21
*** bauzas has joined #tripleo		08:21
*** CaptTofu has quit IRC		08:22
*** viktors has joined #tripleo		08:24
*** jtomasek has joined #tripleo		08:32
rpodolyaka	morning tripleo	08:34
*** gcha has joined #tripleo		08:37
shadower	hey	08:38
*** dtantsur has joined #tripleo		08:39
openstackgerrit	Dougal Matthews proposed a change to openstack/tuskar: Use TripleO Heat Merge to manage the stack https://review.openstack.org/52045	08:41
*** tserong has joined #tripleo		08:47
*** rlandy has joined #tripleo		08:50
*** dtantsur has quit IRC		08:54
*** dtantsur has joined #tripleo		08:54
*** markmc has joined #tripleo		08:56
openstackgerrit	Dougal Matthews proposed a change to openstack/tuskar: Use TripleO Heat Merge to manage the stack https://review.openstack.org/52045	08:59
d0ugal	lsmola: ^	08:59
d0ugal	That should make Jenkins happier.	08:59
lsmola	d0ugal: cool	08:59
*** ifarkas has joined #tripleo		09:01
*** vkozhukalov has joined #tripleo		09:05
openstackgerrit	Ladislav Smola proposed a change to openstack/tuskar-ui: Adding real deleting and creating of overcloud https://review.openstack.org/73343	09:07
d0ugal	dtantsur: Looks like that warning in Python 2.7 only	09:07
*** rdopieralski has joined #tripleo		09:07
d0ugal	dtantsur: So, not sure we can do much about it without effecting 2.6 support.	09:08
d0ugal	lsmola: ^	09:08
lifeless	we're not even hitting _cache_tftp_images	09:08
*** e0ne has joined #tripleo		09:09
*** rcarrillocruz has joined #tripleo		09:09
*** al_ has joined #tripleo		09:10
*** al_ is now known as Guest36489		09:10
*** rcarrillocruz1 has quit IRC		09:10
*** derekh has joined #tripleo		09:13
*** Guest36489 has quit IRC		09:15
*** al0 has joined #tripleo		09:16
*** marun has joined #tripleo		09:16
*** al0 has quit IRC		09:18
*** hashar has joined #tripleo		09:18
*** al has joined #tripleo		09:18
*** al has quit IRC		09:20
*** al has joined #tripleo		09:20
*** marun has quit IRC		09:21
*** ramishra has quit IRC		09:22
openstackgerrit	Radomir Dopieralski proposed a change to openstack/tuskar-ui: Add a deployment scaling dialog https://review.openstack.org/73551	09:27
*** lucasagomes has joined #tripleo		09:28
*** ramishra has joined #tripleo		09:28
derekh	lifeless: test env deploy didn't go smoothly ?	09:31
*** dtantsur has quit IRC		09:32
lifeless	derekh: it did	09:32
*** dtantsur has joined #tripleo		09:32
lifeless	derekh: wrong broker address	09:32
ekarlso	hmm, do I need to do something special with bridges in order to get things to work or ?	09:33
lifeless	derekh: redeploy died	09:33
lifeless	debugging	09:33
*** UtahDave has quit IRC		09:33
openstackgerrit	Radomir Dopieralski proposed a change to openstack/tuskar-ui: Update the deployment design page https://review.openstack.org/73238	09:33
ekarlso	had first a problem with ovs + briding, so removed the bridge module and I deleted the default network now it fails to run devtest_seed.sh due to default network missing p	09:33
derekh	lifeless: ahh ok, bugger, if you want me to look at anything just shout.	09:34
*** athomas has joined #tripleo		09:35
lifeless	2014-02-14 09:32:12,684.684 26712 TRACE nova.network.neutronv2.api [instance: 3ab70541-a728-4f06-a556-659413821638] NeutronClientException: 409-{u'NeutronError': {u'message': u'Unable to complete operation for network e85b44c7-1136-4217-954e-cdf0acdddfe1. The mac address 78:e7:d1:23:62:95 is in use.', u'type': u'MacAddressInUse', u'detail': u''}}	09:35
lifeless	might be related	09:35
lifeless	though WTF that is TRACE?!??!??! is beyond me	09:35
derekh	good questions, seems it should be warning at the very least	09:35
lifeless	we've had BMC's run out of sessions	09:37
lifeless	bunch of ZOMG	09:37
lsmola	lifeless: hola	09:41
lsmola	lifeless: is stack-update working for you? https://bugs.launchpad.net/tripleo/+bug/1279011	09:41
lifeless	lsmola: things may be all broken right now, and we can't really tell - was intending to have CI up about 10 hours ago ... but things happened	09:42
lsmola	lifeless: talked with heat guys, the template seems to be fine, so I have no idea :-)	09:42
lifeless	lsmola: what makes this critical?	09:43
lsmola	lifeless: ok, cool	09:43
lifeless	lsmola: I mean - is stack-update broken, or is scaling up not working. These are two different things	09:43
lsmola	lifeless: well I figured that stack-create and update are kind of most important operations we have	09:43
lsmola	lifeless: stack update is broken when you increase a number of computes	09:44
lifeless	lsmola: is it broken when you don't do that ?	09:44
lsmola	lifeless: other cases seems to be working	09:44
lifeless	ok, so its not a regression	09:44
lifeless	its a use case that hasn't been implemented	09:44
lsmola	lifeless: well not sure, critical should be, when there is no workaround, right?	09:45
lsmola	lifeless: oh, it never worked?	09:45
lifeless	noone tried it that I know of	09:45
lsmola	lifeless: hehe ok	09:45
lsmola	lifeless: so should I keep it critical? it is still 1 of the 2 most important features	09:46
lifeless	lsmola: no, criticals are reserved for firedrills, regressions and the like	09:46
lsmola	lifeless: ok, back to high	09:47
lifeless	lsmola: done already	09:47
lsmola	lifeless: ok cool	09:47
lsmola	lifeless: thank you for the feedback	09:47
lifeless	yw	09:47
lifeless		09:49
lifeless	No filename or root path specified	09:49
lifeless	No more network devices	09:49
lifeless	derekh: ^ bad pxe data from neutron. wtfmoment	09:49
lifeless	and yet	09:50
lifeless	tag:9bbc56c0-60f1-4f8e-a5c7-d008686cbb6e,option:bootfile-name,pxelinux.0	09:50
lifeless	tag:9bbc56c0-60f1-4f8e-a5c7-d008686cbb6e,option:server-ip-address,10.10.16.169	09:50
lifeless	visible in neutron	09:50
lifeless	address handed out is 192.168.1.157/	09:51
lifeless	AHHA	09:51
lifeless	derekh: I know whats happening	09:51
*** saju_m has quit IRC		09:51
derekh	hope so, I wasn't looking forward to digging into neutron :-)	09:52
lifeless	derekh: bug 1272803	09:52
lifeless	derekh: just added a comment to the description. Read and weep	09:53
*** vkozhukalov has quit IRC		09:53
* derekh reading		09:53
derekh	ok, I can see how this would be random on each instance, but if an entirer stack succeeds and then the entirer stack passes, seem a bit unrandom	09:56
lifeless	derekh: its entirely explainable	09:58
lifeless	derekh: I forgot to delete the port before redeploying	09:58
derekh	and why would the overcloud be responding to macs of hosts on the undercloud, we have removed the port allocations, suppose it mightn't have deleted the fully	09:58
lifeless	port/ports/	09:58
lifeless	derekh: if one forgets, boom	09:59
derekh	lifeless: ok, makes sense	09:59
derekh	lifeless: so have you deleted them since then, I don't see them now	09:59
lifeless	once I realised	09:59
lifeless	I took action :)	09:59
derekh	lifeless: ok, I buy the explaination, so redeploy and we should be golden	10:00
lifeless	thats whats happening right now	10:01
lifeless	first box is just about to pxe boot	10:01
lifeless	there it goes	10:02
lifeless	deploy disk	10:02
lifeless	woo	10:02
derekh	woot	10:02
lifeless	what a brain-f*ck	10:02
lifeless	the first deploy 9/10 failed, one worked	10:02
lifeless	so I didn't clue in. - like it wasn't obviously 'this doesn't work'	10:02
derekh	sometimes you just need to stare at something for a few hours :-(	10:03
lifeless	no	10:03
lifeless	one never needs that	10:03
lifeless	right, one active	10:03
derekh	ok, need is the wrong word	10:04
derekh	overcloud controller may need more memory : Feb 14 09:48:49 overcloud-notcompute0-t3bybsd652xw kernel: Out of memory: Kill process 5074 (glance-api) score 208 or sacrifice child	10:04
derekh	in user run devtest	10:04
lifeless	really? lol	10:04
lifeless	I think we should file bugs	10:04
lifeless	cause, 2G should be -tonnes- for non-cache pages	10:04
*** vkozhukalov has joined #tripleo		10:05
derekh	total used free shared buffers cached	10:05
derekh	Mem: 2051192 1649940 401252 16708 7484 61072	10:05
derekh	glance-api got killed whill uploading the user image	10:06
derekh	and again, I wonder has it started storing images in memory all of a suddon	10:08
derekh	will take a look, and see what needs to be filed	10:08
derekh	btw, this worked https://review.openstack.org/#/c/73458/ can you remove the -2	10:08
derekh	lifeless: ^	10:08
lifeless	heh, it survived the -2er - me - uploading a fixed patch	10:10
lifeless	ubuntu@te-broker:~$ echo 'status' \| nc localhost 4730 -q 1 \| grep lockenv	10:15
lifeless	lockenv 5 0 8	10:15
lifeless	ok, now we're gassing with cookery	10:15
lifeless	echo 'status' \| nc localhost 4730 -q 1 \| grep lockenv	10:16
lifeless	lockenv 2 2 11	10:16
derekh	testenvs 1 4 and 7 have registered with overcloud neutron	10:16
*** max_lobur_afk is now known as max_lobur		10:17
*** CaptTofu has joined #tripleo		10:18
lifeless	derekh:	10:20
lifeless	2014-02-14 10:18:33.335 \| [2014/02/14 10:18:33 AM] [ERROR] Could not parse metadata file: /tmp/tmpXuYNcJ	10:20
lifeless	from https://jenkins02.openstack.org/job/check-tripleo-seed-precise/306/console	10:21
lifeless	derekh: did you see StevenK	10:21
lifeless	's fix?	10:21
*** nosnos has quit IRC		10:21
derekh	lifeless: looking	10:22
*** CaptTofu has quit IRC		10:23
*** hashar has quit IRC		10:23
derekh	lifeless: crap, I hadn't seen it, +2'd	10:24
*** hashar has joined #tripleo		10:25
derekh	lifeless: if you want I can redeploy it again, once merged, its probably late there	10:26
*** ccorrigan has joined #tripleo		10:27
lifeless	it is a little	10:27
lifeless	derekh: please do	10:29
lifeless	#./deploy-testenv 10.10.16.135 192.168.1.48	10:29
lifeless	is how I deployed	10:29
lifeless	but	10:29
derekh	lifeless: ok, can you +A the patch https://review.openstack.org/#/c/73491/	10:29
lifeless	actually, nvm , full deploy is needed anyhow	10:29
openstackgerrit	A change was merged to openstack-infra/tripleo-ci: Add more escaping when writing the JSON datafile https://review.openstack.org/73491	10:30
derekh	testenv stack deleted / ports deleted and building new image	10:36
derekh	hmm, so would the fact that ports weren't been cleaned up explain a lot of the issues we've been seeing in general with instances not booting	10:41
*** boris-42_ has quit IRC		10:43
derekh	\| f220235a-3f03-470e-a469-f9d7bb76ef67 \| testenv \| CREATE_IN_PROGRESS \| 2014-02-14T10:45:38Z \|	10:45
lifeless	derekh: yes, that bug really is critical :)	10:46
lifeless	gnight	10:50
derekh	gnight, have a good weekend	10:51
*** ramishra has quit IRC		10:55
*** ramishra has joined #tripleo		11:05
openstackgerrit	Dougal Matthews proposed a change to openstack/tuskar: Use TripleO Heat Merge to manage the stack https://review.openstack.org/52045	11:05
*** gcha has quit IRC		11:06
*** CaptTofu has joined #tripleo		11:17
*** CaptTofu has quit IRC		11:18
*** CaptTofu has joined #tripleo		11:19
*** CaptTofu has quit IRC		11:20
*** CaptTofu has joined #tripleo		11:20
*** CaptTofu has quit IRC		11:21
*** CaptTofu has joined #tripleo		11:21
*** CaptTofu has quit IRC		11:24
openstackgerrit	Dougal Matthews proposed a change to openstack/tuskar: Use TripleO Heat Merge to manage the stack https://review.openstack.org/52045	11:26
*** CaptTofu has joined #tripleo		11:27
*** CaptTofu has quit IRC		11:32
*** CaptTofu has joined #tripleo		11:32
openstackgerrit	Dmitry Tantsur proposed a change to openstack/tuskar-ui: Add CRUD operations for node profiles https://review.openstack.org/73576	11:35
*** lazy_prince has left #tripleo		11:35
*** al has quit IRC		11:37
*** al has joined #tripleo		11:39
*** d0ugal has quit IRC		11:41
*** d0ugal has joined #tripleo		11:41
*** al has quit IRC		11:43
*** al has joined #tripleo		11:44
openstackgerrit	Ladislav Smola proposed a change to openstack/tuskar-ui: Adding real index, deleting and creating of overcloud https://review.openstack.org/73343	11:49
dtantsur	Hi everyone. I have a question about architecture field in node profile UI (tuskar-ui). Do we have a fixed list of possible architectures or we'll allow user to input everything there?	11:55
*** boris-42_ has joined #tripleo		11:56
openstackgerrit	Radomir Dopieralski proposed a change to openstack/tuskar-ui: Update the deployment design page https://review.openstack.org/73238	12:04
ekarlso	lifeless: here boss ? You got a clue on my issue	12:06
dtantsur	question #2: does it really makes sense to make ID for node profile settable by user/editable?	12:06
openstackgerrit	Ladislav Smola proposed a change to openstack/tuskar-ui: Adding real index, deleting and creating of overcloud https://review.openstack.org/73343	12:10
openstackgerrit	A change was merged to openstack/tripleo-incubator: Rename Openstack to OpenStack https://review.openstack.org/73208	12:12
openstackgerrit	A change was merged to openstack/tripleo-image-elements: Rename Openstack to OpenStack https://review.openstack.org/73206	12:12
*** hashar has quit IRC		12:12
ekarlso	I'm unable to get the devtest stuff going	12:12
openstackgerrit	A change was merged to openstack/diskimage-builder: Rename Openstack to OpenStack https://review.openstack.org/72902	12:12
openstackgerrit	Dmitry Tantsur proposed a change to openstack/tuskar-ui: Add CRUD operations for node profiles https://review.openstack.org/73576	12:16
openstackgerrit	A change was merged to openstack/tuskar: Use TripleO Heat Merge to manage the stack https://review.openstack.org/52045	12:16
*** ifarkas has quit IRC		12:18
*** ifarkas has joined #tripleo		12:20
openstackgerrit	A change was merged to openstack/diskimage-builder: Teach cache-url to handle file:// URIs. https://review.openstack.org/72964	12:23
ifarkas	dtantsur, you might want to check the #tuskar channel. tuskar folks doesn't read #tripleo that often	12:33
openstackgerrit	A change was merged to openstack/tripleo-incubator: Rename EXTRA_ELEMENTS to SEED_DIB_EXTRA_ARGS https://review.openstack.org/73032	12:33
*** weshay has joined #tripleo		12:34
*** mrunge has quit IRC		12:35
derekh	anybody else having devtest crash when uploading the user qcow image ?	12:39
derekh	https://bugs.launchpad.net/tripleo/+bug/1280275	12:40
ekarlso	anyone here running a devtest in a cloud ?	12:42
*** gcha has joined #tripleo		12:42
markmc	derekh, this ? https://bugs.launchpad.net/openstack-ci/+bug/1280072	12:44
markmc	derekh, https://review.openstack.org/73585	12:44
derekh	markmc: ahh, I hadn't spotted that, looking	12:45
derekh	markmc: seems to be different (but related to the new swift client release), my problem is extra memory usage, this one is a curruption problem, I'll add a link to the memory bug and try that patch	12:48
markmc	derekh, roger	12:48
*** hashar has joined #tripleo		12:55
lsmola	jprovazn: have a minute?	12:55
jprovazn	lsmola, sure	12:55
lsmola	Waiting for SSH hostkey.	12:55
lsmola	Warning: Permanently added '192.168.122.191' (ECDSA) to the list of known hosts.	12:55
lsmola	Permission denied (publickey,gssapi-keyex,gssapi-with-mic).	12:55
lsmola	jprovazn: after i boot seed, this repeats till timeout	12:55
lsmola	jprovazn: seed is fine, but it cant shh to it?	12:56
lsmola	jprovazn: http://paste.openstack.org/show/65415/	12:56
jprovazn	lsmola, seems you don't use the right key when sshing to the seed machine	12:56
lsmola	jprovazn: ok	12:57
lsmola	jprovazn: any idea how to fix it? this is devtest.sh	12:57
jprovazn	lsmola, can you ssh as stackuser?	13:02
lsmola	jprovazn: I have updated tripleo and runnign again, will tell you in 5min	13:03
jprovazn	if yes, then you can put your ssh key to root's authorized_keys	13:03
lsmola	jprovazn: ok	13:03
lsmola	jprovazn: what script should put it there?	13:03
openstackgerrit	Radomir Dopieralski proposed a change to openstack/tuskar-ui: Add a deployment scaling dialog https://review.openstack.org/73551	13:05
derekh	lsmola: try ssh root@192.168.122.XXX -i ~/.ssh/id_rsa_virt_power	13:05
*** lucasagomes is now known as lucas-hungry		13:06
lsmola	derekh: ok, thanks	13:06
derekh	lsmola: and if that works add a +1 to https://review.openstack.org/#/c/73458/	13:06
derekh	or 2 :-)	13:06
lsmola	derekh: ok, great	13:06
lsmola	derekh: ok, I will rather try with your patch applied :-)	13:08
derekh	lsmola: okie dokie	13:08
jistr	lsmola: i hit the same thing as you described to jprovazn	13:16
jprovazn	jistr, derekh has already patch for the issue ^	13:17
jistr	derekh++	13:17
jistr	it works	13:17
jistr	adding +2	13:17
lsmola	jistr: cool	13:17
jistr	after i read it :)	13:18
lsmola	jistr, derekh : I will try it and approve it	13:18
*** howleyt has joined #tripleo		13:20
*** jdob has joined #tripleo		13:21
howleyt	Hi, I'm trying to get devtest up and running and it's failing at some point in the provisioning of the noncompute node in the overcloud:	13:22
howleyt	root@undercloud-undercloud-b2ue6a46w5ag:~# heat stack-list	13:22
howleyt	+--------------------------------------+------------+---------------+----------------------+	13:22
howleyt	\| id \| stack_name \| stack_status \| creation_time \|	13:22
howleyt	+--------------------------------------+------------+---------------+----------------------+	13:22
howleyt	\| 82eab752-5be7-4f44-8956-3c3c9307c757 \| overcloud \| CREATE_FAILED \| 2014-02-11T17:50:01Z \|	13:22
howleyt	+--------------------------------------+------------+---------------+----------------------+	13:22
howleyt	It fails specifically on this resource:	13:23
howleyt	root@undercloud-undercloud-b2ue6a46w5ag:~# heat resource-list overcloud \| grep FAILED	13:23
howleyt	\| notCompute0CompletionCondition \| AWS::CloudFormation::WaitCondition \| CREATE_FAILED \| 2014-02-11T18:38:50Z \|	13:23
howleyt	Any ideas of what is going wrong?	13:23
openstackgerrit	Dmitry Tantsur proposed a change to openstack/tuskar-ui: Add CRUD operations for node profiles https://review.openstack.org/73576	13:24
*** e0ne has quit IRC		13:24
*** e0ne has joined #tripleo		13:25
openstackgerrit	Radomir Dopieralski proposed a change to openstack/tuskar-ui: Add a deployment scaling dialog https://review.openstack.org/73551	13:28
*** vkozhukalov has quit IRC		13:34
howleyt	I also have errors in the os-collect-config.log on my undercloud:	13:35
howleyt	root@undercloud-undercloud-b2ue6a46w5ag:/var/log/upstart# grep -i error os-collect-config.log	13:35
howleyt	error.log	13:35
howleyt	[2014/02/11 04:23:37 PM] [ERROR] key neutron does not exist in ['/var/lib/os-collect-config/heat_local.json', '/var/lib/os-collect-config/ec2.json']	13:35
howleyt	[2014/02/11 04:23:38 PM] [ERROR] key db-password does not exist in ['/var/lib/os-collect-config/heat_local.json', '/var/lib/os-collect-config/ec2.json']	13:35
howleyt	[2014-02-11 16:23:38,013] (os-refresh-config) [ERROR] during configure phase. [Command '['dib-run-parts', '/opt/stack/os-config-refresh/configure.d']' returned non-zero exit status 1]	13:35
howleyt	[2014-02-11 16:23:38,013] (os-refresh-config) [ERROR] Aborting...	13:35
howleyt	2014-02-11 16:23:38.017 1198 ERROR os-collect-config [-] Command failed, will not cache new data. Command 'os-refresh-config' returned non-zero exit status 1	13:35
howleyt	Hide full text	13:35
*** e0ne has quit IRC		13:38
Ng	morning	13:45
rpodolyaka	o/	13:45
*** vkozhukalov has joined #tripleo		13:47
slagle	morning	13:49
openstackgerrit	Jan Provaznik proposed a change to openstack/tripleo-heat-templates: Add rabbit.cookie attribute https://review.openstack.org/70238	14:01
openstackgerrit	A change was merged to openstack-infra/tripleo-ci: Add cpu-{node,mem,disk} to the testenv JSON https://review.openstack.org/73492	14:03
openstackgerrit	Jan Provaznik proposed a change to openstack/tripleo-heat-templates: Add rabbit.cookie attribute https://review.openstack.org/70238	14:09
*** lucas-hungry is now known as lucasagomes		14:10
*** rlandy has quit IRC		14:10
*** tzumainn has quit IRC		14:14
*** tzumainn has joined #tripleo		14:15
howleyt	Hi, anyone free to help me trying to get devtest up and running? It's failing in the provisioning of the noncompute node in my overcloud.	14:23
* Ng runs out to a dr appt		14:27
*** matty_dubs\|gone is now known as matty_dubs		14:31
openstackgerrit	Ralf Haferkamp proposed a change to openstack/tripleo-image-elements: Enable iscsid on openSUSE https://review.openstack.org/73605	14:33
*** lblanchard has joined #tripleo		14:34
*** dprince has joined #tripleo		14:35
jprovazn	howleyt, Hi, can you check if /var/run/os-collect-config//cfn.json exists on your overcloud node?	14:35
openstackgerrit	A change was merged to openstack/diskimage-builder: Remove tox locale overrides https://review.openstack.org/72208	14:36
howleyt	Hi jprovazn, I certainly. I'm seeing this error for two missing keys:	14:37
howleyt	root@undercloud-undercloud-b2ue6a46w5ag:/var/log/upstart# grep -i error os-collect-config.log	14:37
howleyt	error.log	14:37
howleyt	[2014/02/11 04:23:37 PM] [ERROR] key neutron does not exist in ['/var/lib/os-collect-config/heat_local.json', '/var/lib/os-collect-config/ec2.json']	14:37
howleyt	[2014/02/11 04:23:38 PM] [ERROR] key db-password does not exist in ['/var/lib/os-collect-config/heat_local.json', '/var/lib/os-collect-config/ec2.json']	14:37
howleyt	[2014-02-11 16:23:38,013] (os-refresh-config) [ERROR] during configure phase. [Command '['dib-run-parts', '/opt/stack/os-config-refresh/configure.d']' returned non-zero exit status 1]	14:37
howleyt	[2014-02-11 16:23:38,013] (os-refresh-config) [ERROR] Aborting...	14:37
howleyt	2014-02-11 16:23:38.017 1198 ERROR os-collect-config [-] Command failed, will not cache new data. Command 'os-refresh-config' returned non-zero exit status 1	14:37
howleyt	However, those keys do exist in cfn.json	14:37
*** e0ne has joined #tripleo		14:38
jprovazn	howleyt, well, the grepped errors don't have to be critical - os-collect-config runs before cfn.json is fetched from heat metadata server, if you open os-collect-config.log you should see if next iterations of os-collect-config passed	14:40
jprovazn	howleyt, something like "Completed phase post-configure"	14:41
jprovazn	howleyt, if it finished successfully, you might check on undercloud machine: 1) if all stack's intances are active (nova list), 2) heat/nove logs - there should be a log with reason why stack was marked as failed	14:44
howleyt	jprovazn, Well, /var/log/upstart/os-collect-config.log ends with: [2014-02-13 14:10:49,299] (os-refresh-config) [INFO] Completed phase post-configure	14:44
*** dtantsur has quit IRC		14:45
howleyt	All three instances are active alright.	14:45
jprovazn	howleyt, ok, could you please check heat/nova logs on undercloud?	14:45
howleyt	I can ssh onto the two compute nodes, but can't ssh onto noncompute node.	14:45
howleyt	sure	14:45
jprovazn	howleyt, ah, mmt, derekh mentioned ssh key issue earlier today	14:47
jprovazn	though it was related to seed node	14:48
howleyt	Actually, I lie. I can no longer ssh onto any of the overcloud instances now ( I have run devtest once since).	14:48
howleyt	I'll check the logs	14:48
howleyt	Just remembered I had a cut-down version of my heat-engine log in pastebin yesterday: http://pastebin.com/nptHXCdh	14:50
howleyt	Lots of errors like this:	14:51
howleyt	2014-02-13 14:29:30.305 2785 ERROR heat.openstack.common.rpc.common [req-d10cb384-0109-497f-9813-7da370f3900c None] Returning exception The Resource (NovaCompute0Config) is not available. to caller	14:51
howleyt	2014-02-13 14:29:30.306 2785 ERROR heat.openstack.common.rpc.common [req-d10cb384-0109-497f-9813-7da370f3900c None] ['Traceback (most recent call last):\n', ' File "/opt/stack/venvs/heat/local/lib/python2.7	14:51
howleyt	172, in dispatch\n result = getattr(proxyobj, method)(ctxt, **kwargs)\n', ' File "/opt/stack/venvs/heat/local/lib/python2.7/site-packages/heat/engine/service.py", line 62, in wrapped\n return func(sel	14:51
howleyt	f, ctx, args, *kwargs)\n', ' File "/opt/stack/venvs/heat/local/lib/python2.7/site-packages/heat/engine/service.py", line 749, in describe_stack_resource\n raise exception.ResourceNotAvailable(resource_	14:51
howleyt	name=resource_name)\n', 'ResourceNotAvailable: The Resource (NovaCompute0Config) is not available.\n']	14:51
howleyt	jprovazn /var/log/upstart/nova-compute.log on my undercloud also shows an error:	14:56
howleyt	2014-02-13 14:12:38.595 3322 ERROR nova.compute.manager [req-e49ea18f-b82f-47ca-adae-594d06733aa0 3fda0ffd31f04b2691cf74274be4e51a 5f2fe40d1d8149489b1b84e9b65ad806] [instance: f8da7e08-8110-4939-9855-6e66823ecff3] Error: Insufficient compute resources.	14:56
howleyt	2014-02-13 14:12:38.595 3322 TRACE nova.compute.manager [instance: f8da7e08-8110-4939-9855-6e66823ecff3] Traceback (most recent call last):	14:56
howleyt	2014-02-13 14:12:38.595 3322 TRACE nova.compute.manager [instance: f8da7e08-8110-4939-9855-6e66823ecff3] File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/compute/manager.py", line 1044, in _build_instance	14:56
howleyt	2014-02-13 14:12:38.595 3322 TRACE nova.compute.manager [instance: f8da7e08-8110-4939-9855-6e66823ecff3] with rt.instance_claim(context, instance, limits):	14:56
howleyt	2014-02-13 14:12:38.595 3322 TRACE nova.compute.manager [instance: f8da7e08-8110-4939-9855-6e66823ecff3] File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/openstack/common/lockutils.py", line 249, in inner	14:56
howleyt	2014-02-13 14:12:38.595 3322 TRACE nova.compute.manager [instance: f8da7e08-8110-4939-9855-6e66823ecff3] return f(args, *kwargs)	14:56
howleyt	2014-02-13 14:12:38.595 3322 TRACE nova.compute.manager [instance: f8da7e08-8110-4939-9855-6e66823ecff3] File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/compute/resource_tracker.py", line 136, in instance_claim	14:56
howleyt	2014-02-13 14:12:38.595 3322 TRACE nova.compute.manager [instance: f8da7e08-8110-4939-9855-6e66823ecff3] raise exception.ComputeResourcesUnavailable()	14:56
howleyt	2014-02-13 14:12:38.595 3322 TRACE nova.compute.manager [instance: f8da7e08-8110-4939-9855-6e66823ecff3] ComputeResourcesUnavailable: Insufficient compute resources.	14:56
howleyt	2014-02-13 14:12:38.595 3322 TRACE nova.compute.manager [instance: f8da7e08-8110-4939-9855-6e66823ecff3]	14:56
howleyt	2014-02-13 14:15:32.680 3322 WARNING nova.openstack.common.loopingcall [-] task run outlasted interval by 0.512338 sec	14:56
howleyt	2014-02-13 14:20:48.697 3322 WARNING nova.compute.manager [-] Bandwidth usage not supported by hypervisor.	14:56
*** hashar has quit IRC		14:57
*** edmund has quit IRC		14:58
openstackgerrit	A change was merged to openstack/tripleo-incubator: Restore generation of ssh keypair https://review.openstack.org/73458	14:59
jprovazn	howleyt, hm, "Insufficient compute resources" sounds like a node for an instance couldn't be allocated or there was a node with required HW params	15:01
jprovazn	howleyt, what "nova list" and "nova baremetal-node-list" prints?	15:02
jprovazn	(on undercloud)	15:02
derekh	howleyt: just wondering are you running ./scripts/devtest.sh --trash-my-machine with or without a -c ?	15:05
*** jprovazn is now known as jprovazn_afk		15:07
derekh	woot, first successful ci undercloud job https://jenkins02.openstack.org/job/check-tripleo-undercloud-precise/319/	15:08
derekh	now to figure out the overcloud	15:09
*** hashar has joined #tripleo		15:10
*** ifarkas has quit IRC		15:17
*** boris-42_ has quit IRC		15:17
mordred	derekh: awesome!	15:18
*** ifarkas has joined #tripleo		15:19
Ng	nice	15:19
openstackgerrit	Jiri Tomasek proposed a change to openstack/tuskar-ui: Fixing the nodes table https://review.openstack.org/72648	15:21
*** jdob has quit IRC		15:23
*** jdob has joined #tripleo		15:23
howleyt	derekh: I have run devtest twice, the second time with -c	15:25
howleyt	jprovazn:root@undercloud-undercloud-57spmb6tdxn7:~# nova list	15:26
howleyt	+--------------------------------------+-------------------------------------+--------+------------+-------------+--------------------+	15:26
howleyt	\| ID \| Name \| Status \| Task State \| Power State \| Networks \|	15:26
howleyt	+--------------------------------------+-------------------------------------+--------+------------+-------------+--------------------+	15:26
howleyt	\| c6d3d751-d195-4754-9595-135804e7d53c \| overcloud-NovaCompute0-4djus5fwnmui \| ACTIVE \| - \| Running \| ctlplane=192.0.2.6 \|	15:26
howleyt	\| 14e954ce-cdbd-419b-b670-cc1505cfd0e3 \| overcloud-NovaCompute1-bc2lx4xxhsui \| ACTIVE \| - \| Running \| ctlplane=192.0.2.7 \|	15:26
howleyt	\| f8da7e08-8110-4939-9855-6e66823ecff3 \| overcloud-notCompute0-65jwa3sikh62 \| ACTIVE \| - \| Running \| ctlplane=192.0.2.8 \|	15:26
howleyt	+--------------------------------------+-------------------------------------+--------+------------+-------------+--------------------+	15:26
derekh	howleyt: I think the -c runs to fast, so the scheduler doesn't know yet about baremetal nodes by the time the heat stack-create runs, which might explain the Insufficient resource problem	15:27
derekh	howleyt: if I'm correct this might be a workaround for it http://paste.fedoraproject.org/77284/13923916/	15:27
* derekh has been meaning to look into this but hasn't gotten around to it		15:28
derekh	has anybody else seen this behaviour ^ ?	15:28
*** dtantsur has joined #tripleo		15:33
*** hashar has quit IRC		15:34
*** hashar has joined #tripleo		15:34
*** jdob has quit IRC		15:36
*** jdob has joined #tripleo		15:36
howleyt	Thanks derekh, I'll try that out. I seemed to have been in a better position in terms ssh'ing onto overcloud compute nodes before running with -c	15:36
openstackgerrit	Dirk Mueller proposed a change to openstack/tripleo-image-elements: Install syslinux also in nova-baremetal https://review.openstack.org/73623	15:39
openstackgerrit	Ivan Kolodyazhny proposed a change to openstack/python-tuskarclient: Using tox instead of tuskar_with_venv.sh script https://review.openstack.org/73383	15:41
*** dko has quit IRC		15:46
openstackgerrit	Radomir Dopieralski proposed a change to openstack/tuskar-ui: Add a deployment scaling dialog https://review.openstack.org/73551	15:48
howleyt	derekh I've kicked off devtest.sh -c run with those sleeps inserted. I probably won't get a result before I today, but we'll see. Thanks	15:50
derekh	howleyt: no prob	15:50
howleyt	before I "leave" today, that is.	15:51
openstackgerrit	Dmitry Tantsur proposed a change to openstack/tuskar-ui: Add list/created/delete operations for node profiles https://review.openstack.org/73576	15:52
*** gcha has quit IRC		15:53
openstackgerrit	Ralf Haferkamp proposed a change to openstack/tripleo-image-elements: Remove no longer needed generation of the virtual-power-key https://review.openstack.org/73630	15:53
openstackgerrit	Dmitry Tantsur proposed a change to openstack/tuskar-ui: Add list/created/delete operations for node profiles https://review.openstack.org/73576	16:00
*** edmund has joined #tripleo		16:00
*** e0ne_ has joined #tripleo		16:02
*** e0ne has quit IRC		16:06
derekh	SpamapS: is it ok to zap one of the findbroken-* instances on the undercloud ? wanna being up a test test env and it looks like we're out of hosts	16:06
ccrouch	so we've got 10 RH folks lined up for tripleo	16:06
ccrouch	waiting for hotel information	16:06
ccrouch	tripleo meetup in sunnyvale	16:06
*** rdopieralski has quit IRC		16:07
howleyt	derekh: I'm still waiting for overcloud stack to be ready, but I'm already seeing this error in the undercloud:	16:07
howleyt	root@undercloud-undercloud-xcxtyoq4p3c4:~# sudo grep -i error /var/log/upstart/nova-compute.log	16:07
howleyt	2014-02-14 16:00:10.178 3317 ERROR nova.compute.manager [req-ee311015-4ab3-4c1b-b002-e3fbc6d7c9b1 3d71b5ddbf46445b8676ae0196c9201f d538acdabd22435c9aff9c8565ee2041] [instance: 1510a2e2-39e7-49b0-9b94-ff418dbbe487] Error: Insufficient compute resources.	16:07
*** lblanchard has quit IRC		16:07
ccrouch	cody-somerville: any news on hotels ^ ?	16:07
*** morazi has quit IRC		16:07
derekh	howleyt: doesn't look good so, you can source ~/root/stackrc on the undercloud and run a nova list to confirm	16:08
derekh	howleyt: I meant ~root/stackrc	16:09
*** morazi has joined #tripleo		16:10
*** lblanchard has joined #tripleo		16:11
openstackgerrit	Dmitry Tantsur proposed a change to openstack/tuskar-ui: Add list/created/delete operations for node profiles https://review.openstack.org/73576	16:16
howleyt	derekh: root@undercloud-undercloud-xcxtyoq4p3c4:~# nova list	16:17
howleyt	+--------------------------------------+-------------------------------------+--------+------------+-------------+--------------------+	16:17
howleyt	\| ID \| Name \| Status \| Task State \| Power State \| Networks \|	16:17
howleyt	+--------------------------------------+-------------------------------------+--------+------------+-------------+--------------------+	16:17
howleyt	\| fd57db8d-c69a-40d6-92ab-c1ab450333ee \| overcloud-NovaCompute0-h3v3fvasuuiw \| ACTIVE \| - \| Running \| ctlplane=192.0.2.6 \|	16:17
howleyt	\| 1510a2e2-39e7-49b0-9b94-ff418dbbe487 \| overcloud-NovaCompute1-whydnad4soiy \| ACTIVE \| - \| Running \| ctlplane=192.0.2.8 \|	16:17
howleyt	\| c2b4574f-685c-49af-85dd-7dab878c39ae \| overcloud-notCompute0-q23hiaous33x \| ACTIVE \| - \| Running \| ctlplane=192.0.2.5 \|	16:17
howleyt	+--------------------------------------+-------------------------------------+--------+------------+-------------+--------------------+	16:17
cody-somerville	ccrouch: Hey there. Working on contract. We're working to drive the rate down which is right now artificially inflated due to March break/Sprint break. Will announce details probably early next week.	16:17
openstackgerrit	Derek Higgins proposed a change to openstack-infra/tripleo-ci: Get logs from undercloud https://review.openstack.org/73635	16:18
openstackgerrit	Dmitry Tantsur proposed a change to openstack/tuskar-ui: Add list/create/delete operations for node profiles https://review.openstack.org/73576	16:18
derekh	howleyt: maybe heat retried and was successful the second time, I think it does that, what does heat list say?	16:19
*** hashar has quit IRC		16:20
*** e0ne_ has quit IRC		16:22
*** robsparker has joined #tripleo		16:23
*** dtantsur has quit IRC		16:25
*** e0ne has joined #tripleo		16:25
howleyt	derekh: It's still in progress, but I don't have much hope for it:	16:26
howleyt	root@undercloud-undercloud-xcxtyoq4p3c4:~# heat stack-list	16:26
howleyt	+--------------------------------------+------------+--------------------+----------------------+	16:26
howleyt	\| id \| stack_name \| stack_status \| creation_time \|	16:26
howleyt	+--------------------------------------+------------+--------------------+----------------------+	16:26
howleyt	\| 4dfe1a9b-e9a7-4731-97ec-c180d24c1a6e \| overcloud \| CREATE_IN_PROGRESS \| 2014-02-14T16:00:01Z \|	16:26
howleyt	+--------------------------------------+------------+--------------------+----------------------+	16:26
*** vkozhukalov has quit IRC		16:26
derekh	howleyt: if you want you can ssh to the notcompute node and see what os-collect-config is doing, i.e. see if it has failed	16:27
*** bauzas has quit IRC		16:27
ccrouch	thanks cody-somerville	16:28
*** morazi_ has joined #tripleo		16:30
*** morazi has quit IRC		16:31
*** bnemec is now known as beekneemech		16:32
howleyt	derekh: unfortunately, I've never reached the point where I could ssh to notcompute node, currently, I get: ssh: connect to host 192.0.2.5 port 22: Connection refused	16:32
derekh	howleyt: not good, the only advice I have from there is to try and look at the console with vnc or something and see if you can see any errors that way	16:33
*** peoplemerge has joined #tripleo		16:34
*** rcarrillocruz has quit IRC		16:41
*** rcarrillocruz has joined #tripleo		16:42
*** e0ne has quit IRC		16:42
*** e0ne has joined #tripleo		16:44
rbrady-afk	howleyt: were you able to run "heat event-list overcloud"?	16:52
openstackgerrit	James Slagle proposed a change to openstack/tripleo-image-elements: Fix typo https://review.openstack.org/73655	16:53
openstackgerrit	James Slagle proposed a change to openstack/tripleo-image-elements: Install lvm2 package for cinder https://review.openstack.org/73656	16:53
openstackgerrit	James Slagle proposed a change to openstack/tripleo-image-elements: Common cinder install code https://review.openstack.org/73657	16:53
openstackgerrit	James Slagle proposed a change to openstack/tripleo-image-elements: Use os-svc-restart for cinder-api https://review.openstack.org/73658	16:53
openstackgerrit	James Slagle proposed a change to openstack/tripleo-image-elements: Remove unneeded chown of /var/run/nova https://review.openstack.org/73659	16:53
openstackgerrit	James Slagle proposed a change to openstack/tripleo-image-elements: Add create-dir service for neutron ovs agent https://review.openstack.org/73660	16:53
*** saju_m has joined #tripleo		16:54
howleyt	derekh: heat event-list: http://pastebin.com/KbeqA0KF	16:55
howleyt	also got final confirmation from heat that overcloud stack create failed.	16:55
*** vkozhukalov has joined #tripleo		16:56
*** viktors has quit IRC		16:57
derekh	howleyt: so os-refresh-config on notComputes is supposed to send out a notify to heat to say its ready, in your case that didn't happen, your probably going to have to vnc into the undercloud vm to see what we wrong	16:58
openstackgerrit	A change was merged to openstack/tripleo-image-elements: Look for local.json in preference to config.json. https://review.openstack.org/73401	16:58
derekh	howleyt: you should be looking in the os-collect-config logs, (os-collect-config calls os-refresh-config)	16:59
*** ifarkas has quit IRC		16:59
*** e0ne has quit IRC		17:00
howleyt	derekh: ok, I have to run. I'll look into it next week. Thanks for your help, have a good weekend!	17:04
*** bauzas has joined #tripleo		17:06
*** rpodolyaka has left #tripleo		17:06
derekh	howleyt: okie dokie, you too	17:08
*** boris-42_ has joined #tripleo		17:08
*** markmc has quit IRC		17:15
*** matty_dubs is now known as matty_dubs\|lunch		17:16
*** vkozhukalov has quit IRC		17:17
SpamapS	derekh: Yes you can zap findbrokens. Those are as much to hold on to good machines as they are to find broken ones. :)	17:19
*** pblaho has joined #tripleo		17:20
derekh	SpamapS: btw on broken machine, its possible some of them may have been a result of https://bugs.launchpad.net/tripleo/+bug/1272803 , as neutron ports on the overcloud may have been left behind by old test environments	17:22
derekh	SpamapS: just a hunch but if you saw nodes fail to come up it could be because the got dhcp from the wrong place	17:22
cody-somerville	Wow. upstart is dead!	17:24
cody-somerville	Ubuntu is switching to systemd.	17:24
*** athomas has quit IRC		17:25
SpamapS	derekh: Yeah I agree.	17:27
SpamapS	cody-somerville: yeah... <sniff>	17:27
SpamapS	cody-somerville: best for everyone.. but still.. :(	17:27
SpamapS	the trolls in Debian have been.. entertaining	17:28
SpamapS	if not horrifying	17:28
SpamapS	suggesting forking Debian and/or physically beating Lennart.	17:28
*** shardy is now known as shardy_afk		17:32
Ng	at least it will help us simplify	17:37
openstackgerrit	Dan Prince proposed a change to openstack/tripleo-incubator: Use $TRIPLEO_ROOT/seed.qcow2 as the cache. https://review.openstack.org/73674	17:38
Ng	well, eventually, I imagine 14.04 will have users who care about tripleo for some time to come :/	17:38
SpamapS	Ng: meh	17:45
SpamapS	Ng: Nobody ever cared about upstart in Ubuntu server except me. ;)	17:45
*** ramishra has quit IRC		17:45
Ng	SpamapS: I mean in the sense that we'll need to support upstart for some period of time, so 14.04 users can actually use tripleo :)	17:45
SpamapS	meh meh .. who wants 14.04 ?	17:47
SpamapS	14.10 will be so much better	17:47
SpamapS	and you'll have _9_ whole months to realize that betterness	17:47
*** jistr has quit IRC		17:51
openstackgerrit	A change was merged to openstack/tuskar-ui: Use git.openstack.org instead of github.com https://review.openstack.org/73371	17:54
openstackgerrit	A change was merged to openstack/tuskar-ui: Switch over to oslosphinx https://review.openstack.org/73352	17:54
openstackgerrit	A change was merged to openstack/tuskar-ui: Rename Openstack to OpenStack https://review.openstack.org/73213	17:54
*** epim has joined #tripleo		17:55
openstackgerrit	Dougal Matthews proposed a change to openstack/tuskar: Enable gating on E123 https://review.openstack.org/73678	17:58
openstackgerrit	Dougal Matthews proposed a change to openstack/tuskar: Enable gating on E124 https://review.openstack.org/73679	17:58
openstackgerrit	Dougal Matthews proposed a change to openstack/tuskar: Enable gating on E126 https://review.openstack.org/73680	17:58
openstackgerrit	Dougal Matthews proposed a change to openstack/tuskar: Enable gating on E127 https://review.openstack.org/73681	17:58
openstackgerrit	Dougal Matthews proposed a change to openstack/tuskar: Enable gating on E128 https://review.openstack.org/73682	17:58
openstackgerrit	Dougal Matthews proposed a change to openstack/tuskar: Enable gating on E122 https://review.openstack.org/73400	17:58
d0ugal	jdob: Tidying for this week done. ^	17:58
*** cadenzajon has joined #tripleo		18:01
*** newell has joined #tripleo		18:02
openstackgerrit	A change was merged to openstack/tripleo-image-elements: Move cinder-volume creation to post-configure https://review.openstack.org/71975	18:03
*** derekh has quit IRC		18:04
*** morazi_ is now known as morazi		18:04
*** pblaho has quit IRC		18:05
openstackgerrit	A change was merged to openstack/tuskar-ui: Add missing semicolons in js https://review.openstack.org/73182	18:06
devananda	lifeless: interesting. which machines // what was going on at the time you ran out of sessions?	18:08
*** jistr has joined #tripleo		18:09
*** jistr has quit IRC		18:09
*** w_ has joined #tripleo		18:09
*** rbrady-afk is now known as rbrady		18:10
*** peoplemerge has quit IRC		18:11
*** olaph has quit IRC		18:11
*** epim has quit IRC		18:15
*** matty_dubs\|lunch is now known as matty_dubs		18:16
*** ramishra has joined #tripleo		18:16
openstackgerrit	Ben Nemec proposed a change to openstack/tripleo-incubator: Don't include unsupported distro check in docs https://review.openstack.org/73683	18:16
*** max_lobur is now known as max_lobur_afk		18:20
*** ramishra has quit IRC		18:21
*** lucasagomes has quit IRC		18:28
*** panda_ has joined #tripleo		18:30
*** rpodolyaka1 has joined #tripleo		18:32
*** panda has quit IRC		18:33
*** peoplemerge has joined #tripleo		18:38
openstackgerrit	Ben Nemec proposed a change to openstack/tripleo-incubator: Fix devtest_variables doc formatting https://review.openstack.org/73689	18:40
*** rcarrillocruz1 has joined #tripleo		18:42
*** rcarrillocruz has quit IRC		18:43
*** wusWWW has joined #tripleo		18:51
*** rcarrillocruz has joined #tripleo		18:57
*** rcarrillocruz1 has quit IRC		19:00
*** rcarrillocruz2 has joined #tripleo		19:00
*** rcarrillocruz has quit IRC		19:01
slagle	is there a way to force fail a heat stack-create?	19:01
slagle	e.g., i know it's going to fail, and I don't want to have to wait for stack-create to timeout before i can delete it	19:02
openstackgerrit	A change was merged to openstack/tuskar-ui: Adding real index, deleting and creating of overcloud https://review.openstack.org/73343	19:02
Shrews	slagle: how much snow did you get?	19:03
*** chrisfoster has quit IRC		19:03
slagle	Shrews: 10" with another 3-6" coming according to Ray	19:04
Shrews	gotta love Ray	19:04
Shrews	thinking of doing some boarding up there this Monday	19:04
*** rcarrillocruz has joined #tripleo		19:04
slagle	nice, supposed to be warmer and sunny next week	19:05
slagle	good time to hit the slopes	19:05
openstackgerrit	Dan Prince proposed a change to openstack/tripleo-incubator: Add NeutronPublicInterface arg for the undercloud https://review.openstack.org/72094	19:05
*** Kim-Chi-San has joined #tripleo		19:05
*** Kim-Chi-San is now known as tburnes		19:05
lifeless	morning	19:06
lifeless	devananda: not entirely sure	19:06
*** rcarrillocruz2 has quit IRC		19:06
lifeless	devananda: see hours later though when I tracked the real cause: https://bugs.launchpad.net/tripleo/+bug/1272803	19:06
*** e0ne has joined #tripleo		19:06
lifeless	ekarlso: hi, whats up (and no, not here @1am :)	19:07
*** e0ne has quit IRC		19:10
lifeless	oh yeah, seed and undercloud job success	19:11
lifeless	booyah	19:11
*** saju_m has quit IRC		19:11
*** rcarrillocruz1 has joined #tripleo		19:14
*** rcarrillocruz has quit IRC		19:16
*** ramishra has joined #tripleo		19:16
*** e0ne has joined #tripleo		19:19
*** jprovazn_afk is now known as jprovazn		19:20
jprovazn	SpamapS, Hi	19:21
*** ramishra has quit IRC		19:21
*** sgrasley has quit IRC		19:24
*** rcarrillocruz has joined #tripleo		19:26
SpamapS	jprovazn: howdy	19:26
*** rcarrillocruz1 has quit IRC		19:27
jprovazn	SpamapS, I wanted to discuss this: https://review.openstack.org/#/c/68392/4/elements/rabbitmq-server/install.d/20-rabbitmq-server	19:28
SpamapS	jprovazn: Cool!	19:29
jprovazn	SpamapS, the reason I put symlink there is because /var/lib/rabbitmq/.erlang.cookie is now template, I didn't fine a way how to convince rabbitmq to change path for erlang cookie (though it might be I missed something)	19:30
jprovazn	SpamapS, and based on this I didn't like the fact that then there will be both /var/lib/rabbitmq and /mnt/state/var/lib/rabbitmq	19:31
jprovazn	and both of them will be used	19:31
SpamapS	jprovazn: it is a bug if you can't change the path to the cookie, I think the RabbitMQ devs will agree there.	19:32
SpamapS	jprovazn: I'd rather see just the symlink for just the file that is the problem	19:32
SpamapS	jprovazn: because then it becomes obvious when to remove that symlink.	19:32
SpamapS	(when the bug is fixed ;)	19:32
lifeless	jprovazn: we need to generate precious files on /mnt though	19:32
lifeless	jprovazn: so e.g. /mnt/state/var/lib/rabbitmq/.erland.cookie	19:33
jprovazn	lifeless, SpamapS sure, erlang.cookie doesn\t ahve to be saved - it will be set on each os-apply-config run	19:33
*** CaptTofu_ has joined #tripleo		19:33
*** CaptTofu has quit IRC		19:34
jprovazn	um, so if I'll not be lucky with changing cookie path, are you ok with creating symlink just for .erlang.cookie, right?	19:34
SpamapS	jprovazn: I would insist on that, yes. :)	19:34
*** rcarrillocruz1 has joined #tripleo		19:35
SpamapS	jprovazn: doing the whole dir is more confusing to me, because I ask myself then "why can't this just be on the state dir?"	19:35
openstackgerrit	Ben Nemec proposed a change to openstack/tripleo-incubator: Stop trying to copy out SSH keys from notcompute. https://review.openstack.org/73465	19:35
jprovazn	SpamapS, ok	19:35
hewbrocca	ahh the dreaded erlang.cookie	19:35
*** rcarrillocruz has quit IRC		19:35
jprovazn	SpamapS, and to answer this question: https://review.openstack.org/#/c/68392/4/elements/rabbitmq-server/os-config-applier/etc/rabbitmq/rabbitmq-env.conf	19:37
lifeless	jprovazn: uhm then it should be in /var/run	19:37
lifeless	jprovazn: we certainly don't want it changing from run to run though, that would cause cluster partitions during deploys.	19:37
SpamapS	BTW I went ahead and started tripleo-cd again	19:39
jprovazn	lifeless, I thought that when you reboot one instance of stack (or boot different node instead of the first one), then same heat provides to it same metadata -> same rabbit.cookie value	19:40
*** cd-undercloud has joined #tripleo		19:40
cd-undercloud	************ overcloud complete status=1 **********	19:40
*** cd-undercloud has quit IRC		19:40
jprovazn	s/then same heat/then heat/	19:40
lifeless	jprovazn: I'm not sure about OS::Heat::RandomString for that	19:41
*** rcarrillocruz has joined #tripleo		19:42
lifeless	jprovazn: we need to make sure its a resource that won't get recreated by mistake	19:42
lifeless	jprovazn: I suspect on -create => new string	19:42
lifeless	updates can recreate resources	19:42
jprovazn	pfff	19:42
*** rcarrillocruz1 has quit IRC		19:43
jprovazn	lifeless, then using RandomString looks little bit suboptimal in this case	19:44
jprovazn	because it would be rewritten anyway	19:45
*** morazi has quit IRC		19:45
*** rcarrillocruz1 has joined #tripleo		19:46
*** rcarrillocruz has quit IRC		19:46
pleia2	lifeless:	19:47
pleia2	Feb 14 19:44:23 localhost sshd[577]: Invalid user fedora from 74.95.15.41	19:47
pleia2	Feb 14 19:44:23 localhost sshd[577]: input_userauth_request: invalid user fedora [preauth]	19:47
pleia2	Feb 14 19:44:23 localhost sshd[577]: Connection closed by 74.95.15.41 [preauth]	19:47
pleia2	Feb 14 19:44:41 localhost useradd[635]: new group: name=fedora, GID=1000	19:47
pleia2	Feb 14 19:44:41 localhost useradd[635]: new user: name=fedora, UID=1000, GID=1000, home=/home/fedora, shell=/bin/bash	19:47
pleia2	AAAHHHHAHA	19:47
pleia2	lifeless: but now I can fix it :)	19:47
*** jtomasek has quit IRC		19:48
SpamapS	lifeless: OS::Heat::RandomString stays stable for the life of the stack.	19:48
SpamapS	lifeless: unless you change the 'salt' parameter.	19:48
SpamapS	jprovazn: ^^	19:48
jprovazn	SpamapS, cool, thanks	19:48
lifeless	pleia2: race condition ?	19:49
*** cd-undercloud has joined #tripleo		19:50
cd-undercloud	************ overcloud complete status=1 **********	19:50
*** cd-undercloud has quit IRC		19:50
jprovazn	SpamapS, about this https://review.openstack.org/#/c/68392/4/elements/rabbitmq-server/os-config-applier/etc/rabbitmq/rabbitmq-env.conf - the reason why it's not hostname is that I need a simple way how to get IP and hostname pairs for all nodes on each node in cluster	19:50
pleia2	lifeless: yeah	19:50
pleia2	lifeless: not sure about best solution yet, but I'll investigate a few	19:51
jprovazn	SpamapS, and also nodename in cluster should not change during lifetime	19:51
SpamapS	jprovazn: Right, can't you just use Fn::GetAtt [ OtherServer, instance_name ] the same way you get the network address?	19:51
SpamapS	jprovazn: that instance_name will stay stable as long as the instance is never replaced... which is fine because it will lose its' state if it is replaced anyway.	19:52
lifeless	pleia2: whinge upstream? :)	19:52
jprovazn	SpamapS, ok, can try	19:53
lifeless	jprovazn: btw there is a Gn::Join in heat	19:53
lifeless	jprovazn: it can do the string manioulation to ,join lists for you	19:53
pleia2	lifeless: hah, perhaps	19:54
jprovazn	lifeless, yes, this would be handy for places where just hostnames delimited by ',' are used, but I need IP too for /etc/hostnames, so hostnames list in heat template would be duplicit	19:58
*** spzala has joined #tripleo		19:58
*** morazi has joined #tripleo		19:59
*** jprovazn is now known as jprovazn_afk		19:59
lifeless	jprovazn: there's no rule against duplicating data in the meatadata dict given machines	20:00
*** cd-undercloud has joined #tripleo		20:00
cd-undercloud	************ overcloud complete status=1 **********	20:00
*** cd-undercloud has quit IRC		20:00
lifeless	jprovazn_afk: as long as the source is single-location, heat can format it for nice consumption by instances	20:00
lifeless	2014-02-14 19:05:26.833 \| Removing /opt/stack/new//user.qcow2-new	20:00
lifeless	2014-02-14 19:05:26.880 \| Waiting for the overcloud stack to be ready	20:00
lifeless	2014-02-14 19:43:22.747 \| Build timed out (after 125 minutes). Marking the build as failed.	20:00
lifeless	2014-02-14 19:43:22.921 \| /opt/stack/new//tripleo-incubator/scripts/wait_for: line 55: 6421 Terminated sleep $SLEEPTIME	20:00
lifeless	2014-02-14 19:43:22.926 \| + get_state_from_host seed root@192.168.1.173	20:00
*** vkozhukalov has joined #tripleo		20:02
*** AaronGr is now known as AaronGr_Zzz		20:05
*** AaronGr_Zzz is now known as AaronGr		20:07
*** cd-undercloud has joined #tripleo		20:10
cd-undercloud	************ overcloud complete status=1 **********	20:10
*** cd-undercloud has quit IRC		20:10
SpamapS	+ devtest_overcloud.sh vlan25 138.35.77.4/25 eth2 138.35.77.1 138.35.77.131 138.35.77.254 138.35.77.128/25 /opt/stack/tr	20:11
SpamapS	ipleo-incubator/tripleo-cloud/tripleo-cd-admins /opt/stack/tripleo-incubator/tripleo-cloud/tripleo-cd-users	20:11
SpamapS	[2014/02/14 07:59:40 PM] [ERROR] key arch does not exist in ['/opt/stack/testenv.json']	20:11
lifeless	SpamapS: our testenv definition will need to be updated on tripleo	20:12
lifeless	-undercloud	20:12
SpamapS	not sure I understand.. but I'll look into it a bit later	20:12
SpamapS	have to leave on long errands now :-P	20:12
lifeless	SpamapS: look at the schema dox in devtest_setpu.sh	20:12
SpamapS	Yeah, so I didn't realize we were even writing a "testenv.json"	20:13
SpamapS	since this is not a test env	20:13
SpamapS	it is a real env	20:13
lifeless	not sure tere is a difference :)	20:13
SpamapS	if they're intended to be the same we should call them just "environments" :-P	20:14
SpamapS	my impression of testenv is "thing to run lots of these on one box"	20:14
lifeless	it describes the environment devtest sripts run in	20:14
SpamapS	we don't have one for the cd cloud	20:15
lifeless	so yes, its an environment, and devtest_testenv creates a test environment	20:15
lifeless	right, we need one know, to capture the arch	20:15
SpamapS	NODE_ARCH doesn't push it in?	20:16
SpamapS	gah.. out of time	20:16
lifeless	I've created one	20:16
SpamapS	-> long errands :-P	20:16
SpamapS	bbl	20:17
*** ramishra has joined #tripleo		20:17
ekarlso	lifeless:	20:19
ekarlso	10:33 < ekarlso> hmm, do I need to do something special with bridges in order to get things to work or ?	20:19
ekarlso	10:33 < ekarlso> had first a problem with ovs + briding, so removed the bridge module and I deleted the default network now it fails to run devtest_seed.sh due to default network missing p	20:19
*** rpodolyaka1 has quit IRC		20:19
lifeless	ekarlso: running setup-network might fix that	20:20
*** morazi has quit IRC		20:20
lifeless	ekarlso: I'm persuming you're running devtest on a lcaol machine	20:20
*** morazi has joined #tripleo		20:21
*** ramishra has quit IRC		20:22
*** akuznetsov has quit IRC		20:22
*** akuznetsov has joined #tripleo		20:23
*** cadenzajon has quit IRC		20:29
*** e0ne has quit IRC		20:35
rwsu	lifeless: about https://review.openstack.org/#/c/72524/, so I see that you want to put the MESSAGING_BACKEND into EXTRA_ARGS. I find using EXTRA_ARGS confusing because that element is required for the image to work, not extra	20:36
rwsu	maybe I'm not understanding your intention	20:36
lifeless	rwsu: everything in EXTRA_ARGS is needed for the image to work too	20:36
lifeless	rwsu: but like I said yesterday, do whatever makes sense to you, but fix the thing where you were installing it into the hypervisor, and I'll do a review later.	20:37
lifeless	rwsu: my suggestion was to permit hypervisor vs ctl plane specific extra args.	20:37
lifeless	rwsu: I don't think we should model every variation with new variables.	20:37
lifeless	rwsu: it just becomes a maintenance headache	20:38
openstackgerrit	lifeless proposed a change to openstack/tripleo-image-elements: Fixup tripleo-cd fallout from the CI push. https://review.openstack.org/73708	20:38
rwsu	lifeless: ok, i'm hunged up on the name EXTRA.. I thought it would be where stackuser would eventually go, but i'll push another update to the patch with your suggestion, thanks	20:40
openstackgerrit	lifeless proposed a change to openstack/tripleo-incubator: Make the overcloud-user config identify itself https://review.openstack.org/73462	20:48
openstackgerrit	lifeless proposed a change to openstack/tripleo-incubator: Permit specifying ovs bridge mappings too. https://review.openstack.org/71706	20:48
openstackgerrit	lifeless proposed a change to openstack/tripleo-incubator: Stop trying to copy out SSH keys from notcompute. https://review.openstack.org/73465	20:48
lifeless	rwsu: I suggest there need to be two patches; one that addresses the ability to add elements to ctlplane only, and one that does the messaging choice stuff.	20:49
lifeless	ok, I have to run - ciao!	20:49
*** akuznetsov has quit IRC		20:52
rwsu	later, have a good weekend	20:53
*** dprince has quit IRC		21:03
*** shardy_afk is now known as shardy		21:03
openstackgerrit	Devananda van der Veen proposed a change to openstack/tripleo-incubator: Add Ironic bits to _undercloud https://review.openstack.org/72969	21:03
*** cadenzajon has joined #tripleo		21:11
*** rcarrillocruz has joined #tripleo		21:12
*** rcarrillocruz1 has quit IRC		21:14
*** jprovazn_afk has quit IRC		21:17
*** ramishra has joined #tripleo		21:18
*** ramishra has quit IRC		21:23
*** peoplemerge has quit IRC		21:35
*** cd-undercloud has joined #tripleo		21:41
cd-undercloud	************ overcloud complete status=1 **********	21:41
*** cd-undercloud has quit IRC		21:41
openstackgerrit	A change was merged to openstack/tripleo-image-elements: Remove no longer needed generation of the virtual-power-key https://review.openstack.org/73630	21:44
*** peoplemerge has joined #tripleo		21:44
*** rcarrillocruz1 has joined #tripleo		21:47
*** rcarrillocruz has quit IRC		21:49
*** bauzas has quit IRC		21:52
*** ftcjeff has joined #tripleo		21:53
*** edmund1 has joined #tripleo		21:58
*** edmund has quit IRC		22:00
*** jdob has quit IRC		22:03
*** rcarrillocruz has joined #tripleo		22:03
*** rcarrillocruz1 has quit IRC		22:05
*** rcarrillocruz1 has joined #tripleo		22:07
*** rcarrillocruz has quit IRC		22:09
*** matty_dubs is now known as matty_dubs\|gone		22:09
*** panda has joined #tripleo		22:11
*** panda_ has quit IRC		22:15
*** edmund1 has quit IRC		22:16
*** ramishra has joined #tripleo		22:19
*** lblanchard has quit IRC		22:19
*** ramishra has quit IRC		22:23
*** CaptTofu_ has quit IRC		22:31
*** rwsu has quit IRC		22:34
*** rcarrillocruz has joined #tripleo		22:42
*** rcarrillocruz1 has quit IRC		22:43
ekarlso	lifeless: it's a cloud instance	22:50
*** ftcjeff has quit IRC		23:02
*** cd-undercloud has joined #tripleo		23:03
cd-undercloud	************ overcloud complete status=1 **********	23:03
*** cd-undercloud has quit IRC		23:03
*** rwsu has joined #tripleo		23:04
*** vkozhukalov has quit IRC		23:04
*** ramishra has joined #tripleo		23:20
*** rwsu has quit IRC		23:22
*** ramishra has quit IRC		23:24
*** chrisfoster has joined #tripleo		23:25
*** rbrady has quit IRC		23:25
*** chrisfoster has quit IRC		23:27
*** rcarrillocruz1 has joined #tripleo		23:33
*** rcarrillocruz has quit IRC		23:34
*** rcarrillocruz has joined #tripleo		23:41
*** rcarrillocruz1 has quit IRC		23:43
*** lazy_prince has joined #tripleo		23:51
*** killer_prince has quit IRC		23:52
*** lazy_prince is now known as killer_prince		23:52
*** peoplemerge has quit IRC		23:56
*** peoplemerge has joined #tripleo		23:58
*** killer_prince has quit IRC		23:59

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!