Thursday, 2016-02-04

*** yuanying has quit IRC00:09
*** yuanying has joined #tripleo00:16
*** yuanying has quit IRC00:19
*** yuanying has joined #tripleo00:22
*** chlong has joined #tripleo00:40
*** eil397 has quit IRC01:04
*** shivrao has quit IRC01:06
*** thrash|f00dz is now known as thrash|g0ne01:09
*** yuanying has quit IRC01:11
*** yuanying has joined #tripleo01:12
*** stendulker has joined #tripleo01:13
*** rwsu has joined #tripleo01:14
*** dsneddon_ has quit IRC01:19
*** dsneddon_ has joined #tripleo01:19
*** yuanying has quit IRC01:21
*** yuanying has joined #tripleo01:23
*** crinkle_ has joined #tripleo01:30
*** crinkle has quit IRC01:31
*** Hunner has quit IRC01:31
*** crinkle_ is now known as crinkle01:31
*** penick has quit IRC01:31
*** Hunner has joined #tripleo01:36
*** Hunner has joined #tripleo01:36
*** tiswanso has joined #tripleo01:36
*** alop has quit IRC01:37
*** weshay_xchat has joined #tripleo01:38
*** tiswanso has quit IRC01:43
*** tiswanso has joined #tripleo01:43
*** weshay_xchat is now known as weshay01:48
*** tzumainn has quit IRC02:01
*** stendulker has quit IRC02:25
*** yamahata has quit IRC02:39
*** shivrao has joined #tripleo02:46
*** shivrao has quit IRC02:59
*** weshay has quit IRC03:08
*** yuanying has quit IRC03:15
*** weshay_xchat has joined #tripleo03:18
*** yuanying has joined #tripleo03:19
*** yuanying has quit IRC03:24
*** yuanying has joined #tripleo03:35
*** stendulker has joined #tripleo03:38
*** rbrady has quit IRC03:40
*** weshay_xchat has quit IRC03:41
*** yuanying_ has joined #tripleo03:41
*** yuanying has quit IRC03:44
*** links has joined #tripleo03:49
*** yuanying_ has quit IRC04:06
*** yuanying has joined #tripleo04:07
*** yuanying has quit IRC04:08
*** yuanying has joined #tripleo04:08
*** rbrady has joined #tripleo04:08
*** ryansb has quit IRC04:11
*** ryansb has joined #tripleo04:11
*** ryansb has joined #tripleo04:11
*** Ng has quit IRC04:11
*** SlickNik has quit IRC04:11
*** SlickNik has joined #tripleo04:12
*** trown|outttypeww has quit IRC04:12
*** Nakato has quit IRC04:12
*** trown has joined #tripleo04:12
*** jroll has quit IRC04:13
*** gus has quit IRC04:13
*** davidlenwell_ has joined #tripleo04:13
*** davidlenwell has quit IRC04:13
*** chlong has quit IRC04:13
*** Nakato has joined #tripleo04:13
*** jpeeler has quit IRC04:13
*** bandini has quit IRC04:13
*** bandini has joined #tripleo04:14
*** gus has joined #tripleo04:14
*** jroll has joined #tripleo04:14
*** pblaho has quit IRC04:19
*** pblaho has joined #tripleo04:19
*** jrist has quit IRC04:19
*** cmyster has quit IRC04:19
*** Ng has joined #tripleo04:20
*** ChanServ sets mode: +v Ng04:20
*** jrist has joined #tripleo04:20
*** jrist has joined #tripleo04:20
*** cmyster has joined #tripleo04:20
*** xinwu has quit IRC04:20
*** jpeeler has joined #tripleo04:21
*** xinwu has joined #tripleo04:21
*** xinwu has quit IRC04:21
*** shivrao has joined #tripleo04:21
*** xinwu has joined #tripleo04:23
*** xinwu has quit IRC04:26
*** chlong has joined #tripleo04:28
openstackgerritMerged openstack/diskimage-builder: Print unparsable file in pkg-map  https://review.openstack.org/26730504:47
openstackgerritIan Wienand proposed openstack/diskimage-builder: Fix tar listing in functional tests  https://review.openstack.org/27603704:51
*** jtomasek_ has joined #tripleo05:02
*** davidlenwell_ has quit IRC05:02
*** davidlenwell has joined #tripleo05:05
*** ChanServ sets mode: +v davidlenwell05:05
*** akuznetsov has joined #tripleo05:12
*** xinwu has joined #tripleo05:20
*** anande has joined #tripleo05:20
*** anande has quit IRC05:27
*** akuznetsov has quit IRC05:42
*** oshvartz has quit IRC05:50
*** masco has joined #tripleo06:05
*** akuznetsov has joined #tripleo06:05
*** penick has joined #tripleo06:13
*** tiswanso has quit IRC06:16
*** penick_ has joined #tripleo06:16
*** penick has quit IRC06:18
*** penick_ is now known as penick06:18
*** akuznetsov has quit IRC06:18
*** akuznetsov has joined #tripleo06:19
*** akuznetsov has quit IRC06:29
*** jtomasek_ has quit IRC06:40
*** jprovazn has joined #tripleo06:55
*** Marga_ has joined #tripleo06:56
*** tzumainn has joined #tripleo07:01
*** oshvartz has joined #tripleo07:01
*** penick has quit IRC07:23
*** dshulyak has joined #tripleo07:25
*** akuznetsov has joined #tripleo07:32
*** tzumainn has quit IRC07:38
*** ukalifon has joined #tripleo07:40
*** dsneddon_ has quit IRC07:42
*** hjensas has quit IRC07:50
*** akuznetsov has quit IRC07:55
*** jistr|mobi has joined #tripleo07:59
openstackgerritMerged openstack/diskimage-builder: Add support for gentoo to simple-init  https://review.openstack.org/27379008:00
*** jdob has joined #tripleo08:01
*** bnemec has joined #tripleo08:02
*** ukalifon has quit IRC08:04
*** fgimenez has joined #tripleo08:06
*** shivrao has quit IRC08:06
*** Marga_ has quit IRC08:10
*** Marga_ has joined #tripleo08:10
*** xinwu has quit IRC08:13
*** tosky has joined #tripleo08:14
*** akuznetsov has joined #tripleo08:15
*** hjensas has joined #tripleo08:20
*** hjensas has quit IRC08:20
*** hjensas has joined #tripleo08:20
*** aufi has joined #tripleo08:21
*** dsneddon_ has joined #tripleo08:24
*** gfidente has joined #tripleo08:24
*** jistr has joined #tripleo08:29
*** ifarkas has joined #tripleo08:30
*** jcoufal has joined #tripleo08:30
*** jaosorior has joined #tripleo08:31
*** mkovacik has joined #tripleo08:31
*** akuznetsov has quit IRC08:31
*** olap has joined #tripleo08:32
*** akuznetsov has joined #tripleo08:41
*** jtomasek__ has joined #tripleo08:43
*** akuznetsov has quit IRC08:53
*** liverpooler has joined #tripleo08:53
*** jcoufal has quit IRC08:54
*** bvandenh has joined #tripleo08:56
*** rhallisey has joined #tripleo09:04
*** mkovacik has quit IRC09:06
*** paramite has joined #tripleo09:10
*** paramite is now known as paramite|afk09:10
*** jistr has quit IRC09:16
*** openstackgerrit has quit IRC09:17
*** openstackgerrit has joined #tripleo09:17
*** paramite|afk is now known as paramite09:21
*** dsneddon_ has quit IRC09:22
*** mbound has joined #tripleo09:24
*** devvesa has joined #tripleo09:24
devvesahi lucasagomes: you are the ironic master, right?09:27
*** nico_auv has joined #tripleo09:27
devvesaI have a question for you09:29
*** dsneddon_ has joined #tripleo09:29
*** ukalifon1 has joined #tripleo09:30
*** mcornea has joined #tripleo09:32
devvesaor maybe trown...09:32
lucasagomesdevvesa, I wish, what's up?09:33
devvesaI have a baremetal environment right now09:35
devvesaAnd I can see all the registered nodes as 'manageable'09:35
devvesaBut when I run the './tripleo.sh --introspect-nodes' script, it gets stuck09:36
devvesaIt is safe to run the overcloud? (my first time in a baremetal env)09:36
devvesaI ignore what the '--introspect-nodes' does09:37
lucasagomesdevvesa, right, it get stuck where? Booting the nodes ?09:38
*** morazi has joined #tripleo09:38
devvesaI really don't know. The nodes were already on09:39
devvesaI see the following09:39
devvesahttp://fpaste.org/318087/14545217/09:39
devvesaAnd keeps waiting...09:39
lucasagomesdevvesa, right, do you have access to the console?09:40
lucasagomessee if the nodes booted09:41
*** electrofelix has joined #tripleo09:41
openstackgerritMerged openstack/tripleo-heat-templates: neutron: delete by default router/dhcp namespaces  https://review.openstack.org/27156909:42
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates: neutron: delete by default router/dhcp namespaces  https://review.openstack.org/27611509:43
*** mgould has joined #tripleo09:43
devvesalucasagomes: the nodes are up: http://fpaste.org/318317/45790701/09:44
lucasagomesdevvesa, that's fine yes... but it may not booted the ramdisk09:47
lucasagomesthere are a couple of things that can go wrong at that stage: DHCP problems, PXE/iPXE problems, kernel problems09:48
*** trown has quit IRC09:49
devvesalucasagomes: I had problems in the Virtual Environment with ramdisk as well... Had to upload the old ramdisk09:50
devvesaMaybe you remember09:50
devvesaIs there any log that I can look at?09:51
*** ukalifon1 has quit IRC09:53
lucasagomesdevvesa, logs depend of the type of machine you are booting... I mean for virtual you could redirect the console to a file (we do that in gate)09:56
lucasagomesfor baremetal you have to access the remote controller to see the logs09:56
lucasagomesif it fails to boot, the system will eventually time out and error09:56
lucasagomesremote console to see the logs*09:57
devvesaOk. I'm there09:57
*** apetrich has quit IRC09:58
openstackgerritMerged openstack/instack-undercloud: puppet: fix uchiwa password parameter  https://review.openstack.org/27420209:59
*** dsneddon_ has quit IRC10:01
*** rlandy has quit IRC10:04
openstackgerritEmilien Macchi proposed openstack/instack-undercloud: Deploy Monitoring on the undercloud with Puppet  https://review.openstack.org/27612710:05
*** rlandy has joined #tripleo10:08
*** gfidente has quit IRC10:10
*** apetrich has joined #tripleo10:12
*** gfidente has joined #tripleo10:15
*** gfidente has quit IRC10:15
*** gfidente has joined #tripleo10:15
*** apetrich has quit IRC10:17
*** NobodyCa1 has joined #tripleo10:18
*** dsneddon_ has joined #tripleo10:19
*** NobodyCa1 has quit IRC10:26
*** apetrich has joined #tripleo10:32
*** dshulyak_ has joined #tripleo10:39
*** dshulyak has quit IRC10:39
*** dshulyak_ is now known as dshulyak10:39
*** bnemec has quit IRC10:40
*** fgimenez has quit IRC10:41
*** fgimenez has joined #tripleo10:42
*** bnemec has joined #tripleo10:42
*** paramite is now known as paramite|afk10:45
*** stendulker has quit IRC10:55
*** dsneddon_ has quit IRC10:56
*** bnemec has quit IRC11:01
*** jaosorior has quit IRC11:01
*** mgould has quit IRC11:04
*** yamahata has joined #tripleo11:10
*** paramite|afk is now known as paramite11:16
*** mgould has joined #tripleo11:18
*** mgould has quit IRC11:28
*** mgould has joined #tripleo11:32
*** apetrich has quit IRC11:34
*** bnemec has joined #tripleo11:36
*** panda_ has quit IRC11:37
*** panda_ has joined #tripleo11:37
*** dsneddon_ has joined #tripleo11:37
devvesalucasagomes: Downloading the ironic agent from TripleO CI works and I can run introspect-nodes now11:44
lucasagomesdevvesa, alright. So the previous ramdisk wasn't booting? Did you see anything strange in the console?11:44
devvesathe console entered in some kind of recovery mode and keep stuck there11:46
*** mgould has quit IRC11:46
devvesaNow I have another problem when I deploy the overcloud, though. The nodes stay in 'wait call-back' mode11:47
*** dsneddon_ has quit IRC11:48
devvesaHowever, I see the login in the console and I can ping them from the undercloud host11:48
*** mgould has joined #tripleo11:48
devvesaWhat is actually ironic waiting?11:49
*** fgimenez has quit IRC11:51
*** bvandenh has quit IRC11:51
*** fgimenez has joined #tripleo11:53
*** fgimenez has joined #tripleo11:53
*** jaosorior has joined #tripleo11:55
*** dsneddon_ has joined #tripleo11:56
*** jistr has joined #tripleo11:56
jaosoriorany +A for this? https://review.openstack.org/#/c/275436/11:57
*** apetrich has joined #tripleo12:02
*** paramite is now known as paramite|afk12:03
*** athomas has joined #tripleo12:07
*** apetrich has quit IRC12:07
devvesa@lucasagomes: ^12:11
lucasagomesdevvesa, "wait call-back" means that Ironic is waiting for the ramdisk to ping the Ironic API back saying it's ready for deployment12:12
lucasagomesdevvesa, is it stuck in the login forever? (Cause soon the ironic-python-agent should start and start talking to the ironic api)12:12
devvesalucasagomes: yes, is stuck in the login12:13
devvesaI understand that it has already passed the 'ramdisk' phase12:13
lucasagomesso "ramdisk" is just the type of image you are booting12:14
lucasagomeswe run a service called ironic-python-agent when that image boots up, that service will talk to the ironic api12:15
devvesahttp://i.imgur.com/03hMH1k.png12:15
lucasagomesand ironic will then start instructing ipa how to deploy the node (submitting commands to it)12:15
devvesalucasagomes: For what I see, the node has already the CentOS ready to be configured ...12:16
openstackgerritJames Slagle proposed openstack/python-tripleoclient: Simplify checking for stack complete  https://review.openstack.org/27514912:16
lucasagomesdevvesa, oh wait... You sure this booted the ramdisk? It seems that the boot failed and it falled back to the hard disk then12:16
lucasagomestry to restart that node please, check the console, see if it's really booting from the network and not from the local disk12:16
lucasagomesit may happen that the network boot failed for whatever reason (e.g can't get DHCP) and it fallback'd to the next boot device (which can be the hard disk)12:17
devvesano. it is the node. i've tried a login and now I see the following http://i.imgur.com/ak2Y6aK.png12:19
devvesaThe IP address of the hostname is the one assigned by neutron-undercloud12:19
lucasagomesdevvesa, right, so it was deployed?!12:20
devvesayes12:20
lucasagomesbut ironic is in "wait call-back" ?12:21
devvesayes12:21
*** paramite|afk is now known as paramite12:21
lucasagomesI don't think it's possible off the top of my head12:21
lucasagomesthe harddisk of this node was clean before you started the deployment? Or it had an image written on it already ? (from previous deployments)12:21
devvesait had a RedHat12:22
devvesawell, I've tried the deployment several times.. so just before it had a CentOS12:22
devvesatoo12:22
*** masco has quit IRC12:22
lucasagomesright... yeah cause Ironic will always move the node from "wait call-back" to "deploying" prior to writing the image to the disk12:24
lucasagomesdevvesa, you can look at the ironic-conductor logs, see if it wrote the image in the disk12:24
lucasagomesbut I don't think so... I think it couldn't boot via the network and then booted from the disk12:24
lucasagomesdevvesa, you can restart that node and see if it's failing to boot from the network12:25
devvesaok12:25
devvesalucasagomes: http://i.imgur.com/L06rgbP.png12:29
devvesaIt doesn't12:29
lucasagomesdevvesa, alright let it boot then12:29
devvesalucasagomes: same as before. Login screen. 'wait call-back' state12:30
lucasagomesdevvesa, centos screen?12:31
devvesayes12:31
lucasagomesok... so I may assume that the ironic-python-agent service failed to start on that ramdisk12:31
lucasagomesand it's not talking to the ironic API back12:31
devvesaAnd... I can not see the log anywhere, right?12:32
lucasagomesdevvesa, you can log in that node, if the image was built with the dynamic login element (https://github.com/openstack/diskimage-builder/tree/master/elements/dynamic-login)12:33
lucasagomesyou can change the httpboot/pxelinux.conf/<mac> and add the parameters there12:33
lucasagomesyou can also redirect the journald logs to the console so you can see what's going on without logging in12:33
lucasagomesjust add "systemd.journald.forward_to_console=yes" to the kernel cmdline12:34
lucasagomesyou can modify the kernel cmdline via ironic's configuration file as well (see pxe_append_parameters config option at ironic.conf) but you will need to restart the ironic-conductor service and deployment to that to get effect12:35
devvesathat sounds great12:35
lucasagomesdevvesa, or you can even try another ramdisk, the upstream one is used in our gate12:35
lucasagomesso it's likely that works12:35
devvesaby upstream you mean 'master'?12:36
lucasagomesdevvesa, yup12:36
lucasagomeshttp://tarballs.openstack.org/ironic-python-agent/coreos/files/12:36
devvesabecause I took the current one from the CI12:36
devvesain the stable release12:36
devvesa(BTW, i'm testing stable/liberty)12:36
lucasagomesright, yeah it's backward compatible12:37
lucasagomestry the other steps before, if you can login in the ramdisk check for the "ironic-python-agent" service, see if it's started and running12:37
devvesacoreos?12:38
lucasagomesdevvesa, upstream we build it on top of coreos12:38
*** yamahata has quit IRC12:38
openstackgerritGiulio Fidente proposed openstack-infra/tripleo-ci: Add support for --overcloud-update and use it for HA job  https://review.openstack.org/26046612:39
*** pino|work has quit IRC12:42
*** thrash|g0ne is now known as thrash12:43
*** weshay_xchat has joined #tripleo12:44
*** weshay_xchat is now known as weshay12:44
lucasagomesdevvesa, I will grab some food, pelase try to find out whether the service in the ramdisk has started or not and we can continue from there12:45
devvesaThank you very much lucasagomes12:45
lucasagomesno problem12:45
*** lucasagomes is now known as lucas-hungry12:46
*** tzumainn has joined #tripleo12:49
*** thrash has quit IRC12:51
*** aufi_ has joined #tripleo12:52
*** aufi has quit IRC12:53
*** cmyster has quit IRC12:53
*** cmyster has joined #tripleo12:54
openstackgerritEmilien Macchi proposed openstack/instack-undercloud: iptables: add missing rule for ceilometer/ssl  https://review.openstack.org/27620212:56
*** thrash has joined #tripleo12:58
*** thrash has joined #tripleo12:58
*** dtantsur|afk is now known as dtantsur12:58
openstackgerritEmilien Macchi proposed openstack/instack-undercloud: iptables: add missing rule for ceilometer/ssl  https://review.openstack.org/27620213:02
*** aufi_ is now known as aufi13:10
*** yamahata has joined #tripleo13:12
*** apetrich has joined #tripleo13:16
*** jayg|g0n3 is now known as jayg13:19
*** apetrich has quit IRC13:21
*** yamahata has quit IRC13:24
*** bvandenh has joined #tripleo13:26
*** trown has joined #tripleo13:26
*** yamahata has joined #tripleo13:29
*** paramite is now known as paramite|afk13:30
*** ukalifon1 has joined #tripleo13:32
*** dsneddon_ has quit IRC13:34
*** akuznetsov has joined #tripleo13:34
*** paramite|afk is now known as paramite13:34
*** yamahata has quit IRC13:36
*** david-lyle has quit IRC13:38
*** gfidente has quit IRC13:39
*** ukalifon1 has quit IRC13:39
*** apetrich has joined #tripleo13:40
*** Goneri has quit IRC13:42
*** pradk has joined #tripleo13:42
*** trozet has quit IRC13:46
*** gfidente has joined #tripleo13:47
*** gfidente has quit IRC13:47
*** gfidente has joined #tripleo13:47
*** dsneddon_ has joined #tripleo13:47
*** lucas-hungry is now known as lucasagomes13:48
*** paramite is now known as paramite|afk13:52
devvesalucasagomes: here again?13:52
lucasagomesdevvesa, yup13:52
devvesaI see the following in the console: http://i.imgur.com/p7wYOP7.png13:52
devvesaAnd this in the ironic-api logs: http://fpaste.org/318410/14545938/13:52
*** paramite|afk is now known as paramite13:53
devvesaAre both related?13:53
*** jayg is now known as jayg|g0n313:53
*** jayg|g0n3 is now known as jayg13:53
*** links has quit IRC13:56
EmilienMslagle: https://forge.puppetlabs.com/camptocamp/kmod13:56
lucasagomesdevvesa, yes!!! lookup() is the first method the ramdisk service will call13:58
lucasagomesin order to know which node it is13:58
*** trozet has joined #tripleo13:58
lucasagomesdevvesa, ok, which driver are u using to deploy?13:59
devvesapxe_drac13:59
openstackgerritGiulio Fidente proposed openstack/tripleo-puppet-elements: Add puppet-kmod into image  https://review.openstack.org/27624913:59
devvesaIt is a DELL environment14:00
* lucasagomes checks14:00
*** ayoung has quit IRC14:01
openstackgerritJames Slagle proposed openstack/tripleo-heat-templates: Increase size of connection tracking table  https://review.openstack.org/27576914:01
lucasagomesdevvesa, https://github.com/openstack/ironic/blob/stable/liberty/ironic/drivers/drac.py#L46-L5314:02
lucasagomesdevvesa, seems like the drac driver is not exposing the lookup method14:02
lucasagomeslooks like a bug in the driver driver14:02
lucasagomesdevvesa, can you change that code (1 sec I will give you the snippet), restart the ironic conductor and the node booting?14:03
devvesasure14:03
lucasagomesdevvesa, ok, on that mapping dict, just add a new element14:04
lucasagomesdevvesa, 'lookup': self.iscsi_vendor,14:04
lucasagomesrestart the ironic conductor and the node being deployed14:04
*** barra204 has quit IRC14:04
lucasagomesdevvesa, I'm filing a bug upstream against the drac driver and I if it works we can fix it there and backport to liberty14:05
devvesalucasagomes: Well, since I'm bothering you the whole day, I'm happy to help in the stability of ironic :)14:05
*** shakamunyi has joined #tripleo14:06
*** fgimenez has quit IRC14:06
lucasagomesdevvesa, thanks!14:06
lucasagomesdevvesa, yeah, let's make sure that's the problem14:06
openstackgerritGiulio Fidente proposed openstack/tripleo-puppet-elements: Add puppet-kmod into image  https://review.openstack.org/27624914:06
*** julim has joined #tripleo14:07
*** shadower74 has joined #tripleo14:09
*** shadower74 has quit IRC14:09
*** fgimenez has joined #tripleo14:10
*** fgimenez has joined #tripleo14:10
*** jcoufal has joined #tripleo14:10
*** jcoufal has quit IRC14:10
*** jcoufal has joined #tripleo14:11
lucasagomesdevvesa, https://review.openstack.org/#/c/276256/14:15
lucasagomesif that works I will propose a backport one for liberty14:15
devvesalucasagomes: +114:15
lucasagomesdevvesa, oh actually that may be wrong, 1 sec14:16
lucasagomesdevvesa, cause that's a driver vendor passthru...14:16
devvesaups, ok14:16
lucasagomesdevvesa, ok patch-set #2 should do it https://review.openstack.org/#/c/276256/214:17
*** pino|work has joined #tripleo14:17
*** paramite is now known as paramite|afk14:18
*** paramite|afk is now known as paramite14:20
*** tiswanso has joined #tripleo14:21
*** dshulyak has quit IRC14:21
* devvesa is testing14:22
*** akuznetsov has quit IRC14:24
*** egafford has joined #tripleo14:25
*** lblanchard has joined #tripleo14:25
mariosbnemec: https://devconfcz2016.sched.org/14:25
*** dshulyak has joined #tripleo14:26
*** tiswanso has quit IRC14:33
openstackgerritJames Slagle proposed openstack/python-tripleoclient: Simplify checking for stack complete  https://review.openstack.org/27514914:33
*** tiswanso has joined #tripleo14:34
openstackgerritJames Slagle proposed openstack/python-tripleoclient: Simplify checking for stack complete  https://review.openstack.org/27526314:35
lucasagomesdevvesa, no dice?14:37
devvesait failed, but I remember to restart the ironic-api after run the overcloud14:37
*** gmoro has joined #tripleo14:38
devvesaSo I consider it a false fail, trying again14:38
lucasagomesack14:40
lucasagomesdevvesa, you have to restart openstack-ironic-api and conductor btw14:40
dtantsurhey folks, what is stable/liberty condition now? is it worth rechecking?14:40
lucasagomesbecause conductor is the one that loads the driver14:40
devvesaFirst I restarted only the conductor14:41
devvesaYes, it fails14:42
devvesaAnd it leaves the node in 'error' state14:43
devvesaI could catch the following in the api logs14:43
devvesahttp://fpaste.org/318434/54596382/14:43
devvesalucasagomes: ^14:43
lucasagomesdevvesa, that's on the second try?14:44
* lucasagomes looks14:44
devvesait was the first one. In the second one it left the node on 'error' state and I stopped thinking that I would find another trace in logs14:45
devvesaBut I couldn't14:45
lucasagomesok, hmm aparently the drac driver wasn't tested with the IPA ramdisk at all :-/14:46
*** ayoung has joined #tripleo14:47
devvesaI found an error inside the wsman message in the ironic-conductor14:48
trowndtantsur: last full pass for stable/liberty looks like yesterday, but lots of 'all but ha' passes14:50
trowndtantsur: probably worth rechecking14:50
d0ugalQuick Poll: When you want to check the status of your TripleO deployment, what vital statistics do you check?14:50
dtantsurthanks14:50
lucasagomesdevvesa, can you paste it?14:51
devvesayes. A second14:51
devvesahttp://fpaste.org/318445/97521145/ lucasagomes: ^14:52
devvesawell ^14:52
devvesaThere is a "Import of system configuration XML file operation completed with errors."14:53
devvesaat the end14:53
lucasagomesifarkas, ^ any idea ?14:53
devvesamake it sense to try the pxe_ipmitool?14:53
lucasagomesdevvesa, it does yes14:54
devvesaok. let me try14:54
ifarkaschecking...14:54
lucasagomesdevvesa, tho the lock error seems not a persistent one14:54
*** trown is now known as trown|mtg14:55
ifarkaslucasagomes, devvesa, drac driver handles jobs in 'completed with errors' state, so that shouldn't be an issue14:55
openstackgerritLucas Alvares Gomes proposed openstack/diskimage-builder: Install IPA in a virtual enviroment  https://review.openstack.org/23584314:55
devvesaifakras: ok, thanks!14:56
*** Goneri has joined #tripleo14:56
*** pradk has quit IRC14:56
*** pradk has joined #tripleo14:56
*** tosky has quit IRC14:57
openstackgerritDerek Higgins proposed openstack-infra/tripleo-ci: Source undercloud environment variable from a file  https://review.openstack.org/27566714:57
openstackgerritDerek Higgins proposed openstack-infra/tripleo-ci: Split the deploy script into its own file  https://review.openstack.org/27566814:57
ifarkasdevvesa, does ironic node-show reveals any error message?14:58
devvesa| last_error             | Failed to tear down. Error: not all arguments converted during string   |14:58
devvesa|                        | formatting14:58
ifarkasdevvesa, can you find anything related to this "not all arguments converted during" in conductor log?15:00
devvesaifarkas: No sorry. I trimmed the error log for the last pastie :(15:04
devvesaI'll try now the pxe_ipmitool driver15:04
ifarkasdevvesa, ack15:04
*** morazi has quit IRC15:07
*** paramite is now known as paramite|afk15:09
dtantsurcould someone please approve a trivial DIB change having 2x +2? https://review.openstack.org/#/c/270267/15:13
openstackgerritBen Nemec proposed openstack/tripleo-common: Extend ping test create timeout  https://review.openstack.org/27629315:20
openstackgerritJames Slagle proposed openstack/tripleo-heat-templates: Increase size of connection tracking table  https://review.openstack.org/27576915:24
*** paramite|afk is now known as paramite15:25
*** fgimenez has quit IRC15:28
*** fgimenez has joined #tripleo15:28
openstackgerritGiulio Fidente proposed openstack-infra/tripleo-ci: Deploy using network isolation the HA job  https://review.openstack.org/27342415:32
*** julim has quit IRC15:33
*** julim has joined #tripleo15:35
*** panda_ has quit IRC15:37
*** eggmaster has quit IRC15:37
*** panda_ has joined #tripleo15:38
openstackgerritGiulio Fidente proposed openstack-infra/tripleo-ci: Deploy using network isolation the HA job  https://review.openstack.org/27342415:39
*** xinwu has joined #tripleo15:42
lucasagomesfolks can I get some eyes on https://review.openstack.org/#/c/235843/ ? This will fix the problem of building a fedora ramdisk with IPA15:45
lucasagomesdtantsur, ^ tested, works15:45
dtantsurjust +1ed15:46
lucasagomesty :-)15:46
openstackgerritGiulio Fidente proposed openstack/tripleo-heat-templates: Allow the deployer to pick a predefined IP for VIPs  https://review.openstack.org/27631615:46
*** oshvartz has quit IRC15:49
lucasagomesdevvesa, worked with ipmitool?15:50
*** jistr|mobi has quit IRC15:50
devvesalucasagomes15:59
devvesaNow I have a worse problem. It happened to me sometimes when I deployed VMs16:00
devvesaSomehow, it fails the deployment and tries over and over again with different nodes16:00
devvesait tries one node, the node fails (don't know the reason) and then tries another node16:01
devvesaThis is the trace on the conductor log: http://fpaste.org/318485/54601729/16:02
devvesaI've always solved by creating a new virtual environment16:03
devvesaWhat means not solving anything at all :)16:03
devvesaany idea?16:04
*** trown|mtg is now known as trown16:07
*** morazi has joined #tripleo16:09
devvesalucasagomes: ^16:09
lucasagomesdevvesa, that's the RetryFilter in Nova16:12
lucasagomeswhen an instance fails to deploy it will to deploy on another host (by default it tries 3 times)16:12
devvesaDo you know why it started to fail now?16:13
lucasagomesdevvesa, the error is coming from the ramdisk saying it can't start the iSCSI target (it will expose the local disk to the ironic conductor via iSCSI)16:14
lucasagomese o erro foi "No such file or directory"16:15
lucasagomeshmm... these nodes have a local disk right?16:15
devvesafor sure16:16
devvesaThis is an error that happened to me on VMs eventually16:16
trowndevvesa: liberty?16:16
devvesaNo, it happened more on master16:16
devvesaIn liberty is the first time I see it16:17
devvesaDo you think that is an error that occurs 'after' the previous error?16:17
devvesaI mean, that we have solved the previous one and now we are facing a new one?16:17
*** liverpooler has quit IRC16:18
trowndevvesa: could be multiple issues with similar manifestation16:19
trowndevvesa: I have not seen the  "No such file or directory" error on master16:19
devvesatrown: it can be...16:20
*** aufi has quit IRC16:20
trownI have seen that error on the IPA image in RDO, because tgt is not installed, and LIO support is only in mitaka+16:20
*** rpothier has joined #tripleo16:21
openstackgerritEmilien Macchi proposed openstack/instack-undercloud: iptables: add missing rule for ceilometer/ssl  https://review.openstack.org/27632916:22
*** alop has joined #tripleo16:23
*** david-lyle has joined #tripleo16:24
*** shardy has joined #tripleo16:30
lucasagomesdevvesa, yeah off the top of my head I haven't seem that either... I mean we test IPA in gate with VMs16:32
lucasagomesbut you know can happen... It may be just that the kernel haven't updated the devices tables yet16:32
lucasagomesperhaps we need a explicit "partprobe"16:32
*** hjensas has quit IRC16:33
*** akuznetsov has joined #tripleo16:40
dtantsurdevvesa, trown++ it might be a dependency problem in IPA itself16:44
dtantsurboth times I've seen this error it was the cause16:44
*** fgimenez has quit IRC16:45
*** apetrich has quit IRC16:45
jpeelershardy: could you do a quick container related review https://review.openstack.org/#/c/265366/16:45
trowndevvesa: also, are you doing HA all on a smaller virthost? (32G RAM or less)?16:45
trownI have very inconsistent results packing all the VMs needed for HA on this size host16:46
*** olap has quit IRC16:46
*** fgimenez has joined #tripleo16:47
trownfwiw, so does our CI16:47
*** shivrao has joined #tripleo16:48
trowndtantsur: on a slightly related topic (I saw the double scheduler bug in CI recently), would disabling all Nova filters on the undercloud except for the capabilities filter work for the new advanced matching?16:48
trownI think that would keep us from hitting the scheduler bug16:49
*** shivrao has quit IRC16:49
dtantsurtrown, it would work, but I'm not convinced it will solve the scheduler race (if you're referring to it)16:50
devvesatrown: no. is a baremetal machine of 128g ram16:50
*** rhallisey has quit IRC16:50
trowndtantsur: I saw https://review.openstack.org/273154 merged, and it made me think that only those filters that got an Exact variant were affected16:51
trownmaybe that is a naive assumption16:51
dtantsurtrown, I don't think we're using Exact filters in TripleO16:51
dtantsuror rather, someone told me we don't :)16:51
trowndtantsur: definitely not, that patch is not even in current-passed-ci yet16:52
*** shardy has quit IRC16:52
trowndtantsur: oh... I see, I read that totally wrong16:52
*** jaosorior has quit IRC16:52
trowndtantsur: so that patch does nothing for the tripleo case16:52
trownperfect16:52
trownsince we seem to be the only ones who hit that bug16:53
*** morazi has quit IRC16:53
dtantsurtrown, only tripleo has such a high probability of hitting it, because only tripleo *usually* deploy all the available nodes16:54
dtantsurother use cases assume you still have plenty of free resources16:54
dtantsurthis case is exceptional for them16:54
trownright... I just got hopeful that patch would save us16:54
trownbut it is totally irrelevant to tripleo case16:54
dtantsurtrown, another set of patches to nova is expected to solve it, but it's for mitaka16:54
dtantsurtrown, https://blueprints.launchpad.net/nova/+spec/host-state-level-locking16:55
*** jcoufal has quit IRC16:58
*** jtomasek__ has quit IRC16:58
*** mcornea has quit IRC17:00
*** bnemec has quit IRC17:02
*** jistr has quit IRC17:02
*** akrivoka has joined #tripleo17:03
*** jdob has quit IRC17:06
*** jaosorior_ has joined #tripleo17:08
*** gfidente has quit IRC17:08
*** jaosorior_ is now known as jaosorior17:09
*** dsneddon_ has quit IRC17:09
*** yamahata has joined #tripleo17:14
*** jistr has joined #tripleo17:16
*** fgimenez has quit IRC17:17
*** jistr has quit IRC17:18
*** dtantsur is now known as dtantsur|afk17:18
*** mbound has quit IRC17:21
*** akuznetsov has quit IRC17:23
*** xinwu has quit IRC17:25
*** xinwu has joined #tripleo17:26
*** tiswanso has quit IRC17:32
*** tiswanso has joined #tripleo17:32
*** paramite is now known as paramite|afk17:34
*** trown is now known as trown|lunch17:35
*** athomas has quit IRC17:35
*** paramite|afk is now known as paramite17:39
*** lucasagomes is now known as lucas-dinner17:44
*** mgould is now known as mgould|afk17:44
*** paramite has quit IRC17:50
*** shivrao has joined #tripleo17:51
*** shivrao_ has joined #tripleo17:54
*** jistr|mobi has joined #tripleo17:56
*** shivrao has quit IRC17:56
*** shivrao_ is now known as shivrao17:56
*** ayoung has quit IRC17:58
*** akrivoka has quit IRC17:59
*** Marga_ has quit IRC18:05
*** Marga_ has joined #tripleo18:05
*** olap has joined #tripleo18:07
*** xinwu has joined #tripleo18:08
*** ifarkas has quit IRC18:17
*** trown|lunch is now known as trown18:25
*** trown is now known as trown|mtg18:33
*** Marga_ has quit IRC18:38
*** devvesa has quit IRC18:39
*** yamahata has quit IRC18:40
*** shivrao has quit IRC18:41
*** eil397 has joined #tripleo18:43
*** bvandenh has quit IRC18:50
prometheanfireanyone able to get one more +2? https://review.openstack.org/#/c/273772/ dib18:51
*** dshulyak has quit IRC18:53
*** dshulyak has joined #tripleo18:54
*** dsneddon_ has joined #tripleo18:58
*** trown|mtg is now known as trown18:58
*** tiswanso has quit IRC19:02
*** Marga_ has joined #tripleo19:04
*** Marga_ has quit IRC19:04
*** Marga_ has joined #tripleo19:04
*** tiswanso has joined #tripleo19:05
*** tiswanso has quit IRC19:05
*** tiswanso has joined #tripleo19:05
*** akuznetsov has joined #tripleo19:06
*** david-lyle has quit IRC19:11
*** ayoung has joined #tripleo19:13
*** jprovazn has quit IRC19:15
*** penick has joined #tripleo19:18
*** jaosorior has quit IRC19:19
*** electrofelix has quit IRC19:22
*** pblaho has quit IRC19:36
*** dsneddon_ is now known as dsneddon_brno19:37
*** panda_ has quit IRC19:37
*** panda_ has joined #tripleo19:38
*** shivrao has joined #tripleo19:46
*** trozet has quit IRC19:48
*** ayoung has quit IRC20:04
*** akuznetsov has quit IRC20:26
*** dshulyak has quit IRC20:31
*** eil397 has quit IRC20:38
*** trown is now known as trown|outttypeww20:41
*** SlickN1k has joined #tripleo20:42
*** goneri_ has joined #tripleo20:43
*** gmoro_ has joined #tripleo20:43
*** Goneri has quit IRC20:47
*** SlickNik has quit IRC20:47
*** crinkle has quit IRC20:47
*** gmoro has quit IRC20:47
*** SlickN1k is now known as SlickNik20:47
*** crinkle has joined #tripleo20:49
*** crinkle has quit IRC20:49
*** bswartz has joined #tripleo20:51
*** bswartz has left #tripleo20:51
openstackgerritJames Slagle proposed openstack/tripleo-heat-templates: Increase size of connection tracking table  https://review.openstack.org/27576920:54
*** lblanchard has quit IRC21:04
*** NobodyCa1 has joined #tripleo21:07
*** absubram has quit IRC21:09
*** NobodyCa1 has quit IRC21:14
*** goneri_ has quit IRC21:21
*** dsneddon_brno has quit IRC21:24
*** rpothier has quit IRC21:25
*** dsneddon_brno has joined #tripleo21:25
*** weshay has quit IRC21:29
*** tiswanso has quit IRC21:44
*** pradk has quit IRC21:46
*** pradk has joined #tripleo21:48
*** julim has quit IRC21:55
*** julim has joined #tripleo21:55
*** julim has quit IRC22:02
*** jayg is now known as jayg|g0n322:05
*** apetrich has joined #tripleo22:06
*** absubram has joined #tripleo22:06
*** absubram_ has joined #tripleo22:08
*** nico_auv has quit IRC22:09
*** absubram has quit IRC22:11
*** absubram_ is now known as absubram22:11
*** absubram has quit IRC22:14
*** davidlenwell has quit IRC22:22
*** davidlenwell has joined #tripleo22:23
*** ChanServ sets mode: +v davidlenwell22:23
*** crinkle has joined #tripleo22:26
*** david-lyle has joined #tripleo22:48
*** egafford has quit IRC22:51
*** david-lyle has quit IRC22:52
*** goneri_ has joined #tripleo22:57
*** ayoung has joined #tripleo23:02
*** ayoung_ has joined #tripleo23:03
*** eil397 has joined #tripleo23:07
*** yamahata has joined #tripleo23:10
*** jistr|mobi has quit IRC23:19
*** pradk has quit IRC23:28
*** chlong has quit IRC23:30
*** panda_ has quit IRC23:37
*** panda_ has joined #tripleo23:38
openstackgerritJeff Peeler proposed openstack/tripleo-heat-templates: Minor fixes to allow local docker registry usage  https://review.openstack.org/26536623:39

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!