| * NobodyCam wanders afk to go look at at the fish... | 00:06 | |
| NobodyCam | devananda: I think I have a good path for 96902 should be able to get a new patch up tomorrow | 00:06 |
|---|---|---|
| *** matsuhashi has joined #openstack-ironic | 00:27 | |
| *** eguz has quit IRC | 00:28 | |
| openstackgerrit | OpenStack Proposal Bot proposed a change to openstack/ironic-python-agent: Updated from global requirements https://review.openstack.org/88722 | 00:35 |
| openstackgerrit | OpenStack Proposal Bot proposed a change to openstack/python-ironicclient: Updated from global requirements https://review.openstack.org/96263 | 00:39 |
| openstackgerrit | Ellen Hui proposed a change to openstack/ironic: Add node id to DEBUG message in _exec_ipmitool https://review.openstack.org/98928 | 00:44 |
| jroll | devananda: if you're curious :) https://review.openstack.org/#/c/98930/ | 01:05 |
| jroll | nova spec for configdrive generation | 01:05 |
| jroll | will hit the ironic spec tomorrow | 01:05 |
| *** ellenh has quit IRC | 01:08 | |
| mrda | damn, just realised I missed today's meeting | 01:15 |
| * mrda goes and reads scrollback | 01:15 | |
| *** rloo has quit IRC | 01:17 | |
| *** rloo has joined #openstack-ironic | 01:18 | |
| devananda | rloo: reviewed already | 01:22 |
| devananda | er, jroll .... ^ | 01:22 |
| devananda | rloo: sorry, ignore that | 01:22 |
| *** godp1301 has joined #openstack-ironic | 01:25 | |
| *** godp1301 has quit IRC | 01:27 | |
| *** godp1301 has joined #openstack-ironic | 01:27 | |
| *** nosnos has joined #openstack-ironic | 01:33 | |
| *** godp1301 has quit IRC | 01:35 | |
| *** igordcard has quit IRC | 01:41 | |
| openstackgerrit | A change was merged to openstack/ironic: Add __init__.py for nova scheduler filters https://review.openstack.org/96912 | 01:48 |
| openstackgerrit | A change was merged to openstack/ironic: Cleanup mock patch without `with` part 2 https://review.openstack.org/86536 | 01:48 |
| devananda | \O/ | 01:54 |
| devananda | things are landing. wooot! | 01:54 |
| devananda | ok, /me walks home | 01:54 |
| devananda | ttyl! | 01:54 |
| mrda | :) | 01:55 |
| *** Haomeng has quit IRC | 02:19 | |
| *** godp1301 has joined #openstack-ironic | 02:34 | |
| *** godp1301 has quit IRC | 02:37 | |
| *** rloo has quit IRC | 02:43 | |
| *** rloo has joined #openstack-ironic | 02:43 | |
| *** rloo has quit IRC | 02:48 | |
| *** rloo has joined #openstack-ironic | 02:49 | |
| *** rwsu has quit IRC | 02:52 | |
| *** vinbs has joined #openstack-ironic | 02:57 | |
| jroll | devananda: thanks :) | 03:23 |
| openstackgerrit | A change was merged to openstack/ironic: Log exceptions from deploy and tear_down https://review.openstack.org/97318 | 03:36 |
| *** nosnos has quit IRC | 03:43 | |
| jroll | devananda: updated, thanks again, useful feedback :) | 03:45 |
| *** pcrews_ has quit IRC | 03:52 | |
| openstackgerrit | Ruby Loo proposed a change to openstack/ironic: Work around iscsiadm delete failures https://review.openstack.org/96577 | 03:54 |
| openstackgerrit | Ruby Loo proposed a change to openstack/ironic: Remove 'fake' and 'ssh' drivers from default enabled list https://review.openstack.org/88711 | 03:54 |
| *** eghobo has joined #openstack-ironic | 03:55 | |
| *** eghobo has quit IRC | 03:55 | |
| *** eghobo has joined #openstack-ironic | 03:56 | |
| *** coolsvap|afk is now known as coolsvap | 03:57 | |
| *** rloo has quit IRC | 03:58 | |
| *** rloo has joined #openstack-ironic | 03:59 | |
| *** rloo has quit IRC | 04:10 | |
| *** rloo has joined #openstack-ironic | 04:11 | |
| *** nosnos has joined #openstack-ironic | 04:24 | |
| *** Mikhail_D_wk has quit IRC | 04:25 | |
| *** Mikhail_D_wk has joined #openstack-ironic | 04:25 | |
| openstackgerrit | lifeless proposed a change to openstack/ironic: Draft a 'pxe_logpower' driver https://review.openstack.org/98950 | 04:30 |
| *** rloo has quit IRC | 04:33 | |
| *** eghobo has quit IRC | 04:36 | |
| *** k4n0 has joined #openstack-ironic | 04:38 | |
| lifeless | devananda: ^ it was small enough I figured pushing it up to gerrit was better than sending folk a patch directly | 04:39 |
| *** eghobo has joined #openstack-ironic | 04:50 | |
| *** Poornima has joined #openstack-ironic | 04:53 | |
| *** eghobo has quit IRC | 04:54 | |
| *** eghobo has joined #openstack-ironic | 04:54 | |
| *** harlowja is now known as harlowja_away | 04:59 | |
| *** Mikhail_D_ltp has joined #openstack-ironic | 05:07 | |
| *** Mikhail_D_ltp has quit IRC | 05:17 | |
| *** lazy_prince has joined #openstack-ironic | 05:17 | |
| *** lazy_prince has quit IRC | 05:21 | |
| *** rakesh_hs has joined #openstack-ironic | 05:26 | |
| *** lazy_prince has joined #openstack-ironic | 05:28 | |
| vinbs | Hello Ironic :) | 05:34 |
| vinbs | I'm trying to launch an instance on baremetal using ironic | 05:35 |
| *** pradipta_away is now known as pradipta | 05:36 | |
| vinbs | I have progressed till the stage where my baremetal gets an ip assigned from dnsmasq running in my openstack node | 05:36 |
| vinbs | but when the baremetal tries to pxe boot, I get this error: | 05:37 |
| vinbs | PXE-E11: ARP TIMEOUT | 05:37 |
| vinbs | followed by PXE-E38: tftp cannot open connection | 05:38 |
| vinbs | What might be the mistake I have done with my pxe boot setup? | 05:38 |
| *** eguz has joined #openstack-ironic | 05:50 | |
| *** eghobo has quit IRC | 05:50 | |
| *** sysexit has joined #openstack-ironic | 05:50 | |
| *** rameshg87 has joined #openstack-ironic | 05:56 | |
| openstackgerrit | Adam Gandelman proposed a change to openstack/ironic: Update Nova driver's list_instance_uuids() https://review.openstack.org/98268 | 05:56 |
| *** Poornima has quit IRC | 06:01 | |
| openstackgerrit | OpenStack Proposal Bot proposed a change to openstack/ironic: Imported Translations from Transifex https://review.openstack.org/96063 | 06:01 |
| *** Poornima has joined #openstack-ironic | 06:08 | |
| *** Poornima has quit IRC | 06:13 | |
| *** Poornima has joined #openstack-ironic | 06:13 | |
| *** Poornima has quit IRC | 06:16 | |
| *** Poornima has joined #openstack-ironic | 06:16 | |
| *** Poornima has quit IRC | 06:20 | |
| *** max_lobur has joined #openstack-ironic | 06:20 | |
| *** rameshg87 has quit IRC | 06:25 | |
| *** Kai14 has joined #openstack-ironic | 06:28 | |
| *** eguz has quit IRC | 06:41 | |
| *** rameshg87 has joined #openstack-ironic | 06:42 | |
| *** lazy_prince has quit IRC | 06:49 | |
| *** Mikhail_D_ltp has joined #openstack-ironic | 06:51 | |
| *** r0j4z0 has quit IRC | 06:56 | |
| *** athomas has joined #openstack-ironic | 06:56 | |
| *** max_lobur has quit IRC | 06:56 | |
| *** jcoufal has joined #openstack-ironic | 06:59 | |
| *** nosnos has quit IRC | 06:59 | |
| *** matsuhashi has quit IRC | 07:01 | |
| *** matsuhas_ has joined #openstack-ironic | 07:01 | |
| *** nosnos has joined #openstack-ironic | 07:05 | |
| *** lazy_prince has joined #openstack-ironic | 07:06 | |
| *** foexle has joined #openstack-ironic | 07:17 | |
| Mikhail_D_ltp | Good morning folks! :) | 07:19 |
| mrda | Morning Mikhail_D_ltp | 07:24 |
| openstackgerrit | Rakesh H S proposed a change to openstack/ironic: ipmi double bridging functionality https://review.openstack.org/95775 | 07:39 |
| *** ifarkas has joined #openstack-ironic | 07:40 | |
| *** max_lobur has joined #openstack-ironic | 07:43 | |
| *** jistr has joined #openstack-ironic | 08:00 | |
| *** max_lobur has quit IRC | 08:04 | |
| *** athomas has quit IRC | 08:07 | |
| openstackgerrit | A change was merged to openstack/ironic: Fix concurrent deletes in virt driver https://review.openstack.org/98184 | 08:08 |
| *** gcb has joined #openstack-ironic | 08:08 | |
| *** athomas has joined #openstack-ironic | 08:09 | |
| *** martyntaylor has joined #openstack-ironic | 08:16 | |
| *** ndipanov has joined #openstack-ironic | 08:17 | |
| *** derekh_ has joined #openstack-ironic | 08:17 | |
| *** lucasagomes has joined #openstack-ironic | 08:29 | |
| *** max_lobur has joined #openstack-ironic | 08:37 | |
| *** rameshg87 has quit IRC | 08:37 | |
| *** matsuhas_ has quit IRC | 08:38 | |
| *** nosnos has quit IRC | 08:40 | |
| *** pelix has joined #openstack-ironic | 08:41 | |
| *** nosnos has joined #openstack-ironic | 08:43 | |
| *** nosnos has quit IRC | 08:44 | |
| *** matsuhashi has joined #openstack-ironic | 08:44 | |
| *** athomas has quit IRC | 08:49 | |
| *** jcoufal has quit IRC | 08:55 | |
| *** jcoufal has joined #openstack-ironic | 08:56 | |
| vinbs | Hello lucasagomes :) | 08:59 |
| lucasagomes | vinbs, hey good morning | 08:59 |
| *** romcheg has joined #openstack-ironic | 09:01 | |
| vinbs | lucasagomes, I was trying to launch an instance on baremetal | 09:02 |
| vinbs | lucasagomes, I'm stuck at probably the last stage | 09:02 |
| *** nosnos has joined #openstack-ironic | 09:03 | |
| vinbs | lucasagomes, I'm using flat networking and the baremetal node gets an ip assigned from dnsmasq running on my openstack node | 09:03 |
| vinbs | but I get an error when the baremetal node tries to do pxe boot, PXE E-11: ARP timeout | 09:04 |
| vinbs | any idea where I might have gone wrong? | 09:04 |
| lucasagomes | vinbs, hmmm.... I haven't seem it | 09:04 |
| romcheg | Good morning vinbs lucasagomes and everyone else! | 09:04 |
| lucasagomes | but lemme try to check online to see if I get some idea | 09:04 |
| agordeev | good morning Ironic | 09:06 |
| vinbs | good morning romcheg :) | 09:06 |
| lucasagomes | vinbs, tftp server is up? can you try to access it locally? tftp localhost | 09:07 |
| lucasagomes | and try to get <file> | 09:07 |
| lucasagomes | romcheg, morning! | 09:08 |
| *** athomas has joined #openstack-ironic | 09:09 | |
| *** rameshg87 has joined #openstack-ironic | 09:09 | |
| vinbs | lucasagomes, I'm able to do tftp localhost | 09:11 |
| vinbs | lucasagomes, but i'm not able to get any file from /tftpboot | 09:11 |
| lucasagomes | vinbs, right hmm it seems to be a common issue, google returns a bunch of results for that error | 09:12 |
| lucasagomes | causes vary a bit | 09:12 |
| lucasagomes | http://www.bootix.com/support/problems_solutions/pxe_e11_arp_timeout.html | 09:12 |
| vinbs | lucasagomes, let me see if I can fix it using those results | 09:13 |
| vinbs | thanks :) | 09:13 |
| lucasagomes | np | 09:13 |
| *** pradipta is now known as pradipta_away | 09:18 | |
| *** pradipta_away is now known as pradipta | 09:20 | |
| *** pradipta is now known as pradipta_away | 09:21 | |
| *** Haomeng has joined #openstack-ironic | 09:22 | |
| *** rameshg87 has quit IRC | 09:32 | |
| *** max_lobur has quit IRC | 09:36 | |
| *** max_lobur has joined #openstack-ironic | 09:44 | |
| openstackgerrit | Aleksandr Gordeev proposed a change to openstack/ironic-python-agent: Improve GenericHardwareManager https://review.openstack.org/92847 | 09:49 |
| openstackgerrit | Aleksandr Gordeev proposed a change to openstack/ironic-python-agent: Add hardware_utils https://review.openstack.org/92399 | 09:49 |
| *** rameshg87 has joined #openstack-ironic | 09:50 | |
| *** rameshg87 has left #openstack-ironic | 09:54 | |
| *** loki184 has joined #openstack-ironic | 09:56 | |
| *** gcb has quit IRC | 10:04 | |
| Shrews | Morning all. Just FYI, I will not be around today. Have fun with the review jam! | 10:07 |
| openstackgerrit | A change was merged to openstack/ironic: PXE driver to validate the requested image in Glance https://review.openstack.org/96710 | 10:14 |
| *** gcb has joined #openstack-ironic | 10:16 | |
| *** igordcard has joined #openstack-ironic | 10:16 | |
| openstackgerrit | OpenStack Proposal Bot proposed a change to openstack/ironic: Updated from global requirements https://review.openstack.org/96228 | 10:18 |
| *** romcheg has quit IRC | 10:20 | |
| *** vinbs_ has joined #openstack-ironic | 10:25 | |
| *** vinbs has quit IRC | 10:26 | |
| *** vinbs_ is now known as vinbs | 10:26 | |
| *** rameshg87 has joined #openstack-ironic | 10:31 | |
| rameshg87 | hello lucasagomes | 10:32 |
| openstackgerrit | Anusha Ramineni proposed a change to openstack/ironic-specs: iLO Power Driver for Ironic https://review.openstack.org/97455 | 10:40 |
| vinbs | lucasagomes, good news :) | 10:41 |
| vinbs | lucasagomes, pxe boot went through | 10:41 |
| lucasagomes | rameshg87, morning | 10:42 |
| lucasagomes | vinbs, nice!!! what was it? | 10:42 |
| lucasagomes | some network config? | 10:42 |
| rameshg87 | good morning lucasagomes | 10:42 |
| vinbs | lucasagomes, I didn't have tftp server running :P | 10:42 |
| lucasagomes | vinbs, ah hah | 10:43 |
| vinbs | lucasagomes, silliest mistake i cud have made | 10:43 |
| lucasagomes | vinbs, well, happens :D | 10:43 |
| rameshg87 | a request to take a stab at https://review.openstack.org/#/c/89500/ | 10:43 |
| rameshg87 | :-) | 10:43 |
| lucasagomes | rameshg87, oh true, I will take a look in a bit | 10:43 |
| vinbs | lucasagomes, since this is a baremetal server how do i get the right image to boot. the cirros image i used panics on boot | 10:43 |
| lucasagomes | vinbs, you can create another image with diskimage-builder | 10:44 |
| lucasagomes | vinbs, have you used it before? | 10:44 |
| lucasagomes | should be pretty straight forward https://github.com/openstack/diskimage-builder | 10:45 |
| vinbs | lucasagomes, that is the image that panics.. i think I need an image with right drivers for my hardware | 10:45 |
| lucasagomes | vinbs, a standard ubuntu/fedora image won't boot ? | 10:46 |
| vinbs | lucasagomes, I have tried with only cirros so far | 10:47 |
| vinbs | lucasagomes, i will try with ubuntu image then | 10:47 |
| lucasagomes | right, cause with dib you can generate an fedora or ubuntu image | 10:47 |
| lucasagomes | vinbs, cool | 10:47 |
| *** romcheg has joined #openstack-ironic | 10:48 | |
| *** martyntaylor has quit IRC | 10:51 | |
| *** rakesh_hs has quit IRC | 10:55 | |
| rameshg87 | lucasagomes, sometime back we had talked about adding support in diskimage-builder to provision images from the baremetal node itself (rather than iscsi and then do it from conductor node) | 10:57 |
| *** ekarlso has quit IRC | 10:57 | |
| rameshg87 | lucasagomes: i have submitted a tripleo design spec for that: https://review.openstack.org/97736 | 10:58 |
| rameshg87 | lucasagomes, you can have a stab at that as well .. :-) | 10:58 |
| *** ekarlso has joined #openstack-ironic | 10:58 | |
| lucasagomes | rameshg87, nice, are u suggesting to fetch the image directly from glance? | 10:58 |
| lucasagomes | rameshg87, ah right sorry wil read the spec | 10:58 |
| rameshg87 | lucasagomes, proposing to fetch from swift using temp url | 10:59 |
| *** rakesh_hs has joined #openstack-ironic | 11:02 | |
| *** coolsvap is now known as coolsvap|afk | 11:04 | |
| *** martyntaylor has joined #openstack-ironic | 11:04 | |
| *** bigjools has quit IRC | 11:05 | |
| *** bigjools has joined #openstack-ironic | 11:05 | |
| *** bigjools has joined #openstack-ironic | 11:05 | |
| *** romcheg has quit IRC | 11:24 | |
| openstackgerrit | Stig Telfer proposed a change to openstack/ironic-specs: Power driver for SNMP-enabled smart PDUs https://review.openstack.org/96838 | 11:27 |
| *** rameshg87 has left #openstack-ironic | 11:28 | |
| Kai14 | hi! Since I deployed my first vm with ironic I'm wondering if neutron already is capable of sending a DHCP Offer? because I had to use dhcp-server for this. | 11:29 |
| *** romcheg has joined #openstack-ironic | 11:40 | |
| *** gcb has quit IRC | 11:47 | |
| agordeev | Kai14: hi, surely it is capable. Few month ago merged into Devstack fakebaremetal env support is relies on that. | 11:47 |
| agordeev | Kai14: http://docs.openstack.org/developer/ironic/dev/dev-quickstart.html#deploying-ironic-with-devstack | 11:48 |
| *** sysexit has quit IRC | 11:49 | |
| Kai14 | agordeev, yes I did this. But unfortunately it didn't work with neutron. Therefore I used the dhcp-server for assigning IPs. | 11:49 |
| Kai14 | agordeev, I also used tcpdump and saw the dhcp boot requests in my devstack VM. Could it be that neutron only replies to dhcp request which come over a specific network interface? | 11:51 |
| agordeev | Kai14: neutron starts dnsmasq in separate network namespace for DHCP managing connected into `private` neutron network by default. It replies to incoming requests only for specific MAC addresses assigned to neutron ports | 11:56 |
| Kai14 | agordeev, okay thanks. seems that I have to look into the neutron configuration. But this "neutron ports" normally have the same mac address like my ironic-port, right? | 11:59 |
| *** sysexit has joined #openstack-ironic | 12:01 | |
| agordeev | Kai14: right, it should has exactly the same mac | 12:04 |
| *** vinbs has quit IRC | 12:07 | |
| *** Alexei_987 has joined #openstack-ironic | 12:15 | |
| *** lucasagomes is now known as lucas-hungry | 12:16 | |
| openstackgerrit | Hans Lindgren proposed a change to openstack/ironic: Fix host manager node detection logic https://review.openstack.org/97447 | 12:17 |
| openstackgerrit | Hans Lindgren proposed a change to openstack/ironic: Update IronicHostManager tests to better match how code works https://review.openstack.org/99026 | 12:17 |
| openstackgerrit | Hans Lindgren proposed a change to openstack/ironic: Add missing stats to IronicNodeState https://review.openstack.org/99027 | 12:17 |
| *** lucas-hungry is now known as lucasagomes | 12:20 | |
| *** jdob has joined #openstack-ironic | 12:22 | |
| *** loki184 has quit IRC | 12:24 | |
| openstackgerrit | A change was merged to openstack/ironic: Fix AttributeError when calling create_engine() https://review.openstack.org/98425 | 12:26 |
| openstackgerrit | OpenStack Proposal Bot proposed a change to openstack/ironic: Updated from global requirements https://review.openstack.org/96228 | 12:28 |
| *** dhellmann has quit IRC | 12:28 | |
| *** romcheg1 has joined #openstack-ironic | 12:37 | |
| *** romcheg has quit IRC | 12:39 | |
| *** lucasagomes is now known as lucas-hungry | 12:44 | |
| *** rloo has joined #openstack-ironic | 12:53 | |
| jroll | mornin y'all | 12:56 |
| *** linggao has joined #openstack-ironic | 12:56 | |
| jroll | 03:59:15 rameshg87 | lucasagomes, proposing to fetch from swift using temp url <- have you seen this? https://review.openstack.org/#/c/98904/ | 12:59 |
| rloo | morning jroll | 13:01 |
| jroll | hey rloo :) | 13:02 |
| romcheg1 | Morning jroll rloo! | 13:02 |
| *** rloo has quit IRC | 13:02 | |
| jroll | hiya romcheg1! | 13:02 |
| *** romcheg1 is now known as romcheg | 13:03 | |
| *** rloo has joined #openstack-ironic | 13:03 | |
| JoshNang | good morning ironic! | 13:04 |
| *** rloo has quit IRC | 13:04 | |
| *** rloo has joined #openstack-ironic | 13:04 | |
| jroll | a wild JoshNang appears | 13:04 |
| linggao | good morning JoshNang jroll | 13:05 |
| *** rloo has quit IRC | 13:05 | |
| *** rloo has joined #openstack-ironic | 13:05 | |
| *** k4n0 has quit IRC | 13:06 | |
| devananda | g'morning, all! | 13:06 |
| *** godp1301 has joined #openstack-ironic | 13:06 | |
| *** godp1301 has left #openstack-ironic | 13:07 | |
| jroll | morning devananda | 13:07 |
| devananda | for those new to reviewdays, we coordinate here: https://etherpad.openstack.org/p/IronicReviewDay | 13:08 |
| devananda | there's still some left overs from the last one ... heh | 13:08 |
| devananda | once a thing is merged, it gets deleted | 13:08 |
| devananda | first thing that we should do - add a list of reviews for crit/high bug fixes that need reviews | 13:09 |
| linggao | good morning devananda | 13:12 |
| romcheg | Morning devananda! | 13:15 |
| rloo | lucas-hungry: this needs to be rebased: https://review.openstack.org/#/c/91097/. | 13:16 |
| *** romcheg has quit IRC | 13:19 | |
| *** romcheg has joined #openstack-ironic | 13:20 | |
| *** sseago has quit IRC | 13:20 | |
| rloo | devananda: are you going to have a 'want-for-j1' section, or is that the crit/high bug fixes? | 13:20 |
| *** sseago has joined #openstack-ironic | 13:20 | |
| *** rloo has quit IRC | 13:21 | |
| *** rloo has joined #openstack-ironic | 13:21 | |
| *** Kai14 has quit IRC | 13:22 | |
| devananda | rloo: i want everything :) let's just land what we can | 13:22 |
| *** rloo has quit IRC | 13:22 | |
| *** rloo has joined #openstack-ironic | 13:22 | |
| romcheg | It's been verified so slooooowly... | 13:22 |
| devananda | rloo: i'd rather have a perspective of what's important overall | 13:22 |
| rloo | FWIW, I'm going to rebase https://review.openstack.org/#/c/96498/ unless Shrews is here? | 13:23 |
| rloo | devananda: sounds good to me | 13:23 |
| devananda | also, for anyone not familiar with it -- http://status.openstack.org/zuul/ <-- status of the check/gate queues | 13:23 |
| rloo | romcheg: IF it even successfully gets verified :-( | 13:23 |
| devananda | right now, there's only a ~4hr gate queue :-D | 13:23 |
| devananda | though with the rate of random failures in the gate, it's really more likely double that | 13:23 |
| romcheg | rloo: I tried to approve some patches carefully to check it out | 13:23 |
| romcheg | devananda: should we approve patches now or be very gentle and let the queue to become smaller? | 13:24 |
| devananda | also, elastic-recheck is not matching ironic right now, even though we've submitted several E-R queries in the last week | 13:24 |
| devananda | status of E-R can be seen here: http://status.openstack.org/elastic-recheck/ | 13:24 |
| rloo | romcheg: I'm finding a lot of failures wrt bug 1326754 - Ubuntu CI failures : Unable to locate package ubuntu-cloud-keyring | 13:24 |
| devananda | note especially these, which I've seen hit our merge attemts several times | 13:25 |
| devananda | Bug 1286635 - "Packages cannot be authenticated" when doing an apt-get in the gate | 13:25 |
| devananda | Bug 1326754 - Ubuntu CI failures : Unable to locate package ubuntu-cloud-keyring | 13:25 |
| devananda | Bug 1321494 - NodeLocked causing random test failures | 13:25 |
| romcheg | I also faced "apache2ctl command not found" | 13:25 |
| devananda | romcheg: that is actually Bug 1326754 | 13:26 |
| romcheg | devananda: Got it | 13:27 |
| devananda | romcheg: I've seen several of those -- if you look earlier in the log, it failed to install apache because of 1326754 in all cases that i've seen so far | 13:27 |
| devananda | if anyone feels like hacking on E-R itself, this is why we're not getting any comments from E-R on our test failures: https://bugs.launchpad.net/openstack-ci/+bug/1328318 | 13:29 |
| devananda | lucas-hungry: the mgmt interface patch series -- those ready or close to? | 13:31 |
| *** lucas-hungry is now known as lucasagomes | 13:34 | |
| lucasagomes | devananda, sorry was having lunch | 13:34 |
| lucasagomes | devananda, 1 sec I'm on a call | 13:34 |
| *** jgrimm has joined #openstack-ironic | 13:34 | |
| *** matty_dubs|gone is now known as matty_dubs | 13:36 | |
| *** martyntaylor1 has joined #openstack-ironic | 13:38 | |
| *** matsuhashi has quit IRC | 13:39 | |
| *** martyntaylor has quit IRC | 13:41 | |
| *** rakesh_hs has quit IRC | 13:46 | |
| *** nosnos has quit IRC | 13:50 | |
| openstackgerrit | Jarrod Johnson proposed a change to stackforge/pyghmi: Fix infinitely recursing custom keepalives https://review.openstack.org/99060 | 13:50 |
| openstackgerrit | Jarrod Johnson proposed a change to stackforge/pyghmi: Fix infinitely recursing custom keepalives https://review.openstack.org/99060 | 13:51 |
| * devananda steps away for a few minutes | 13:52 | |
| jroll | ok, I think we've got enough things to look at | 13:55 |
| jroll | let's get started when deva returns | 13:55 |
| rloo | jroll: you can just get started? | 13:56 |
| openstackgerrit | Jarrod Johnson proposed a change to stackforge/pyghmi: Fix infinitely recursing custom keepalives https://review.openstack.org/99060 | 13:56 |
| jroll | rloo: in previous jams (that I've been in) we've tended to look at them together, but ok :) | 13:59 |
| romcheg | Have to go for a while. Will be back later | 13:59 |
| rloo | jroll: oh. this is the first one I'm participating in ;) I'm rebasing something so no need for everyone to look at what I'm rebasing. | 13:59 |
| jroll | cool | 14:00 |
| lucasagomes | devananda, so those are pretty much ready (needs rebase), except from the REST api patch for it, I would like to hold it until we get the async api thing ironed out | 14:00 |
| lucasagomes | rloo, thank you will do | 14:00 |
| lucasagomes | and morning all :) | 14:00 |
| rloo | morning lucasagomes! (I tried to rebase that last night and couldn't get it to work. I might have been too tired.) | 14:00 |
| *** lazy_prince has quit IRC | 14:00 | |
| lucasagomes | rloo, :( will rebase it here | 14:01 |
| jroll | lucasagomes: hey, on https://review.openstack.org/#/c/98120/ | 14:03 |
| * devananda is back | 14:03 | |
| jroll | does that need to depend on the wip review? | 14:03 |
| *** jcoufal has quit IRC | 14:03 | |
| jroll | I understand there will be failures if the unique constraint lands first, but it's just failing things early so I think I'm ok with that | 14:04 |
| devananda | lucasagomes: async api spec seemed to have derailed into a discussion of several things I didn't propose. lifeless and I revisited it here: https://etherpad.openstack.org/p/ironic-and-fragile-hardware but I Haven't reworked that into the spec yet | 14:04 |
| *** romcheg1 has joined #openstack-ironic | 14:04 | |
| lucasagomes | jroll, yeah it should, we first have to make sure that the instance get's unassociated with the node in the tear down | 14:04 |
| lucasagomes | and then we can add the unique constraint | 14:04 |
| lucasagomes | jroll, right, it's a bit risky no? | 14:04 |
| jroll | lucasagomes: well, how I see it: today it fails in horrible strange ways. if the unique constraint lands, it will fail earlier in less strange ways. | 14:05 |
| lucasagomes | I would prefer to do some tests first see if I don't break anything before adding the uc before | 14:05 |
| lucasagomes | jroll, hah | 14:05 |
| lucasagomes | well that's true | 14:05 |
| lucasagomes | devananda, ^? | 14:05 |
| jroll | :) | 14:05 |
| lucasagomes | devananda, right, yeah I read the etherpad I like the way it's going | 14:05 |
| openstackgerrit | A change was merged to stackforge/pyghmi: Fix infinitely recursing custom keepalives https://review.openstack.org/99060 | 14:05 |
| lucasagomes | devananda, but right now my patch adding the REST api to the management interface does things sync *e.g setting boot device | 14:06 |
| lucasagomes | so it really goes against the effort we are doing to have things async | 14:06 |
| lucasagomes | I'd like to rework that patch before landing it | 14:06 |
| *** romcheg has quit IRC | 14:06 | |
| lucasagomes | but the rest of the mgmt interface, for internal use, is pretty much ready to go | 14:06 |
| lucasagomes | needs rebase | 14:06 |
| devananda | lucasagomes: gotcha. so that's an interesting question. expose very nicetohave feature in a sync way // make more work for async patch. | 14:07 |
| lucasagomes | devananda, yeah, exactly... | 14:07 |
| lucasagomes | tho reworking it in an async way won't take that long | 14:08 |
| devananda | lucasagomes: and i see your point, though i'd be ok landing it and including it in the refactor // async work. if you want to hold it off, please WIP them all | 14:08 |
| lucasagomes | and we don't have to wait for the rest of the api work | 14:08 |
| lucasagomes | devananda, ack will WIP it | 14:08 |
| lucasagomes | devananda, it's just the last patch of the series | 14:09 |
| openstackgerrit | A change was merged to openstack/ironic: Fix the logging message to identify node by uuid https://review.openstack.org/98319 | 14:09 |
| devananda | \o/ | 14:09 |
| lucasagomes | the interface itself, and calling it internal (<driver>.management.<method>) can land in with no problem | 14:09 |
| * lucasagomes start rebasing stuff | 14:09 | |
| * devananda deletes a review from the etherpad! | 14:09 | |
| devananda | lucasagomes: cool | 14:10 |
| lucasagomes | devananda, any comments on the unique constraint thing that jroll raised? | 14:10 |
| devananda | lucasagomes: yea, just posted | 14:11 |
| jroll | devananda: I see your comment there, but wdyt specifically about landing it without the review it depends on | 14:13 |
| jroll | 07:05:06 jroll | lucasagomes: well, how I see it: today it fails in horrible strange ways. if the unique constraint lands, it will fail earlier in less strange ways. | 14:13 |
| openstackgerrit | Jarrod Johnson proposed a change to stackforge/pyghmi: Don't defer custom keepalive expiry on all payloads https://review.openstack.org/99067 | 14:13 |
| *** romcheg1 has quit IRC | 14:13 | |
| devananda | ah | 14:14 |
| NobodyCam | uggg .. I am very sorry | 14:14 |
| NobodyCam | I just awoke | 14:14 |
| NobodyCam | :( | 14:14 |
| *** jdob has quit IRC | 14:15 | |
| *** jdob has joined #openstack-ironic | 14:15 | |
| devananda | jroll: I think they both need to land to fix this bug | 14:15 |
| lucasagomes | the reason why the first one is WIP | 14:15 |
| devananda | jroll: adding the uuid will make it fail faster, but i'm not sure it'll make it any better | 14:16 |
| lucasagomes | is because I need to modify a bit the driver | 14:16 |
| devananda | jroll: because nova still may not be able to clean up after a failed deploy | 14:16 |
| devananda | lucasagomes: right - adam_g had some very good feedback there, i think | 14:16 |
| lucasagomes | the driver try to find the node by looking at the instance uuid, when we tear down, the instance is not unassociated with the node | 14:16 |
| lucasagomes | so the driver would fail in the wait() condition | 14:16 |
| devananda | NobodyCam: g'morning! | 14:16 |
| lucasagomes | devananda, nice will talk to him | 14:17 |
| NobodyCam | Good morning devananda ,and all! | 14:17 |
| openstackgerrit | Lucas Alvares Gomes proposed a change to openstack/ironic: Remove 'node' parameter from the Console and Rescue interfaces https://review.openstack.org/91097 | 14:17 |
| openstackgerrit | Lucas Alvares Gomes proposed a change to openstack/ironic: Remove 'node' parameter from the validate() methods https://review.openstack.org/91104 | 14:17 |
| lucasagomes | mornin NobodyCam | 14:17 |
| lucasagomes | rloo, ^ rebased | 14:17 |
| rloo | lucasagomes: thx! | 14:17 |
| jroll | devananda: sure, but I don't think there's any problem with landing the unique constraint patch now | 14:17 |
| jroll | mornin' NobodyCam :) | 14:18 |
| * jroll passes NobodyCam a cup of coffee | 14:18 | |
| devananda | jroll: i agree in as much as i dont think it would make the situation worse :) | 14:18 |
| jroll | devananda: sure. so let's fix up the comment you made and land it :) | 14:19 |
| lucasagomes | jroll, you want me to remove the dependency on the first patch? | 14:19 |
| jroll | lucasagomes: yes | 14:20 |
| linggao | devananda lucasagomes and all could you guys look at the commnet I put in this bug see if you agree with the proposal? https://bugs.launchpad.net/ironic/+bug/1301279 | 14:20 |
| jroll | lucasagomes: and deva had a comment there | 14:20 |
| *** dhellmann has joined #openstack-ironic | 14:20 | |
| NobodyCam | morning lucasagomes rloo jroll and linggao | 14:20 |
| lucasagomes | jroll, sure lemme do it | 14:20 |
| lucasagomes | and look at the comment as well | 14:20 |
| linggao | morning NobodyCam | 14:21 |
| jroll | thanks :) the comment is a quick thing | 14:21 |
| rloo | morning NobodyCam! | 14:21 |
| jroll | although... | 14:21 |
| jroll | I guess you'd have to add to the migration as well | 14:21 |
| *** agordeev has quit IRC | 14:21 | |
| lucasagomes | yeah will drop the index on the migration | 14:22 |
| openstackgerrit | Chris Krelle proposed a change to openstack/ironic: Rework make_partitions logic when preserve_ephemeral is set https://review.openstack.org/97590 | 14:23 |
| * jroll brb | 14:23 | |
| openstackgerrit | Chris Krelle proposed a change to openstack/ironic: Wipe any metadata from a nodes disk https://review.openstack.org/93133 | 14:26 |
| *** agordeev has joined #openstack-ironic | 14:27 | |
| *** jistr has quit IRC | 14:29 | |
| openstackgerrit | Jarrod Johnson proposed a change to stackforge/pyghmi: Don't defer custom keepalive expiry on all payloads https://review.openstack.org/99067 | 14:30 |
| openstackgerrit | A change was merged to openstack/ironic: Return the HTTP Location for newly created resources https://review.openstack.org/92615 | 14:31 |
| *** jistr has joined #openstack-ironic | 14:31 | |
| *** jdob_ has joined #openstack-ironic | 14:31 | |
| *** jdob has quit IRC | 14:31 | |
| devananda | linggao: lucasagomes: repleid on https://bugs.launchpad.net/ironic/+bug/1301279 | 14:33 |
| *** Mikhail_D_ltp has quit IRC | 14:33 | |
| lucasagomes | will take a look | 14:33 |
| openstackgerrit | OpenStack Proposal Bot proposed a change to openstack/ironic: Updated from global requirements https://review.openstack.org/96228 | 14:34 |
| openstackgerrit | OpenStack Proposal Bot proposed a change to openstack/ironic-python-agent: Updated from global requirements https://review.openstack.org/88722 | 14:34 |
| linggao | devananda, lucasagomes: our current Ironic scheduler filters are using "exact match" algorithm. So I do not think nova shceduler will schedule a second instance on the same bm node. So that should not be a problem | 14:36 |
| openstackgerrit | Ruby Loo proposed a change to openstack/ironic: Set instance default_ephemeral_device https://review.openstack.org/96498 | 14:37 |
| openstackgerrit | Lucas Alvares Gomes proposed a change to openstack/ironic: Add unique constraint to instance_uuid https://review.openstack.org/98120 | 14:37 |
| lucasagomes | jroll, ^ removed the dependecy and dropped the index | 14:37 |
| *** pcrews_ has joined #openstack-ironic | 14:38 | |
| linggao | devananda, lucasagomes : so even nova sees that node has available resources, but it cannot schedule a second instance. maybe this one is not a bug? | 14:38 |
| lucasagomes | linggao, will read the bug now, 1 sec | 14:38 |
| jroll | thanks lucasagomes | 14:39 |
| openstackgerrit | OpenStack Proposal Bot proposed a change to openstack/python-ironicclient: Updated from global requirements https://review.openstack.org/96263 | 14:39 |
| openstackgerrit | Josh Gachnang proposed a change to openstack/ironic: Allow noauth for Neutron https://review.openstack.org/98922 | 14:39 |
| jroll | devananda: why don't we just have a filter that looks at instance_uuid; if it is not None, the node is not available | 14:39 |
| jroll | (wrt that bug) | 14:40 |
| *** jgrimm has quit IRC | 14:40 | |
| * jroll makes a note in the bug | 14:40 | |
| lucasagomes | devananda, linggao so the bug was about nova reporting the available resource... I didn't test, but I also don't think nova will try to deploy a new instance onto that node | 14:40 |
| lucasagomes | even because that node is already associated with an instance | 14:40 |
| *** jdob_ is now known as jdob | 14:40 | |
| JoshNang | i think that's right. i had that exception come up a few times in testing when tear down failed and I didn't fully reset the node in the db. | 14:42 |
| devananda | lucasagomes: so i think the scheduler might try to select it, if avail resources == some flavor. however nova.virt.ironic driver will refuse to associate a second instance to it | 14:43 |
| linggao | devananda, true. | 14:43 |
| lucasagomes | hmm right | 14:43 |
| devananda | lucasagomes: so if the scheduler didn't pick a node that has non-null instance_uuid, then i dont see what problem that bug would actually pose | 14:44 |
| devananda | simply reporting available resources doesn't == a bug IMHO | 14:44 |
| lucasagomes | perhaps should the get_available_nodes() method in the driver, filter out the nodes which are already associated? | 14:44 |
| lucasagomes | and prevent that from happening? | 14:44 |
| jroll | devananda: but the scheduler *can* pick a node today, AIUI | 14:44 |
| jroll | lucasagomes: +1 | 14:44 |
| devananda | lucasagomes: no... we discussed that a while back. we shouldn't hide them. we should show them as totally consumed | 14:44 |
| lucasagomes | right | 14:45 |
| openstackgerrit | Jarrod Johnson proposed a change to stackforge/pyghmi: Don't defer custom keepalive expiry on all payloads https://review.openstack.org/99067 | 14:45 |
| devananda | if get_available_nodes() doesn't return a node that is associated, noav will delete the compute_host record for that node | 14:45 |
| lucasagomes | oh ew, yeah it rings me a bell | 14:45 |
| lucasagomes | devananda, so here's another idea | 14:46 |
| lucasagomes | devananda, there's another function called get_available_resource, which receives a node as parameter | 14:46 |
| lucasagomes | if that node is associated | 14:46 |
| lucasagomes | we might just want to return to nova saying "look this node has no more available resources" | 14:46 |
| jroll | why not use scheduler filters to accomplish this? | 14:46 |
| lucasagomes | or we do that already? lemme check the code | 14:47 |
| jroll | or is the goal to report no available resources on the node? | 14:47 |
| lucasagomes | jroll, I think we should report no more available resources on that node | 14:47 |
| jroll | ok | 14:47 |
| lucasagomes | because we can't slice the resources in the baremetal for multiple instances | 14:47 |
| lucasagomes | so as I see it, if it's already associated, all the resources are already taken | 14:48 |
| jroll | right | 14:48 |
| devananda | lucasagomes: https://github.com/openstack/ironic/blob/master/ironic/nova/virt/ironic/driver.py#L185 | 14:48 |
| devananda | we alrady do that :) | 14:49 |
| lucasagomes | devananda, yeah, just saw it | 14:49 |
| lucasagomes | :/ hmm | 14:49 |
| lucasagomes | why does nova then detect the changes in the properties and audit it | 14:49 |
| lucasagomes | :/ | 14:49 |
| devananda | lucasagomes: so taht bug was reported ~2mo ago. we should verify that it still exists | 14:49 |
| * lucasagomes is thinking loud | 14:49 | |
| lucasagomes | devananda, yeah, +1! | 14:49 |
| lucasagomes | linggao, did you verified that bug recently? it's still happen? | 14:50 |
| lucasagomes | I can run a quick test here | 14:50 |
| linggao | lucasagomes, no. I have not. I just try to see how to fix it. | 14:51 |
| lucasagomes | ic, will run a quick test here | 14:51 |
| *** dwalleck has joined #openstack-ironic | 14:54 | |
| devananda | ifarkas: hi! I see you started working on https://bugs.launchpad.net/ironic/+bug/1314961 | 14:56 |
| openstackgerrit | Yuriy Zveryanskyy proposed a change to openstack/ironic: Use temporary dir for image conversion https://review.openstack.org/98338 | 14:56 |
| devananda | ifarkas: have you seen the spec https://review.openstack.org/94923 and etherpad https://etherpad.openstack.org/p/ironic-and-fragile-hardware ? | 14:56 |
| devananda | ifarkas: I think these are very nearly overlapping with that bug -- the discussion became much longer than my initial report though | 14:57 |
| jroll | NobodyCam: how far is this from being mergeable? https://review.openstack.org/#/c/85529/ | 14:59 |
| devananda | can we get a few more eyes on the disk partitioner fixes? https://review.openstack.org/#/c/97590/ and https://review.openstack.org/#/c/93133/ | 15:00 |
| devananda | first one LGTM, i'm reviewing second now | 15:00 |
| devananda | (fix for critical bug...) | 15:00 |
| rloo | am looking at 97590 now ;) | 15:01 |
| jroll | yep, looking at second now | 15:01 |
| *** jcoufal has joined #openstack-ironic | 15:01 | |
| NobodyCam | devananda: as your doing review-y type things have a free second to look at http://paste.openstack.org/show/njC3I1WB1g9EBEiOX4EW/ and see if thats what you where thinking when we talked lastnight? | 15:02 |
| *** jcoufal has quit IRC | 15:02 | |
| lucasagomes | will do, looking at the first one | 15:03 |
| *** jcoufal has joined #openstack-ironic | 15:03 | |
| rloo | I approved 97590 | 15:03 |
| rloo | ^^ lucasagomes | 15:04 |
| devananda | NobodyCam: not what i meant | 15:04 |
| NobodyCam | jroll: humm, let me refresh myself on that one | 15:04 |
| lucasagomes | rloo, ah just sent my review | 15:04 |
| lucasagomes | rloo, but cool thanks | 15:05 |
| lucasagomes | devananda, linggao the bug still there | 15:05 |
| rloo | lucasagomes: as long as you didn't -1? | 15:05 |
| lucasagomes | 2014-06-10 16:05:15.446 AUDIT nova.compute.resource_tracker [req-71b190c5-81a9-4ee2-b631-42fe3fe226de None None] Free ram (MB): 512 | 15:05 |
| lucasagomes | it had 512 of ram before, after deploy I upgraded to 1024 | 15:05 |
| lucasagomes | now it's reporting 512 free even tho the node is deployed | 15:05 |
| lucasagomes | rloo, hah no I +2'd :) | 15:05 |
| devananda | lucasagomes: wait a few more minutes, don't delete the instance, see what happens | 15:06 |
| lucasagomes | devananda, ack | 15:06 |
| devananda | lucasagomes: i am wondering if it's a race between differen tperioeidic tasks ... | 15:06 |
| lucasagomes | devananda, I restarted the n-cpu after I updated the properties tho | 15:06 |
| lucasagomes | ok | 15:06 |
| linggao | lucasagomes, where did you see the extra available resoruces? in the log fle or using a nova command? | 15:06 |
| lucasagomes | linggao, n-cpu will log the available resources | 15:07 |
| NobodyCam | jroll: all the debug's from service.py would need to removed theres a odd error log in pxe I'm not sure why is there | 15:08 |
| openstackgerrit | Josh Gachnang proposed a change to openstack/ironic: Adding swift temp url support https://review.openstack.org/81391 | 15:08 |
| jroll | NobodyCam: right | 15:09 |
| openstackgerrit | Devananda van der Veen proposed a change to openstack/ironic: Stop ipmitool.validate from touching the BMC https://review.openstack.org/99121 | 15:09 |
| jroll | NobodyCam: do you want some +1's on the review before you do that, or? | 15:09 |
| devananda | NobodyCam: this: if "illegal option" in err.stderr | 15:11 |
| devananda | NobodyCam: isn't going to work | 15:11 |
| *** jistr has quit IRC | 15:12 | |
| *** jistr has joined #openstack-ironic | 15:12 | |
| NobodyCam | devananda: that block is missing the = FAlse \n else:\n TIMING_SUPPORT = True | 15:13 |
| devananda | NobodyCam: a) different binaries use different wording (I see "invalid option" locally) and b) different locales may use a different language | 15:13 |
| devananda | NobodyCam: string matching the error output of a command is not portable | 15:14 |
| NobodyCam | oh different language | 15:14 |
| devananda | and openipmi vs (the real) ipmitool may use different strings even in english | 15:14 |
| *** jcoufal has quit IRC | 15:15 | |
| openstackgerrit | A change was merged to openstack/ironic: Update Nova driver's list_instance_uuids() https://review.openstack.org/98268 | 15:16 |
| rloo | lucasagomes: should I rebase this then? https://review.openstack.org/#/c/93731/ | 15:16 |
| lucasagomes | rloo, +1, also idk if that dependency makes sense | 15:16 |
| lucasagomes | it seems unrelated | 15:16 |
| devananda | \o/ | 15:17 |
| * devananda removes another review from the page | 15:17 | |
| lucasagomes | :D | 15:17 |
| rloo | lucasagomes: ok, let me see about the dependency. although it probably doesn't matter, we want to land them both :-) | 15:17 |
| * lucasagomes starts rebasing the mgmt interface and looking at the comments | 15:17 | |
| lucasagomes | rloo, yeah that's correct :) | 15:17 |
| NobodyCam | 97590 just passed test :-p | 15:17 |
| NobodyCam | 93313 will need a recheck :-p | 15:18 |
| rloo | NobodyCam: this could be your big day! | 15:18 |
| NobodyCam | :) | 15:18 |
| devananda | lucasagomes: http://logs.openstack.org/04/98504/1/gate/gate-tempest-dsvm-ironic/d2d912f/console.html#_2014-06-10_14_25_24_620 | 15:19 |
| rloo | NobodyCam: wrt a recheck for 93313. What failed? Or do we just do a recheck no bug? | 15:19 |
| devananda | lucasagomes: looks like bug https://bugs.launchpad.net/ironic/+bug/1321494 is nt fixed | 15:19 |
| *** Mikhail_D_ltp has joined #openstack-ironic | 15:19 | |
| lucasagomes | :( | 15:20 |
| NobodyCam | recheck bug 1326754 | 15:20 |
| lucasagomes | devananda, indeed, so the return code doesn't actually help us here | 15:20 |
| rloo | NobodyCam: ok, I did a recheck. but you could have done it too, right? | 15:20 |
| devananda | it solved part of the issue | 15:20 |
| *** martyntaylor1 has quit IRC | 15:20 | |
| devananda | lucasagomes: we're seeing it /less/ -- but also in a different place | 15:21 |
| openstackgerrit | Jay Faulkner proposed a change to openstack/ironic-python-agent: Perform manual error checks instead of set -e https://review.openstack.org/97860 | 15:21 |
| devananda | lucasagomes: in the virtual-ironic job, it was failing (and is no longer) when NodeLocked showed up in the api log | 15:21 |
| devananda | lucasagomes: now we're actually returning that error to the tempest client, which then fails a test | 15:21 |
| lucasagomes | devananda, right, it might be because the virtual-ironic job actually retry the request | 15:22 |
| lucasagomes | cause the driver will retry it if it returnes conflict | 15:22 |
| devananda | lucasagomes: right | 15:22 |
| lucasagomes | where in tempest it doesn't have any retry mechanism | 15:22 |
| devananda | whereas our temepst client is not, i think | 15:22 |
| lucasagomes | that's a problem | 15:22 |
| lucasagomes | yeah | 15:22 |
| devananda | yea | 15:22 |
| NobodyCam | rloo: I've always waited until the current run reports back as failed before running the recheck | 15:22 |
| openstackgerrit | Jay Faulkner proposed a change to openstack/ironic-python-agent: Perform manual error checks instead of set -e https://review.openstack.org/97860 | 15:22 |
| rloo | NobodyCam: so why did you think it needed a recheck? | 15:23 |
| *** jcoufal has joined #openstack-ironic | 15:24 | |
| devananda | lucasagomes: did you find out when dtantsur|pto would be back? | 15:25 |
| NobodyCam | check-tempest-dsvm-virtual-ironic failed with the bug i pasted above | 15:25 |
| lucasagomes | devananda, oh, true lemme ask his manager | 15:25 |
| devananda | adam_g: you reported https://bugs.launchpad.net/ironic/+bug/1327392 recently -- were you able to work around it by deleting that instance and retrying? | 15:26 |
| rloo | NobodyCam: ahh. ok. the random failure :-( | 15:27 |
| NobodyCam | :( | 15:27 |
| openstackgerrit | Ruby Loo proposed a change to openstack/ironic: Workaround for conflicting volume labels https://review.openstack.org/93133 | 15:27 |
| openstackgerrit | Ruby Loo proposed a change to openstack/ironic: Allow more time for API requests to be completed https://review.openstack.org/93731 | 15:27 |
| *** jgrimm has joined #openstack-ironic | 15:28 | |
| NobodyCam | new version? | 15:28 |
| jroll | NobodyCam: this lgtm, can you rebase from master and push a new set up? https://review.openstack.org/#/c/85529/ | 15:29 |
| *** viktors is now known as viktors|afk | 15:30 | |
| jroll | rloo: is https://review.openstack.org/93133 just a rebase? | 15:30 |
| NobodyCam | jroll: one sec | 15:30 |
| rloo | jroll, NobodyCam: I rebased 93731. Which was dependent on 93133. Not sure what happened. | 15:30 |
| jroll | rloo: wait, that looks completely different :/ | 15:31 |
| lucasagomes | devananda, sent him a message on IRC, will let u know as soon as I get the answer | 15:32 |
| rloo | jroll: it looks like it reverted to some earlier patch. Yikes. | 15:32 |
| NobodyCam | oh wow | 15:33 |
| jroll | rloo: yeah :/ | 15:33 |
| devananda | lucasagomes: ack. I'll wait a bit before unassigning that bug then | 15:33 |
| rloo | how the #%$#% did that happen. | 15:33 |
| NobodyCam | rloo: want me to repush it up? | 15:33 |
| rloo | NobodyCam: yes please. | 15:33 |
| rloo | I wonder what that did to the rebase of 93731. | 15:34 |
| openstackgerrit | Chris Krelle proposed a change to openstack/ironic: Wipe any metadata from a nodes disk https://review.openstack.org/93133 | 15:34 |
| devananda | NobodyCam: looks like https://review.openstack.org/#/c/98510/ needs a rebase too ? | 15:34 |
| lucasagomes | devananda, he should be back on Thursday of this week | 15:36 |
| lucasagomes | devananda, sorry, I was unsure whether he said he would be on PTO for 1 or 2 weeks | 15:36 |
| ifarkas | devananda, hi, I have read an earlier version of the spec but I haven't seen the etherpad | 15:36 |
| ifarkas | devananda, I will read the discussion. Does the bug description need to be changed significantly? | 15:37 |
| openstackgerrit | Chris Krelle proposed a change to openstack/ironic: Add retry logic to iscsiadm commands https://review.openstack.org/98510 | 15:37 |
| devananda | lucasagomes: np. will leave bug assigned to him, then | 15:39 |
| lucasagomes | ack | 15:39 |
| *** rameshg87 has joined #openstack-ironic | 15:39 | |
| openstackgerrit | Ruby Loo proposed a change to openstack/ironic: Allow more time for API requests to be completed https://review.openstack.org/93731 | 15:41 |
| devananda | ifarkas: the bug describes an early (and scope limited) version of the spec proposal | 15:41 |
| devananda | ifarkas: the etherpad is approaching a more complete proposal | 15:41 |
| *** rameshg87 has quit IRC | 15:42 | |
| devananda | ifarkas: I haven't reviewed your patch yet, but we should probably focus on making the whole API async, not just driver validation | 15:42 |
| devananda | ifarkas: now that i see this is an area you have been working on, i wanted bring you into the discussion // see if you will continue working on it as the scope has changed | 15:44 |
| rloo | lucasagomes: do you want to take a look at https://review.openstack.org/#/c/93731/, get rid of your -1? | 15:45 |
| lucasagomes | rloo, will do | 15:45 |
| lucasagomes | rloo, ah that's fine for me, the -1 is from the previous patch... will +2 and approve it | 15:45 |
| lucasagomes | 1 line change yay quick review :D | 15:45 |
| jroll | are we using https://bugs.launchpad.net/devstack/+bug/1327516 or https://bugs.launchpad.net/devstack/+bug/1326209 for the 'apache2ctl failed' thing | 15:45 |
| jroll | ? | 15:46 |
| jroll | or just choose at random? :P | 15:46 |
| rloo | thx lucasagomes! | 15:46 |
| ifarkas | devananda, yeah, thanks for that. making the whole api async makes a lot of sense. I thought from the beginning that my patch will be a part of bigger solution, as an earlier version of your spec also describe making other methods async | 15:46 |
| ifarkas | devananda, I definitely want to continue work on it, and probably can help with other methods as well | 15:47 |
| devananda | jroll: looking | 15:47 |
| devananda | jroll: 1327516 -- address already in use??? | 15:47 |
| devananda | jroll: that seems totally unrelated | 15:47 |
| jroll | oh wow | 15:48 |
| * jroll runs off to fill his coffee cup | 15:48 | |
| jroll | I see apache2ctl issues there, and I recall someone mentioning it's actually a keystone issue | 15:48 |
| jroll | but yes, agree | 15:48 |
| devananda | jroll: 1326754 is the bug i've been using | 15:49 |
| devananda | jroll: but pls check the logs for that error beforehand | 15:49 |
| openstackgerrit | A change was merged to stackforge/pyghmi: Don't defer custom keepalive expiry on all payloads https://review.openstack.org/99067 | 15:49 |
| devananda | jroll: it's possible different issues are manifestign with the apache2ctl message | 15:49 |
| jroll | devananda: for 2014-06-05 22:20:16.218 | sudo: /usr/sbin/apache2ctl: command not found | 15:49 |
| jroll | ? | 15:49 |
| devananda | jroll: link to the log file pls | 15:49 |
| jroll | http://logs.openstack.org/22/96922/2/check/check-tempest-dsvm-full/6c88c67/console.html.gz | 15:49 |
| devananda | jroll: taht error is a symptom, not a cause | 15:50 |
| jroll | ah | 15:50 |
| *** dwalleck has quit IRC | 15:50 | |
| devananda | jroll: http://logs.openstack.org/22/96922/2/check/check-tempest-dsvm-full/6c88c67/logs/screen-key.txt.gz#_2014-06-05_22_24_12_113 | 15:52 |
| openstackgerrit | Chris Krelle proposed a change to openstack/ironic: Fix for tripleO undercloud gate tests https://review.openstack.org/85529 | 15:52 |
| devananda | issue is "address already in use" | 15:52 |
| devananda | jroll: i haven't seen this before | 15:53 |
| openstackgerrit | Russell Haering proposed a change to openstack/ironic-python-agent: Fix HardwareManager loading order bug https://review.openstack.org/98678 | 15:54 |
| NobodyCam | jroll: I've seen that the last two days | 15:54 |
| NobodyCam | ues recheck bug 1326754 | 15:54 |
| jroll | devananda: interesting, so it was that first bug :P | 15:55 |
| jroll | devananda: how did you find that? I don't see screen-key in the logs dir but maybe overlooking things | 15:55 |
| jroll | NobodyCam: I think it is https://bugs.launchpad.net/devstack/+bug/1327516 | 15:55 |
| derekh_ | NobodyCam: If you want to run the undercloud ci job on ^ you'll need to run check experimental | 15:56 |
| devananda | jroll: it's there | 15:56 |
| jroll | devananda: I'm looking at something completely different (somehow), sorry | 15:58 |
| devananda | jroll: http://logs.openstack.org/22/96922/2/check/check-tempest-dsvm-full/6c88c67/logs/ | 15:58 |
| jroll | devananda: right, I know. I got to the log dir from a different patchset :/ | 15:59 |
| devananda | jroll: when was that failure? | 15:59 |
| devananda | jroll: oh ... | 15:59 |
| jroll | this is a heat patch | 15:59 |
| devananda | right | 15:59 |
| jroll | wtf am I doing | 15:59 |
| devananda | that's why i'm confused | 15:59 |
| jroll | I may have clicked through to the logs from the bug rather than the patch I was looking at | 16:00 |
| jroll | rather, given you the wrong link that way | 16:00 |
| devananda | ack | 16:00 |
| devananda | right link? | 16:00 |
| jroll | http://logs.openstack.org/20/98120/3/check/check-tempest-dsvm-virtual-ironic/c60a674/ | 16:00 |
| devananda | jroll: bingo -- exactly the bug i thought it ws -- http://logs.openstack.org/20/98120/3/check/check-tempest-dsvm-virtual-ironic/c60a674/console.html#_2014-06-10_15_06_23_782 | 16:00 |
| devananda | 1326754 | 16:01 |
| *** coolsvap|afk is now known as coolsvap | 16:01 | |
| jroll | right | 16:01 |
| jroll | thanks | 16:01 |
| *** hemna has joined #openstack-ironic | 16:04 | |
| devananda | NobodyCam: https://review.openstack.org/#/c/93133/64/ironic/drivers/modules/deploy_utils.py is still logging the error multiple times | 16:05 |
| openstackgerrit | Lucas Alvares Gomes proposed a change to openstack/ironic: SSH virsh to use the new ManagementInterface https://review.openstack.org/89884 | 16:06 |
| openstackgerrit | Lucas Alvares Gomes proposed a change to openstack/ironic: IPMITool to use the new ManagementInterface https://review.openstack.org/86092 | 16:06 |
| openstackgerrit | Lucas Alvares Gomes proposed a change to openstack/ironic: SeaMicro to use the new ManagementInterface https://review.openstack.org/86328 | 16:06 |
| openstackgerrit | Lucas Alvares Gomes proposed a change to openstack/ironic: IPMINative to use the new ManagementInterface https://review.openstack.org/86588 | 16:06 |
| lucasagomes | rebased/fixed | 16:06 |
| *** jgrimm has quit IRC | 16:07 | |
| openstackgerrit | Ellen Hui proposed a change to openstack/ironic-python-agent: Tries to advertise valid default IP https://review.openstack.org/96980 | 16:07 |
| devananda | NobodyCam: it looks like it will log three times: in deploy_utils destroy_disk_metadata, then in deploy_utils deploy, then in pxe.py _continue_deploy | 16:07 |
| NobodyCam | devananda: I have the logging in destroy_disk_metadata so we get the node id | 16:12 |
| * devananda looks at the logging in pxe.py and feels sadness | 16:13 | |
| devananda | NobodyCam: my point is not that you're logging the wrong thing, but that we're going to log the same error three times | 16:13 |
| devananda | anyone want to take an action to do a massive cleanup of logging in the pxe driver and associated deploy utils? | 16:14 |
| jroll | rloo: since Shrews isn't around today, could you try to answer my question here? https://review.openstack.org/#/c/96498/11/ironic/nova/tests/virt/ironic/test_driver.py | 16:15 |
| NobodyCam | ack.. I can change the logging | 16:15 |
| jroll | rloo: (it's ok if you can't answer that) | 16:15 |
| lucasagomes | comstud, ping | 16:16 |
| lucasagomes | comstud, 91419 lgtm, but needs rebase :( | 16:17 |
| devananda | NobodyCam: commented on https://review.openstack.org/#/c/85529/ | 16:18 |
| NobodyCam | devananda: I will remove the logging from destroy_disk_metadata, but kinda feel the pxe / deploy_utils logging should be another patch. do you agree? | 16:18 |
| devananda | NobodyCam: pls hold off on changign the logging in the deploy patch | 16:18 |
| devananda | NobodyCam: exactly | 16:18 |
| NobodyCam | just removing the loggin in destroy_disk_metadata | 16:18 |
| devananda | NobodyCam: I think the patch is fien to land as is (with redundant logging) because we need to clean that up *anyway* | 16:19 |
| NobodyCam | ok ... I will hold off | 16:19 |
| devananda | NobodyCam: on the metadata wipe patch, how / where has that been tested? | 16:23 |
| *** ellenh has joined #openstack-ironic | 16:23 | |
| *** dividehex has joined #openstack-ironic | 16:23 | |
| NobodyCam | devananda: here with tripleo devtest | 16:24 |
| lucasagomes | comstud, will rebase that since we want it to land (hope u don't mind) | 16:27 |
| openstackgerrit | Lucas Alvares Gomes proposed a change to openstack/ironic: TaskManager: Only support single node locking https://review.openstack.org/91419 | 16:27 |
| devananda | NobodyCam: should we invoke "check experimental" to get a tripleo undercloud run on it? is there a chance that'll pass? | 16:27 |
| openstackgerrit | Chris Krelle proposed a change to openstack/ironic: Default newly enrolled nodes to power_off https://review.openstack.org/85529 | 16:28 |
| NobodyCam | we could but I dont think it will pass | 16:29 |
| *** jgrimm has joined #openstack-ironic | 16:29 | |
| devananda | NobodyCam: ack | 16:30 |
| devananda | NobodyCam: any reason not to land https://review.openstack.org/#/c/85529/ now (assuming it passes all tests)? | 16:30 |
| *** matty_dubs is now known as matty_dubs|afk | 16:31 | |
| NobodyCam | opt atm | 16:32 |
| devananda | Shrews: ping | 16:32 |
| jroll | devananda: Shrews said he's out today :) | 16:33 |
| devananda | jroll: ah, thx | 16:33 |
| jroll | devananda: also, typos in commit messages worth a -1 or no? (can't remember) | 16:33 |
| devananda | jroll: i'm curious about your comment on 96498 too | 16:33 |
| devananda | jroll: depends on how bad they are | 16:33 |
| jroll | not bad :) | 16:33 |
| * jroll looks | 16:33 | |
| devananda | jroll: you can also edit commit messages in gerrit directly now | 16:33 |
| devananda | jroll: helpful for fixing those little typos ;) | 16:33 |
| derekh_ | btw, on that patch, looks like nodes could be in a power on state, looks like we thought ironic powered off during register http://git.openstack.org/cgit/openstack/tripleo-incubator/tree/scripts/boot-seed-vm#n150 | 16:34 |
| JayF | Does it generate a new patchset? | 16:34 |
| jroll | devananda: oh, I'll just do that :) | 16:34 |
| jroll | JayF: yes | 16:34 |
| jroll | devananda: ah, you can't edit someone else's message | 16:34 |
| devananda | jroll: oh | 16:34 |
| jroll | devananda: anyway, on 96498, it's just that there's extra code that's not very related to the patch. like, the code to trigger validation failure and test that the failure happened could be removed IMO | 16:35 |
| devananda | derekh_: right. the point is, they are being enrolled with no known state, so ironic should force them off initially | 16:36 |
| devananda | derekh_: that is what that patch does, AFAICT | 16:36 |
| devananda | jroll: you think that whole test case is unrelated to the patch and should be split out? | 16:36 |
| derekh_ | devananda: yup, was just confirming that it was the case that nodes could be running | 16:36 |
| devananda | derekh_: ack | 16:36 |
| jroll | devananda: not the entire test case, I just think it can be reduced down to a few lines | 16:36 |
| derekh_ | devananda: for nova-bm we explicitly shut them down | 16:37 |
| jroll | devananda: the bit that runs spawn() and `self.assertEqual('/dev/sda1', instance['default_ephemeral_device'])` should definitely be there | 16:37 |
| devananda | derekh_: right. but you're not doing that for ironic because we planned to fix it -- though, really, you SHOULD be doing that for ironic UNTIL we fix it | 16:37 |
| jroll | devananda: but the other asserts, the triggering validation failure, etc, are not necessary IMO | 16:37 |
| devananda | derekh_: not breaking support for ironic while the patch has been in flight.... | 16:38 |
| devananda | derekh_: i may be kicking a dead horse, but that seems like it was unhelpful, when the code is right there to prevent this issue in tripleo :) | 16:38 |
| derekh_ | devananda: I wasn't under the impression there was anything to fix, I'm pretty sure information at the time was that ironic shuts down nodes when they are registered | 16:38 |
| *** igordcard has quit IRC | 16:39 | |
| derekh_ | *I was under... | 16:39 |
| devananda | derekh_: nope. we never did. thus the bug, and the following fix that we're trying to land now | 16:39 |
| devananda | derekh_: perhaps just a miscommunication. anyway. i hope that fix lands today | 16:39 |
| * devananda looks at the gate backlog again | 16:40 | |
| jroll | devananda: all that said, I'm fine with landing it, since it clearly works. I just think there's extra code there that could be removed, and was curious to learn if it is indeed necessary or doing something I don't realize. :) | 16:40 |
| devananda | under 3 hours! crazy! | 16:40 |
| lucasagomes | someone? https://review.openstack.org/#/c/91097/ two +2 already | 16:40 |
| derekh_ | devananda: cool, yup looks like a miscommunication, iirc somebody said it would be ok in ironic and we wen't doing a undercloud job at the time | 16:40 |
| devananda | lucasagomes: /me looks | 16:41 |
| devananda | lucasagomes: are you tracking that patch series on the etherpad? | 16:41 |
| lucasagomes | devananda, adding now, only 2 more patches to go | 16:41 |
| devananda | lucasagomes: humm, https://review.openstack.org/#/c/91104/ failed jenkins but not immediately obvious to me why | 16:44 |
| rloo | jroll, wrt the test in 96498. I don't know. I rebased it but I didn't review it because it had already been +2 and approved. | 16:44 |
| lucasagomes | devananda, 2014-06-10 14:37:19.893 | ERROR: the main setup script run by this job failed - exit code: 1 | 16:44 |
| *** vinbs_ has joined #openstack-ironic | 16:44 | |
| jroll | rloo: ok, cool. like I told devananda, I'm fine with it landing as-is, was just curious if the extra code was needed | 16:45 |
| lucasagomes | tho this is also confusing: 2014-06-10 14:30:25.401 | sudo: /usr/sbin/apache2ctl: command not found | 16:45 |
| *** vinbs_ is now known as vinbs | 16:46 | |
| devananda | lucasagomes: yea. that's a symptom, not a cause, but good enough for now | 16:46 |
| devananda | lucasagomes: it looks like devstack got most services started, then something failed but didn't log in devstacklog.txt | 16:46 |
| devananda | lucasagomes: worth earmarking that failure and looking into more later, perhaps | 16:47 |
| lucasagomes | lemme see if I can dig more into that logs | 16:47 |
| devananda | lucasagomes: is https://review.openstack.org/#/c/91419/ going to conflict with https://review.openstack.org/#/c/91097/15 ? | 16:48 |
| devananda | lucasagomes: yes, it is. ugh | 16:48 |
| lucasagomes | >.< | 16:48 |
| devananda | lucasagomes: which order do you want to merge these? 91419 is much much larger | 16:49 |
| lucasagomes | damn, I can make 91097 to depend on 91419 | 16:49 |
| devananda | lucasagomes: that'd be great | 16:49 |
| lucasagomes | devananda, I will add the dependency, I prefer to change my own patch | 16:49 |
| lucasagomes | cool 1 sec will do it | 16:49 |
| * NobodyCam is back | 16:49 | |
| * devananda re-reviews 91419 | 16:50 | |
| NobodyCam | sorry about conf call | 16:50 |
| lucasagomes | tho 91097 is already approved | 16:50 |
| lucasagomes | devananda, if I submit another patch I cancel the merge? | 16:50 |
| devananda | lucasagomes: yep | 16:52 |
| openstackgerrit | Josh Gachnang proposed a change to openstack/ironic: Adding swift temp url support https://review.openstack.org/81391 | 16:52 |
| lucasagomes | cooleo | 16:52 |
| *** jcoufal has quit IRC | 16:55 | |
| *** dwalleck has joined #openstack-ironic | 17:01 | |
| *** dwalleck has quit IRC | 17:02 | |
| openstackgerrit | Lucas Alvares Gomes proposed a change to openstack/ironic: Remove 'node' parameter from the Console and Rescue interfaces https://review.openstack.org/91097 | 17:03 |
| openstackgerrit | Lucas Alvares Gomes proposed a change to openstack/ironic: Remove 'node' parameter from the validate() methods https://review.openstack.org/91104 | 17:03 |
| lucasagomes | rebased on top of 91419 | 17:03 |
| *** dwalleck has joined #openstack-ironic | 17:05 | |
| devananda | thanks! | 17:07 |
| *** dwalleck has quit IRC | 17:08 | |
| *** harlowja_away is now known as harlowja | 17:08 | |
| rloo | hey, what's the difference between recheck and reverify? | 17:13 |
| devananda | rloo: not much | 17:14 |
| rloo | devananda: so it doesn't matter which one we use? | 17:14 |
| *** derekh_ has quit IRC | 17:15 | |
| lucasagomes | I use reverify after the patch is approved and failed on gate | 17:15 |
| devananda | rloo: last time i checked with clarkb, my understanding was they are basicaly no different and we can just use recheck | 17:15 |
| devananda | reverify was originally meant for ^^ what lucas said | 17:16 |
| rloo | thx devananda, lucasagomes. Simple is better, so I'll just use recheck from now on ;) | 17:16 |
| lucasagomes | heh me too :D | 17:16 |
| lucasagomes | if that works for both cases that would be better indeed | 17:16 |
| lucasagomes | devananda, thanks for the info | 17:16 |
| devananda | fwiw, they used to be different, then infra changed something | 17:17 |
| * rloo thinks that once we all start using nly recheck, someone will modify reverify! | 17:17 | |
| *** sysexit has quit IRC | 17:17 | |
| NobodyCam | woo hoo Mr. Jenkins +1'd 93313 | 17:18 |
| *** jgrimm has quit IRC | 17:19 | |
| rloo | NobodyCam: but you still need it to be approved AND for Jenkins to +1 again! | 17:19 |
| NobodyCam | :-p | 17:20 |
| rloo | lucasagomes: for https://review.openstack.org/#/c/98120/, are those the migration tests that don't work? | 17:20 |
| lucasagomes | rloo, they are not tested in gate :( | 17:21 |
| *** Alexei_987 has quit IRC | 17:21 | |
| lucasagomes | u can enable mysql to test it locally tho | 17:21 |
| devananda | NobodyCam: i'd like to triage https://bugs.launchpad.net/ironic/+bug/1327392 -- can this issue be worked around by simply deleting the instance and re-provisioning? or does it wedge the node badly? or ...? | 17:22 |
| lucasagomes | rloo, yeah, they work but currently are not being tested | 17:22 |
| *** sysexit has joined #openstack-ironic | 17:24 | |
| lucasagomes | alright I'll take a break | 17:29 |
| *** dwalleck has joined #openstack-ironic | 17:29 | |
| lucasagomes | have a good night rloo devananda NobodyCam everyone | 17:30 |
| *** lucasagomes is now known as lucas-dinner | 17:30 | |
| devananda | lucas-dinner: g'night! thanks!! | 17:31 |
| devananda | I'm going to take a break in a bit too -- been at this >4hrs now | 17:31 |
| devananda | everyoen who's been around doing reviews -- jroll, rloo, GheRivero, NobodyCam -- thanks a lot! | 17:31 |
| rloo | lucas-dinner: ciao. thx, i'll test locally then ;) | 17:31 |
| devananda | I'm really happy with what we've accomplished | 17:31 |
| devananda | etherpad has a good list of what's in flight // soon to come | 17:32 |
| jroll | devananda: thank you for leading :) | 17:33 |
| * jroll goes afk to eat lunch, head to the office, etc | 17:34 | |
| NobodyCam | sorry another phone call ... looking | 17:35 |
| NobodyCam | devananda: I am not sure, I have not actually encountered that issue | 17:36 |
| NobodyCam | I got the report via hipchat and put up the retry patch, which was then tested with another full run | 17:37 |
| adam_g | devananda, sorry, re 1327392.. yes, deleting the instance and re-creating it seemed to work. the issue was random and seemed only to happen during a large number of concurrent deployments | 17:37 |
| NobodyCam | so I am not sure how baddly the node is stuck | 17:37 |
| adam_g | devananda, i think i have lost access to the hardware for the time being but im happy to play more with it once i get it back | 17:37 |
| devananda | adam_g: thanks. i'll set pri to medium then | 17:38 |
| NobodyCam | ty adam_g :) | 17:38 |
| openstackgerrit | Ellen Hui proposed a change to openstack/ironic: Add node id to DEBUG messages in impitool https://review.openstack.org/98928 | 17:38 |
| devananda | adam_g: we have a fix in flight, so it'd be great to get validation of that | 17:38 |
| NobodyCam | devananda: adam_g: I believe Nicholas tested with the patch I put up | 17:39 |
| adam_g | devananda, one issue with that work around is that the scheduler should automatically try to reschedule it after failure, but then we hit https://bugs.launchpad.net/ironic/+bug/1326364 which makes matters worse. | 17:39 |
| devananda | adam_g: right, also have fixes in flight for that, but may take a bit longer to land those | 17:40 |
| *** dwalleck has quit IRC | 17:40 | |
| *** jistr has quit IRC | 17:40 | |
| devananda | https://review.openstack.org/#/c/93133 has had a lot of folks weigh in over the course of its history -- I'd like to land it today, but also give a little more time for others to review | 17:41 |
| *** zdiN0bot has joined #openstack-ironic | 17:47 | |
| russell_h | agordeev: https://review.openstack.org/#/c/98678/ | 17:49 |
| *** vinbs has quit IRC | 17:50 | |
| *** zdiN0bot has left #openstack-ironic | 17:52 | |
| openstackgerrit | Josh Gachnang proposed a change to openstack/ironic: Factoring out PXE and TFTP functions https://review.openstack.org/90233 | 17:53 |
| devananda | rloo: this one https://review.openstack.org/#/c/97558/1 is goign to conflict with a big refactorign that i'd like to land https://review.openstack.org/#/c/91419/ | 17:53 |
| devananda | rloo: would you mind rebasing on top of that? | 17:53 |
| rloo | devananda: no worries, i can do that. but in meeting right now. well, i'll see if i can multi task ;) | 17:54 |
| devananda | rloo: thanks :) | 17:54 |
| *** Mikhail_D_ltp has quit IRC | 17:58 | |
| NobodyCam | devananda: is this more like what you were thinking? http://paste.openstack.org/show/u0pLjdtNbcemFI0RE3dr/ | 17:59 |
| devananda | NobodyCam: better | 17:59 |
| lifeless | morning | 18:01 |
| NobodyCam | good morning lifeless | 18:01 |
| *** rwsu has joined #openstack-ironic | 18:02 | |
| * NobodyCam reboots | 18:03 | |
| *** dwalleck has joined #openstack-ironic | 18:04 | |
| rloo | I seem to be having issues doing any git operations. Getting Permission denied (public key). Is it working for you? | 18:08 |
| devananda | lifeless: mornin! | 18:08 |
| * NobodyCam is back | 18:09 | |
| devananda | rloo: oh, hum, yep, me too | 18:09 |
| NobodyCam | I was getting 500 errors | 18:09 |
| rloo | devananda: thx for confirming. Was wondering if it was a ploy to make me focus on this meeting I'm supposed to be listening to ;) | 18:10 |
| NobodyCam | s/was/am/ | 18:10 |
| NobodyCam | fatal: unable to access 'https://github.com/openstack/diskimage-builder/': The requested URL returned error: 500 | 18:10 |
| devananda | fwiw, you should all use git.openstack.org :) | 18:10 |
| devananda | rloo, NobodyCam ^ | 18:10 |
| rloo | devananda: oh yeah, the backup plan ;) | 18:10 |
| NobodyCam | devananda: the command I am running is "devtest.sh --trash-my-machine" | 18:11 |
| devananda | git remote set-url origin git://git.openstack.org/openstack/ironic | 18:11 |
| devananda | ^^^ | 18:11 |
| devananda | NobodyCam: right, so tripleo should be using git.openstack.org instead of github :p | 18:11 |
| devananda | lifeless: ^ | 18:11 |
| lifeless | devananda: ironic git /opt/stack/ironic https://git.openstack.org/openstack/ironic | 18:12 |
| lifeless | is whats in tripleo-image-elements/elements/ironic/source-repository-ironic | 18:12 |
| lifeless | tripleo-incubator$ grep github.com scripts -r | 18:14 |
| lifeless | returns no hits | 18:14 |
| devananda | lifeless: ++ | 18:14 |
| devananda | NobodyCam: see ^ | 18:14 |
| lifeless | so, we are. | 18:14 |
| devananda | lifeless: i was responding to NobodyCam's claim otherwise. apologies for the misdirection | 18:14 |
| devananda | all - please hold off approving anything new at this point -- we've got enough fixes in flight that it will take the rest of the day to land them, probably | 18:15 |
| *** sysexit has quit IRC | 18:16 | |
| devananda | and infra just approved https://review.openstack.org/#/c/98886/1 so we should see a lot less gate failures in about an hour (once that propagates) | 18:16 |
| *** coolsvap is now known as coolsvap|afk | 18:17 | |
| *** devananda changes topic to "OpenStack Bare Metal Provisioning. NOTE: Please refrain from approving additional patch sets (rechecks are fine) until the current merge queue settles down. Thanks!~" | 18:18 | |
| *** max_lobur has quit IRC | 18:24 | |
| *** harlowja is now known as harlowja_away | 18:25 | |
| lifeless | adam_g: https://review.openstack.org/#/c/98886/1 - did you put a similar fix up for diskimage-builder ? | 18:27 |
| adam_g | lifeless, https://review.openstack.org/#/c/98896/ | 18:27 |
| adam_g | lifeless, .. is the equiv bug i saw affecting the tripleo gate yesterday | 18:28 |
| adam_g | der | 18:29 |
| openstackgerrit | Josh Gachnang proposed a change to openstack/ironic: Factoring out PXE and TFTP functions https://review.openstack.org/90233 | 18:32 |
| *** jgrimm has joined #openstack-ironic | 18:33 | |
| *** romcheg has joined #openstack-ironic | 18:34 | |
| openstackgerrit | Josh Gachnang proposed a change to openstack/ironic: Factoring out PXE and TFTP functions https://review.openstack.org/90233 | 18:36 |
| NobodyCam | brb | 18:42 |
| *** igordcard has joined #openstack-ironic | 18:43 | |
| *** dwalleck has quit IRC | 18:46 | |
| * NobodyCam is back | 18:46 | |
| *** pelix has quit IRC | 18:48 | |
| *** romcheg has quit IRC | 18:49 | |
| *** ellenh has quit IRC | 18:54 | |
| *** harlowja_away is now known as harlowja | 19:03 | |
| openstackgerrit | Josh Gachnang proposed a change to openstack/ironic: Factoring out PXE and TFTP functions https://review.openstack.org/90233 | 19:03 |
| *** martyntaylor has joined #openstack-ironic | 19:05 | |
| *** martyntaylor has left #openstack-ironic | 19:15 | |
| openstackgerrit | A change was merged to openstack/ironic: Rework make_partitions logic when preserve_ephemeral is set https://review.openstack.org/97590 | 19:19 |
| *** max_lobur has joined #openstack-ironic | 19:19 | |
| NobodyCam | woo hoo :-p | 19:20 |
| *** sysexit has joined #openstack-ironic | 19:20 | |
| NobodyCam | one down | 19:20 |
| openstackgerrit | Josh Gachnang proposed a change to openstack/ironic: Adding swift temp url support https://review.openstack.org/81391 | 19:22 |
| openstackgerrit | Josh Gachnang proposed a change to openstack/ironic: Drivers determine acceptable power states https://review.openstack.org/86744 | 19:25 |
| * lifeless checks the latest version did indeed have the needed fixes ;) | 19:26 | |
| lifeless | NobodyCam: it would be good at this point to add a rebuild --preserve-ephemeral test to tempest | 19:27 |
| NobodyCam | lifeless: yes I can see that | 19:27 |
| NobodyCam | just about to jump on a conf call | 19:28 |
| lifeless | 'enjoy' | 19:28 |
| lifeless | adam_g: thanks, got that merged | 19:28 |
| adam_g | lifeless, cool. seems to have helped the ironic gate so far http://no-carrier.net/~adam/openstack/ironic_gate_status.html | 19:29 |
| adam_g | strange the issue popped up in two places at once | 19:29 |
| devananda | lifeless: we have one proposed alraedy :) | 19:29 |
| lifeless | devananda: oh, but missing what - review love? | 19:30 |
| lifeless | adam_g: its been happening for weeks | 19:30 |
| lifeless | adam_g: just folk paying attention to the metrics | 19:30 |
| lifeless | adam_g: (see how long the bug was open on tripleo) | 19:30 |
| devananda | lifeless: https://review.openstack.org/#/c/94439/ | 19:31 |
| adam_g | lifeless, derek opened 1326754 on 2014-06-05, i opened 1325844 on 2014-06-03. are there others that should be closed? | 19:32 |
| openstackgerrit | Ruby Loo proposed a change to openstack/ironic: VendorPassthru.validate()s call _parse_driver_info https://review.openstack.org/97558 | 19:32 |
| lifeless | adam_g: huh | 19:33 |
| lifeless | adam_g: possibly we filed it late :/ | 19:33 |
| adam_g | oh. anyway, hope that helps | 19:33 |
| devananda | looks like tempest-dsvm-ironic-* failing on NodeLocked races is now the top bug hitting out check/gate queues | 19:43 |
| devananda | latest example: https://jenkins06.openstack.org/job/gate-tempest-dsvm-ironic-postgres/77/consoleText | 19:43 |
| devananda | lifeless: iirc, you'd suggested we put retry logic natively into the client at one point. | 19:43 |
| *** ellenh has joined #openstack-ironic | 19:43 | |
| devananda | lifeless: given the async work, i don't think that will be necessary long-term. | 19:43 |
| devananda | thoughts? | 19:44 |
| devananda | mitigate the gate issues by having tempest retry, basically copy the retry code from the nova driver as a temporary solution? | 19:44 |
| devananda | adam_g ^ ? | 19:44 |
| *** zdiN0bot has joined #openstack-ironic | 19:44 | |
| *** dwalleck has joined #openstack-ironic | 19:46 | |
| adam_g | devananda, seems like it would work. i dont know if there are similar things in tempest that do that, or how tempest cores would feel about it if not. | 19:47 |
| lifeless | adam_g: you very much helped. | 19:48 |
| lifeless | devananda: if we move away from '409 is something you should expect a lot of' | 19:48 |
| lifeless | devananda: then i'm cool with the client not directly aiding users with it | 19:48 |
| lifeless | devananda: w.r.t. tempest, I think you'll get pushback about the API quality its testing :) | 19:49 |
| lifeless | devananda: so my suggestion is we get the spec in shape and reference that as long term explanation | 19:49 |
| *** dwalleck has quit IRC | 19:49 | |
| *** ellenh has quit IRC | 19:55 | |
| *** ellenh has joined #openstack-ironic | 20:07 | |
| NobodyCam | brb | 20:09 |
| *** pcrews_ has quit IRC | 20:11 | |
| *** pcrews_ has joined #openstack-ironic | 20:13 | |
| openstackgerrit | A change was merged to openstack/ironic: Return the HTTP Location for accepted requestes https://review.openstack.org/92616 | 20:32 |
| openstackgerrit | A change was merged to openstack/ironic: Work around iscsiadm delete failures https://review.openstack.org/96577 | 20:32 |
| *** BadCub has quit IRC | 20:33 | |
| *** jrist has quit IRC | 20:33 | |
| *** dguerri has quit IRC | 20:33 | |
| *** beecee has quit IRC | 20:33 | |
| *** ryanpetrello has quit IRC | 20:33 | |
| *** beecee has joined #openstack-ironic | 20:33 | |
| *** dguerri has joined #openstack-ironic | 20:33 | |
| *** BadCub has joined #openstack-ironic | 20:33 | |
| *** jrist has joined #openstack-ironic | 20:33 | |
| *** ryanpetrello has joined #openstack-ironic | 20:33 | |
| *** dhellmann_ has joined #openstack-ironic | 20:41 | |
| NobodyCam | oh so nice to see patches landing | 20:43 |
| *** romcheg has joined #openstack-ironic | 20:46 | |
| *** dhellmann_ has quit IRC | 20:48 | |
| *** dhellmann_ has joined #openstack-ironic | 20:48 | |
| *** zdiN0bot has quit IRC | 20:49 | |
| *** romcheg has quit IRC | 20:49 | |
| *** lsmola__ has joined #openstack-ironic | 20:54 | |
| *** beecee has quit IRC | 20:54 | |
| *** hemna has quit IRC | 20:54 | |
| *** dhellmann has quit IRC | 20:54 | |
| *** athomas has quit IRC | 20:54 | |
| *** Mikhail_D_wk has quit IRC | 20:54 | |
| *** coolsvap|afk has quit IRC | 20:54 | |
| *** openstackgerrit has quit IRC | 20:54 | |
| *** lsmola_ has quit IRC | 20:54 | |
| *** dhellmann_ is now known as dhellmann | 20:54 | |
| *** hemna has joined #openstack-ironic | 20:55 | |
| *** athomas has joined #openstack-ironic | 20:56 | |
| *** dhellmann has quit IRC | 20:56 | |
| *** Mikhail_D_wk has joined #openstack-ironic | 20:57 | |
| *** dwalleck has joined #openstack-ironic | 21:03 | |
| *** foexle has quit IRC | 21:03 | |
| *** dwalleck has quit IRC | 21:11 | |
| *** beecee has joined #openstack-ironic | 21:11 | |
| *** coolsvap|afk has joined #openstack-ironic | 21:11 | |
| *** openstackgerrit has joined #openstack-ironic | 21:11 | |
| *** dwalleck has joined #openstack-ironic | 21:11 | |
| *** athomas has quit IRC | 21:19 | |
| *** linggao has quit IRC | 21:20 | |
| NobodyCam | devananda: with the info you dig up about ipmitool and number of times a command is tired(sent) do we still want to loop in the code for versions that do not support -N -R | 21:25 |
| devananda | NobodyCam: definitely not | 21:25 |
| NobodyCam | :) | 21:25 |
| jroll | eh? | 21:27 |
| jroll | that makes them... essentially not supported. | 21:27 |
| NobodyCam | well there are fixed unchangeable values built in | 21:27 |
| NobodyCam | oh I do have a question. | 21:28 |
| NobodyCam | https://review.openstack.org/#/c/96902/8/ironic/drivers/modules/ipmitool.py #L343 lucas asks a i18n question | 21:28 |
| NobodyCam | I believe the andswer is yes we should, but I am not 100% sure | 21:31 |
| devananda | jroll: ipmitool <= 1.8.11 doesn't allow the user to determine retry count or interval. it does it for you. and sends 18 packets in ~45 seconds. | 21:31 |
| NobodyCam | devananda: do you know the answer? to the i18n question? | 21:35 |
| NobodyCam | s/answer?/answer/ | 21:36 |
| jroll | devananda: welp. | 21:36 |
| *** jdob has quit IRC | 21:36 | |
| jroll | devananda: cool software. | 21:37 |
| devananda | jroll: right. also, that was addressed ~2yr ago upstream. i blame packagers for not updating | 21:37 |
| jroll | devananda: of course, which is why I said before that we shouldn't support it :) | 21:38 |
| jroll | "complain to your distro" | 21:38 |
| openstackgerrit | Adam Gandelman proposed a change to openstack/ironic: Wait for Neutron port updates when using SSHPower https://review.openstack.org/91719 | 21:38 |
| NobodyCam | adam_g: ohh sleep 15 :-p | 21:40 |
| devananda | jroll: right. which is why i'msuggesting we write the code to behave reasonably on old versions, and transparently be better on newer versions | 21:42 |
| devananda | rather than send 48 packets for a single "power on" command ... | 21:42 |
| lifeless | devananda: yes, but again, we've not seen this issue with ipmitool in nova-bm | 21:42 |
| lifeless | devananda: its not great, but existing BMCs all cope | 21:42 |
| mrda | Morning Ironic! | 21:42 |
| lifeless | devananda: btw I've got a contact in ilo (via wan-yen) and am escalating the root cause to her) | 21:42 |
| jroll | devananda: sure, totally fine with what y'all are doing :) | 21:43 |
| lifeless | devananda: +1 on transparently better on newer versions | 21:43 |
| lifeless | jroll: since devananda and I are here, did you want to talk about the power assertion default ? | 21:43 |
| devananda | lifeless: great. I'd really like to know /what/ is causing the error you're seeing in ironic then | 21:43 |
| devananda | lifeless: as i'm not clear on what behavioral difference there is w.r.t. sending ipmi commands for power on/off | 21:43 |
| NobodyCam | morning mrda | 21:43 |
| lifeless | devananda: so would I! | 21:44 |
| jroll | lifeless: tbh, I'm not very opinionated on it, it's just something that surprised me while setting up an environment and I thought it might surprise other deployers as well. | 21:44 |
| devananda | lifeless: can you packet sniff udp 623 when ironic knocks over a BMC? :) | 21:44 |
| lifeless | devananda: possibly; we'd need to sniff it for a day or so... | 21:45 |
| *** dhellmann has joined #openstack-ironic | 21:55 | |
| *** sysexit has quit IRC | 22:03 | |
| openstackgerrit | Josh Gachnang proposed a change to openstack/ironic-specs: Swift Temporary URLs Spec https://review.openstack.org/98904 | 22:03 |
| *** max_lobur has quit IRC | 22:15 | |
| *** klindgren_ has joined #openstack-ironic | 22:20 | |
| *** hemna has quit IRC | 22:31 | |
| *** hemna has joined #openstack-ironic | 22:38 | |
| adam_g | devananda, anything come up at summit about potential integration points where this would be useful this in juno? https://blueprints.launchpad.net/ironic/+spec/event-callback-api wondering if i should propose a spec for it, or if the feature is not needed atm | 22:44 |
| devananda | adam_g: to help with the neutron port update, yes | 22:46 |
| devananda | adam_g: and to propagate node status changes faster | 22:46 |
| devananda | adam_g: and help alleviate some of the polling by nova | 22:46 |
| *** igordcard has quit IRC | 22:50 | |
| * devananda wanders afk for a bit | 22:57 | |
| NobodyCam | :) | 22:57 |
| openstackgerrit | Ghe Rivero proposed a change to openstack/ironic-python-agent: Fix IPA standby extension test https://review.openstack.org/99230 | 22:57 |
| adam_g | devananda, okay cool, ill propose a spec on that soon | 22:57 |
| lifeless | adam_g: quick thoughts for you | 23:00 |
| lifeless | adam_g: allow multiple folk to be notified | 23:01 |
| lifeless | adam_g: and don't require config settings for it - make it via the API | 23:01 |
| lifeless | adam_g: the async API discussion we've been having makes me think of something like 'here is a webhook to call when this node changes state' | 23:01 |
| lifeless | there's plenty of prior art out there, of course (e.g. PSHB, which is particularly aimed at things with structured representations, so while it won't be an exact match perhaps inspiration can be drawn) | 23:03 |
| adam_g | lifeless, this wouldn't be about nodes changing states, but things outside of ironic changing state (eg, neutron port created). ideally, ironic would be able to receive notifications from multiple folk. ironic sending notifications is different | 23:05 |
| lifeless | adam_g: oh | 23:05 |
| openstackgerrit | A change was merged to openstack/ironic: Allow more time for API requests to be completed https://review.openstack.org/93731 | 23:06 |
| lifeless | adam_g: so there the same thing applies but you need to convince other people :) | 23:09 |
| *** radsy has joined #openstack-ironic | 23:09 | |
| lifeless | adam_g: I don't think it will help with propogating node changes, nor with nova's polling, unless we get Ironic sending events too. | 23:09 |
| lifeless | adam_g: PSHB basically defines an HTTP protocol for subscribing to things at urls and being notified on change, in a scalable, secure manner. | 23:10 |
| lifeless | adam_g: its pretty much an exact fit for what we need IMNSHO. Just need to toss the XML away. | 23:10 |
| *** dwalleck has quit IRC | 23:11 | |
| klindgren_ | Is the ironic scheduler_host_manger jsut a patched version of the normal hostmanager scheduler and as such should be able to handle scheduling for normal libvirt vm's as well? | 23:13 |
| klindgren_ | Reading the deploy guide and trying to figure out what all I need to do to try out ironic in an existing environment without blowing that environment up for a while | 23:14 |
| jroll | klindgren_: I believe ironic needs its own nova-scheduler instance at the moment | 23:15 |
| NobodyCam | yep | 23:15 |
| NobodyCam | klindgren_: are you able to setup ironic in its own az | 23:15 |
| klindgren_ | ok - thats what I was thinking. I was hoping that I wouldn't have to | 23:16 |
| jroll | you can share a nova-api, but it needs its own scheduler and compute(s) | 23:16 |
| jroll | I guess that host manager might work alongside normal VMs, if you're willing to only provision one VM per compute host | 23:17 |
| devananda | klindgren_: you can use a single az with separate host aggregates or separate cells | 23:17 |
| jroll | :P | 23:17 |
| devananda | but yes, ironic needs a separate scheduler process, and its own compute processes | 23:18 |
| *** pcrews_ has quit IRC | 23:20 | |
| klindgren_ | If I already have host aggregates setup for the existing servers, under a single az how does one define which scheduler to use for the host aggregate? Or did I just miss that completely | 23:20 |
| *** lucas-dinner has quit IRC | 23:24 | |
| devananda | klindgren_: tbh, i'm not sure. more of a nova question. googling continues to point me at this: http://docs.openstack.org/grizzly/openstack-compute/admin/content/host-aggregates.html | 23:24 |
| devananda | which is now quite old :( | 23:25 |
| klindgren_ | devananda, yea - I was getting pointed to the same doc | 23:26 |
| openstackgerrit | Josh Gachnang proposed a change to openstack/ironic: Adding swift temp url support https://review.openstack.org/81391 | 23:26 |
| devananda | klindgren_: ah, this is better: http://docs.openstack.org/trunk/openstack-ops/content/scaling.html | 23:27 |
| klindgren_ | so reading that doc, what I need is a new region for ironic. As AZ's and Host aggregates share the same nova services | 23:37 |
| klindgren_ | http://docs.openstack.org/trunk/openstack-ops/content/scaling.html#segragation_methods | 23:37 |
| klindgren_ | only regions or cells allow you to provide a different scheduler | 23:38 |
| openstackgerrit | Jim Rollenhagen proposed a change to openstack/ironic-specs: Expose configdrive to instances https://review.openstack.org/99235 | 23:38 |
| klindgren_ | Previously, all services had an availability zone. Currently, only the nova-compute service has its own availability zone. Services such as nova-scheduler, nova-network, and nova-conductor have always spanned all availability zones. -- Does that sound correct? | 23:38 |
| jroll | hmm, I don't know anything about availability zones - we use regions and cells for VM-land at rackspace | 23:40 |
| jroll | or nova-land, rather | 23:40 |
| devananda | alright, i think i'm done for the day ... | 23:47 |
| devananda | g'night ,all! | 23:47 |
| NobodyCam | night devananda | 23:52 |
| *** rloo has quit IRC | 23:58 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!