*** mjturek has quit IRC | 00:19 | |
*** rh-jelabarre has quit IRC | 00:47 | |
openstackgerrit | Julia Kreger proposed openstack/ironic master: Fix IPv6 iPXE support https://review.openstack.org/628008 | 00:52 |
---|---|---|
openstackgerrit | Julia Kreger proposed openstack/ironic master: WIP ipv6 job https://review.openstack.org/613169 | 01:02 |
*** sthussey has quit IRC | 01:05 | |
TheJulia | rpioso: fyi https://storyboard.openstack.org/#!/story/2004519 | 01:11 |
rpioso | TheJulia: y, I’m well aware of that. | 01:14 |
TheJulia | rpioso: okay, thanks! | 01:14 |
rpioso | TheJulia: Thx for checkin :) | 01:17 |
rpioso | On a different topic, I may have found you a system, as we discussed previously. I expect to connect you with the POC tomorrow. | 01:21 |
TheJulia | rpioso: oh, awesome! Thanks! | 01:28 |
TheJulia | I'm going to call it a night shortly, have a wonderful evening | 01:28 |
*** rloo has quit IRC | 01:56 | |
rpioso | U2! | 02:17 |
openstackgerrit | Merged openstack/ironic master: Allow disabling TFTP image cache https://review.openstack.org/625120 | 02:21 |
*** jrist has quit IRC | 03:07 | |
openstackgerrit | Debayan Ray proposed openstack/ironic master: Fix SushyError namespacing in Redfish inspection https://review.openstack.org/628095 | 03:21 |
openstackgerrit | Debayan Ray proposed openstack/ironic master: Allow non-deterministic ``local_gb`` property https://review.openstack.org/628096 | 03:43 |
openstackgerrit | Debayan Ray proposed openstack/ironic master: Fix port creation post Redfish inspect https://review.openstack.org/628097 | 04:05 |
*** codyhill has joined #openstack-ironic | 05:29 | |
codyhill | Hey folks. I’m coming along pretty well with Ironic (with nova). And now I’m trying to tackle using swap. I have added 64GB of Swap to my nova flavor. I then subtracted that space from the disk option. My node gets chosen and tries to boot. But I never get to the point in the boot process where it requests the config drive. And the system just hangs a the login prompt from the ramdisk. I have tested with Flavo | 05:39 |
codyhill | with 1GB larger and 1GB smaller disks. And The schedulure won’t select the node. So I think I have the flavor correct. I see documentation a lot of places on how to debug the inspector ram disk by passing a root password hash or ssh key. But how can I do that for normal system provisions? I’d like to login and see where this thing is hung up. Any help would be appricated. | 05:39 |
codyhill | Ha! Well… I about the 10th time I tried this… I decided to just work. (I tried again to collect logs to post a link to here) Hmm… | 06:08 |
codyhill | And it’s late so I can’t type properly… It* decided to work. | 06:09 |
*** bfournie has quit IRC | 06:15 | |
codyhill | Well… Trying again failed. Here are the conductor logs: https://ghostbin.com/paste/mxhq8 | 06:40 |
codyhill | A little more info about the flavor, node, & instance… https://ghostbin.com/paste/edbzn | 06:46 |
codyhill | Also this is OpenStack Newton | 06:46 |
*** jtomasek has joined #openstack-ironic | 06:49 | |
*** rcernin has quit IRC | 07:09 | |
iurygregory | good morning all =D | 07:23 |
etingof | iurygregory, o/ | 07:25 |
iurygregory | etingof, o/ cold day yay | 07:26 |
* etingof is not sure if the weather is good for biking today | 07:26 | |
iurygregory | -4 feels like -9 lol | 07:27 |
*** hwoarang has quit IRC | 07:47 | |
*** hwoarang has joined #openstack-ironic | 07:50 | |
*** hjensas has quit IRC | 07:54 | |
*** moshele has joined #openstack-ironic | 08:05 | |
*** codyhill has quit IRC | 08:07 | |
openstackgerrit | Iury Gregory Melo Ferreira proposed openstack/networking-baremetal master: [DNM] Tests https://review.openstack.org/624671 | 08:29 |
openstackgerrit | Digambar proposed openstack/ironic stable/queens: Fix OOB introspection to use pxe_enabled flag in idrac driver https://review.openstack.org/624388 | 08:35 |
*** dtantsur|afk is now known as dtantsur | 08:59 | |
*** robbbe has joined #openstack-ironic | 09:01 | |
openstackgerrit | paresh sao proposed openstack/ironic master: OOB RAID implementation for ilo5 based HPE Proliant servers. https://review.openstack.org/555664 | 09:04 |
dtantsur | morning ironic | 09:04 |
iurygregory | morning dtantsur o/ | 09:05 |
*** derekh has joined #openstack-ironic | 09:18 | |
*** hjensas has joined #openstack-ironic | 09:27 | |
*** e0ne has joined #openstack-ironic | 09:42 | |
*** dkehn has quit IRC | 11:04 | |
*** bfournie has joined #openstack-ironic | 12:03 | |
*** S4ren has joined #openstack-ironic | 12:18 | |
openstackgerrit | Ilya Etingof proposed openstack/sushy-tools master: Add configuration file to sushy-emulator https://review.openstack.org/616178 | 12:18 |
openstackgerrit | Ilya Etingof proposed openstack/sushy-tools master: Add memoization to expensive emulator calls https://review.openstack.org/612758 | 12:20 |
openstackgerrit | Ilya Etingof proposed openstack/sushy-tools master: Limit instances exposure https://review.openstack.org/616516 | 12:22 |
openstackgerrit | Ilya Etingof proposed openstack/sushy-tools master: Add configurable libvirt firmware https://review.openstack.org/620605 | 12:23 |
*** rh-jelabarre has joined #openstack-ironic | 12:49 | |
*** jcoufal has joined #openstack-ironic | 13:11 | |
*** trown|outtypewww is now known as trown | 13:11 | |
*** ianychoi has joined #openstack-ironic | 13:17 | |
*** derekh has quit IRC | 13:24 | |
TheJulia | good morning | 13:42 |
iurygregory | TheJulia, morning o/ | 13:43 |
iurygregory | i have news about networking-baremetal for you \o/ | 13:44 |
iurygregory | i made it work in a devstack env and tempest is green \o/ | 13:44 |
etingof | iurygregory++ | 13:44 |
iurygregory | =D | 13:44 |
etingof | good morning, TheJulia o/ | 13:44 |
iurygregory | snow helped today lol | 13:45 |
* etingof is interested in the details of the fix (though has to run now) | 13:47 | |
iurygregory | variables missing in the local.conf | 13:48 |
iurygregory | jsut double checking if all i added are really necessary now XD | 13:48 |
dtantsur | TheJulia: morning! would you be fine with approving https://review.openstack.org/#/c/624119/ as it is? | 13:49 |
patchbot | patch 624119 - ironic - Allocation API: database and RPC - 6 patch sets | 13:49 |
TheJulia | I think so... | 13:52 |
openstackgerrit | Julia Kreger proposed openstack/ironic master: Fix IPv6 iPXE support https://review.openstack.org/628008 | 13:54 |
openstackgerrit | Julia Kreger proposed openstack/ironic master: WIP ipv6 job https://review.openstack.org/613169 | 13:54 |
openstackgerrit | Hamdy Khader proposed openstack/ironic master: Add support for Smart NICs https://review.openstack.org/583573 | 13:54 |
TheJulia | dtantsur: I got ipxe to get the proper next filename on ipv6 :) \o/ | 13:55 |
dtantsur | \o/ | 13:55 |
TheJulia | v6 routing seems to be... problematic though, but one step closer | 13:56 |
TheJulia | I think uefi boot from url is largely going to be us trying to figure out how we're going to test it more than anything else.... | 13:57 |
dtantsur | it has always been the case with UEFI, indeed | 13:57 |
TheJulia | yeah, I looked at the firmware image out there and it appears not to support it by default, and of course it doesn't have the best documentation :( | 14:02 |
TheJulia | but, we've got the key elements, delineation and identification is the key next | 14:02 |
*** mmethot has joined #openstack-ironic | 14:03 | |
openstackgerrit | Dmitry Tantsur proposed openstack/ironic master: Allocation API: minor fixes to DB and RPC https://review.openstack.org/628187 | 14:03 |
dtantsur | TheJulia, kaifeng: follow-up posted ^^ | 14:03 |
*** jistr is now known as jistr|mtg | 14:04 | |
*** derekh has joined #openstack-ironic | 14:04 | |
*** sthussey has joined #openstack-ironic | 14:09 | |
*** rloo has joined #openstack-ironic | 14:11 | |
openstackgerrit | Iury Gregory Melo Ferreira proposed openstack/networking-baremetal master: [DNM] Tests https://review.openstack.org/624671 | 14:13 |
TheJulia | dtantsur: parent patch wf+1 | 14:16 |
dtantsur | thnx | 14:16 |
*** dkehn has joined #openstack-ironic | 14:16 | |
openstackgerrit | Hamdy Khader proposed openstack/ironic master: Add support for Smart NICs https://review.openstack.org/583573 | 14:17 |
openstackgerrit | Mark Goddard proposed openstack/bifrost master: Always use a virtualenv in CI testing https://review.openstack.org/628196 | 14:51 |
TheJulia | dtantsur: can you run the meeting on the 14th? | 14:53 |
TheJulia | I will be in metal tube on approach to an airport during the meeting | 14:53 |
TheJulia | which I've had problems with the wifi on the aircraft before | 14:53 |
*** mjturek has joined #openstack-ironic | 14:54 | |
*** jistr|mtg is now known as jistr | 14:55 | |
dtantsur | TheJulia: I think I can, yes | 14:56 |
TheJulia | I will likely also need someone to run the 28th | 14:56 |
TheJulia | since I'll be in Brno | 14:57 |
*** Goneri has joined #openstack-ironic | 14:57 | |
*** hjensas has quit IRC | 14:59 | |
dtantsur | TheJulia: I'll probably be back (returning on the 27th) | 15:05 |
*** moshele has quit IRC | 15:06 | |
*** Bhujay has joined #openstack-ironic | 15:15 | |
*** Bhujay has quit IRC | 15:16 | |
iurygregory | CI seems happy yay https://review.openstack.org/#/c/624671/ \o/ | 15:22 |
patchbot | patch 624671 - networking-baremetal - [DNM] Tests - 10 patch sets | 15:22 |
openstackgerrit | Ruby Loo proposed openstack/ironic master: Fix typo in release note https://review.openstack.org/628203 | 15:26 |
*** e0ne has quit IRC | 15:28 | |
openstackgerrit | Iury Gregory Melo Ferreira proposed openstack/networking-baremetal master: Change networking-baremetal to zuulv3/python3 https://review.openstack.org/614026 | 15:28 |
openstackgerrit | Iury Gregory Melo Ferreira proposed openstack/networking-baremetal master: Change networking-baremetal to zuulv3/python3 https://review.openstack.org/614026 | 15:38 |
openstackgerrit | Mark Goddard proposed openstack/bifrost master: Use a virtualenv in openSUSE CI testing https://review.openstack.org/628196 | 15:43 |
*** hjensas has joined #openstack-ironic | 15:44 | |
openstackgerrit | Mark Goddard proposed openstack/bifrost master: Use a virtualenv in openSUSE CI testing https://review.openstack.org/628196 | 15:45 |
openstackgerrit | Mark Goddard proposed openstack/bifrost master: Use a virtualenv in openSUSE CI testing https://review.openstack.org/628196 | 15:48 |
openstackgerrit | Merged openstack/ironic master: Fix typo in release note https://review.openstack.org/628203 | 16:07 |
openstackgerrit | Mark Goddard proposed openstack/ironic master: WIP: Deploy templates: data model, DB API & objects https://review.openstack.org/627663 | 16:17 |
*** codyhill has joined #openstack-ironic | 16:21 | |
codyhill | Good Morning. | 16:21 |
codyhill | I posted an issue last night. But my client quite due to a Mac update… So I’m not sure if I got a response. Is this channel archied anywhere that would allow me to see history? | 16:23 |
codyhill | Anyhow I’ll ask my question again. | 16:25 |
codyhill | I’m trying to get the swap partion created using Ironic and Nova. I’ve created the nova flavor with the correct swap. And subtracted that swap from the Disk in the flavor. The swap and disk add up to local_gb. When I boot this system it never gets past booting into the ram disk and getting an IP. The next action I’d expect to see is the config-drive get mounted and copied over… But that doesn’t happen. | 16:32 |
codyhill | (Although I did get this to happen randomly when I was trying to reporduce the issue. But I couldn’t get it to work again.) Here are the logs: https://ghostbin.com/paste/7pcn3 Here are is some info on the flavor, node, & instance: https://ghostbin.com/paste/jy5wd And this is OpenStack Newton by the way. | 16:32 |
TheJulia | good morning codyhill | 16:33 |
TheJulia | codyhill: eavesdrop.openstack.org | 16:33 |
codyhill | What I’d really like to know how to do is log into the RAM disk while this thing is hung so that I can debug. I see a way to do this for the ironic inspector by modifying the pxe command line flags. But not during a provision. | 16:33 |
codyhill | Good Moring TheJulia! | 16:34 |
TheJulia | It is just never network booting for the deployment? | 16:34 |
codyhill | We are copying to disk and then booting local after. But it isn’t getting that far. | 16:35 |
TheJulia | yeah, looks like the ramdisk was never able to reach the API to heartbeat based upon the logs you provided. | 16:36 |
openstackgerrit | Mark Goddard proposed openstack/bifrost master: Use a virtualenv in openSUSE CI testing https://review.openstack.org/628196 | 16:36 |
codyhill | It is PXE booting. And hangs at the screen that show the login prompt and IPs | 16:36 |
TheJulia | so yeah, sounds like purely networking. Your using flat networking right? | 16:36 |
codyhill | Yes. And if i change the flavor to not include swap... | 16:36 |
codyhill | It works great. | 16:36 |
codyhill | This is Ubuntu-16.04 BTW. | 16:37 |
codyhill | Let me see If I can ping the instance when it’s in this state. Normally I can. | 16:39 |
TheJulia | codyhill: do you see anything reported to the console by IPA? | 16:39 |
codyhill | It only shows the IPs that the interfaces get and the login prompt. | 16:40 |
codyhill | Normally I see a bunch of things flashing. | 16:40 |
codyhill | ANd it prints things. | 16:40 |
codyhill | But not when I hit this. | 16:40 |
TheJulia | For some reason I thought swap was GB. I could only guess that maybe IPA is booting up, performing its lookup, and there maybe an exception in the properties. | 16:44 |
TheJulia | somehow, your going to need to troubleshoot from with-in the ramdisk | 16:44 |
codyhill | Yes. How do I troubleshoot within the Ramdisk? I see how to do this via inspector. But I couldn’t find documentation on debugging this with a normal provision. | 16:45 |
iurygregory | TheJulia, newbie question, tempest tests in python3 take more time than in python2? | 16:46 |
iurygregory | networking-baremetal-multitenant-vlans-python2 is green in https://review.openstack.org/#/c/614026/ but python3 i got a timeout =( | 16:46 |
patchbot | patch 614026 - networking-baremetal - Change networking-baremetal to zuulv3/python3 - 16 patch sets | 16:46 |
TheJulia | its mb | 16:47 |
TheJulia | codyhill: typically you make a special ramdisk with a user built in, and possibly even perform a manual execution of ironic-python-agent starting up in the ramdisk so you can see on a terminal what it is doing. Sure looks like it is on a short deployment timeout before it gave up and powered off the node. You might want to also check your API log to see what, if anything is occurring ironic side, but as far as I can | 16:50 |
TheJulia | see from what you provided, IPA is simply never heartbeating... and there doesn't appear to be any reason that would cause that centered around swap in the newton-eol tagged code | 16:50 |
codyhill | TheJulia: The newton documentation doens’t specify MB or GB. But Pike specifies MB. | 16:50 |
TheJulia | codyhill: yeah, I just confirmed it was MB | 16:50 |
TheJulia | iurygregory: it really shouldn't take longer. There may be something happening out of immediate view which is taking longer though :\ | 16:51 |
iurygregory | 20min =( and boom timeout hehe | 16:52 |
iurygregory | maybe the host was slow i dont know | 16:52 |
TheJulia | repeatedly, it shouldn't be | 16:53 |
codyhill | The Julia: I’m using: `DIB_CLOUD_INIT_DATASOURCES="Ec2, ConfigDrive, OpenStack" DIB_RELEASE=xenial disk-image-create baremetal grub2 ubuntu dhcp-all-interfaces cloud-init-datasources -o ubuntu-16.04` to create my image for this. Do I need to mount one of these disk and inject a Key? Or can I pass an ENV variable to get one inected for the custom ram disk? | 16:53 |
TheJulia | where is it timing out, same place? repeatable? | 16:53 |
TheJulia | codyhill: That would be for the image your deploying, not the deployment ramdisk that performs the deployment | 16:54 |
codyhill | This generates a ram disk as well. | 16:54 |
iurygregory | in python2 didnt timeout with the fix in local.conf maybe python3 will need more time or not | 16:54 |
codyhill | I then link them together in Glance. | 16:55 |
TheJulia | you'll need something like DIB_RELEASE=xenial disk-image-create ubuntu ironic-agent devuser -o ipa | 16:55 |
TheJulia | you could edit your ironic-agent element to disable ipa start-up upon boot, and there are other devuser element flags you'll want to set to give yourself a user to login to | 16:55 |
iurygregory | http://paste.openstack.org/show/739878/ | 16:55 |
codyhill | Ok. My cleaning and ironic inspector image was built like that... | 16:55 |
codyhill | TheJulia: Thanks. I’ll look at docs and see what I can figure out for those flags. | 16:56 |
NobodyCam | Good Morning Ironic'ers | 17:00 |
iurygregory | morning NobodyCam | 17:00 |
NobodyCam | morning iurygregory :) | 17:03 |
iurygregory | TheJulia, do you think i should try a recheck since only the BasicOps failed and was timeout? before was BasicOps and Multienancy failling =) http://logs.openstack.org/26/614026/16/check/networking-baremetal-multitenant-vlans/6b14293/testr_results.html.gz | 17:08 |
openstackgerrit | Dmitry Tantsur proposed openstack/ironic master: [WIP] Allocation API: implementation without HA and take over https://review.openstack.org/628223 | 17:09 |
iurygregory | and python2 all green \o/ we almost there yay | 17:09 |
*** robbbe has quit IRC | 17:17 | |
*** gyee has joined #openstack-ironic | 17:25 | |
TheJulia | iurygregory: I'd prefer that you try to understand why it timed out, what was state things were at when it timed out. Was it writing the image, or just bad timing between a status check and the test and deployment | 17:27 |
* TheJulia steps away to deal with some family drama for a few minutes | 17:27 | |
* dtantsur steps away for some exercising | 17:32 | |
*** dkehn has left #openstack-ironic | 17:33 | |
openstackgerrit | Ruby Loo proposed openstack/ironic stable/rocky: Allow disabling instance image cache https://review.openstack.org/628229 | 17:33 |
TheJulia | exercise+++++++ | 17:37 |
*** trown is now known as trown|lunch | 17:41 | |
TheJulia | lunch, migraine meds, and some exercise sound like a perfect idea | 17:42 |
openstackgerrit | Mark Goddard proposed openstack/bifrost master: Perform ironic online data migrations https://review.openstack.org/627513 | 17:53 |
*** S4ren has quit IRC | 18:03 | |
openstackgerrit | Derek Higgins proposed openstack/ironic master: Testing UEFI job on bionic https://review.openstack.org/626755 | 18:03 |
*** derekh has quit IRC | 18:03 | |
*** robbbe has joined #openstack-ironic | 18:08 | |
*** MattMan has quit IRC | 18:09 | |
*** MattMan has joined #openstack-ironic | 18:09 | |
*** bnemec has quit IRC | 18:13 | |
*** bnemec has joined #openstack-ironic | 18:26 | |
*** bnemec has quit IRC | 18:30 | |
*** bnemec has joined #openstack-ironic | 18:34 | |
*** bnemec has quit IRC | 18:39 | |
openstackgerrit | Dmitry Tantsur proposed openstack/ironic master: [WIP] Allocation API: implementation without HA and take over https://review.openstack.org/628223 | 18:43 |
*** dtantsur is now known as dtantsur|afk | 18:43 | |
dtantsur|afk | see you tomorrow | 18:43 |
*** bnemec has joined #openstack-ironic | 18:52 | |
*** bnemec has quit IRC | 18:58 | |
*** trown|lunch is now known as trown | 19:00 | |
openstackgerrit | Julia Kreger proposed openstack/ironic master: WIP ipv6 job https://review.openstack.org/613169 | 19:00 |
TheJulia | hopefully that solves it. Better aligns with my local config | 19:01 |
*** bnemec has joined #openstack-ironic | 19:14 | |
codyhill | TheJulia: The issue looks to have been caused by the second Nic in the server getting a DHCP address after the primary NIC. Which lead the system to not beable to route to the ironic-api. I’ve reconfigured the switches to disable the PVID on NIC2 and only have it on NIC1 and that seems to have solved the issue. {fingers crossed} I’m gonig to test a few more times becuase it randomly worked a few times as wellâ | 19:24 |
*** irclogbot_1 has quit IRC | 19:24 | |
codyhill | And it always worked without swap. I’m not sure if adding a swap partion allowed the race condition to happen? But looks like this is resolved. | 19:24 |
*** irclogbot_1 has joined #openstack-ironic | 19:26 | |
TheJulia | codyhill: ahh, yeah that is a fun nice race condition because the default gw would be changed to who knows what | 19:30 |
codyhill | Yep! | 19:30 |
TheJulia | it shouldn't of, but race conditions can sometimes be like that | 19:30 |
codyhill | Thanks for the help! I wouldn’t have figured it out without the new ramdisk taht I could loginto. | 19:36 |
TheJulia | :) | 19:51 |
TheJulia | codyhill: happy to help | 19:51 |
*** e0ne has joined #openstack-ironic | 20:21 | |
*** robbbe has quit IRC | 20:34 | |
*** e0ne has quit IRC | 20:37 | |
*** e0ne has joined #openstack-ironic | 20:54 | |
*** moshele has joined #openstack-ironic | 21:01 | |
rpioso | Good late morning :) | 21:05 |
mjturek | TheJulia: fyi looked a bit at the job today, and from a deployment standpoint it looks good.. | 21:14 |
mjturek | pkvmci job that is | 21:14 |
mjturek | so, need to log into the dsvm and see if something's going wrong there | 21:14 |
TheJulia | mjturek: ack. It looked like it might have been devstack-gate's doing | 21:18 |
rpioso | mjturek: Congratulations! | 21:33 |
mjturek | thank you rpioso!! | 21:33 |
rpioso | :) | 21:33 |
mjturek | TheJulia: hmmm didn't notice that | 21:33 |
TheJulia | mjturek: entirely possible that was one of the other ci systems | 21:50 |
mjturek | fair enough! | 21:50 |
*** e0ne has quit IRC | 21:50 | |
TheJulia | rpioso: Got your email, thanks! I don't have the mental spoons to reply today, but will tomorrow | 21:52 |
*** e0ne has joined #openstack-ironic | 21:52 | |
*** e0ne has quit IRC | 21:57 | |
rpioso | TheJulia: yw. Hope it helps. | 21:58 |
*** rloo has quit IRC | 22:07 | |
*** rloo has joined #openstack-ironic | 22:10 | |
*** trown is now known as trown|outtypewww | 22:13 | |
*** hjensas is now known as hjensas|brb | 22:16 | |
*** rcernin has joined #openstack-ironic | 22:42 | |
*** rloo has quit IRC | 22:43 | |
openstackgerrit | Julia Kreger proposed openstack/ironic master: WIP ipv6 job https://review.openstack.org/613169 | 22:43 |
*** rloo has joined #openstack-ironic | 22:44 | |
TheJulia | it occurs to me that I think derek is kind of fighting the same issue in the neutron plugin init stuffs :( | 22:48 |
*** jcoufal has quit IRC | 22:49 | |
TheJulia | My brain has reached friend. I guess I'll keep trying on that later so hopefully we can get a successful CI run of it :\ | 22:51 |
TheJulia | fried | 22:54 |
TheJulia | reached fried | 22:54 |
* TheJulia just gets a laugh out of it | 22:54 | |
openstackgerrit | Ilya Etingof proposed openstack/sushy master: Add System<->Manager linkage https://review.openstack.org/621663 | 23:08 |
*** moshele has quit IRC | 23:17 | |
*** mmethot has quit IRC | 23:22 | |
openstackgerrit | Ilya Etingof proposed openstack/sushy master: Add System<->Manager linkage https://review.openstack.org/621663 | 23:23 |
*** jcoufal has joined #openstack-ironic | 23:47 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!