| *** ociuhandu has joined #openstack-ironic | 00:01 | |
| *** rcernin has quit IRC | 00:03 | |
| *** gyee has quit IRC | 00:03 | |
| *** ianychoi has quit IRC | 00:03 | |
| *** yoctozepto has quit IRC | 00:03 | |
| *** gudrutis has quit IRC | 00:03 | |
| *** mushrushu has quit IRC | 00:03 | |
| *** trandles has quit IRC | 00:03 | |
| *** benj_ has quit IRC | 00:03 | |
| *** bburns has quit IRC | 00:03 | |
| *** openstackgerrit has quit IRC | 00:03 | |
| *** hjensas has quit IRC | 00:03 | |
| *** eagereagle1 has quit IRC | 00:03 | |
| *** bdodd has quit IRC | 00:03 | |
| *** irclogbot_3 has quit IRC | 00:03 | |
| *** antonym has quit IRC | 00:03 | |
| *** spotz has quit IRC | 00:03 | |
| *** baha has quit IRC | 00:03 | |
| *** arne_wiebalck has quit IRC | 00:03 | |
| *** alexmcleod has quit IRC | 00:03 | |
| *** tzumainn has quit IRC | 00:03 | |
| *** rh-jelabarre has quit IRC | 00:03 | |
| *** pmannidi has quit IRC | 00:03 | |
| *** tkajinam has quit IRC | 00:03 | |
| *** rajinir has quit IRC | 00:03 | |
| *** buhman has quit IRC | 00:03 | |
| *** andrein has quit IRC | 00:03 | |
| *** guilhermesp has quit IRC | 00:03 | |
| *** jungleboyj has quit IRC | 00:03 | |
| *** donnyd has quit IRC | 00:03 | |
| *** persia has quit IRC | 00:03 | |
| *** v1k0d3n has quit IRC | 00:03 | |
| *** seongsoocho has quit IRC | 00:03 | |
| *** portdirect has quit IRC | 00:03 | |
| *** phrobb has quit IRC | 00:03 | |
| *** coreycb has quit IRC | 00:03 | |
| *** gmann has quit IRC | 00:03 | |
| *** melwitt has quit IRC | 00:03 | |
| *** rloo has quit IRC | 00:03 | |
| *** mkowalski has quit IRC | 00:03 | |
| *** larsks has quit IRC | 00:03 | |
| *** rh-jelabarre has joined #openstack-ironic | 00:04 | |
| *** tzumainn has joined #openstack-ironic | 00:05 | |
| *** pmannidi has joined #openstack-ironic | 00:05 | |
| *** tkajinam has joined #openstack-ironic | 00:05 | |
| *** rajinir has joined #openstack-ironic | 00:05 | |
| *** buhman has joined #openstack-ironic | 00:05 | |
| *** andrein has joined #openstack-ironic | 00:05 | |
| *** guilhermesp has joined #openstack-ironic | 00:05 | |
| *** jungleboyj has joined #openstack-ironic | 00:05 | |
| *** donnyd has joined #openstack-ironic | 00:05 | |
| *** persia has joined #openstack-ironic | 00:05 | |
| *** v1k0d3n has joined #openstack-ironic | 00:05 | |
| *** seongsoocho has joined #openstack-ironic | 00:05 | |
| *** portdirect has joined #openstack-ironic | 00:05 | |
| *** phrobb has joined #openstack-ironic | 00:05 | |
| *** coreycb has joined #openstack-ironic | 00:05 | |
| *** gmann has joined #openstack-ironic | 00:05 | |
| *** melwitt has joined #openstack-ironic | 00:05 | |
| *** rloo has joined #openstack-ironic | 00:05 | |
| *** mkowalski has joined #openstack-ironic | 00:05 | |
| *** larsks has joined #openstack-ironic | 00:05 | |
| *** rcernin has joined #openstack-ironic | 00:06 | |
| *** gyee has joined #openstack-ironic | 00:06 | |
| *** bburns has joined #openstack-ironic | 00:06 | |
| *** ianychoi has joined #openstack-ironic | 00:06 | |
| *** hjensas has joined #openstack-ironic | 00:06 | |
| *** eagereagle1 has joined #openstack-ironic | 00:06 | |
| *** spotz has joined #openstack-ironic | 00:06 | |
| *** bdodd has joined #openstack-ironic | 00:06 | |
| *** yoctozepto has joined #openstack-ironic | 00:06 | |
| *** irclogbot_3 has joined #openstack-ironic | 00:06 | |
| *** gudrutis has joined #openstack-ironic | 00:06 | |
| *** antonym has joined #openstack-ironic | 00:06 | |
| *** mushrushu has joined #openstack-ironic | 00:06 | |
| *** trandles has joined #openstack-ironic | 00:06 | |
| *** baha has joined #openstack-ironic | 00:06 | |
| *** arne_wiebalck has joined #openstack-ironic | 00:06 | |
| *** alexmcleod has joined #openstack-ironic | 00:06 | |
| *** benj_ has joined #openstack-ironic | 00:06 | |
| *** mkowalski has quit IRC | 00:07 | |
| *** ociuhandu has quit IRC | 00:07 | |
| *** tosky has quit IRC | 00:07 | |
| *** bburns_ has joined #openstack-ironic | 00:09 | |
| *** alexmcleod_ has joined #openstack-ironic | 00:09 | |
| *** andrein has quit IRC | 00:09 | |
| *** guilhermesp has quit IRC | 00:09 | |
| *** jungleboyj has quit IRC | 00:09 | |
| *** donnyd has quit IRC | 00:09 | |
| *** coreycb has quit IRC | 00:09 | |
| *** gmann has quit IRC | 00:09 | |
| *** jungleboyj has joined #openstack-ironic | 00:09 | |
| *** bburns has quit IRC | 00:10 | |
| *** alexmcleod has quit IRC | 00:10 | |
| *** donnyd has joined #openstack-ironic | 00:10 | |
| *** coreycb has joined #openstack-ironic | 00:10 | |
| *** andrein has joined #openstack-ironic | 00:10 | |
| *** guilhermesp has joined #openstack-ironic | 00:10 | |
| *** irclogbot_3 has quit IRC | 00:10 | |
| *** gmann has joined #openstack-ironic | 00:11 | |
| *** irclogbot_2 has joined #openstack-ironic | 00:11 | |
| *** anuradha1904 has quit IRC | 00:42 | |
| *** mkowalski has joined #openstack-ironic | 00:43 | |
| *** mkowalski has quit IRC | 00:52 | |
| *** dsneddon has quit IRC | 00:54 | |
| *** rloo has quit IRC | 01:01 | |
| *** mkowalski has joined #openstack-ironic | 01:03 | |
| eandersson | If I need Ironic to make an API call to upgrade firmware (api call + restart hardware) as part of a deploy what would be the ideal place to implement that? | 01:50 |
|---|---|---|
| eandersson | e.g. make sure firmware is on x.y.z (if it isn't apply and reboot) and then deploy | 01:51 |
| eandersson | My initial thought was to just override the deploy interface | 01:59 |
| *** ianychoi has quit IRC | 02:42 | |
| *** ociuhandu has joined #openstack-ironic | 02:43 | |
| *** ociuhandu has quit IRC | 02:47 | |
| lbragstad | TheJulia i was buried in a few things today - but i have ironic on my plate for tomorrow | 02:49 |
| *** Qianbiao has joined #openstack-ironic | 03:06 | |
| *** rcernin has quit IRC | 03:11 | |
| *** rcernin has joined #openstack-ironic | 03:12 | |
| *** mkrai has joined #openstack-ironic | 03:33 | |
| *** ricolin_ has joined #openstack-ironic | 03:52 | |
| *** gyee has quit IRC | 04:12 | |
| *** ianychoi has joined #openstack-ironic | 04:29 | |
| *** Qianbiao has quit IRC | 04:46 | |
| *** openstackgerrit has joined #openstack-ironic | 04:56 | |
| openstackgerrit | ankit proposed openstack/ironic master: Fixes issue of redfish firmware update https://review.opendev.org/c/openstack/ironic/+/764914 | 04:56 |
| TheJulia | eandersson: depends on version, management interface most likely | 05:26 |
| TheJulia | lbragstad: no worries | 05:26 |
| * TheJulia goes to the land of ZzZzZzZz | 05:26 | |
| *** tzumainn has quit IRC | 05:47 | |
| *** dsneddon has joined #openstack-ironic | 05:54 | |
| *** dsneddon has quit IRC | 06:00 | |
| eandersson | Thanks - will try that | 06:14 |
| arne_wiebalck | Good morning, ironic! | 06:31 |
| janders | good morning arne_wiebalck, Happy Friday! o/ | 06:31 |
| arne_wiebalck | Hey janders o/ | 06:32 |
| openstackgerrit | sonali bhausaheb borkar proposed x/sushy-oem-idrac master: Add get pxe port macs for bios mode https://review.opendev.org/c/x/sushy-oem-idrac/+/770724 | 07:01 |
| *** rcernin has quit IRC | 07:05 | |
| openstackgerrit | sonali bhausaheb borkar proposed x/sushy-oem-idrac master: Add get pxe port macs for bios mode https://review.opendev.org/c/x/sushy-oem-idrac/+/770904 | 07:29 |
| *** rcernin has joined #openstack-ironic | 07:43 | |
| *** rcernin has quit IRC | 07:48 | |
| *** rcernin has joined #openstack-ironic | 07:49 | |
| *** rcernin has quit IRC | 07:54 | |
| *** nam-est has joined #openstack-ironic | 07:58 | |
| openstackgerrit | Merged openstack/ironic master: Fix redfish-virtual-media boot mode handling ordering https://review.opendev.org/c/openstack/ironic/+/770833 | 07:58 |
| *** Nisha_Agarwal has joined #openstack-ironic | 08:07 | |
| *** rpittau|afk is now known as rpittau | 08:09 | |
| rpittau | good morning ironic! o/ | 08:09 |
| janders | good morning rpittau | 08:10 |
| janders | Happy Friday | 08:10 |
| rpittau | hey janders :) | 08:11 |
| rpittau | Happy Friday! | 08:12 |
| openstackgerrit | Kaifeng Wang proposed openstack/ironic-python-agent master: Remove lldp-timeout support https://review.opendev.org/c/openstack/ironic-python-agent/+/701207 | 08:14 |
| *** anuradha1904 has joined #openstack-ironic | 08:20 | |
| *** Qianbiao has joined #openstack-ironic | 08:22 | |
| *** rcernin has joined #openstack-ironic | 08:26 | |
| *** rcernin has quit IRC | 08:31 | |
| *** tosky has joined #openstack-ironic | 08:42 | |
| *** Nisha_Agarwal has quit IRC | 08:43 | |
| *** lucasagomes has joined #openstack-ironic | 09:04 | |
| *** rcernin has joined #openstack-ironic | 09:28 | |
| *** ociuhandu has joined #openstack-ironic | 09:33 | |
| *** rcernin has quit IRC | 09:33 | |
| *** derekh has joined #openstack-ironic | 09:37 | |
| *** ociuhandu has quit IRC | 09:44 | |
| *** mkrai has quit IRC | 09:50 | |
| *** mkrai_ has joined #openstack-ironic | 09:50 | |
| *** ociuhandu has joined #openstack-ironic | 09:50 | |
| janders | anyone up for the SPUC? arne_wiebalck rpittau? | 10:02 |
| janders | https://bluejeans.com/772893798 | 10:02 |
| rpittau | janders: coming! :) | 10:02 |
| *** Qianbiao has quit IRC | 10:03 | |
| *** Qianbiao has joined #openstack-ironic | 10:07 | |
| *** mkrai_ has quit IRC | 10:18 | |
| *** mkrai has joined #openstack-ironic | 10:19 | |
| *** Nisha_Agarwal has joined #openstack-ironic | 10:20 | |
| openstackgerrit | Dmitry Tantsur proposed openstack/ironic stable/victoria: Fix redfish-virtual-media boot mode handling ordering https://review.opendev.org/c/openstack/ironic/+/770922 | 10:31 |
| *** dtantsur|afk is now known as dtantsur | 10:38 | |
| dtantsur | morning folks, sorry for missing the SPUC | 10:38 |
| * dtantsur is now in possession of a permanent residency permit \o/ | 10:38 | |
| janders | congratulations dtantsur!!! \o/ | 10:40 |
| janders | (and no worries about the SPUC :) | 10:40 |
| janders | we finished early cause rpittau had a clashing meeting but happy to jump back in if you like? | 10:42 |
| dtantsur | I'm afraid I need to eat something urgently, been up since 7am | 10:44 |
| janders | first things first | 10:44 |
| janders | next week! :) | 10:44 |
| dtantsur | yeah | 10:44 |
| janders | I think the PR grant calls for a champagne (or prosecco or beer) breakfast! :) | 10:45 |
| *** rcernin has joined #openstack-ironic | 10:45 | |
| Nisha_Agarwal | Hey dtantsur , congratulations!!! | 10:46 |
| *** ociuhandu has quit IRC | 10:46 | |
| dtantsur | janders++ | 10:47 |
| dtantsur | thank you Nisha! | 10:47 |
| *** Qianbiao has quit IRC | 10:50 | |
| *** anuradha1904 has quit IRC | 10:59 | |
| janders | dtantsur rpittau for NVMe cleaning change, would you be comfortable with code that checks this output http://paste.openstack.org/show/801655/ for what is supported and takes action accordingly? I'm a little concerned with potential future changes to nvme-cli output but haven't found a better way so far (json output seems useless for sanitize/format | 11:02 |
| janders | features) | 11:02 |
| janders | I'm thinking trying sanitize if supported and falling back to secure format if not | 11:03 |
| janders | (and then to shred if none of the two is supported / completes successfully provided that shred fallback is enabled in ironic.conf) | 11:03 |
| *** ociuhandu has joined #openstack-ironic | 11:17 | |
| *** rcernin has quit IRC | 11:17 | |
| janders | dtantsur rpittau alternatively, for the first iteration, I can make it only use secure format instead (this feature seems to have much more consistent support - the only NVMe that supports sanitize that I've seen so far is the boot drive of my laptop :) ). What do you think? | 11:24 |
| *** ociuhandu has quit IRC | 11:26 | |
| *** ociuhandu has joined #openstack-ironic | 11:27 | |
| iurygregory | good morning janders dtantsur rpittau and Ironic! happy friday! | 11:28 |
| janders | hey iurygregory o/ | 11:28 |
| janders | Happy Friday! | 11:28 |
| *** rcernin has joined #openstack-ironic | 11:30 | |
| rpittau | hey iurygregory :) | 11:31 |
| rpittau | sorry janders, just reading now, I think that's ok, for the output I mean, we'll nened to keep an eye on the nvme-cli stuff | 11:33 |
| rpittau | and yeah, I think I saw the same with my laptop :D | 11:33 |
| rpittau | janders: in short, your approach sounds good to me | 11:33 |
| janders | thank you rpittau :) | 11:34 |
| * rpittau goes to lunch, carbonara awaits | 11:34 | |
| janders | as I'm typing all this up I'm getting a little worried I made it overcomplicated (and hence prone to bugs/failures), I might create two versions of this change for further discussion | 11:34 |
| iurygregory | carbonara *-* | 11:34 |
| janders | thanks rpittau, enjoy! | 11:35 |
| janders | yum! | 11:35 |
| janders | (not dnf in this case :) ) | 11:35 |
| iurygregory | hahahaha | 11:36 |
| iurygregory | janders, nice one! | 11:36 |
| *** mkrai has quit IRC | 11:37 | |
| *** Nisha_Agarwal has quit IRC | 11:48 | |
| dtantsur | janders: I think trying sanitize is still a good idea | 11:51 |
| dtantsur | we should expect it to become more popular in the enterprise field as the standard gets adopted | 11:51 |
| janders | dtantsur agreed. Should we make it experimental (and possibly have a config option making either format or sanitize the default NVMe cleaning method)? | 11:52 |
| janders | or is failing cleaning if nvme-cli output isn't convincing sufficient? | 11:53 |
| dtantsur | janders: we can learn in advance which method is supported, right? | 11:53 |
| dtantsur | so I'd try sanitize if it's supported, but an option to choose may be nice as well | 11:53 |
| janders | yeah the way I'm doing it right now is interrogating the NVMe and than I have some logic to pick the preferred solution | 11:54 |
| janders | idea is - if sanitize is supported, use it, if not, go to secure format (crypto variety if available, standard if not) | 11:54 |
| openstackgerrit | Jacob Anders proposed openstack/ironic-python-agent master: [WIP] Add support for using NVMe specific cleaning https://review.opendev.org/c/openstack/ironic-python-agent/+/770237 | 12:00 |
| janders | ^^ this is very crude, to be continued next week | 12:01 |
| janders | (not worth looking at yet) | 12:01 |
| janders | see you next week Ironic o/ | 12:06 |
| janders | have a great weekend everyone | 12:06 |
| janders | once again congrats on your PR dtantsur! | 12:06 |
| dtantsur | thank you and have a great weekend too | 12:07 |
| *** jamesdenton has quit IRC | 12:07 | |
| *** jamesdenton has joined #openstack-ironic | 12:08 | |
| arne_wiebalck | janders: rpittau sorry, busy morning | 12:25 |
| arne_wiebalck | dtantsur: \o/ for the permit! | 12:26 |
| dtantsur | :) | 12:26 |
| *** rcernin has quit IRC | 12:31 | |
| *** ricolin_ has quit IRC | 12:42 | |
| *** rcernin has joined #openstack-ironic | 12:45 | |
| openstackgerrit | Dmitry Tantsur proposed openstack/ironic-lib master: Import mounted context manager from IPA https://review.opendev.org/c/openstack/ironic-lib/+/770048 | 12:47 |
| *** zzzeek has quit IRC | 13:00 | |
| *** rcernin has quit IRC | 13:00 | |
| *** zzzeek has joined #openstack-ironic | 13:00 | |
| *** rcernin has joined #openstack-ironic | 13:05 | |
| *** Qianbiao has joined #openstack-ironic | 13:14 | |
| *** JamesBenson has joined #openstack-ironic | 13:19 | |
| openstackgerrit | sonali bhausaheb borkar proposed x/sushy-oem-idrac master: Add export system configuration https://review.opendev.org/c/x/sushy-oem-idrac/+/710327 | 13:27 |
| *** derekh has quit IRC | 13:29 | |
| *** derekh has joined #openstack-ironic | 13:30 | |
| *** rcernin has quit IRC | 13:30 | |
| openstackgerrit | sonali bhausaheb borkar proposed x/sushy-oem-idrac master: Add get pxe port macs for bios mode https://review.opendev.org/c/x/sushy-oem-idrac/+/770932 | 13:40 |
| openstackgerrit | Verification of a change to openstack/ironic failed: Update `cleaning_error_handler` https://review.opendev.org/c/openstack/ironic/+/770257 | 13:44 |
| openstackgerrit | Verification of a change to openstack/ironic failed: Do not enter maintenance if cleaning fails before running the 1st step https://review.opendev.org/c/openstack/ironic/+/770258 | 13:44 |
| *** paras333_ has joined #openstack-ironic | 13:57 | |
| *** jdandrea has joined #openstack-ironic | 13:57 | |
| dtantsur | anyone has an idea why stable/victoria is so unstable? | 13:58 |
| *** rloo has joined #openstack-ironic | 14:02 | |
| rpittau | mmm the multinode job has roughly a failure every 7-8 runs | 14:02 |
| dtantsur | okay, I checked one job, all nodes on the subnode failed to PXE boot | 14:04 |
| rpittau | fantastic | 14:04 |
| dtantsur | are we seeing the same on master? | 14:04 |
| shadower | just saw it, congrats dtantsur! \o/ | 14:05 |
| dtantsur | thx :) | 14:05 |
| * dtantsur hopes it was not about PXE booting :D | 14:05 | |
| rpittau | not really, I see a failure every now and then, but way less frequent | 14:05 |
| dtantsur | ooookay, so something we forgot to backport? | 14:05 |
| shadower | haha nope, about the permit :) | 14:05 |
| dtantsur | :) | 14:05 |
| dtantsur | see, we can barely maintain the current stable, I don't understand why we still pretend to support Train and older.. | 14:06 |
| rpittau | oh I'm still struggling to get ussuri in shape and btw, before I forget for the Nth time, grenade in inspector is 100% broken in victoria | 14:07 |
| dtantsur | \o/ | 14:07 |
| *** Qianbiao has quit IRC | 14:08 | |
| rpittau | dtantsur: I meant to talk you earlier this week but... well! TL;DR nodes are not starting because they get the wrong ipxe config file | 14:08 |
| dtantsur | I'll take a look after multinode okay? | 14:08 |
| rpittau | sure | 14:08 |
| *** Qianbiao has joined #openstack-ironic | 14:08 | |
| rpittau | if it can help for the multinode and the backports, I went through the patches recently to backport the focal related ones, I think I got them all, this was 10 days ago | 14:11 |
| iurygregory | rpittau, all patches are merged? | 14:11 |
| rpittau | yeah | 14:12 |
| iurygregory | do we have a link in the whiteboard with the failures we are seeing? | 14:12 |
| rpittau | I mentioned just the grenade inspector as it's been a while now | 14:13 |
| iurygregory | grenade inspector is the puzzle I coulnd't do anything D: | 14:14 |
| * dtantsur has put a harsh proposal on https://etherpad.opendev.org/p/ironic-wallaby-midcycle | 14:17 | |
| iurygregory | no bootable device (not good) - kernel panic would tell it could be memory problems hehe | 14:18 |
| dtantsur | I think it's DHCP | 14:18 |
| dtantsur | for a kernel problem we'd see the kernel/ramdisk downloaded first | 14:19 |
| dtantsur | there don't seem to be any meaningful diff between stable/victoria and master in zuul.d and devstack | 14:22 |
| dtantsur | okay, anyone has a link to an inspector problem handy? | 14:24 |
| * dtantsur found | 14:25 | |
| dtantsur | RAMDISK: Couldn't find valid RAM disk image starting at 0 | 14:27 |
| dtantsur | this *may* actually be a memory problem | 14:27 |
| *** ociuhandu_ has joined #openstack-ironic | 14:28 | |
| iurygregory | funny... | 14:31 |
| *** ociuhandu has quit IRC | 14:31 | |
| openstackgerrit | Dmitry Tantsur proposed openstack/ironic-inspector stable/victoria: Increase memory for the grenade job https://review.opendev.org/c/openstack/ironic-inspector/+/770941 | 14:32 |
| dtantsur | I think that's it ^^ | 14:32 |
| rpittau | dtantsur: not sure that's the problem, if you look at the ipa ramdisk logs you'll see that it gets the config from boot.ipxe, but it should take it from ironic-inspector.ipxe instead | 14:33 |
| rpittau | or am I wrong ? | 14:33 |
| dtantsur | rpittau: maybe it uses managed boot? | 14:33 |
| iurygregory | lmao it should be 512 in inspector O.o | 14:34 |
| dtantsur | also in the run I've found there is no IPA, the kernel crashes | 14:34 |
| rpittau | ok, mmm, I'm talking about this https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_761/767536/2/check/ironic-inspector-grenade/761e24d/controller/logs/ironic-bm-logs/node-0_no_ansi_2021-01-05-09%3A19%3A30_log.txt | 14:34 |
| dtantsur | well, right, we never even get to running inspector tests | 14:37 |
| rpittau | memory definitely needs to be increased, I'm not saying it doesn't | 14:37 |
| dtantsur | which raises a question: what the actual hell is this? https://zuul.opendev.org/t/openstack/build/761e24d5d4a2435fa01cac12a0336860/log/controller/logs/grenade.sh_log.txt#1995 | 14:37 |
| dtantsur | why is it creating a nova server? | 14:38 |
| rpittau | uh.... | 14:38 |
| dtantsur | (because grenade, I know) | 14:38 |
| dtantsur | what I think is happening is nova trying to create a server, hits timeout, retries a couple of times and fails | 14:38 |
| dtantsur | yeah, 40 minutes inside 'server create' | 14:39 |
| dtantsur | fixing the memory issue will likely make it pass, although the question remains valid: how do we ask nova to stop? | 14:39 |
| dtantsur | (it's probably okay to have it in the ironic CI) | 14:39 |
| dtantsur | it happens in normal runs too: https://zuul.openstack.org/build/7cb6b446f74047a79a9360b41ffd76b5/log/controller/logs/grenade.sh_log.txt#2005 | 14:40 |
| * dtantsur shakes fist at grenade | 14:41 | |
| iurygregory | we can disble nova in grenade if we want | 14:41 |
| rpittau | alright, I think that's the same reason why it's failing in ussuri | 14:41 |
| dtantsur | likely. lemme propose a backport | 14:41 |
| *** tzumainn has joined #openstack-ironic | 14:42 | |
| * dtantsur wonders if he needs to squash it with the database migrations fix | 14:42 | |
| openstackgerrit | Dmitry Tantsur proposed openstack/ironic-inspector stable/ussuri: Increase memory for the jobs https://review.opendev.org/c/openstack/ironic-inspector/+/770946 | 14:43 |
| rpittau | that might be a good idea | 14:45 |
| openstackgerrit | Dmitry Tantsur proposed openstack/ironic master: redfish-virtual-media: make fewer calls when preparing boot https://review.opendev.org/c/openstack/ironic/+/764323 | 14:47 |
| dtantsur | it becomes increasingly annoying to rebase this patch ^^^ | 14:47 |
| rpittau | mmm I thought I gave +2 already to that, oh well | 14:48 |
| rpittau | I need to split now, I'll check more stuff later or tomorrow | 14:48 |
| rpittau | have a great weekend everyone! o/ | 14:48 |
| *** rpittau is now known as rpittau|afk | 14:48 | |
| TheJulia | there is no slepeing in with a corgi overlord | 15:03 |
| TheJulia | dtantsur: congrats! | 15:04 |
| dtantsur | thanks and good morning :) | 15:04 |
| TheJulia | intermittent CI failures? | 15:06 |
| dtantsur | all sorts of, yeah | 15:08 |
| TheJulia | ugh, are we keeping a list? | 15:08 |
| iurygregory | TheJulia, the corgi still wants to participate in meetings? XD | 15:08 |
| dtantsur | on the whiteboard, yes | 15:09 |
| TheJulia | iurygregory: he sure participated on my call with my intern yesterday | 15:09 |
| iurygregory | hehehe | 15:09 |
| TheJulia | he was being extremely vocal | 15:09 |
| *** ociuhandu_ has quit IRC | 15:09 | |
| iurygregory | he was helping with debug? :D | 15:10 |
| TheJulia | it was more a mash up of corgi demanding the cat be removed from my desk | 15:13 |
| * TheJulia is unsure of this morning's coffee | 15:17 | |
| TheJulia | and the cat is back on my desk | 15:17 |
| TheJulia | There is potential! | 15:18 |
| trandles | cat on desk == ok | 15:18 |
| trandles | cat standing between me and monitor == not ok | 15:18 |
| trandles | unfortunately the cat seems to think that standing on the desk howling for attention is a good use of his and my time | 15:19 |
| TheJulia | So the cat is attracted to my desk because of the flying fiends... or birds visiting the Palm tree and orange tree outside my home office window | 15:21 |
| TheJulia | we *think* one of the other trees is a lemon tree but it doesn't seem to like the climate. | 15:21 |
| *** ociuhandu has joined #openstack-ironic | 15:22 | |
| iurygregory | hehehehe | 15:23 |
| iurygregory | and the corig is mad because the ca is on the desk | 15:23 |
| iurygregory | s/ca/cat | 15:23 |
| TheJulia | yup | 15:23 |
| trandles | between the cat and the window is a set of wire shelves covered in soon-to-be sprouting seedlings for the garden | 15:25 |
| trandles | he doesn't like that he can't stand on the shelves | 15:25 |
| iurygregory | awww =( | 15:25 |
| TheJulia | heh, smart | 15:25 |
| trandles | so I put a piece of cardboard on it so he can lay there for now, in the sun, watching the birds | 15:25 |
| TheJulia | Cats will eat leaves, so it is a good thing... espescialy if any of the plants are poisonous to them | 15:26 |
| trandles | I think trying to take that away when I start my seeds is going to piss him off | 15:26 |
| TheJulia | maybe make a "catainment box" | 15:26 |
| trandles | There are a few basil plants there now but he leaves them alone. | 15:26 |
| trandles | "catainment box" is a good idea! | 15:26 |
| TheJulia | spuc in 1.5 hours? | 15:29 |
| *** Qianbiao has quit IRC | 15:31 | |
| TheJulia | Interesting, driver libs being loaded illicit a failure in the acl tests :\ | 15:31 |
| TheJulia | Guess I need to look at that today :( | 15:31 |
| TheJulia | Has everyone voted on the bylaws changes? | 15:32 |
| openstackgerrit | Derek Higgins proposed openstack/ironic master: Add a delay/retry is vmedia insert fails https://review.opendev.org/c/openstack/ironic/+/770270 | 15:40 |
| dtantsur | I may end up falling asleep before SPUC | 15:40 |
| *** anuradha1904 has joined #openstack-ironic | 15:42 | |
| TheJulia | my body has already filed official protest to being awake | 15:44 |
| TheJulia | so... | 15:44 |
| openstackgerrit | Aija Jauntēva proposed openstack/ironic master: Add 'deploy steps' parameter for provisioning API https://review.opendev.org/c/openstack/ironic/+/768353 | 15:49 |
| *** juanoterocas has joined #openstack-ironic | 15:52 | |
| TheJulia | juanoterocas: did you get your question answered yesterday? | 15:55 |
| TheJulia | hmm, my acl test issue feels like the runner is getting slightly polluted :\ | 16:01 |
| iurygregory | SPUC is in 51min right? | 16:08 |
| TheJulia | yes | 16:08 |
| iurygregory | last friday I was at the doctor D: | 16:08 |
| iurygregory | today I will join \o/ | 16:09 |
| TheJulia | \o/ | 16:09 |
| TheJulia | I'll promise not to run... many unit tests while on the call today | 16:09 |
| iurygregory | let the unit tests run while you are on the call =D | 16:10 |
| openstackgerrit | Aija Jauntēva proposed openstack/ironic master: Add 'deploy steps' parameter for provisioning API https://review.opendev.org/c/openstack/ironic/+/768353 | 16:10 |
| TheJulia | it makes the computer cry | 16:10 |
| juanoterocas | Hi TheJulia, no I didn[t, still understading and struggling with : CRITICAL neutron.plugins.ml2.managers [-] The following mechanism drivers were not found: {'baremetal'} | 16:12 |
| iurygregory | I think you need a new computer TheJulia =( | 16:14 |
| TheJulia | juanoterocas: is networking-baremetal installed? | 16:15 |
| TheJulia | iurygregory: likely | 16:15 |
| juanoterocas | yes, up and running | 16:16 |
| TheJulia | networking-baremetal has two parts | 16:20 |
| TheJulia | one an ml2 plugin, the other a daemon. To install the ml2 plugin, it needs to be installed along side neutron-api if memory serves | 16:20 |
| TheJulia | Yeah, really seeming like the issue I have hit is one of those fun test runner issues | 16:22 |
| juanoterocas | some docs might be helpful about ml2 plugin, I have followed this doc https://docs.openstack.org/networking-baremetal/latest/install/index.html#install-the-networking-baremetal-plugin-and-agent | 16:23 |
| TheJulia | and your able to start ironic-neutron-agent? | 16:25 |
| juanoterocas | yes | 16:26 |
| juanoterocas | ● ironic-neutron-agent.service - OpenStack Ironic Neutron Agent | 16:27 |
| juanoterocas | Loaded: loaded (/etc/systemd/system/ironic-neutron-agent.service; enabled; vendor preset: enabled) | 16:27 |
| juanoterocas | Active: active (running) since Fri 2021-01-15 08:28:00 -05; 2h 51min ago | 16:27 |
| TheJulia | how exactly did you isntall networking-baremetal? | 16:28 |
| juanoterocas | pip command | 16:29 |
| TheJulia | with-in the neutron venv? | 16:29 |
| *** paras333_ has quit IRC | 16:31 | |
| juanoterocas | no, perhaps that is the issue, how I userneutron venv? | 16:31 |
| TheJulia | typically it is isntalled in a virtualenv with a specific path | 16:31 |
| TheJulia | so say, if the bonary running in is located in /opt/stack/neutron/bin/blah/blah | 16:32 |
| TheJulia | there should be an /opt/stack/neutron/bin/activate file to source to enter the venv | 16:32 |
| juanoterocas | ok, I'm using microstack (snap), so Iĺl assume is a virtual env as well and need to locate that activate executable. | 16:35 |
| *** ricolin_ has joined #openstack-ironic | 16:35 | |
| *** ricolin_ has quit IRC | 16:35 | |
| TheJulia | ahh, well good luck | 16:36 |
| openstackgerrit | Derek Higgins proposed openstack/ironic master: Add a delay/retry is vmedia insert fails https://review.opendev.org/c/openstack/ironic/+/770270 | 16:40 |
| * dtantsur slowly fades into the weekend | 16:41 | |
| *** dtantsur is now known as dtantsur|afk | 16:41 | |
| *** iurygregory has quit IRC | 16:41 | |
| *** ociuhandu_ has joined #openstack-ironic | 16:53 | |
| TheJulia | I think i figured out why my test was failing | 16:53 |
| *** ociuhandu has quit IRC | 16:56 | |
| *** ociuhandu_ has quit IRC | 16:58 | |
| *** lucasagomes has quit IRC | 17:00 | |
| TheJulia | is today a spuc? | 17:02 |
| openstackgerrit | Julia Kreger proposed openstack/ironic master: Start populating existing policy tests https://review.opendev.org/c/openstack/ironic/+/768136 | 17:05 |
| *** iurygregory has joined #openstack-ironic | 17:06 | |
| *** ociuhandu has joined #openstack-ironic | 17:11 | |
| *** ociuhandu has quit IRC | 17:16 | |
| arne_wiebalck | bye everyone, have a good week-end o/ | 17:31 |
| openstackgerrit | Merged openstack/ironic stable/victoria: Update `cleaning_error_handler` https://review.opendev.org/c/openstack/ironic/+/770257 | 17:36 |
| iurygregory | bye arne_wiebalck | 17:38 |
| derekh | sorry SPUC network problems got me at the end | 17:45 |
| viks____ | TheJulia: Now dhcp request from baremetal is hitting the neutron dhcp and i see some ack message going back via tcpdump... but in my baremetal KVM console, i see `PXE-E11: ARP timeout`? any idea what might be happening? | 17:56 |
| *** dsneddon has joined #openstack-ironic | 17:59 | |
| *** derekh has quit IRC | 18:03 | |
| *** gyee has joined #openstack-ironic | 18:14 | |
| *** jdandrea has quit IRC | 18:30 | |
| *** jdandrea has joined #openstack-ironic | 18:38 | |
| TheJulia | viks____: it sounds like the node doesn't have the parameters, can you confirm that the dhcp server reply (and thus the neutron port) have the parameters for network booting? | 19:12 |
| openstackgerrit | Julia Kreger proposed openstack/ironic master: Duplicate testing for system scoped ACL testing https://review.opendev.org/c/openstack/ironic/+/770002 | 19:17 |
| openstackgerrit | Julia Kreger proposed openstack/ironic master: Bump oslo.log requirement to 4.3.0 https://review.opendev.org/c/openstack/ironic/+/763256 | 19:18 |
| openstackgerrit | Julia Kreger proposed openstack/ironic master: Introduce common personas for secure RBAC https://review.opendev.org/c/openstack/ironic/+/763255 | 19:18 |
| viks____ | TheJulia: the ack from dhcp looks something like below: | 19:23 |
| viks____ | https://www.irccloud.com/pastebin/lknU4hpL/ | 19:24 |
| TheJulia | viks____: so Ironic, if controlling things, injects the various optionsf or network booting | 19:24 |
| TheJulia | fwiw, those parameters are a reply that can't start PXE | 19:25 |
| TheJulia | i.e. ironic has not been told or engaging that yet from what I can see | 19:26 |
| *** rcernin has joined #openstack-ironic | 19:27 | |
| viks____ | TheJulia: Do i need to have the ironic inspector running? how does PXE get the ramdisk from glance via PXE booting? am i doing something wrong? | 19:27 |
| *** rcernin has quit IRC | 19:31 | |
| openstackgerrit | Julia Kreger proposed openstack/ironic master: Introduce common personas for secure RBAC https://review.opendev.org/c/openstack/ironic/+/763255 | 19:31 |
| openstackgerrit | sonali bhausaheb borkar proposed x/sushy-oem-idrac master: Add get pxe port macs for bios mode https://review.opendev.org/c/x/sushy-oem-idrac/+/770904 | 19:34 |
| browzing | hello there *waves*. I'm curious what generates the deploy_ramdisk when a machine starts the 'provisioning' state. Our inspection phase is working properly w/ the initrd w/ proper firmware files symlinked in /shared/html/images but when a node starts provisioning something generates /shared/html/<node_uuid>/ with deploy_initrd that is missing all the required firmware | 20:01 |
| *** pmannidi has quit IRC | 20:03 | |
| TheJulia | viks____: Ironic does it, OR if your trying to do inspection you need a dedicated dhcp server for that *or* to have ironic trigger managed inspection (starting in ?train?... I think) | 20:13 |
| TheJulia | browzing: so ironic keeps a cached copy if the file doesn't change but there are cases where it won't. I'd kill/replace your ironic-conductor container | 20:14 |
| browzing | Eeek, killing it causes it to re-download the broken initrds from rdoproject.org and replaces my symlink fixes for the inspection phase | 20:20 |
| browzing | alternate question: is there some guaranteed stability or "don't break userspace" around releasing these images? | 20:20 |
| browzing | rough feedback: unfortunately we've burned 6 full working days and 2 engineers on this and it's not a great experience... | 20:21 |
| *** jdandrea has quit IRC | 20:29 | |
| TheJulia | so it gets cached in like /var/lib/ironic/master_images | 20:29 |
| TheJulia | you can nuke it from with-in the container | 20:29 |
| browzing | interesting, that's empty | 20:30 |
| browzing | ttps://imgur.com/a/JmvKtEv | 20:30 |
| TheJulia | I don't know about RDO specifically for their builds, but upstream we use the images we build for CI | 20:31 |
| TheJulia | symlink fixes? | 20:31 |
| TheJulia | if you take a look in /etc/ironic/ironic.conf it will have the paths to the folders used | 20:32 |
| TheJulia | the image it should be drawing from is the shared folder across the metal3 containers if memory serves | 20:32 |
| browzing | Yea we had a "workaround" for the inspection phase that gets inspection working by downloading the 2020-11-18 initrd and fixing all the symlinks to point at that in /shared/html/images | 20:32 |
| TheJulia | oh | 20:32 |
| TheJulia | heh | 20:32 |
| browzing | but it's using _something_ else and there's nothing else in /shared/html/images besides the initrd we want it to use... | 20:33 |
| TheJulia | so yeah, ironic isbasically copying that same file but you've hotwired inspection to draw from a different file it sounds like | 20:33 |
| TheJulia | ironic literally just copies the files from the shared folder and drops them in place | 20:34 |
| TheJulia | which reminds me... has that fix merged | 20:34 |
| TheJulia | ugh | 20:34 |
| TheJulia | no | 20:34 |
| browzing | interesting... yea our experience is not that | 20:35 |
| TheJulia | well, after first copy, it should be from the cache if memory serves | 20:35 |
| browzing | inspecting is definitely using the files in the shared folder, provisioning definitely is not | 20:35 |
| browzing | if i compare each ramdisk, the inspection ones has the firmware, the generated "provisioning" ones do not have required firmware | 20:36 |
| browzing | digging around for a cache... | 20:36 |
| TheJulia | It would really depend on what the baremetal operator is "populating" | 20:36 |
| TheJulia | if you can query ironic directly, look in the node driver_info field | 20:36 |
| TheJulia | it will have a deploy_kernel and deploy_ramdisk field which may be a file path or url | 20:37 |
| TheJulia | Unfortunately not conciously aware of the baremetal operator is doing | 20:37 |
| TheJulia | zaneb: o/ | 20:37 |
| zaneb | howdy | 20:37 |
| TheJulia | hmm image build failed on rdo | 20:37 |
| TheJulia | zaneb: is there any insight you can provide for browzing ? | 20:39 |
| zaneb | reading scrollback... | 20:39 |
| TheJulia | I don't know what the operator is putting in place at this point | 20:39 |
| zaneb | browzing: openshift or upstream metal³? | 20:41 |
| browzing | upstream metal3 | 20:41 |
| browzing | we have _most_ things pinned to a specific version but i think we didn't pin ironic-ipa-downloader and we've been having nothing but trouble because of that :p | 20:42 |
| *** zbitter has joined #openstack-ironic | 20:42 | |
| zaneb | best place to ask is probably #cluster-api-baremetal on k8s slack | 20:44 |
| browzing | done! https://kubernetes.slack.com/archives/CHD49TLE7/p1610742059008400 | 20:44 |
| browzing | we often ask questions there it's mostly crickets | 20:44 |
| trandles | I haven't gone looking around in the ironic database. Does it store any record of past deployments? Or would I have to gleen that info from logs? | 20:45 |
| *** jdandrea has joined #openstack-ironic | 20:46 | |
| TheJulia | trandles: sadly we don't store that past data. Nova *does* but i don't think you'r front ending it with nova | 20:47 |
| zaneb | browzing: it's pretty quiet. there's some folks in EU time zone who might have a better idea | 20:47 |
| trandles | I realized I can `baremetal node show` and see what is currently in instance_info, but that doesn't necessarily reflect what is currently provisioned on the node. | 20:47 |
| browzing | zaneb: yea, not too much hope though since it's friday there already :p | 20:47 |
| TheJulia | trandles: well, if it is provisioned... but if it is not | 20:47 |
| zaneb | most folks in US time are more familiar with openshift, and stuff like images and container versions work a bit differently in openshift because of reasons | 20:48 |
| trandles | TheJulia: the question was asked of me earlier this week if Ironic knows what a node is currently running. I said I didn't think so... | 20:48 |
| TheJulia | trandles: yes, someone could change the record :\ | 20:49 |
| TheJulia | someone could also change the host once deployed too | 20:49 |
| TheJulia | browzing: https://github.com/metal3-io/ironic-image/blob/master/ironic.conf.j2#L153 | 20:50 |
| browzing | 👀 | 20:51 |
| browzing | yea that path doesn't exist in the container :) however, /shared/html/images does and it's populated by stuff downloaded from the initContainer ironic-ipa-downloader script | 20:51 |
| browzing | (this baby https://github.com/metal3-io/ironic-ipa-downloader/blob/master/get-resource.sh) | 20:52 |
| TheJulia | check /shared/html/tmp too | 20:52 |
| browzing | nada `ls: cannot access /shared/html/tmp: No such file or directory` | 20:53 |
| * TheJulia blinks | 20:55 | |
| browzing | aah /shared/tmp based on the get-resource.sh script | 20:55 |
| *** rcernin has joined #openstack-ironic | 20:57 | |
| *** juanoterocas has quit IRC | 21:00 | |
| *** juanoterocas has joined #openstack-ironic | 21:00 | |
| *** rcernin has quit IRC | 21:01 | |
| *** JamesBenson has quit IRC | 21:03 | |
| TheJulia | lbragstad: you wouldn't happen to have an example system member token... would you? | 21:15 |
| TheJulia | also system project member would be helpful | 21:15 |
| *** juanoterocas has quit IRC | 21:17 | |
| *** juanoterocas has joined #openstack-ironic | 21:18 | |
| *** juanoterocas has quit IRC | 21:18 | |
| TheJulia | hmm, I think I see what is going on | 21:23 |
| *** rcernin has joined #openstack-ironic | 21:53 | |
| *** rcernin has quit IRC | 21:58 | |
| *** jdandrea has quit IRC | 22:01 | |
| lbragstad | TheJulia you want the raw token response? | 22:02 |
| lbragstad | TheJulia system admin: http://paste.openstack.org/show/801679/ | 22:04 |
| TheJulia | lbragstad: it is the templating, having tomodify things around some | 22:04 |
| *** rcernin has joined #openstack-ironic | 22:04 | |
| lbragstad | http://paste.openstack.org/show/801680/ system-member | 22:04 |
| lbragstad | http://paste.openstack.org/show/801681/ is system-reader | 22:05 |
| lbragstad | those are the raw token response bodies | 22:05 |
| lbragstad | if you have a devstack laying around you can use `openstack token issue --os-cloud devstack-system-admin --debug` to get one | 22:06 |
| TheJulia | it has been a while since I've had a devstack setup :( | 22:18 |
| *** ociuhandu has joined #openstack-ironic | 22:20 | |
| *** ociuhandu has quit IRC | 22:25 | |
| TheJulia | getting there | 22:29 |
| *** sdanni has quit IRC | 22:37 | |
| *** erbarr has quit IRC | 22:38 | |
| *** erbarr has joined #openstack-ironic | 22:38 | |
| *** logan- has quit IRC | 22:40 | |
| *** logan- has joined #openstack-ironic | 22:53 | |
| *** yoctozepto5 has joined #openstack-ironic | 22:54 | |
| *** yoctozepto has quit IRC | 22:55 | |
| *** yoctozepto5 is now known as yoctozepto | 22:55 | |
| *** nandita has quit IRC | 22:57 | |
| *** vdrok has quit IRC | 22:57 | |
| *** nandita has joined #openstack-ironic | 22:57 | |
| *** vdrok has joined #openstack-ironic | 22:58 | |
| *** TheJulia has quit IRC | 22:58 | |
| *** TheJulia has joined #openstack-ironic | 22:58 | |
| *** rh-jelabarre has quit IRC | 22:59 | |
| *** rh-jelabarre has joined #openstack-ironic | 22:59 | |
| *** mkowalski has quit IRC | 23:00 | |
| *** coreycb has quit IRC | 23:01 | |
| *** coreycb has joined #openstack-ironic | 23:02 | |
| *** ftarasenko has quit IRC | 23:03 | |
| *** logan- has quit IRC | 23:04 | |
| *** mkowalski has joined #openstack-ironic | 23:13 | |
| *** yoctozepto6 has joined #openstack-ironic | 23:16 | |
| *** yoctozepto has quit IRC | 23:17 | |
| *** yoctozepto6 is now known as yoctozepto | 23:17 | |
| *** logan- has joined #openstack-ironic | 23:19 | |
| *** eandersson has quit IRC | 23:20 | |
| *** ddurst has quit IRC | 23:20 | |
| *** ddurst has joined #openstack-ironic | 23:21 | |
| *** eandersson has joined #openstack-ironic | 23:21 | |
| *** rloo has quit IRC | 23:27 | |
| *** yoctozepto4 has joined #openstack-ironic | 23:42 | |
| *** yoctozepto has quit IRC | 23:44 | |
| *** yoctozepto4 is now known as yoctozepto | 23:44 | |
| *** yoctozepto4 has joined #openstack-ironic | 23:58 | |
| *** yoctozepto has quit IRC | 23:59 | |
| *** yoctozepto4 is now known as yoctozepto | 23:59 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!