| *** irclogbot_2 has joined #openstack-ironic | 00:22 | |
| *** irclogbot_2 has quit IRC | 00:39 | |
| *** kaifeng has joined #openstack-ironic | 00:47 | |
| *** kaifeng has left #openstack-ironic | 00:47 | |
| *** kaifeng has joined #openstack-ironic | 00:47 | |
| *** stevebaker_ has joined #openstack-ironic | 00:47 | |
| *** irclogbot_2 has joined #openstack-ironic | 01:26 | |
| *** stevebaker_ has quit IRC | 01:33 | |
| *** stevebaker_ has joined #openstack-ironic | 01:41 | |
| *** irclogbot_2 has quit IRC | 01:48 | |
| *** ddurst has quit IRC | 01:50 | |
| *** ddurst has joined #openstack-ironic | 01:52 | |
| *** irclogbot_2 has joined #openstack-ironic | 02:47 | |
| *** bobmel has joined #openstack-ironic | 03:17 | |
| *** bobmel has quit IRC | 03:22 | |
| *** chenhaw has joined #openstack-ironic | 03:25 | |
| *** mkrai has joined #openstack-ironic | 03:52 | |
| *** ricolin has joined #openstack-ironic | 04:02 | |
| goldyfruit_ | Ok, my issue was definitely due to Python 2.7 | 05:03 |
|---|---|---|
| goldyfruit_ | I'm using Debian 10 to build IPA which has 2.7 by default | 05:03 |
| goldyfruit_ | Python 2.7 support has been dropped from ironic-python-agent and ironic-python-agent-builder | 05:05 |
| mkrai | Hi goldyfruit_ good morning | 05:44 |
| mkrai | goldyfruit_, What issue are you facing? | 05:44 |
| *** rcernin has quit IRC | 07:12 | |
| *** bobmel has joined #openstack-ironic | 07:18 | |
| *** bobmel has quit IRC | 07:23 | |
| *** belmoreira has joined #openstack-ironic | 07:35 | |
| arne_wiebalck | Good morning, ironic! | 07:37 |
| mkrai | good morning ironic | 07:54 |
| mkrai | arne_wiebalck, Hi gm o/ | 07:54 |
| arne_wiebalck | Hey mkrai o/ | 07:55 |
| mkrai | arne_wiebalck, which deploy images do you use for your deployment? | 07:55 |
| arne_wiebalck | mkrai: you mean for the instances? | 07:55 |
| mkrai | arne_wiebalck, yes | 07:55 |
| arne_wiebalck | mkrai: the majority is centos | 07:56 |
| mkrai | arne_wiebalck, did you build them? | 07:57 |
| mkrai | arne_wiebalck, I have tried multiple deploy images but my provisioning is unsuccesful | 07:57 |
| arne_wiebalck | mkrai: not personally, but yes they have been rebuilt | 07:57 |
| arne_wiebalck | mkrai: I've tried a Ubuntu cloud image the other day, worked fine | 07:58 |
| mkrai | arne_wiebalck, can you share the link? | 07:58 |
| arne_wiebalck | mkrai: still issues with the networking? | 07:58 |
| openstackgerrit | Ankit Kumar proposed openstack/ironic master: Fixes issue with checking whether ISO is passed https://review.opendev.org/697953 | 07:59 |
| mkrai | arne_wiebalck, I am facing 2 issues at the moment | 07:59 |
| mkrai | arne_wiebalck, One one setup the node is active but can't ssh | 07:59 |
| mkrai | arne_wiebalck, on other setup, the cleaning fails because of some image format issue | 07:59 |
| arne_wiebalck | mkrai: I was using a the uefi image from here: https://cloud-images.ubuntu.com/xenial/current/ | 08:00 |
| mkrai | arne_wiebalck, these are user images, right? | 08:00 |
| arne_wiebalck | mkrai: yes | 08:00 |
| mkrai | arne_wiebalck, in my case, the issue is IPA is not running. SO I tried to use different deploy images. | 08:00 |
| mkrai | arne_wiebalck, which deploy images did you use? | 08:01 |
| arne_wiebalck | mkrai: ah, so you mean the deploy RAM disk image? | 08:01 |
| *** igordc has joined #openstack-ironic | 08:01 | |
| mkrai | arne_wiebalck, yes :) | 08:01 |
| arne_wiebalck | mkrai: ours is based on centos as well | 08:01 |
| arne_wiebalck | mkrai: and we built it ourselves, yes | 08:01 |
| mkrai | arne_wiebalck, ok | 08:03 |
| mkrai | arne_wiebalck, http://paste.openstack.org/show/787298/ here's the error | 08:03 |
| arne_wiebalck | mkrai: with diskimage-builder | 08:03 |
| arne_wiebalck | mkrai: can the conductor ping the node that has booted the IPA image? | 08:04 |
| mkrai | No | 08:05 |
| mkrai | arne_wiebalck, No | 08:05 |
| arne_wiebalck | mkrai: but the node boots into the IPA image? | 08:06 |
| mkrai | arne_wiebalck, one question i find something weird | 08:06 |
| mkrai | arne_wiebalck, HTTPConnectionPool(host='10.228.161.154', port=9999) | 08:06 |
| mkrai | arne_wiebalck, what IP is this supposed to be? | 08:06 |
| mkrai | arne_wiebalck, Yes node booted IPA | 08:07 |
| mkrai | arne_wiebalck, so I understood what's the issue here. | 08:08 |
| arne_wiebalck | mkrai: I'm not familiar with that particular error message, but I think this should be the IP of the node with the IPA | 08:08 |
| mkrai | arne_wiebalck, right | 08:08 |
| arne_wiebalck | mkrai: what I'd suggest is to put the node into maintenance, send it into cleaning, log into the node, tap the log, remove the maintenance, and see what happens | 08:09 |
| mkrai | arne_wiebalck, there are two interfaces with the instance, ironic booted with interface 1 on neutron network, deployment was succesful. | 08:09 |
| mkrai | arne_wiebalck, on final poweron, interface 1 tried to do PXE boot again and it failed with "No boot filename received" | 08:10 |
| *** sshnaidm|off is now known as sshnaidm | 08:11 | |
| mkrai | arne_wiebalck, hence it tried to boot from interface 2 which is from non-neutron dhcp and got IP "10.228.161.154". ANd Ironic tried to check agent on this interface which is wrong. | 08:11 |
| mkrai | arne_wiebalck, sorry for the long messages :( | 08:11 |
| arne_wiebalck | mkrai: ah, ok | 08:11 |
| arne_wiebalck | mkrai: so the instance is sucessfully deployed, but comes up with the wrong IP? | 08:12 |
| mkrai | arne_wiebalck, yes | 08:13 |
| * arne_wiebalck has to join a meeting in 2 mins | 08:13 | |
| mkrai | arne_wiebalck, ok please ping me whenever you're free :) | 08:13 |
| mkrai | arne_wiebalck, I need help to fix this issue. Thanks alot for your help | 08:13 |
| arne_wiebalck | mkrai: so, I'd think either noone should reply to the 2nd dhcp request, or the node needs to be configured for local boot, no? | 08:14 |
| * arne_wiebalck has to go, ttyl | 08:14 | |
| mkrai | arne_wiebalck, right. Bye! | 08:15 |
| *** jtomasek has joined #openstack-ironic | 08:15 | |
| *** tesseract has joined #openstack-ironic | 08:30 | |
| *** iurygregory has joined #openstack-ironic | 08:31 | |
| *** tesseract has quit IRC | 08:31 | |
| iurygregory | good morning Ironic! | 08:31 |
| *** tesseract has joined #openstack-ironic | 08:31 | |
| *** igordc has quit IRC | 08:36 | |
| *** khansa has joined #openstack-ironic | 09:01 | |
| mkrai | good morning iurygregory o/ | 09:04 |
| iurygregory | mkrai, o/ | 09:04 |
| *** dougsz has joined #openstack-ironic | 09:05 | |
| *** yolanda has joined #openstack-ironic | 09:09 | |
| *** rcernin has joined #openstack-ironic | 09:11 | |
| *** rpittau|afk is now known as rpittau | 09:15 | |
| rpittau | good morning ironic! o/ | 09:15 |
| iurygregory | rpittau, o/ | 09:15 |
| rpittau | hey iurygregory :) | 09:15 |
| khansa | good morning guys o/ | 09:15 |
| rpittau | hey khansa, good morning :) | 09:16 |
| iurygregory | morning khansa | 09:16 |
| *** dougsz has quit IRC | 09:21 | |
| rpittau | goldyfruit_: hope everything's ok just switching to Python 3 :) | 09:29 |
| *** dougsz has joined #openstack-ironic | 09:34 | |
| *** derekh has joined #openstack-ironic | 09:38 | |
| *** ijw has joined #openstack-ironic | 09:39 | |
| * rpittau insert more coffee to continue... | 09:40 | |
| *** ijw has quit IRC | 09:43 | |
| *** mgoddard has joined #openstack-ironic | 09:46 | |
| *** belmoreira has quit IRC | 09:53 | |
| *** belmoreira has joined #openstack-ironic | 09:57 | |
| *** mkrai has quit IRC | 09:58 | |
| *** mkrai has joined #openstack-ironic | 09:59 | |
| *** dtantsur|afk is now known as dtantsur | 10:02 | |
| *** mkrai has quit IRC | 10:04 | |
| *** rcernin has quit IRC | 10:14 | |
| *** csmart has quit IRC | 10:16 | |
| *** mkowalski has quit IRC | 10:16 | |
| *** ajya[m] has quit IRC | 10:16 | |
| *** belmoreira has quit IRC | 10:18 | |
| *** belmoreira has joined #openstack-ironic | 10:21 | |
| openstackgerrit | Rachit Kapadia proposed openstack/ironic master: Change to bug fix instead of features to follow-on idrac job patch. https://review.opendev.org/697972 | 10:48 |
| openstackgerrit | Rachit Kapadia proposed openstack/ironic master: Changed to bug fix to follow-on idrac job patch. https://review.opendev.org/697972 | 10:54 |
| *** jtomasek has quit IRC | 10:54 | |
| *** jtomasek has joined #openstack-ironic | 10:56 | |
| *** jtomasek has quit IRC | 10:57 | |
| *** jtomasek has joined #openstack-ironic | 10:57 | |
| *** mkrai has joined #openstack-ironic | 11:17 | |
| *** Lucas_Gray has joined #openstack-ironic | 11:20 | |
| *** bobmel has joined #openstack-ironic | 11:20 | |
| *** bobmel has quit IRC | 11:26 | |
| * etingof is waking up his Dell machine by pouring some coffee into it | 11:34 | |
| arne_wiebalck | etingof: non-decaf == turbo button | 11:39 |
| dtantsur | ++ | 11:39 |
| *** mkrai has quit IRC | 11:39 | |
| * etingof goes straight with Biohazard Coffee | 11:42 | |
| *** pcaruana has joined #openstack-ironic | 11:46 | |
| *** mkowalski has joined #openstack-ironic | 11:48 | |
| *** ajya[m] has joined #openstack-ironic | 11:48 | |
| *** csmart has joined #openstack-ironic | 11:48 | |
| *** stevebaker has quit IRC | 11:57 | |
| *** stevebaker has joined #openstack-ironic | 12:05 | |
| *** mkrai has joined #openstack-ironic | 12:14 | |
| *** belmoreira has quit IRC | 12:22 | |
| *** tzumainn has joined #openstack-ironic | 12:25 | |
| iurygregory | dtantsur, if you have some time can you check Bob's comment on https://review.opendev.org/#/c/696914/2/ironic_python_agent/extensions/image.py I'm not sure if I fully understood =) | 12:31 |
| patchbot | patch 696914 - ironic-python-agent - Avoid grub2-install when on UEFI boot mode - 2 patch sets | 12:31 |
| dtantsur | iurygregory: you mean, re chroot? | 12:47 |
| iurygregory | dtantsur, nope, about the efibootmgr | 12:48 |
| iurygregory | about the chroot TheJulia said she doesn't think we need | 12:49 |
| dtantsur | iurygregory: then I'm not sure which comment you have in mind | 12:49 |
| *** bfournie has quit IRC | 12:50 | |
| dtantsur | about installing efibootmgr on the ramdisk? yes, you need it (otherwise how will it work?) | 12:50 |
| iurygregory | yeah | 12:50 |
| dtantsur | not sure if it's available for tinycorelinux, but we should add it to DIB | 12:50 |
| iurygregory | so i need to add to ipa-builder ? | 12:50 |
| dtantsur | iurygregory: yep | 12:52 |
| iurygregory | dtantsur, tks going to look how to do this o/ | 12:52 |
| dtantsur | iurygregory: see pkg-map/package-installs.yaml | 12:53 |
| dtantsur | https://docs.openstack.org/diskimage-builder/latest/elements/package-installs/README.html | 12:53 |
| iurygregory | dtantsur++ | 12:53 |
| dtantsur | and https://docs.openstack.org/diskimage-builder/latest/elements/pkg-map/README.html | 12:53 |
| goldyfruit_ | mkrai, rpittau I don't I'll have to remove all Python 2.7 related stuff on the Debian | 12:58 |
| dtantsur | TheJulia: I wonder if any 3rd party CI is working atm.. | 13:11 |
| dtantsur | TheJulia: it feels like only Dell and IBM are running at all, and both are red :( | 13:11 |
| *** rh-jelabarre has joined #openstack-ironic | 13:16 | |
| openstackgerrit | Dmitry Tantsur proposed openstack/ironic-python-agent master: DNM testing the CI https://review.opendev.org/698010 | 13:17 |
| *** goldyfruit_ has quit IRC | 13:19 | |
| *** Lucas_Gray has quit IRC | 13:26 | |
| openstackgerrit | Dmitry Tantsur proposed openstack/ironic-tempest-plugin master: Actually test rescue in the standalone job https://review.opendev.org/698011 | 13:27 |
| dtantsur | TheJulia: also w00t ^^^ :( | 13:27 |
| rpittau | iurygregory: if it can help you, efibootmgr exists as package in tinycore as efibootmgr.tcz | 13:34 |
| openstackgerrit | Dmitry Tantsur proposed openstack/ironic master: CI: disable IPv6 in neutron https://review.opendev.org/698012 | 13:35 |
| dtantsur | double sigh at ^^ | 13:35 |
| iurygregory | rpittau, tks! | 13:35 |
| *** Goneri has joined #openstack-ironic | 13:35 | |
| rpittau | dtantsur: was going to ask about the failures :/ | 13:36 |
| dtantsur | well.. devstack uses dual-stack v4-v6 networking now. we're not ready for that. | 13:36 |
| dtantsur | hjensas if one day you could help us untangle the mess ^^ it would be awesome | 13:37 |
| *** bfournie has joined #openstack-ironic | 13:42 | |
| *** ociuhandu has joined #openstack-ironic | 13:51 | |
| *** xXraphXx has joined #openstack-ironic | 13:55 | |
| TheJulia | good morning | 13:56 |
| TheJulia | I'm not sure how we can peel that back short of making ironic grok v4/46 suubnets in parallel | 13:57 |
| * TheJulia needs... lots of coffee coffee | 13:57 | |
| * rpioso suggests to etingof devstack local boot as Dell server’s preferred Monday grind | 13:58 | |
| TheJulia | misbehaving mirrors this morning | 13:59 |
| TheJulia | maybe I should just go back to bed | 13:59 |
| openstackgerrit | Iury Gregory Melo Ferreira proposed openstack/ironic-python-agent master: Avoid grub2-install when on UEFI boot mode https://review.opendev.org/696914 | 14:00 |
| TheJulia | ugh, 4 meetings this morning | 14:01 |
| rpittau | I'm also seeing jobs failing because of password not allowed as authentication method in ipa: | 14:01 |
| rpittau | https://89a28d92de3b4c8c3017-1438e56e418f3d4087dd94ee6330f7d7.ssl.cf5.rackcdn.com/697953/1/check/ironic-tempest-ipa-partition-bios-agent_ipmitool-tinyipa-indirect/c311892/controller/logs/tempest_log.txt.gz | 14:01 |
| TheJulia | rpioso: if in rescue testing, it is another symptom of ipa never exiting | 14:02 |
| iurygregory | good morning TheJulia | 14:03 |
| TheJulia | err | 14:03 |
| TheJulia | rppi | 14:03 |
| TheJulia | rpittau: | 14:03 |
| * TheJulia just focuses on on coffee first | 14:03 | |
| iurygregory | ^ ++ | 14:03 |
| rpittau | TheJulia: yeah, so far I saw it in rescue testing only | 14:04 |
| TheJulia | the bug I filed on that only covered the console log because it is fairly clear what actually happened once you find the vm log | 14:05 |
| TheJulia | and then double check the code :( | 14:05 |
| *** rloo has joined #openstack-ironic | 14:06 | |
| dtantsur | morning TheJulia | 14:14 |
| dtantsur | yeah, "thanks" to standalone jobs not actually testing rescue (see above).. | 14:14 |
| iurygregory | Am I the only one who can't access the logs from Dell CI on https://review.opendev.org/#/c/698012/ or in any other patches? XD I'm getting 404 | 14:24 |
| patchbot | patch 698012 - ironic - CI: disable IPv6 in neutron - 1 patch set | 14:24 |
| TheJulia | dtantsur: I don't blame them, with the neutron port flips it is the most unstable feature we have because neturon can be super slow about completing port changes | 14:31 |
| *** goldyfruit has joined #openstack-ironic | 14:44 | |
| *** cdearborn has joined #openstack-ironic | 14:44 | |
| *** kaifeng_ has joined #openstack-ironic | 14:51 | |
| *** jerrywan1 has joined #openstack-ironic | 14:54 | |
| *** jerrywan1 has quit IRC | 14:59 | |
| TheJulia | #startmeeting ironic | 15:00 |
| openstack | Meeting started Mon Dec 9 15:00:13 2019 UTC and is due to finish in 60 minutes. The chair is TheJulia. Information about MeetBot at http://wiki.debian.org/MeetBot. | 15:00 |
| iurygregory | o/ | 15:00 |
| TheJulia | o/ | 15:00 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 15:00 |
| *** openstack changes topic to " (Meeting topic: ironic)" | 15:00 | |
| openstack | The meeting name has been set to 'ironic' | 15:00 |
| *** beekneemech is now known as bnemec | 15:00 | |
| kaifeng_ | o/ | 15:00 |
| rpittau | o/ | 15:00 |
| * iurygregory was too fast | 15:00 | |
| *** root has joined #openstack-ironic | 15:00 | |
| TheJulia | heh | 15:00 |
| arne_wiebalck | o/ | 15:00 |
| TheJulia | \o | 15:00 |
| *** root is now known as Guest23212 | 15:00 | |
| TheJulia | Good morning everyone! | 15:00 |
| Guest23212 | o/ | 15:01 |
| rpioso | o/ | 15:01 |
| mgoddard | \o | 15:01 |
| Guest23212 | jerrywang1 | 15:01 |
| cdearborn | o/ | 15:01 |
| khansa | o/ | 15:01 |
| *** Guest23212 is now known as jerrywang1 | 15:01 | |
| TheJulia | Our meeting agenda can be found on the wiki! | 15:01 |
| TheJulia | #link https://wiki.openstack.org/wiki/Meetings/Ironic#Agenda_for_next_meeting | 15:01 |
| jerrywang1 | o/ | 15:01 |
| TheJulia | #topic Announcements/Reminders | 15:02 |
| *** openstack changes topic to "Announcements/Reminders (Meeting topic: ironic)" | 15:02 | |
| bfournie | o/ | 15:02 |
| hjensas | o/ | 15:03 |
| TheJulia | On the announcements front, It is fairly clear CI is very broken right now and the default IPv6 in the flat networks in devstack have sort of made things less reliable. | 15:03 |
| TheJulia | dtantsur: do we have a bug filed for ^ | 15:03 |
| dtantsur | not from me, sorry | 15:03 |
| TheJulia | dtantsur: c'est la vie | 15:03 |
| dtantsur | but I have a patch | 15:03 |
| dtantsur | #link https://review.opendev.org/698012 potential CI fix | 15:03 |
| patchbot | patch 698012 - ironic - CI: disable IPv6 in neutron - 1 patch set | 15:03 |
| TheJulia | On the reminder front, CERN has offered to host a midcycle. There is a sign-up page. | 15:03 |
| dtantsur | let's recheck it once before approving though | 15:03 |
| TheJulia | arne_wiebalck: if you hav the link handy | 15:04 |
| TheJulia | dtantsur: thanks | 15:04 |
| arne_wiebalck | https://indico.cern.ch/event/863986/ | 15:04 |
| rpioso | arne_wiebalck: Thank you! | 15:04 |
| TheJulia | #link https://indico.cern.ch/event/863986/ | 15:04 |
| arne_wiebalck | please register if you plan to attend | 15:04 |
| TheJulia | Does anyone have anything else to announce or remind us of? | 15:04 |
| * rpioso thinks about his holiday shopping | 15:04 | |
| iurygregory | Dell CI is not with accessible logs =( | 15:04 |
| TheJulia | rajinir: ^^^ | 15:05 |
| dtantsur | I'm not sure any 3rd party CI is working atm :( | 15:05 |
| iurygregory | HP CI I haven't saw reports in a while since they were with POST_FAILURE and no answer to my email =( | 15:05 |
| rpioso | iurygregory: I've informed rajinir thru our internal IM system. | 15:05 |
| iurygregory | dtantsur, yeah | 15:05 |
| dtantsur | I've mostly see Dell and IBM recently | 15:05 |
| dtantsur | would be great to get the status of the others | 15:05 |
| TheJulia | I'll see if I can find some time to follow-up on 3rd party CIs this week. | 15:05 |
| TheJulia | iurygregory: if you can forward that email to me, that would be appreciated. | 15:06 |
| TheJulia | rpittau: thanks | 15:06 |
| iurygregory | TheJulia, sure! | 15:06 |
| TheJulia | err | 15:06 |
| TheJulia | rpioso: thanks | 15:06 |
| TheJulia | Okay! sounds like we can move on then! | 15:06 |
| rpioso | TheJulia: :-) | 15:06 |
| TheJulia | oh, wait | 15:06 |
| TheJulia | rpioso raises a good point about holidays. Around the 15th people tend to start disappearing for two to three weeks as end of year deadlines, training, and time off begin. | 15:07 |
| TheJulia | If there are any cores that will be available during for the next few weeks, even if it is "I have to wave a special flag to get attention" | 15:08 |
| dtantsur | I'm out 16th-1st | 15:08 |
| TheJulia | That would be helpful to at least get CI fixes merged without blocking the entire project until the end of the first week of Janurary | 15:08 |
| TheJulia | dtantsur: k | 15:08 |
| TheJulia | I've got no specific time off planned, but I'll surely become less responsive the 24th-1st | 15:08 |
| dtantsur | Can be pinged mid-holidays after 22nd via personal means | 15:08 |
| rpittau | I will be available for the next 2 weeks, except on fridays, and then available also 23rd 24th, then out until 1st | 15:09 |
| TheJulia | dtantsur: thanks | 15:09 |
| openstackgerrit | Iury Gregory Melo Ferreira proposed openstack/ironic-python-agent-builder master: Add efibootmgr https://review.opendev.org/698026 | 15:09 |
| arne_wiebalck | I should be around most of the time. | 15:09 |
| TheJulia | That at least helps. I suspect our next meeting will be the last for a couple weeks since it may not make sense to hold them but perhaps that is better off as an open discussion topic | 15:09 |
| TheJulia | #topic Review Action Items from the prior week | 15:10 |
| *** openstack changes topic to "Review Action Items from the prior week (Meeting topic: ironic)" | 15:10 | |
| TheJulia | I didn't get to my one and only action item, and it seems I have two now. :) | 15:10 |
| TheJulia | #action TheJulia follow-up on third party CI | 15:10 |
| TheJulia | #action TheJulia email the foundation regarding case studies | 15:10 |
| *** ociuhandu has quit IRC | 15:11 | |
| TheJulia | ^^^ arne_wiebalck I have more motivation to do that since redhat's submission fell in my lap this past week. | 15:11 |
| TheJulia | #topic Review subteam status reports | 15:11 |
| *** openstack changes topic to "Review subteam status reports (Meeting topic: ironic)" | 15:11 | |
| TheJulia | #link https://etherpad.openstack.org/p/IronicWhiteBoard | 15:12 |
| TheJulia | Starting at line 273 | 15:12 |
| TheJulia | looks like we have some todo's from last week | 15:13 |
| * TheJulia suspects everyone was really busy last week | 15:13 | |
| arne_wiebalck | TheJulia: :) | 15:14 |
| TheJulia | Node retirement spec merged \o/ | 15:14 |
| iurygregory | \o/ | 15:15 |
| arne_wiebalck | \o/ | 15:15 |
| TheJulia | tzumainn: I think I saw you were going to take on adding policy code around the node.owner field? | 15:16 |
| dtantsur | this ^^ plus I suggest banning changing node.owner if an owned allocation exists | 15:17 |
| dtantsur | * for this node | 15:17 |
| TheJulia | I think that really covers it for things that are getting attention right now. | 15:17 |
| tzumainn | TheJulia, yep! | 15:17 |
| TheJulia | dtantsur: reasonable... I think | 15:18 |
| *** stendulker has joined #openstack-ironic | 15:18 | |
| TheJulia | o/ stendulker | 15:18 |
| stendulker | o/ | 15:18 |
| stendulker | Sorry, I'm late. | 15:18 |
| TheJulia | No worries | 15:18 |
| TheJulia | Are we good to proceed on from the reviewing statuses and shift to priorities for this coming week? | 15:19 |
| iurygregory | I think we are | 15:19 |
| rpittau | let's | 15:19 |
| rajinir | iurygregory,TheJulia> ,rpioso, dtansur Dell CI is down, I have updated the Ironic Whiteboard. There seems to be some networking issue with our labs. We are working on it. | 15:20 |
| dtantsur | thanks rajinir | 15:20 |
| iurygregory | rajinir, tks! | 15:20 |
| rloo | o/ sheepish... | 15:20 |
| dtantsur | stendulker: hi, do you know the HPE CI status? could you update the whitboard if so? | 15:20 |
| TheJulia | Okay, onward to Priorites! | 15:21 |
| TheJulia | #topic Priorities for the coming week | 15:21 |
| *** openstack changes topic to "Priorities for the coming week (Meeting topic: ironic)" | 15:21 | |
| stendulker | yes, its down. Will update. There seems some issue with the devstack image creation. Newly cerated images do not work. It does not take static image. | 15:21 |
| TheJulia | #link https://etherpad.openstack.org/p/IronicWhiteBoard | 15:21 |
| stendulker | Need thsi patch for iLO CI https://review.opendev.org/697953 | 15:22 |
| patchbot | patch 697953 - ironic - Fixes issue with checking whether ISO is passed - 1 patch set | 15:22 |
| TheJulia | Starting at line 169 | 15:22 |
| dtantsur | I could use reviews on https://review.opendev.org/#/c/697451/ | 15:24 |
| patchbot | patch 697451 - ironic - Correct power state handling for managed in-band i... - 1 patch set | 15:24 |
| TheJulia | Looks like the attributeerror fix is now into stable backports. link added | 15:24 |
| * dtantsur added under managed inspection | 15:24 | |
| TheJulia | dtantsur: thanks | 15:24 |
| dtantsur | also https://review.opendev.org/#/c/698011/ will hopefully prevent us from breaking rescue again | 15:25 |
| patchbot | patch 698011 - ironic-tempest-plugin - Actually test rescue in the standalone job - 1 patch set | 15:25 |
| dtantsur | (also added) | 15:25 |
| * iurygregory adds links for ipa and ipa-builder patches =) | 15:25 | |
| TheJulia | dtantsur: thanks! | 15:27 |
| TheJulia | I went ahead and listed a needing feedback item on a WIP I have up | 15:27 |
| TheJulia | Adding IPA<-> Conductor authentication, the very first patch.. Once the gate is happier, it should actually pass CI without any issues. | 15:28 |
| TheJulia | Does anyone have anything else that needs to go on this list? | 15:28 |
| kaifeng_ | i have a patch on the stable branch | 15:29 |
| kaifeng_ | https://review.opendev.org/697775 | 15:29 |
| patchbot | patch 697775 - ironic (stable/train) - Explicitly enable ipxe as boot interface when it's... - 1 patch set | 15:29 |
| rloo | kaifeng_: is there a bug/story associated with that? it isn't a backport? | 15:30 |
| dtantsur | CI change, so a story is optional | 15:31 |
| kaifeng_ | it's not a backport, but as I understand it, is required for the grenade on master | 15:31 |
| rloo | dtantsur: ah, i didn't look 'below the fold' (Just read the commit stuff.) | 15:31 |
| TheJulia | Seems reasonable, but I'm wondering if master has been changed | 15:31 |
| dtantsur | kaifeng_: it seems that master requires the same change | 15:31 |
| TheJulia | oh, that actually makes sense then because of grenade | 15:31 |
| kaifeng_ | the change on master is https://review.opendev.org/697634 | 15:32 |
| patchbot | patch 697634 - ironic - Fix ipxe interface to perform ipxe boot without ip... - 2 patch sets | 15:32 |
| dtantsur | kaifeng_: could you split them apart then? | 15:32 |
| kaifeng_ | which actually fix the issue in the commit message | 15:32 |
| dtantsur | I mean, apply the CI change to master, backport it, then apply this? | 15:32 |
| dtantsur | (and probably also backport) | 15:33 |
| TheJulia | the ironic change was already made | 15:33 |
| TheJulia | in the patch kaifeng_ just linked | 15:33 |
| dtantsur | TheJulia: yep, but it needs to pass the CI | 15:33 |
| dtantsur | and it won't until we patch train. which needs patching master, then backporting. | 15:33 |
| dtantsur | am I missing something? | 15:33 |
| rloo | i'm confused. why doesn't the change in master (https://review.opendev.org/#/c/697634/) have a story/bug associated with it? | 15:33 |
| patchbot | patch 697634 - ironic - Fix ipxe interface to perform ipxe boot without ip... - 2 patch sets | 15:33 |
| TheJulia | It is a chicken/egg situation with grenade, the fix to the older branch needs to land first | 15:34 |
| rloo | it has a release note? | 15:34 |
| dtantsur | TheJulia: not if you split out the CI part only, I think | 15:34 |
| dtantsur | at least I wonder if it has been tried | 15:34 |
| TheJulia | Oh yeah, possibly | 15:34 |
| TheJulia | as the first of two changes | 15:34 |
| rloo | i'm ok with a fix in older branch being needed, to support a new change. but i don't know why the new change doesn't have much info... | 15:34 |
| rloo | if there is a story, the fix in older branch can reference it | 15:35 |
| kaifeng_ | the thing is ipxe without ipxe_enabled=True is not working | 15:35 |
| rloo | kaifeng_: is there a story/bug to track that? | 15:36 |
| kaifeng_ | so my thought is to make decouple that dependency | 15:36 |
| rpittau | kaifeng_: I think it would be easier if we could reference a Story/Task to it | 15:36 |
| dtantsur | I think I understand what kaifeng_ is trying to do, but I'd still prefer https://review.opendev.org/#/c/697775/ applied to master first, then to train. | 15:36 |
| patchbot | patch 697775 - ironic (stable/train) - Explicitly enable ipxe as boot interface when it's... - 1 patch set | 15:36 |
| TheJulia | Yeah, a story as a bug would be good because it shouldn't be a broken case, but we likely merged something along the way that broke it | 15:36 |
| rloo | and i am more confused. if ipxe_enabled = False, why should ipxe work? | 15:36 |
| kaifeng_ | no problem, i will file a story for the patches | 15:36 |
| dtantsur | rloo: they're orthogonal | 15:36 |
| dtantsur | ipxe_enabled only affects the older pxe interface (because of backward compatibility) | 15:37 |
| rpittau | kaifeng_: thanks | 15:37 |
| rloo | dtantsur: oh. i need to refresh my memory wrt ipxe_enabled... | 15:37 |
| TheJulia | Or at least, that is how it is supposed to work :) | 15:37 |
| dtantsur | driver composition is my speciality :D | 15:37 |
| rloo | dtantsur: ah, thx for explaining. | 15:37 |
| TheJulia | Okay, I think we can move on | 15:37 |
| rloo | kaifeng_: ^^ that info in a story/bug would have helped me :D | 15:37 |
| kaifeng_ | the patch on master won't pass unless the patch merged in train.. | 15:38 |
| TheJulia | Are we good to proceed | 15:38 |
| TheJulia | arne_wiebalck: anything baremetal sig wise to bring up? | 15:38 |
| arne_wiebalck | TheJulia: no | 15:38 |
| kaifeng_ | unless we add additional migration path | 15:38 |
| rloo | kaifeng_: open a story, link the patches to the story so it is clear why the train patch is needed. | 15:38 |
| dtantsur | kaifeng_: why won't it? | 15:38 |
| TheJulia | Lets continue this in open discussion | 15:38 |
| dtantsur | https://review.opendev.org/#/c/697775/ does not break any compatibility? | 15:38 |
| patchbot | patch 697775 - ironic (stable/train) - Explicitly enable ipxe as boot interface when it's... - 1 patch set | 15:38 |
| dtantsur | yeah, okay | 15:38 |
| TheJulia | So jumping directly to RFE review if there are no objections | 15:39 |
| kaifeng_ | ok | 15:39 |
| TheJulia | #topic RFE Review | 15:39 |
| *** openstack changes topic to "RFE Review (Meeting topic: ironic)" | 15:39 | |
| TheJulia | We had a new RFE proposed, to enable scoping of introspection rules | 15:39 |
| TheJulia | #link https://storyboard.openstack.org/#!/story/2006995 | 15:39 |
| arne_wiebalck | We filed an RFE to scope inspection rules. This is meant to ease the handling od multiple deliveries and to avoid purging the rules all the time. | 15:40 |
| dtantsur | I influences this proposal, so consider me +1 | 15:40 |
| dtantsur | * influenced | 15:40 |
| TheJulia | This seems reasonable | 15:40 |
| arne_wiebalck | s/od/of/ | 15:40 |
| arne_wiebalck | We put this up here to see if there are any major concerns. | 15:41 |
| rpittau | seems like a needed change | 15:41 |
| arne_wiebalck | The other reason is: does this require a spec? | 15:41 |
| kaifeng_ | does the new field in properties interfere with scheduling? | 15:41 |
| arne_wiebalck | kaifeng_: it should not | 15:42 |
| TheJulia | that is a really good question kaifeng_ | 15:42 |
| TheJulia | If someone is using the json matching scheduler, I suspect it would... but I think that is not advisable | 15:42 |
| dtantsur | given that node.properties is user-updateable, we cannot guarantee anything about it | 15:43 |
| TheJulia | I don't really see an issue with the RFE. I feel like I'm lacking a piece of the puzzle | 15:43 |
| TheJulia | dtantsur: exactly | 15:43 |
| dtantsur | I'm fine with using node.extra as well, but we tend not to interpret its values anyhow | 15:43 |
| * arne_wiebalck wasn't aware that users can update node.properties | 15:44 | |
| TheJulia | I'd prefer we leave node.extra to operators | 15:44 |
| rloo | i don't think we should use node.extra. isn't that for the user? I don't think we should code assuming anything in .extra | 15:44 |
| TheJulia | arne_wiebalck: users as in the running user or user as in admin api user | 15:44 |
| arne_wiebalck | TheJulia: ah, that makes sense, thanks | 15:44 |
| arne_wiebalck | Shouldn't scheduling be based entirely on the resource_class (and traits)? | 15:45 |
| kaifeng_ | one exception is inspector would put an autodiscoverd=True to the node.extra during discovery, looks like we are short of fields :) | 15:45 |
| mgoddard | arne_wiebalck: can you do this without code changes? | 15:45 |
| arne_wiebalck | mgoddard: ? | 15:46 |
| mgoddard | just add an additional first condition, node.properties == 'constant' | 15:46 |
| mgoddard | missing some bits there... | 15:46 |
| mgoddard | node.properties['inspection_scope'] == 'constant' | 15:47 |
| arne_wiebalck | we could do this by tweaking the rules, seems awkward, though | 15:47 |
| mgoddard | ok | 15:47 |
| arne_wiebalck | with inspection of active nodes, there might be additional use cases | 15:48 |
| dtantsur | if you allow some bikeshedding, I'd just the property to inspection_rules_scope | 15:48 |
| dtantsur | s/just/change/ (wut) | 15:48 |
| arne_wiebalck | like regularly checking the nodes | 15:48 |
| * TheJulia thinks we've reached Open Discussion | 15:49 | |
| arne_wiebalck | dtantsur: sounds good | 15:49 |
| *** ijw has joined #openstack-ironic | 15:49 | |
| TheJulia | Moving on then! | 15:49 |
| TheJulia | #topic Open Discussion | 15:49 |
| *** openstack changes topic to "Open Discussion (Meeting topic: ironic)" | 15:49 | |
| rpioso | I have an open discussion thing. Has anyone successfully used devstack to deploy on real hardware with local boot? | 15:49 |
| TheJulia | So there was the previous discussion about grenade + ipxe | 15:49 |
| * rpioso has wrestled with that the past two weeks-ish. | 15:50 | |
| dtantsur | I'd also follow-up on previously discussed https://storyboard.openstack.org/#!/story/2006910 and https://storyboard.openstack.org/#!/story/2006936 | 15:50 |
| TheJulia | I also wonder if we should call next week's meeting the last for the year | 15:50 |
| dtantsur | TheJulia++ | 15:50 |
| rpittau | TheJulia: we probably should | 15:50 |
| *** ociuhandu has joined #openstack-ironic | 15:51 | |
| rloo | ++ to last meeting next week | 15:51 |
| TheJulia | dtantsur: that latter story, they would expect us to magically make work to account to handle someone else's behavior change. I'm okay with the change, but... ugh | 15:52 |
| TheJulia | well, changes, it is going to be in multiple places | 15:52 |
| dtantsur | yeah, it's not ideal | 15:52 |
| TheJulia | rpioso: to your question, it has been... a while for me. I'm actually 3d printing a new case for a new lab router so I can rebuild the home lab.... #geekswithcircuitboardsandnocases.... | 15:52 |
| dtantsur | I'm also concerned about cost/profit balance of this work | 15:52 |
| *** ijw has quit IRC | 15:53 | |
| dtantsur | one benefit is that we can probably save some bandwidth when streaming images (with the direct deploy) | 15:53 |
| *** ijw has joined #openstack-ironic | 15:53 | |
| TheJulia | dtantsur: internally, we have way too much on our plate to guarentee that in a short term anyway | 15:53 |
| dtantsur | also true | 15:53 |
| dtantsur | I think the deployment API brings much more to the project than that | 15:53 |
| dtantsur | so I'd love some feedback on it as well | 15:54 |
| TheJulia | absolutely agree | 15:54 |
| kaifeng_ | well i am not sure i can finish the ipxe in 5 minutes | 15:55 |
| *** ociuhandu has quit IRC | 15:55 | |
| *** mkrai has quit IRC | 15:56 | |
| dtantsur | rpioso: I thinks you have two separate problems: devstack and local boot. They seem orthogonal to me. | 15:57 |
| rpioso | dtantsur: I'm all eyes OO | 15:57 |
| TheJulia | kaifeng_: I likely need a little more context, but I suspect your kind of heading in the right path because without the pre-existing config then the "upgraded" code needed post upgrade will fail. | 15:58 |
| dtantsur | rpioso: well, without even knowing symptoms it's hard to say what you have. But you can at least try to bisect the problem. | 15:58 |
| TheJulia | kaifeng_: but if you merge devstack changes in separately, it will liekly also be needed | 15:58 |
| dtantsur | If you have problems with local boot, you can likely rule out the "devstack" side of your question | 15:58 |
| rpioso | dtantsur: I believe I've gotten successful devstack ironic node deployments with network boot. | 15:59 |
| kaifeng_ | TheJulia: i am raising for some reviews so i can make sure taking the right path. | 15:59 |
| TheJulia | well, "devstack" all depends on the settings and image content being used too... | 15:59 |
| kaifeng_ | the issue is ipxe didn't work without the ipxe configuration option enabled, change in master branch fixed that, but nodes enrolled in train was with the pxe interface | 15:59 |
| kaifeng_ | while we actually used pxe+pxe_enabled as ipxe by default | 16:00 |
| dtantsur | rpioso: does the image you're using have grub2? | 16:00 |
| TheJulia | kaifeng_: but you also have two grenade jobs to pass. One Train->Master+patch, the other master->master | 16:00 |
| rpioso | dtantsur; The most recent symptom is that the tinycore IPA ramdisk can't get the config. TheJulia believes tinycore may not have the needed Intel NIC driver and that tinycore is used in VM environments, not real HW. | 16:00 |
| dtantsur | correct | 16:00 |
| TheJulia | err, master->master+patch | 16:00 |
| dtantsur | I think we can wrap up the meeting and continue with discussions | 16:01 |
| * etingof just used tinycore on Dell R640 | 16:01 | |
| kaifeng_ | no, i didn't turn ipxe_enabled=False, so either ipxe, or pxe works | 16:01 |
| kaifeng_ | sounds a bit tricky | 16:01 |
| *** goldyfruit_ has joined #openstack-ironic | 16:01 | |
| rpioso | dtantsur: Which images should I use? And which devstack settings are needed. | 16:01 |
| *** ijw_ has joined #openstack-ironic | 16:01 | |
| TheJulia | I'm going to end the meeting, we can carry on with discussions | 16:01 |
| dtantsur | etingof: it depends on your luck. if no fancy drivers are needed - cool. | 16:01 |
| TheJulia | Thanks everyone! | 16:01 |
| TheJulia | #endmeeting | 16:01 |
| *** openstack changes topic to "Bare Metal Provisioning | Status: http://bit.ly/ironic-whiteboard | Docs: http://docs.openstack.org/ironic/ | Bugs: https://storyboard.openstack.org/#!/project_group/75 | Contributors are generally present between 6 AM and 12 AM UTC, If we do not answer, please feel free to pose questions to openstack-discuss mailing list." | 16:01 | |
| openstack | Meeting ended Mon Dec 9 16:01:57 2019 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 16:02 |
| rpioso | TheJulia: Thank you! | 16:02 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/ironic/2019/ironic.2019-12-09-15.00.html | 16:02 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/ironic/2019/ironic.2019-12-09-15.00.txt | 16:02 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/ironic/2019/ironic.2019-12-09-15.00.log.html | 16:02 |
| rpittau | thanks! | 16:02 |
| etingof | dtantsur, define luck | 16:02 |
| * TheJulia jumps into #nextmeeting | 16:02 | |
| dtantsur | etingof: set(your_hardware.drivers) == set(tinycore.drivers) | 16:02 |
| rpioso | etingof: Which NIC port is being used for pxe boot? | 16:02 |
| dtantsur | ooops, s/==/in/ | 16:02 |
| dtantsur | rpioso: so, for an IPA image, I'd recommend you stick with DIB-built. You can download one from RDO if you don't want to build yours. | 16:03 |
| dtantsur | for a final (user image), anything that has grub2 should work. Cirros does NOT have grub2. | 16:03 |
| kaifeng_ | TheJulia: okay, I'll file a story to have more context then. | 16:03 |
| TheJulia | kaifeng_: ++ | 16:03 |
| etingof | rpioso, in my setup tinycore reports the NIC as eth1, I've not looked any further yet | 16:03 |
| kaifeng_ | but I am not following this: grenade for master -> master | 16:04 |
| etingof | also, I do not PXE anymore | 16:04 |
| *** goldyfruit has quit IRC | 16:04 | |
| rpioso | dtantsur: Could you suggest specific local.conf settings for those two things? I've been using a CentOS 7 prebuilt instance image. | 16:04 |
| *** ijw has quit IRC | 16:04 | |
| rpioso | etingof: Oh yeah. Right! And that's the answer to your luck question. | 16:04 |
| rpioso | No PXE! | 16:05 |
| etingof | dtantsur, as for the drivers, it's stock IPA kernel/ramdisk on a Dell R640 (I have no idea if hw config can differ on this model) | 16:05 |
| rpioso | dtantsur: TheJulia suggested there's no local.conf/devstack knob for specifying an IPA image. | 16:05 |
| dtantsur | rpioso: if you have a local file already, use https://opendev.org/openstack/ironic/src/branch/master/devstack/lib/ironic#L249-L250 | 16:06 |
| * rpioso is a relatively new devstack user. | 16:06 | |
| dtantsur | or these two to download https://opendev.org/openstack/ironic/src/branch/master/devstack/lib/ironic#L269 | 16:06 |
| etingof | rpioso, does PXE play any role in tinycore hardware interfaces? | 16:06 |
| etingof | I mean, tinycore fires up after PXE happens (or not) | 16:06 |
| TheJulia | the danger is there are also many different knobs and without clarity into the configuration it is hard to have discussions | 16:07 |
| *** jerrywang1 has quit IRC | 16:07 | |
| rpioso | dtantsur: Could you suggest RDO prebuilt IPA image links to use? | 16:07 |
| rpioso | TheJulia: Entirely understandable :-) | 16:08 |
| dtantsur | rpioso: https://images.rdoproject.org/master/rdo_trunk/current-tripleo/ I guess | 16:08 |
| rpioso | dtantsur, etingof, TheJulia: Thank you! I'll give the RDO IPA images a try :-) | 16:09 |
| rpioso | etingof: It does fire up after PXE happens. And then it can't obtain the config :-( | 16:10 |
| etingof | rpioso, so I guess it's not relevant to PXE then | 16:10 |
| *** stendulker has quit IRC | 16:11 | |
| rpittau | rpioso: doesn't look like an issue with PXE, what you mean it can't obtain the config ? | 16:11 |
| *** mkrai has joined #openstack-ironic | 16:12 | |
| * etingof reads it as tinycore kernel can't do networking | 16:12 | |
| TheJulia | etingof: as did I | 16:13 |
| rpioso | etingof: ^^ That's the hypothesis. | 16:15 |
| rpioso | rpittau: It outputs a message to that effect at the console. | 16:15 |
| etingof | rpioso, can you log through console and look around? | 16:16 |
| etingof | if it's booting at all | 16:16 |
| mkrai | Hi Ironicer, I am facing an issue were deployment was succesful, IP was assigned to node but on final power-on the node tries to do a PXE boot again and fails with error "No boot filename received" | 16:18 |
| kaifeng_ | TheJulia: wrt the grenade jobs you mentioned above, i am not sure i understand the master -> master, do we have such a upgrade path? | 16:18 |
| iurygregory | dtantsur, can you take a look if i missed something on https://review.opendev.org/#/c/698026/ ? o/ | 16:18 |
| patchbot | patch 698026 - ironic-python-agent-builder - Add efibootmgr - 1 patch set | 16:18 |
| mkrai | Can someone tell me what could be the reason that node tried to do PXE boot again and not tried to boot from disk? | 16:18 |
| dtantsur | kaifeng_: I think TheJulia means the multinode grenade (hopelessly broken atm) that does only a partial upgrade? | 16:19 |
| kaifeng_ | mkrai: do you set boot_option to local? | 16:19 |
| dtantsur | although no, wait.. I think we never implemented master->master+patch | 16:19 |
| dtantsur | iurygregory: I'll put on my queue | 16:19 |
| mkrai | kaifeng, not explicitly | 16:19 |
| TheJulia | we did I believe.... | 16:20 |
| TheJulia | but... so many different knobs | 16:20 |
| iurygregory | dtantsur, tks! | 16:20 |
| kaifeng_ | mkrai: i remember there was a default value if not configured, and if it's unfortunately netboot.. | 16:20 |
| dtantsur | TheJulia: I think we ended up with stable->master (grenade) and stable->partial master (multinode grenade) | 16:21 |
| * TheJulia really needs to push through that change | 16:21 | |
| dtantsur | I think master->master+patch has remained a dream. probably because of grenade. | 16:21 |
| mkrai | kaifeng_, it initially tries to do default netboot but after deployment it should boot from disk, right? | 16:21 |
| kaifeng_ | :D | 16:21 |
| kaifeng_ | anyway, for the ipxe change, i don't think it needs some change for master->master | 16:22 |
| *** mmethot has quit IRC | 16:22 | |
| *** mmethot has joined #openstack-ironic | 16:23 | |
| kaifeng_ | mkrai: if the node is specified as netbook, then it will not boot from disk | 16:23 |
| kaifeng_ | netboot (autocorrection is evil) | 16:23 |
| mkrai | kaifeng_, ok so netboot should also work, what does the failure "No boot filename received" mean? | 16:24 |
| rpioso | etingof: No, I haven't gotten to that, yet. Does it need a local.conf setting? Or do I need to modify the images? | 16:24 |
| mkrai | kaifeng_, is it related to IPA failure or ironic issue? | 16:24 |
| kaifeng_ | mkrai: that needs to investigate your tftp service | 16:24 |
| mkrai | kaifeng_, ok the deployment succedeed (PXE booting), doesn't this mean that tftp is working fine? | 16:25 |
| openstackgerrit | Merged openstack/python-ironicclient stable/train: Fix AttributeError in negotiate_version https://review.opendev.org/697269 | 16:26 |
| openstackgerrit | Merged openstack/python-ironicclient stable/stein: Fix AttributeError in negotiate_version https://review.opendev.org/697270 | 16:26 |
| * etingof is deploying his Dell sloth in boot_option:local way | 16:26 | |
| kaifeng_ | mkrai: which network interface do you use? | 16:26 |
| mkrai | kaifeng_, neutron | 16:26 |
| kaifeng_ | mkrai: well, for the multi tenancy model, you need pxe service on the tenant network | 16:27 |
| etingof | rpioso, once the kernel is booted, you should be able to log into it via virtual console. that should not depend on devstack/IPA image config | 16:27 |
| kaifeng_ | but i have never used such scenario before | 16:27 |
| mkrai | kaifeng, neutron-dhcp is running on tenant network | 16:28 |
| kaifeng_ | mkrai: but neutron doesn't provide tftp service | 16:29 |
| mkrai | kaifeng_, right | 16:29 |
| mkrai | kaifeng_, tftp running on conductor node | 16:30 |
| rpioso | etingof: I couldn't do that with cirros and I'm not familiar with tinycore's creds. Do you happen to know them? And what are RDO IPA ramdisk's creds? | 16:30 |
| etingof | rpioso, IPA image with tinycore boots into shell prompt | 16:31 |
| kaifeng_ | mkrai: it might need some network topology change, because the tenant network is managed by neutron, you may need to create a tap on the ovs/linux bridge | 16:31 |
| rpioso | etingof: Oh! So, it's the big Enter key? | 16:32 |
| etingof | rpioso, yes, just sudo -s <enter> | 16:33 |
| iurygregory | ^ and the magic happens | 16:35 |
| rpioso | etingof, iurygregory: Thank you! | 16:36 |
| *** rpioso is now known as rpioso|afk | 16:37 | |
| * rpioso|afk drives to the office after WFH, and will be back in a bit. | 16:37 | |
| etingof | obviously, ironic bear armed with two magic wands does the node revival magic | 16:38 |
| etingof | (unless magic backfires, what's not uncommon) | 16:38 |
| mkrai | kaifeng, it is already created, I tested it in the past and it used to work | 16:38 |
| mkrai | kaifeng, but I am seeing this failure recently | 16:39 |
| * iurygregory agrees with etingof | 16:39 | |
| *** ociuhandu has joined #openstack-ironic | 16:41 | |
| *** kaifeng_ has quit IRC | 16:42 | |
| rpittau | good night! o/ | 16:46 |
| *** rpittau is now known as rpittau|afk | 16:46 | |
| *** iurygregory has quit IRC | 16:53 | |
| *** ricolin has quit IRC | 16:56 | |
| *** ociuhandu has quit IRC | 16:58 | |
| mkrai | TheJulia, dtantsur etingof Hi can you please help me with the issue? ^^ | 16:58 |
| dtantsur | I think we tend to recommend using local boot and configdrives with the neutron network interface | 17:00 |
| TheJulia | what is using tcl in tinycore | 17:00 |
| dtantsur | tcl ugh | 17:01 |
| *** primeministerp has joined #openstack-ironic | 17:01 | |
| *** dhellmann_ has joined #openstack-ironic | 17:02 | |
| *** dhellmann has quit IRC | 17:02 | |
| *** dhellmann_ is now known as dhellmann | 17:02 | |
| TheJulia | and why is x11/xorg... | 17:03 |
| mkrai | dtantsur, should I try local boot option? | 17:03 |
| TheJulia | the glib2 expansion | 17:03 |
| dtantsur | mkrai: I'd make it work with local boot first, yes | 17:03 |
| dtantsur | you can consider netboot later if you need it | 17:04 |
| dtantsur | make sure you're using a user image that has grub2 (i.e. not cirros) | 17:04 |
| mkrai | dtantsur, Ok I will try now. Thanks! | 17:04 |
| *** priteau has joined #openstack-ironic | 17:05 | |
| *** gyee has joined #openstack-ironic | 17:16 | |
| openstackgerrit | Julia Kreger proposed openstack/ironic-python-agent-builder master: Shuffle/update tinycore mirrors https://review.opendev.org/698047 | 17:26 |
| TheJulia | ironic-cores ^^^ may resolve our tinycore issues for the day... Maybe | 17:26 |
| mkrai | dtantsur, the deployment failed with error http://paste.openstack.org/show/787334/ | 17:27 |
| dtantsur | mkrai: can your provisioning network access control plane? at least swift? | 17:28 |
| mkrai | dtantsur, yes it can | 17:30 |
| dtantsur | mkrai: it doesn't seem to, according to your paste: No route to host | 17:30 |
| mkrai | dtantsur, I am using centos cloud image | 17:31 |
| *** pcaruana has quit IRC | 17:31 | |
| mkrai | dtantsur, i tried "sudo ip netns exec qdhcp-5c179c83-3d38-4d17-aa37-0991d473b244 ping 10.228.161.155" and it works | 17:31 |
| dtantsur | mkrai: try checking ramdisk logs (you may need enabling them) to see what is happening inside | 17:33 |
| * dtantsur has to go now, sorry | 17:33 | |
| *** dtantsur is now known as dtantsur|afk | 17:33 | |
| *** mmethot has quit IRC | 17:35 | |
| arne_wiebalck | TheJulia: why is the (cedia) mirror newer than the official tc repo? Or is http://repo.tinycorelinux.net not the official repo? | 17:41 |
| *** ijw has joined #openstack-ironic | 17:42 | |
| arne_wiebalck | bye everyone o/ | 17:44 |
| TheJulia | arne_wiebalck: it is not... it seems to only sync once a week | 17:44 |
| TheJulia | arne_wiebalck: so... shouldn't be broken | 17:45 |
| *** ijw_ has quit IRC | 17:45 | |
| arne_wiebalck | TheJulia: from what I see http://repo.tinycorelinux.net has only up to 10.1, no? | 17:46 |
| * arne_wiebalck is maybe looking the wrong place | 17:46 | |
| arne_wiebalck | TheJulia: ah, no: the 11.x dir seems empty | 17:48 |
| arne_wiebalck | TheJulia: well, not for arm | 17:48 |
| *** dsneddon has joined #openstack-ironic | 17:48 | |
| TheJulia | wut? I was just looking at it | 17:48 |
| arne_wiebalck | which one? | 17:48 |
| TheJulia | http://mirror.cedia.org.ec/tinycorelinux/10.x/x86_64/tcz/ | 17:49 |
| *** igordc has joined #openstack-ironic | 17:49 | |
| arne_wiebalck | that's 10.x | 17:49 |
| TheJulia | we're on 10.x now | 17:49 |
| arne_wiebalck | right | 17:50 |
| arne_wiebalck | I was confused by the 11.x dir on cedia | 17:50 |
| TheJulia | ahh | 17:50 |
| arne_wiebalck | which does not seem to exist on http://repo.tinycorelinux.net, but maybe I did just not look well enough ... happens all the time :/ | 17:51 |
| * arne_wiebalck really has to go now, see you all tmrw! | 17:52 | |
| *** priteau has quit IRC | 17:54 | |
| *** priteau has joined #openstack-ironic | 17:55 | |
| *** priteau has quit IRC | 17:58 | |
| *** mkrai has quit IRC | 18:00 | |
| *** derekh has quit IRC | 18:01 | |
| *** rpioso|afk is now known as rpioso | 18:10 | |
| *** dougsz has quit IRC | 18:14 | |
| *** primeministerp has quit IRC | 18:17 | |
| *** ijw has quit IRC | 18:21 | |
| *** pcaruana has joined #openstack-ironic | 18:23 | |
| *** igordc has quit IRC | 18:37 | |
| *** ijw has joined #openstack-ironic | 18:42 | |
| *** ociuhandu has joined #openstack-ironic | 19:10 | |
| TheJulia | rpioso: etingof: re sushy-oem-idrac, do we need to get a tagged release soon-ish for packaging pipelines? | 19:13 |
| etingof | TheJulia, perhaps, assuming it's a packaging requirement | 19:14 |
| *** igordc has joined #openstack-ironic | 19:14 | |
| rpioso | etingof, TheJulia: Doesn't one already exist -- https://pypi.org/project/sushy-oem-idrac/? | 19:15 |
| *** ociuhandu has quit IRC | 19:15 | |
| *** ociuhandu has joined #openstack-ironic | 19:15 | |
| *** dsneddon has quit IRC | 19:16 | |
| TheJulia | rpioso: we can't consume pypi for downstream packages | 19:16 |
| etingof | rpioso, it's not released via openstack upstream I think | 19:16 |
| * rpioso sees the light. | 19:16 | |
| *** ociuhandu has quit IRC | 19:16 | |
| etingof | might be a miracle | 19:17 |
| *** gmann is now known as gmann_afk | 19:17 | |
| rpioso | etingof: No! It's still daylight here :-) | 19:18 |
| *** dsneddon has joined #openstack-ironic | 19:19 | |
| *** dsneddon has quit IRC | 19:20 | |
| *** dsneddon has joined #openstack-ironic | 19:20 | |
| rpioso | etingof: May I help? Instructions welcome :-) | 19:21 |
| etingof | rpioso, would be great, it's LO here already. I have no instructions except reading up openstack release doc ;-/ | 19:22 |
| rpioso | etingof: Do you have the link handy? | 19:23 |
| rpioso | TheJulia: In devstack funland, I'm now seeing a glanceclient error during deployment -- http://paste.openstack.org/show/787344/ | 19:25 |
| etingof | rpioso, may be down the rabbit hole from here? -- https://docs.openstack.org/infra/manual/creators.html | 19:25 |
| rpioso | etingof: Thank you! | 19:25 |
| rpioso | etingof: Have you seen that error ^^ in your wanderings? | 19:26 |
| TheJulia | rpioso: glance didn't start up, Apache lots should have errors | 19:26 |
| rpioso | TheJulia: Where can I find the Apache logs? | 19:27 |
| TheJulia | Systemd? | 19:28 |
| TheJulia | Or /var/log/httpd/ | 19:29 |
| rpioso | TheJulia: I see devstack@g-api.service and devstack@g-reg.service. Those don't seem to be the one I'm looking for. Do you recall its name? | 19:33 |
| TheJulia | rpioso: I do not | 19:34 |
| rpioso | TheJulia: Must glance be configured in local.conf? | 19:35 |
| TheJulia | When you go to restack, have you been running the clean script? | 19:35 |
| rpioso | TheJulia: I start from a clean slate via VM snapshot. | 19:35 |
| *** strigazi has quit IRC | 19:37 | |
| *** tesseract has quit IRC | 19:37 | |
| *** gudrutis2 has quit IRC | 19:38 | |
| *** strigazi has joined #openstack-ironic | 19:40 | |
| *** khansa has quit IRC | 19:42 | |
| *** strigazi has quit IRC | 19:46 | |
| rpioso | TheJulia: "openstack image show UUID" succeeds, but "openstack image save ... UUID" fails. | 20:01 |
| *** bobmel has joined #openstack-ironic | 20:17 | |
| openstackgerrit | Tzu-Mainn Chen proposed openstack/ironic master: Restrict ability to change owner on provisioned node https://review.opendev.org/698078 | 20:19 |
| *** Lucas_Gray has joined #openstack-ironic | 20:21 | |
| *** bobmel has quit IRC | 20:21 | |
| *** ociuhandu has joined #openstack-ironic | 20:43 | |
| *** ociuhandu has quit IRC | 20:48 | |
| TheJulia | I think that would be the glance registery service then | 20:48 |
| *** Lucas_Gray has quit IRC | 21:04 | |
| *** Lucas_Gray has joined #openstack-ironic | 21:08 | |
| openstackgerrit | Julia Kreger proposed openstack/ironic-python-agent-builder master: Update tinycore mirrors https://review.opendev.org/698047 | 21:20 |
| TheJulia | so looks like ^ that the other mirror I was hoping for got updated. Between this morning and this afternoon the last updated times on the tcl8.6 package updated :( | 21:20 |
| TheJulia | BUT! the main repository mirror and md5 checksum matches | 21:20 |
| *** ociuhandu has joined #openstack-ironic | 21:21 | |
| *** ociuhandu has quit IRC | 21:26 | |
| stevebaker_ | heads up, I've proposed this to get gerritbot notfications for Verified -2 in this channel https://review.opendev.org/698091 | 21:29 |
| patchbot | patch 698091 - project-config - IRC #openstack-ironic gerritbot CI failed messages - 1 patch set | 21:29 |
| *** pcaruana has quit IRC | 21:31 | |
| antonym | i'm playing around with getting an efi image working in ironic using diskimage-builder with the block-device-efi and grub2 elements, when i launch a build, it fails during the grub2 config with: Installing GRUB2 boot loader to device /dev/sda failed with Unexpected error while running command.\nCommand: mount /dev/sda2 /tmp/tmphH8cGm\nExit code: 32\nStdout: u''\nStderr: u'mount: /tmp/tmphH8cGm: | 21:32 |
| antonym | wrong fs type, bad option, bad superblock on /dev/sda2 any suggestions on what to look for? | 21:32 |
| TheJulia | antonym: what are you putting in as properties for the glance image | 21:33 |
| TheJulia | stevebaker_: awesome | 21:33 |
| rpioso | TheJulia: I reverted to deploying devstack to perform netboot, instead of local, deployment. It didn't fetch a partition image (CentOS 7) nor RDO IPA images. And it successfully deployed a real hardware node, just like it was a couple of weeks ago. It's using a dib ramdisk. | 21:33 |
| TheJulia | stevebaker_: comments posted | 21:34 |
| TheJulia | rpioso: so what OS did it deploy then? | 21:35 |
| antonym | TheJulia: for the image, --disk-format qcow2 --container-format bare, along with the kernel_id and ramdisk_id do i need to specify some additional settings for grub there? | 21:35 |
| rpioso | TheJulia: I believe cirros. | 21:35 |
| TheJulia | antonym: kernel_id and ramdisk_id's presence cause the image to be marked as a partition image, but ot make a UEFI bootable iamge you need to do a whole disk image | 21:35 |
| antonym | ah, so i should drop those then? | 21:36 |
| TheJulia | antonym: yea, that is likely where things are going moderately sideways on your entire image build use | 21:36 |
| antonym | k, yeah, i had legacy working and booting, but then i decided to shift to UEFI only | 21:37 |
| antonym | so trying to track down all of those steps | 21:37 |
| rpioso | TheJulia: I can't seem to log into the instance from the iDRAC virtual console. It doesn't appear to accept keystrokes. I'm still using UEFI boot mode. | 21:38 |
| TheJulia | rpioso: so, realistically it sounds like it deployed a partition image in that case since your saying cirros and network booting | 21:38 |
| rpioso | TheJulia: Ok | 21:39 |
| stevebaker_ | TheJulia: thanks, replied | 21:39 |
| rpioso | Even if the boot mode is UEFI? | 21:39 |
| TheJulia | rpioso: yes, uefi firmware still knows how to netboot and netboot is the default operating mode | 21:40 |
| TheJulia | stevebaker_: awesome, thanks | 21:41 |
| rpioso | TheJulia: Should I fire off a new devstack deployment for local boot? At least I have a baseline I've confirmed still works on my lab setup. | 21:41 |
| TheJulia | snap it, unstack it, change the parameter for local boot default mode, and change the image that your trying to deploy... and restack? | 21:42 |
| stevebaker_ | TheJulia: do you think we should *also* have a bot which watches storyboard for tasks with a borked_ci tag? | 21:43 |
| rpioso | TheJulia: And download RDO IPA images to devstack/files, too? | 21:43 |
| *** jtomasek has quit IRC | 21:45 | |
| TheJulia | stevebaker_: could be useful | 21:46 |
| TheJulia | rpioso: trying to do that, your far outside the matrix that is tested | 21:47 |
| rpioso | TheJulia: Which IPA images should be used for local boot/UEFI? | 21:47 |
| *** gmann_afk is now known as gmann | 21:48 | |
| TheJulia | well, if you have an IPA image that is working for networking, I would think that it should work if you only change the deployment settings and the actual image to be deployed to disk | 21:48 |
| rpioso | TheJulia: I thought cirros/dib IPA doesn't have the needed grub2. | 21:50 |
| TheJulia | cirros | 21:50 |
| TheJulia | Cirros as an image being deployed | 21:50 |
| TheJulia | and the problem ultimately is that the "wholedisk" image that they have basically is nothing of the sort | 21:51 |
| TheJulia | it is an empty filesystem, to which IPA goes "can't find grub!" | 21:51 |
| rpioso | TheJulia: Oh! So, dib IPA knows how to do the right local thing for UEFI? | 21:51 |
| TheJulia | in theory, it should | 21:52 |
| TheJulia | We've got an open bug that we squash on secureboot signed bootloader though | 21:52 |
| TheJulia | but... working on that | 21:52 |
| openstackgerrit | Julia Kreger proposed openstack/ironic-python-agent-builder master: Revert "Update tinycore from 9.x to 10.x" https://review.opendev.org/698098 | 21:54 |
| TheJulia | Hey! Lets do that instead! | 21:54 |
| *** ijw_ has joined #openstack-ironic | 21:55 | |
| rpioso | TheJulia: Should I unstack.sh or clean.sh? | 21:56 |
| *** Goneri has quit IRC | 21:56 | |
| TheJulia | rpioso: both | 21:57 |
| TheJulia | in order | 21:57 |
| *** ijw has quit IRC | 21:58 | |
| rpioso | TheJulia: I'm going to apply the following settings in local.conf. | 22:00 |
| rpioso | IRONIC_DEFAULT_BOOT_OPTION=local | 22:00 |
| rpioso | IMAGE_URLS="http://cloud.centos.org/centos/7/images/CentOS-7-x86_64-GenericCloud.qcow2" | 22:01 |
| rpioso | Any others needed? | 22:01 |
| antonym | TheJulia: thanks for your help, removing the kernel entries got the server to boot | 22:04 |
| *** ociuhandu has joined #openstack-ironic | 22:04 | |
| *** ociuhandu has quit IRC | 22:10 | |
| *** rcernin has joined #openstack-ironic | 22:16 | |
| *** Lucas_Gray has quit IRC | 22:17 | |
| TheJulia | antonym: \o/ | 22:20 |
| TheJulia | rpioso: give it a spin | 22:21 |
| * rpioso 's eyes are spinning as he watches it restack ;-) | 22:22 | |
| rpittau|afk | TheJulia: hey, just noticed this https://review.opendev.org/698098 , what's the reason behind that? | 22:23 |
| patchbot | patch 698098 - ironic-python-agent-builder - Revert "Update tinycore from 9.x to 10.x" - 1 patch set | 22:23 |
| TheJulia | rpittau|afk: 10.x is actively being changed and we're hitting checksum failures on packages. Where as 9 is... not | 22:23 |
| rpittau|afk | TheJulia: I noticed the issue with the checksum, thought it was just a quick thing, guess we'll wait somre more to switch to 10.x then | 22:27 |
| * rpittau|afk goes back to sleep | 22:28 | |
| *** rcernin has quit IRC | 22:36 | |
| TheJulia | nini | 22:36 |
| *** bfournie has quit IRC | 22:51 | |
| rpioso | TheJulia: Well, the deployment succeeded from an ironic perspective. Ironic asked the server to boot from the hard disk; however, the server did not boot from the hard disk. Instead, it's trying and trying to PXE boot. | 22:54 |
| TheJulia | fun :( | 22:56 |
| rpioso | TheJulia: In the logs, "Configuring local boot for node 54f3ef6c-6ee7-474d-9007-5ebebeca7649" is followed by "Local boot successfully configured for node ..." | 23:01 |
| rpioso | TheJulia: There's no indication that it "Installing the bootloader for node ... on partition, EFI system partition ..." | 23:03 |
| * TheJulia downloads the image to take it apart | 23:03 | |
| rpioso | TheJulia: Which image? | 23:04 |
| rpioso | TheJulia: driver_internal_info states it's a whole disk image. There's also a root_uuid_or_disk_id. | 23:04 |
| TheJulia | the centos image | 23:05 |
| rpioso | +1 | 23:05 |
| rpioso | http://cloud.centos.org/centos/7/images/CentOS-7-x86_64-GenericCloud.qcow2 | 23:06 |
| TheJulia | already downloading | 23:07 |
| TheJulia | what is your instance_info field look like in ironic? | 23:07 |
| TheJulia | oh, its a dual boot image | 23:11 |
| TheJulia | hmmm | 23:11 |
| rpioso | | instance_info | {'image_source': '3385cda1-0610-461a-a7c2-e42eabb17588', 'root_gb': 100, 'capabilities': {'boot_mode': 'uefi', 'boot_option': 'local'}} | 23:12 |
| TheJulia | well, maybe not | 23:13 |
| rpioso | | driver_internal_info | {'is_whole_disk_image': True, 'last_power_state_change': '2019-12-09T22:47:34.009558', 'deploy_steps': None, 'agent_version': '4.0.1.dev47', 'agent_last_heartbeat': '2019-12-09T22:44:26.032692', 'root_uuid_or_disk_id': '0x000b6061'} | 23:14 |
| *** ociuhandu has joined #openstack-ironic | 23:15 | |
| TheJulia | so... that image is not looking UEFI bootable | 23:15 |
| TheJulia | it is purely a single partition xfs file system | 23:15 |
| *** rcernin has joined #openstack-ironic | 23:17 | |
| TheJulia | and it is mbr bootable | 23:19 |
| TheJulia | so no point in doing anything else | 23:19 |
| TheJulia | but... that won't work on uefi mode | 23:19 |
| rpioso | How about something at https://images.rdoproject.org/master/rdo_trunk/current-tripleo/ | 23:19 |
| rpioso | Or elsewhere? | 23:19 |
| TheJulia | give me a couple of minutes | 23:20 |
| *** ociuhandu has quit IRC | 23:20 | |
| TheJulia | so yeah, also missing an EFI bootloader in the contents | 23:20 |
| rpioso | TheJulia: Thank you so much. I'm going to catch a very, very late meal. Then, I'll be back. Will have phone on me, though. | 23:20 |
| TheJulia | https://www.irccloud.com/pastebin/ByuNDnHr/ | 23:21 |
| TheJulia | pulling down one of those | 23:21 |
| *** ociuhandu has joined #openstack-ironic | 23:24 | |
| *** ociuhandu has quit IRC | 23:33 | |
| *** ociuhandu has joined #openstack-ironic | 23:34 | |
| *** ociuhandu has quit IRC | 23:39 | |
| *** ijw_ has quit IRC | 23:41 | |
| TheJulia | rpioso: overcloud-full is just a partition :\ | 23:42 |
| TheJulia | so no uefi boot assets | 23:42 |
| *** ociuhandu has joined #openstack-ironic | 23:44 | |
| *** ociuhandu has quit IRC | 23:49 | |
| *** goldyfruit_ has quit IRC | 23:50 | |
| rpioso | TheJulia: A year plus ago, I believe I uploaded those into glance running on an RDO undercloud. I imported nodes via tripleo and created a server with nova. That workflow was used to test new idrac driver UEFI boot mode support. Local boot deployments worked like champs. | 23:52 |
| rpioso | TheJulia: Might tripleo and/or nova build the instance image on the fly? | 23:53 |
| TheJulia | build instance yes, rework disk contents no | 23:53 |
| TheJulia | well, there went my free disk space | 23:54 |
| TheJulia | the undercloud is a 53.7 GB raw disk image | 23:54 |
| TheJulia | also not uefi bootable | 23:54 |
| *** ociuhandu has joined #openstack-ironic | 23:54 | |
| * TheJulia wonders if there is an edge case here that has magically worked | 23:55 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!