| opendevreview | Iury Gregory Melo Ferreira proposed openstack/ironic-specs master: Zed Themes https://review.opendev.org/c/openstack/ironic-specs/+/841061 | 03:27 |
|---|---|---|
| arne_wiebalck | Good morning, Ironic! | 06:20 |
| rpittau | good morning ironic! o/ | 07:22 |
| opendevreview | likui proposed openstack/ironic master: The Python 3.6 and Python 3.7Support has been dropped since yaga https://review.opendev.org/c/openstack/ironic/+/841071 | 07:25 |
| opendevreview | likui proposed openstack/ironic-inspector master: The Python 3.6 and Python 3.7Support has been dropped since yaga https://review.opendev.org/c/openstack/ironic-inspector/+/841072 | 07:27 |
| rpittau | iurygregory: re inspector CI fix, grenade is fixed in my patch, I will make non-standalone job non-voting for the time being | 07:29 |
| opendevreview | likui proposed openstack/ironic-python-agent master: The Python 3.6 and Python 3.7Support has been dropped since yaga https://review.opendev.org/c/openstack/ironic-python-agent/+/841073 | 07:30 |
| opendevreview | likui proposed openstack/ironic master: The Python 3.6 and Python 3.7Support has been dropped since yaga https://review.opendev.org/c/openstack/ironic/+/841071 | 07:30 |
| opendevreview | likui proposed openstack/ironic-inspector master: The Python 3.6 and Python 3.7Support has been dropped since yaga https://review.opendev.org/c/openstack/ironic-inspector/+/841072 | 07:31 |
| opendevreview | Riccardo Pittau proposed openstack/ironic-inspector master: Fix CI https://review.opendev.org/c/openstack/ironic-inspector/+/840588 | 07:32 |
| opendevreview | likui proposed openstack/ironic master: The Python 3.6 and Python 3.7Support has been dropped since zed https://review.opendev.org/c/openstack/ironic/+/841071 | 07:39 |
| rpittau | TheJulia: thanks! I will have a look today | 07:44 |
| opendevreview | likui proposed openstack/ironic-inspector master: Drop lower-constraints.txt and its testing https://review.opendev.org/c/openstack/ironic-inspector/+/841078 | 08:00 |
| opendevreview | Riccardo Pittau proposed openstack/ironic-python-agent master: Multipath Hardware path handling https://review.opendev.org/c/openstack/ironic-python-agent/+/837039 | 08:03 |
| dtantsur | Slightly-worse-than-average Monday folks! | 09:10 |
| janders | hey arne_wiebalck rpittau dtantsur and Ironic o/ | 10:18 |
| arne_wiebalck | hey janders o/ | 10:18 |
| rpittau | hey janders :) | 10:20 |
| iurygregory | morning Ironic! | 10:36 |
| iurygregory | tks rpittau! | 10:36 |
| opendevreview | OpenStack Release Bot proposed openstack/ironic-python-agent bugfix/8.6: Update .gitreview for bugfix/8.6 https://review.opendev.org/c/openstack/ironic-python-agent/+/841098 | 10:43 |
| opendevreview | OpenStack Release Bot proposed openstack/ironic bugfix/20.2: Update .gitreview for bugfix/20.2 https://review.opendev.org/c/openstack/ironic/+/841099 | 10:44 |
| opendevreview | Merged openstack/ironic-python-agent bugfix/8.6: Update .gitreview for bugfix/8.6 https://review.opendev.org/c/openstack/ironic-python-agent/+/841098 | 11:25 |
| opendevreview | Merged openstack/ironic bugfix/20.2: Update .gitreview for bugfix/20.2 https://review.opendev.org/c/openstack/ironic/+/841099 | 11:35 |
| adam-rozman | Hi all! I have an issue | 12:38 |
| adam-rozman | and i'd like to discuss it with you | 12:38 |
| dtantsur | hi, ask away | 12:41 |
| * TheJulia wipes sleep from her eyes | 12:42 | |
| * dtantsur secretly puts more sleep in TheJulia's eyes | 12:43 | |
| TheJulia | ZzZzZz | 12:43 |
| opendevreview | Merged openstack/sushy stable/xena: Re-trying InsertMedia call with TransferProtocolType if required https://review.opendev.org/c/openstack/sushy/+/839845 | 12:43 |
| opendevreview | Merged openstack/sushy stable/xena: Attempt using Redfish settings resource to get URI https://review.opendev.org/c/openstack/sushy/+/839843 | 12:44 |
| opendevreview | Merged openstack/sushy stable/xena: Follow up to Attempt using Redfish settings resource to get URI https://review.opendev.org/c/openstack/sushy/+/839844 | 12:44 |
| adam-rozman | I have been debugging an in teresting problem reported by some of my coworkers who are deploying ironic as part of k8s baremetal-operator. The main problem for them is that when IPA checks the storage device configuration here: https://opendev.org/openstack/ironic-python-agent/src/branch/master/ironic_python_agent/hardware.py#L522 the SERIAL_SHORT aka ID_SERIAL_SHORT does not contain the actual disk serial number but the | 12:44 |
| adam-rozman | wwn/wwid | 12:44 |
| TheJulia | adam-rozman: what kind of block device and what is the access to the block device? | 12:45 |
| adam-rozman | there are 2 disks in the machine as far as I know at least based on what ssacli tells me, and the two discs are conneted throughan HP Smary Array cargen 10 P816i I car | 12:47 |
| adam-rozman | d | 12:47 |
| adam-rozman | so HP Smart Array gen 10 card | 12:48 |
| dtantsur | wow, interesting. what about lsblk? | 12:51 |
| adam-rozman | I have compiled both then latest smarpqi and hpsa drivers for our IPA but it didn't help, for some reason the ID_SERIAL_SHORT is always shows the wwn value (checked with this udevadm info --query=all --name=/dev/sda ). I don't have direct acces to that environment but from what I remember with lsblk the two disk is shown as two regular hard drive. The disks are usable as /dev/sda and /dev/sdb | 12:53 |
| janders | see you tomorrow Ironic o/ | 12:53 |
| adam-rozman | This issue is blocking provisioning with BMO if they use the disk serial number in root device hints but if they use the as serial number then it works | 12:53 |
| dtantsur | adam-rozman: "sudo lsblk -P -o KNAME,SERIAL <disk>"? | 12:54 |
| dtantsur | ahh, you don't have an access. that's a pity | 12:54 |
| dtantsur | TheJulia: as a morning exercise, could you check https://review.opendev.org/c/openstack/ironic-python-agent/+/839939 please? will be cool to have a complete output recorded. | 12:54 |
| TheJulia | dtantsur: sure | 12:55 |
| TheJulia | adam-rozman: both disks are doing this? | 12:55 |
| adam-rozman | yes both disks, I have saved udevadm output, ssacli output, ironic inspector endpoint, i can give info from those right away | 12:56 |
| TheJulia | dtantsur: any intent to backport that patch? | 12:58 |
| TheJulia | adam-rozman: my *guess* is that the controller is doing it | 12:58 |
| dtantsur | TheJulia: all of the intent! | 12:59 |
| adam-rozman | TheJulia: you mean that the Smart Array controller is giving the wwn number as serial to the drive on purpose ? | 13:00 |
| TheJulia | yes, because once it passes through the controller, it is a volume, not an individual disk | 13:00 |
| TheJulia | I would actually bet the WWN stamped on the disk is different | 13:00 |
| TheJulia | stendulker: o/ you around? | 13:01 |
| stendulker | Hi TheJulia | 13:01 |
| * iurygregory is back o/ | 13:02 | |
| opendevreview | Riccardo Pittau proposed openstack/ironic-python-agent master: Multipath Hardware path handling https://review.opendev.org/c/openstack/ironic-python-agent/+/837039 | 13:02 |
| adam-rozman | Yeah this makes sense as I have checked HP's iLO 4 and iLO 5 documentation and a bit about how things are going with these drives and I have seen that with thse RAID controllers the wwn is the source of truth, but sadly i would need some support from IPA to be able to do the provisioning based on serial number | 13:02 |
| rpittau | hey iurygregory wb :) | 13:02 |
| TheJulia | stendulker: Do you happen to be aware of smart controller behavior as it relates to serial numbers reported? | 13:02 |
| TheJulia | adam-rozman: is wwn blank in the data as well? | 13:03 |
| stendulker | TheJulia: I'm not aware of any such behavior | 13:03 |
| TheJulia | stendulker: thanks | 13:03 |
| adam-rozman | It would be enough to support our use case IF there would be an option to do the root device selection based on ID_SCSI_SERIAL, no nothing is blank wwn fields are correct the only issue is that the ID_SERIAL and ID_SERIAL_SHORT fields also contain the wwn | 13:04 |
| adam-rozman | but the ID_SCSI_SERIAL also from udevadm has the correct disk serial number | 13:04 |
| TheJulia | Interesting... | 13:04 |
| TheJulia | adam-rozman: can you share the command outputs? | 13:05 |
| dtantsur | wow | 13:05 |
| stendulker | TheJulia:But I'm not sure you can rely on that behavior. I had not seen any such behavior being documented | 13:05 |
| TheJulia | stendulker: good to know | 13:05 |
| dtantsur | we should probably include the udev output in the ramdisk logs as well | 13:06 |
| TheJulia | ++ | 13:06 |
| dtantsur | heh, for me ID_SERIAL_SHORT matches the serial number in lsblk | 13:07 |
| dtantsur | fun | 13:07 |
| dtantsur | TheJulia: a thought: maybe we should have a new root device hint, something like "identity" that matches serial, short serial, wwn or wwn with extension? | 13:09 |
| dtantsur | (SCSI serial for disks that have it) | 13:09 |
| TheJulia | I think that makes a lot of sense | 13:09 |
| * dtantsur notices how different udev output for NVMe is | 13:10 | |
| dtantsur | on another machine, I have SCSI_IDENT_SERIAL that matches ID_SERIAL_SHORT and SERIAL in lsblk | 13:11 |
| dtantsur | this is a mess | 13:11 |
| TheJulia | I basically have consistency on my main machine.... I don't think the lab machines at home have disks anymore | 13:13 |
| * TheJulia needs to wake up | 13:17 | |
| TheJulia | okay, we need to throttle memcache logging in ironic's api | 13:27 |
| * TheJulia looks at nova and asks "why are you sending fields parameter twice" | 13:29 | |
| dtantsur | "Because who are you to forbid me that" - proudly responds nova | 13:33 |
| TheJulia | Le sigh | 13:35 |
| TheJulia | Looks like grenade has a legit upgrade bug | 13:35 |
| TheJulia | Well, found a | 13:35 |
| iurygregory | I think ironic-specs jobs are broken .-. | 13:40 |
| rpittau | \o/ | 13:40 |
| iurygregory | locally linters and docs are green for me, but red on CI | 13:40 |
| adam-rozman | TheJulia: I will get back to you with the lsblk output and some udevadm output I will sanitize the output and put it to pastebin or something where it is easier to handle | 13:44 |
| adam-rozman | And thanks for the quick response everybody. | 13:44 |
| TheJulia | sounds good | 13:44 |
| TheJulia | rutro, this might be an sdk bug | 13:59 |
| opendevreview | Merged openstack/ironic-python-agent master: Collect a full lsblk output in the ramdisk logs https://review.opendev.org/c/openstack/ironic-python-agent/+/839939 | 14:21 |
| opendevreview | Dmitry Tantsur proposed openstack/ironic-python-agent stable/yoga: Collect a full lsblk output in the ramdisk logs https://review.opendev.org/c/openstack/ironic-python-agent/+/840937 | 14:31 |
| opendevreview | Dmitry Tantsur proposed openstack/ironic-python-agent bugfix/8.6: Collect a full lsblk output in the ramdisk logs https://review.opendev.org/c/openstack/ironic-python-agent/+/840938 | 14:31 |
| dtantsur | folks, looking for a 2nd +2 on https://review.opendev.org/c/openstack/ironic-python-agent/+/840892 and https://review.opendev.org/c/openstack/ironic-python-agent/+/840670 please | 14:33 |
| rpittau | looking for approval for inspector CI fix if anyone has a moment https://review.opendev.org/c/openstack/ironic-inspector/+/840588 | 14:43 |
| iurygregory | I was about to ask a review on it :D | 14:52 |
| rpittau | :) | 14:52 |
| TheJulia | dtantsur: Got a minute to look at sdk weirdness? | 14:59 |
| iurygregory | #startmeeting ironic | 15:00 |
| opendevmeet | Meeting started Mon May 9 15:00:00 2022 UTC and is due to finish in 60 minutes. The chair is iurygregory. Information about MeetBot at http://wiki.debian.org/MeetBot. | 15:00 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 15:00 |
| opendevmeet | The meeting name has been set to 'ironic' | 15:00 |
| iurygregory | Hello everyone, welcome to our weekly meeting o/ | 15:00 |
| ajya | o/ | 15:00 |
| rpioso | \o | 15:00 |
| TheJulia | o/ | 15:00 |
| matfechner | o/ | 15:00 |
| rpittau | 0/ | 15:00 |
| iurygregory | you can find our agenda in the wiki | 15:00 |
| hjensas | o/ | 15:00 |
| iurygregory | #link https://wiki.openstack.org/wiki/Meetings/Ironic#Agenda_for_next_meeting | 15:00 |
| iurygregory | #topic Announcements / Reminder | 15:01 |
| iurygregory | #info Bugfix branch for Ironic and IPA (inspector needs a CI fix) | 15:01 |
| iurygregory | #info 4 weeks for the OIS \o/ | 15:02 |
| iurygregory | #info Zed Themes is up for review | 15:02 |
| iurygregory | #link https://review.opendev.org/c/openstack/ironic-specs/+/841061 | 15:02 |
| iurygregory | CI seems to be broken I'm testing a fix locally | 15:02 |
| iurygregory | does anyone have anything else to add for announcements/reminder? | 15:03 |
| iurygregory | ok, moving on | 15:05 |
| TheJulia | not on my end | 15:05 |
| iurygregory | #topic Review action items from previous meeting | 15:05 |
| iurygregory | no action items, skipping | 15:05 |
| iurygregory | #topic Review subteam status reports | 15:05 |
| iurygregory | #link https://etherpad.opendev.org/p/IronicWhiteBoard | 15:06 |
| iurygregory | Starting around L65 | 15:06 |
| arne_wiebalck | o/ | 15:07 |
| TheJulia | so I found what appears to be a legitimate upgrade issue between stable/yoga and the master branch at present | 15:11 |
| TheJulia | somehow the fields from nova's list request through the SDK are resulting duplicated | 15:11 |
| iurygregory | ouch! | 15:11 |
| TheJulia | which Ironic's API goes "nope, do not like, not at all" | 15:12 |
| TheJulia | It *doesn't* do it anymore | 15:12 |
| TheJulia | but something is afoot | 15:12 |
| iurygregory | so likely an SDK bug? | 15:12 |
| TheJulia | Looks like it | 15:12 |
| TheJulia | I could use dtantsur's brain cells on the sdk for a little while since I'm still trying to figure out what might be going on | 15:13 |
| iurygregory | I can lend some brain cells also | 15:14 |
| iurygregory | I think we can move to the next topic | 15:15 |
| iurygregory | #topic Deciding on priorities for the coming week | 15:16 |
| iurygregory | #link https://review.opendev.org/q/status:open+hashtag:ironic-week-prio | 15:16 |
| iurygregory | Does anyone have any patches that needs attention? | 15:16 |
| iurygregory | I've added the Zed Themes, to the list | 15:17 |
| TheJulia | any objection to adding https://review.opendev.org/c/openstack/ironic/+/818641 ? | 15:17 |
| rpittau | I'm adding some CI/jobs update patches, quite fast reviews | 15:18 |
| iurygregory | none from my side | 15:18 |
| iurygregory | re objections =) | 15:18 |
| iurygregory | In the Zed Themes if anyone has a topic that will be developed during the cycle and probably needs big attention it would be good to have in the Themes patch | 15:20 |
| rpittau | we should probably take into account that we may add jammy at some point, we're using it already for python 3.10 unit tests | 15:20 |
| iurygregory | this can go into CI Health I think | 15:21 |
| TheJulia | Yeah, we're going to need to go to Jammy | 15:21 |
| iurygregory | but is a good point ++ | 15:21 |
| TheJulia | Eh... it is a bit more beyond just keeping CI working | 15:21 |
| iurygregory | and more complicated... | 15:21 |
| TheJulia | indeed | 15:21 |
| iurygregory | I will add that | 15:21 |
| iurygregory | fair point =) | 15:22 |
| rpittau | you can add me to that, I started already having a look at some stuff :) | 15:22 |
| iurygregory | ack | 15:22 |
| iurygregory | shall we move on? | 15:24 |
| TheJulia | sure | 15:24 |
| iurygregory | #topic Baremetal SIG | 15:24 |
| iurygregory | #link https://etherpad.opendev.org/p/bare-metal-sig | 15:24 |
| iurygregory | arne_wiebalck, o/ | 15:24 |
| TheJulia | So arne mentioned to me that apparently we'll have a SIG session at the Summit | 15:25 |
| arne_wiebalck | presentation tomorrow by iurygregory on Xena and Zed | 15:25 |
| arne_wiebalck | right, we will have a session at the summit | 15:25 |
| iurygregory | session at the Summit? \o/ | 15:25 |
| arne_wiebalck | mostly to present the SIG | 15:25 |
| rpittau | arne_wiebalck: will that be recorded? | 15:26 |
| iurygregory | nice! | 15:26 |
| arne_wiebalck | I guess ... not sure if forum sessions are recorded? | 15:26 |
| TheJulia | Forum sessions are not recorded | 15:26 |
| arne_wiebalck | not sure tbh | 15:26 |
| arne_wiebalck | ok | 15:26 |
| TheJulia | we *can* opt to stream it potentially | 15:26 |
| rpittau | mmm ok :/ | 15:26 |
| TheJulia | or to join in remote folks | 15:26 |
| iurygregory | I can probably try to record if possible | 15:26 |
| TheJulia | ... do I need to pack a big microphone? | 15:26 |
| arne_wiebalck | what we plan to do is look back at what the SIG has done since its inception and what we do atm | 15:26 |
| arne_wiebalck | and see if we can get more people joining (the usual thing :)) | 15:27 |
| arne_wiebalck | I will see if I manage to work on a slide deck and can then share it with you before | 15:28 |
| arne_wiebalck | then, we have a session tomorrow where iurygregory will talk about xena and zed for Ironic | 15:28 |
| arne_wiebalck | 2pm UTC as usual | 15:28 |
| arne_wiebalck | thanks iurygregory for volunteering! | 15:28 |
| iurygregory | np! o/ | 15:28 |
| arne_wiebalck | there won't be a SIG session in June | 15:29 |
| arne_wiebalck | b/c of the summit | 15:29 |
| arne_wiebalck | afterwards we have to see what we do for July and August | 15:29 |
| iurygregory | ++ cool! | 15:29 |
| arne_wiebalck | (we took a break last year I think due to holidays) | 15:29 |
| arne_wiebalck | anyway, I think that is it for the SIG | 15:29 |
| iurygregory | thank you arne_wiebalck | 15:30 |
| iurygregory | #topic RFE review | 15:30 |
| iurygregory | we don't have any topics, but I would like to double check if https://storyboard.openstack.org/#!/story/2009805 and https://storyboard.openstack.org/#!/story/2010007 are duplicated | 15:30 |
| iurygregory | I think the first one was the initial idea arne_wiebalck had, and the second one TheJulia wrote based on the PTG discussions | 15:31 |
| TheJulia | I filed the latter, and in discussion with Arne I believe the consensus was the second one had more substance | 15:31 |
| iurygregory | yeah ++ | 15:31 |
| iurygregory | I just wanted to be sure about it :D tks! | 15:32 |
| iurygregory | #topic Open discussion | 15:32 |
| iurygregory | Does anyone have something we should discuss? | 15:32 |
| TheJulia | Only thing I have is the mysterious SDK issue | 15:33 |
| admiyo | Just a note that I have bifrost deploy on aarch64 | 15:33 |
| TheJulia | \o/ | 15:33 |
| iurygregory | admiyo, WOW YAY! | 15:33 |
| iurygregory | admiyo, are you going to the OIS? :D | 15:33 |
| admiyo | 3 ansible changes that I have hard coded, need to figure out the right way to make arch sensitive | 15:33 |
| admiyo | Not going to the summit | 15:33 |
| iurygregory | gotcha | 15:34 |
| arne_wiebalck | TheJulia: ++ | 15:34 |
| TheJulia | admiyo: maybe upload the changes and let folks see? | 15:34 |
| rpittau | admiyo: if you have a WIP patch we can discuss there | 15:34 |
| admiyo | Will do | 15:34 |
| admiyo | At a minimum I can fpaste the patch so its in the log here...1 sec | 15:34 |
| admiyo | https://paste.openstack.org/show/biZQeDIk0LFynC6DPRVr/ | 15:35 |
| admiyo | Its not perfect. For example, although I say to use the snponly.efi binary, I had to actually build that myself, as I couldn't get the one from ipxe.org to boot. And that is in no .deb that I can find | 15:36 |
| TheJulia | ugh, you can't really conditional some of those in lists | 15:37 |
| TheJulia | but you can develop separate lists and then choose the list based upon architecture | 15:37 |
| rpittau | admiyo: I think we had the same issue with Openshift | 15:37 |
| admiyo | I'd expect that rpittau | 15:37 |
| admiyo | But at least we have the postive thread path illuminated | 15:38 |
| hjensas | I guess the ipxe_efi_binary could just default to snponly.efi ? | 15:38 |
| admiyo | THat seems to be coded in there elsewhere, yes | 15:38 |
| admiyo | I also need to work through getting the IPA images built. I am currently reusing old known-good ones, but not the current debian stable packages | 15:39 |
| admiyo | so...WIP, but Progress | 15:39 |
| * TheJulia starts the coffee maker | 15:40 | |
| * TheJulia notices the wife watching Adam Savage Tested videos... rutro | 15:40 | |
| admiyo | Thats all I got | 15:41 |
| rpittau | admiyo: thanks | 15:42 |
| iurygregory | TheJulia, do you want to talk about the SDK issue now? | 15:44 |
| iurygregory | ok, we can discuss in the channel | 15:50 |
| iurygregory | #topic Who is going to run the next meeting? | 15:50 |
| TheJulia | iurygregory: sure | 15:50 |
| TheJulia | did we end the meeting? | 15:50 |
| TheJulia | oh, heh | 15:50 |
| iurygregory | I was about to end XD | 15:50 |
| iurygregory | since we only have 10 min (not sure if is enough) | 15:50 |
| TheJulia | I think I can run next week's meeting | 15:51 |
| iurygregory | tks TheJulia =) | 15:51 |
| iurygregory | thanks everyone! | 15:51 |
| iurygregory | #endmeeting | 15:51 |
| opendevmeet | Meeting ended Mon May 9 15:51:48 2022 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 15:51 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/ironic/2022/ironic.2022-05-09-15.00.html | 15:51 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/ironic/2022/ironic.2022-05-09-15.00.txt | 15:51 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/ironic/2022/ironic.2022-05-09-15.00.log.html | 15:51 |
| TheJulia | iurygregory: https://meet.google.com/xvr-rngx-hfk | 15:51 |
| iurygregory | let me just grab some water o/ | 15:52 |
| TheJulia | I only have 8 minutes, fwiw | 15:52 |
| iurygregory | oh, we can also talk later if works for you | 15:52 |
| TheJulia | it shoudl be quick | 15:52 |
| iurygregory | ok | 15:52 |
| TheJulia | well, at least the start | 15:52 |
| iurygregory | back o/ | 15:53 |
| iurygregory | I can't hear you for some reason | 15:53 |
| admiyo | Hrm...git review not letting me in. Is there a permission per project thing, or am I just misconfigured? | 15:54 |
| admiyo | git review -s | 15:55 |
| admiyo | ssh://ayoung@review.opendev.org:29418/openstack/bifrost.git did not work. Description: ayoung@review.opendev.org: Permission denied (publickey). | 15:55 |
| admiyo | username is still ayoung.... | 15:55 |
| admiyo | and ssh key looks good | 15:57 |
| rpittau | good night! o/ | 15:57 |
| opendevreview | Takashi Kajinami proposed openstack/ironic-tempest-plugin master: Remove six https://review.opendev.org/c/openstack/ironic-tempest-plugin/+/841138 | 16:01 |
| opendevreview | Merged openstack/ironic-python-agent stable/xena: Do not try to guess EFI partition path by its number https://review.opendev.org/c/openstack/ironic-python-agent/+/840892 | 16:05 |
| admiyo | Ok, new key generated...interesting thing there. | 16:11 |
| admiyo | RSA is deprecated. | 16:11 |
| admiyo | ssh-keygen -t ed25519 .... | 16:11 |
| opendevreview | Adam Young proposed openstack/bifrost master: temp to use the snponly efi https://review.opendev.org/c/openstack/bifrost/+/841142 | 16:14 |
| admiyo | HEY! Lookat that! I can follow directions given enough time. | 16:14 |
| admiyo | rpittau, TheJulia ^^ better than in a paste doc | 16:15 |
| dtantsur | TheJulia: can look in sdk stuffs if needed | 16:28 |
| opendevreview | Merged openstack/ironic-python-agent stable/yoga: Do not try to guess EFI partition path by its number https://review.opendev.org/c/openstack/ironic-python-agent/+/840670 | 16:34 |
| iurygregory | I'm puzzled about the failures in the ironic-specs CI... | 16:57 |
| iurygregory | I can't reproduce locally even in a fresh OS O.o | 16:57 |
| TheJulia | dtantsur: Well, if anything stands out at https://storyboard.openstack.org/#!/story/2010029 it would be helpful | 17:09 |
| TheJulia | I think it is on y oga, could be xena | 17:17 |
| * TheJulia is brain scrambled | 17:17 | |
| dtantsur | I wonder what exactly changed, our API or the SDK | 17:23 |
| * dtantsur will also bet on the latter for now | 17:23 | |
| dtantsur | lemme get some tea and check the recent commits. TheJulia, do you know when it started? | 17:23 |
| TheJulia | no idea, it is gone now | 17:24 |
| TheJulia | hmm | 17:24 |
| TheJulia | Wallaby change -> https://5c12b42ba8ed0193e240-44b52704d8201eff523711187c4a84bb.ssl.cf2.rackcdn.com/839985/3/check/ironic-tempest-ipa-partition-pxe_ipmitool/638a666/controller/logs/apache/tls-proxy_access_log.txt only lists fields once | 17:26 |
| TheJulia | xena job log from same change https://1c8439a39624b0fb94b5-6e47dc903bf8882d80da1b9238bea9de.ssl.cf1.rackcdn.com/839984/3/check/ironic-tempest-ipa-partition-pxe_ipmitool/de41619/controller/logs/apache/tls-proxy_access_log.txt | 17:27 |
| TheJulia | no duplication there | 17:28 |
| TheJulia | Well, this is interesting.... looking at a log from yoga... | 17:30 |
| TheJulia | I see /v2.0/security-groups?id=4fed6749-d3aa-4fc1-9e70-6ef18852122d&fields=id&fields=name | 17:30 |
| TheJulia | I wonder if given grenade results in some updates locally, if maybe we're just in a weird edge case | 17:31 |
| TheJulia | like if openstacksdk got updated and somehow it got duplicated?! | 17:31 |
| TheJulia | so in multinode grenade... n-cpu does not get restarted | 17:41 |
| TheJulia | but I bet the sdk gets updated while the proess is running | 17:41 |
| TheJulia | process | 17:41 |
| TheJulia | no duplication in master->master | 17:42 |
| dtantsur | so, yoga version? | 17:47 |
| TheJulia | eh, this actually looks like it starts mid-upgrade | 17:48 |
| TheJulia | correlating times on grenade logs | 17:48 |
| TheJulia | https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_f8d/840530/6/check/ironic-grenade-multinode-multitenant/f8d16fd/controller/ | 17:52 |
| dtantsur | multinode grenade. what exactly is upgraded there? | 17:52 |
| TheJulia | looks like it went from openstacksdk stable/yoga (whatever that ends up being, to 0.61. Basically everything except services on the subnode, don't get restarted to their new full versions | 17:58 |
| TheJulia | https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_f8d/840530/6/check/ironic-grenade-multinode-multitenant/f8d16fd/controller/logs/screen-n-cpu.txt 22:50:49 is when it starts without a service restart | 17:59 |
| dtantsur | oO | 18:01 |
| TheJulia | and... we see the corresponding log entries on compute1's access_log | 18:01 |
| TheJulia | starting then... | 18:01 |
| dtantsur | both yoga and master have openstacksdk 0.61.0, so it shouldn't be related to its version? | 18:04 |
| TheJulia | "nova-compute keystoneauth1/4.5.0 python-requests/2.27.1 CPython/3.8.10" <-- hmmmm | 18:06 |
| dtantsur | what surprises you? | 18:06 |
| dtantsur | meanwhile, nothing seems to have changed in nova/virt/ironic since yoga | 18:07 |
| TheJulia | yeah | 18:07 |
| dtantsur | if the bug was already present in yoga, why don't we see it in master? | 18:07 |
| * dtantsur confused doggo | 18:08 | |
| * TheJulia confused cat | 18:08 | |
| dtantsur | well, it definitely does not reproduce in a naive attempt on the recent bifrost | 18:10 |
| TheJulia | yeah, something really really weird is going on | 18:10 |
| dtantsur | magic... | 18:17 |
| TheJulia | okay | 18:18 |
| TheJulia | I think I see it | 18:18 |
| TheJulia | well | 18:18 |
| TheJulia | what starts it | 18:18 |
| TheJulia | try doing a 307 redirect with the sdk | 18:18 |
| dtantsur | I wonder how | 18:18 |
| TheJulia | 22:50:49 first request logged.... 22:40:23 is when the controller compute node entered a 307 redirect into the configuration to redirect clients away from itself | 18:19 |
| TheJulia | https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_f8d/840530/6/check/ironic-grenade-multinode-multitenant/f8d16fd/controller/logs/apache_config/ironic-api-wsgi_conf.txt | 18:19 |
| dtantsur | ehmm, wtf is this | 18:19 |
| TheJulia | the job shuts down the new API and forces all traffic to the older API | 18:19 |
| TheJulia | so it swaps the config over | 18:20 |
| TheJulia | I bet if it was a proxypass instead, it would just work and we wouldnt' see this... | 18:20 |
| * dtantsur googles how to make a redirect with nginx | 18:20 | |
| * dtantsur has to do setenforce 0, send help! | 18:23 | |
| opendevreview | Julia Kreger proposed openstack/ironic master: DNM: Multinode Grenade https://review.opendev.org/c/openstack/ironic/+/840530 | 18:24 |
| dtantsur | TheJulia: brilliant! I'vee reproduced it | 18:24 |
| TheJulia | \o/ | 18:24 |
| TheJulia | so I'm not (too) crazy! | 18:24 |
| TheJulia | woohoo! | 18:25 |
| dtantsur | not too much, exactly the right amount of crazy :D | 18:25 |
| * dtantsur is curious if all arguments are duplicated | 18:25 | |
| TheJulia | good question | 18:25 |
| dtantsur | yep | 18:25 |
| dtantsur | same behavior with provision_state | 18:26 |
| dtantsur | may be down to keystoneauth even | 18:26 |
| TheJulia | so it looks like list only | 18:26 |
| TheJulia | https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_f8d/840530/6/check/ironic-grenade-multinode-multitenant/f8d16fd/compute1/logs/apache/access_log.txt | 18:26 |
| TheJulia | oh | 18:27 |
| TheJulia | no, they all are | 18:27 |
| TheJulia | very bottom of that | 18:27 |
| TheJulia | GET /baremetal/v1/nodes?associated=True&fields=instance_uuid&associated=True&fields=instance_uuid | 18:27 |
| dtantsur | yeah | 18:27 |
| dtantsur | same in my testing | 18:27 |
| dtantsur | is it about the time we move to #openstack-sdks? | 18:28 |
| TheJulia | yeah | 18:28 |
| opendevreview | Iury Gregory Melo Ferreira proposed openstack/ironic-specs master: Fix Multiple top-level packages discovered in flat-layout https://review.opendev.org/c/openstack/ironic-specs/+/841162 | 19:27 |
| opendevreview | Iury Gregory Melo Ferreira proposed openstack/ironic-specs master: Zed Themes https://review.opendev.org/c/openstack/ironic-specs/+/841061 | 19:30 |
| iurygregory | yay CI fixed in ironic-specs XD | 19:36 |
| iurygregory | at least the linters job | 19:36 |
| opendevreview | Iury Gregory Melo Ferreira proposed openstack/ironic-specs master: Fix Multiple top-level packages discovered in flat-layout https://review.opendev.org/c/openstack/ironic-specs/+/841162 | 19:42 |
| opendevreview | Merged openstack/ironic master: Fix names of two jobs https://review.opendev.org/c/openstack/ironic/+/838661 | 19:45 |
| opendevreview | Julia Kreger proposed openstack/ironic master: DNM: Multinode Grenade https://review.opendev.org/c/openstack/ironic/+/840530 | 21:38 |
| admiyo | What are the right docs to follow for setting up cleaning on a bifrost (standalone) setup? | 22:42 |
| admiyo | I know I did this before.... | 22:43 |
| janders | good morning Ironic o/ | 23:01 |
| opendevreview | Takashi Kajinami proposed openstack/ironic-tempest-plugin master: Remove six https://review.opendev.org/c/openstack/ironic-tempest-plugin/+/841138 | 23:20 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!