| opendevreview | Boushra Sondos Bettir proposed openstack/ironic master: Support both OVN switches, logical and physical switches. https://review.opendev.org/c/openstack/ironic/+/900568 | 02:40 |
|---|---|---|
| rpittau | good morning ironic! o/ | 08:21 |
| masghar | Good morning! | 08:58 |
| dtantsur | JayF: https://launchpad.net/~virtualpdu-bugs/+members#active is now us, should we ask someone to fix it or should we drop virtualpdu from the triaging dashboard? | 10:28 |
| dtantsur | JayF: also please check my comment on https://bugs.launchpad.net/ironic/+bug/2040236 | 10:28 |
| iurygregory | good morning Ironic | 11:24 |
| iurygregory | dtantsur, if I recall correctly he will be off this week | 11:24 |
| dtantsur | ah, thanksgiving | 11:24 |
| iurygregory | ok, time to go back to multipath bug in IPA \o/ | 12:28 |
| iurygregory | and drink my second coffee (is not even 10 am) | 12:28 |
| iurygregory | I've created https://bugs.launchpad.net/ironic-python-agent/+bug/2043992 for the bug I'm working on | 14:05 |
| opendevreview | Mark Goddard proposed openstack/bifrost stable/2023.2: ironic: Perform online data migrations with localhost DB https://review.opendev.org/c/openstack/bifrost/+/901296 | 14:25 |
| opendevreview | Mark Goddard proposed openstack/bifrost stable/2023.1: ironic: Perform online data migrations with localhost DB https://review.opendev.org/c/openstack/bifrost/+/901297 | 14:25 |
| opendevreview | Mark Goddard proposed openstack/bifrost stable/zed: ironic: Perform online data migrations with localhost DB https://review.opendev.org/c/openstack/bifrost/+/901298 | 14:26 |
| opendevreview | Mark Goddard proposed openstack/bifrost stable/yoga: ironic: Perform online data migrations with localhost DB https://review.opendev.org/c/openstack/bifrost/+/901299 | 14:26 |
| dtantsur | Do we even have the meeting today? | 14:58 |
| iurygregory | dtantsur, now I'm wondering the same thing | 15:00 |
| * dtantsur checks IRC logs | 15:00 | |
| dtantsur | TheJulia to run the 11/20 meeting | 15:01 |
| dtantsur | okay, let's do it | 15:01 |
| dtantsur | #startmeeting ironic | 15:02 |
| opendevmeet | Meeting started Mon Nov 20 15:02:00 2023 UTC and is due to finish in 60 minutes. The chair is dtantsur. Information about MeetBot at http://wiki.debian.org/MeetBot. | 15:02 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 15:02 |
| opendevmeet | The meeting name has been set to 'ironic' | 15:02 |
| dtantsur | #chair TheJulia | 15:02 |
| opendevmeet | Current chairs: TheJulia dtantsur | 15:02 |
| TheJulia | o/ | 15:02 |
| iurygregory | o/ | 15:02 |
| dtantsur | TheJulia: wanna take it from here? | 15:02 |
| TheJulia | sorry, slightly distracted this morning | 15:02 |
| TheJulia | sure | 15:02 |
| TheJulia | #topic Announcements / Reminders | 15:02 |
| dtantsur | #info https://github.com/dtantsur/ironic-bug-dashboard is revived and can be used for triaging (run locally) | 15:03 |
| TheJulia | A standing reminder to review patches tagged with ironic-week-prio. The dashboard will be linked shortly. | 15:03 |
| TheJulia | #link https://tinyurl.com/ironic-weekly-prio-dash | 15:03 |
| TheJulia | Does anyone else have anything to announce or remind us of? | 15:03 |
| dtantsur | We need to add a section for bug triaging, don't we? | 15:03 |
| TheJulia | It appears we don't have any action items | 15:03 |
| iurygregory | dtantsur, ++ | 15:04 |
| TheJulia | Has the policy proposal change merged? | 15:04 |
| dtantsur | I don't believe so (if you mean re bugs) | 15:04 |
| dtantsur | #link https://review.opendev.org/c/openstack/ironic/+/900449 Bug deputy proposal | 15:05 |
| dtantsur | But I was a deputy nonetheless and ready to provide an update :) | 15:05 |
| TheJulia | okay, fair! :) | 15:05 |
| dtantsur | #topic Bug deputy update | 15:06 |
| dtantsur | First, as I've mentioned already, the bug dashboard is functional again | 15:06 |
| dtantsur | We don't have a place to host it yet, but it can be trivially run locally either with `tox -erun` or with the provided Dockerfile | 15:06 |
| dtantsur | #link https://github.com/dtantsur/ironic-bug-dashboard bug dashboard | 15:06 |
| dtantsur | Second, I've done a major bug clean up: | 15:07 |
| dtantsur | Ironic: 184 bugs (-31) + 189 wishlist items (+9). 29 new (-29), 96 in progress (-19), 3 critical (-1), 23 high (+2) and 13 incomplete (+1) | 15:07 |
| dtantsur | Nova bugs with Ironic tag: 23. 0 new, 0 critical, 1 high | 15:07 |
| dtantsur | A lot of bugs got stuck in the open state despite being merged, probably because of the transition to storyboard back then | 15:07 |
| dtantsur | I don't think I've cleaned up everything, but I made some progress | 15:08 |
| dtantsur | Third, and this is a reminder: | 15:08 |
| dtantsur | #info Please triage your bugs if you're a constant member of the team. This includes setting status to Triaged, setting priority and updating tags. | 15:08 |
| dtantsur | Forth, we have virtualpdu on our radar, but nobody here has ACL for its bugs. | 15:09 |
| dtantsur | rpittau: was it you show worked on the virtualpdu ownership the last time? | 15:09 |
| dtantsur | s/show/who/ | 15:09 |
| TheJulia | I believe it was rpittau | 15:09 |
| rpittau | o/ | 15:09 |
| rpittau | yes | 15:10 |
| rpittau | it was me | 15:10 |
| dtantsur | rpittau: can you ping your contacts again re launchpad ownership? | 15:10 |
| rpittau | of course! | 15:10 |
| dtantsur | #action rpittau to ask to change the launchpad ownership for virtualpdu to us | 15:11 |
| dtantsur | Last but not least, does anyone want to be the bug deputy this week? | 15:11 |
| TheJulia | My schedule is moderately crazy this week and I've got a long weekend | 15:12 |
| dtantsur | The crickets tell me that it's on me again :) | 15:12 |
| TheJulia | I likely can next week | 15:12 |
| iurygregory | I can give a try dtantsur | 15:12 |
| dtantsur | nice! | 15:12 |
| iurygregory | if i have questions I will ping you =P | 15:12 |
| dtantsur | iurygregory: something I never got to: check the storyboard for things that we might want transfered | 15:13 |
| dtantsur | and yes, never hesitate to ping me | 15:13 |
| dtantsur | #action iurygregory is the bug deputy this week (and TheJulia potentially next time) | 15:13 |
| iurygregory | ack | 15:13 |
| TheJulia | Onward? | 15:13 |
| dtantsur | Unless there are questions for me already | 15:13 |
| iurygregory | yes | 15:13 |
| TheJulia | #topic Caracal release schedule | 15:13 |
| TheJulia | #link https://releases.openstack.org/caracal/schedule.html | 15:14 |
| TheJulia | Last week was Caracal-1. Caracal-2 is the week of January 8th. | 15:14 |
| TheJulia | #topic Review Ironic CI status & update whiteboard if needed | 15:14 |
| dtantsur | Hmm, is it about the time we make our intermediary release? | 15:14 |
| TheJulia | I suspect it was last week or is around now | 15:14 |
| TheJulia | that being said we had the metal3-integration job broken last week. | 15:15 |
| rpittau | I'm going for a bugfix release next week | 15:15 |
| TheJulia | rpittau: ack, thanks | 15:15 |
| dtantsur | Bobcat was Oct 02, so we should aim for around Dec 02, I assume? | 15:15 |
| TheJulia | What is the word on the metal3-integration CI job? | 15:15 |
| rpittau | dtantsur: yeah, I'll propose beginning next week, and should merge by the end of the week | 15:15 |
| dtantsur | but yeah, next week will make the math somewhat better down the road | 15:16 |
| rpittau | TheJulia: it's fixed | 15:16 |
| TheJulia | cool, so we should be in a better place this week. That was the only issue I was aware of last week. | 15:16 |
| TheJulia | Onward? | 15:16 |
| dtantsur | ++ | 15:16 |
| rpittau | go! | 15:16 |
| TheJulia | #topic Bug Deputy role proposal | 15:17 |
| TheJulia | #link https://review.opendev.org/c/openstack/ironic/+/900449 | 15:17 |
| TheJulia | The document is still outstanding, please review this week. | 15:17 |
| TheJulia | I've rechecked it to hopefully clear the -1 on it | 15:18 |
| TheJulia | Since we have no RFEs to review, we shall proceed to Open Discussion if there is no further discussion ? | 15:18 |
| dtantsur | ++ | 15:19 |
| TheJulia | #topic Open Discussion | 15:19 |
| TheJulia | so, httpboot is looking good, grub just acts weird though | 15:19 |
| dtantsur | stop giving catnip to your grub! | 15:19 |
| dtantsur | (sorry) | 15:20 |
| TheJulia | lol | 15:20 |
| iurygregory | not first time grub acts weird lol | 15:20 |
| TheJulia | well, I pinged some maintainers, got asked questions which I had already tried to answer | 15:20 |
| dtantsur | absolutely unheard of! | 15:20 |
| dtantsur | anything we can help with? | 15:20 |
| TheJulia | and then opened the code, looks like it is edk2 or substrate networking | 15:20 |
| TheJulia | so it might be okay, I think I'll try to polish the patches | 15:20 |
| TheJulia | I just need to get the redfish sushy and sushy-tools changes merged since I'll need to update requirements.txt | 15:21 |
| dtantsur | links? | 15:21 |
| TheJulia | sure | 15:21 |
| iurygregory | I will try to review them this week | 15:22 |
| TheJulia | https://review.opendev.org/c/openstack/sushy-tools/+/901208 <-- this really fixes the prior change | 15:22 |
| TheJulia | #link https://review.opendev.org/c/openstack/sushy/+/718276 | 15:22 |
| TheJulia | Once I can get the base ironic change updated with a released sushy-tools, the jobs will go green | 15:23 |
| dtantsur | nice | 15:23 |
| TheJulia | I did deviate a little from the spec on the redfish and dhcp driven network booting paths, but nothing horrible | 15:23 |
| TheJulia | and ultimately gives us the ability to still sort of do the ipxe logic dance for those who really want openstack integrated ipxe | 15:24 |
| TheJulia | I'll keep sorting out grub, it does the needful though, it is something in grub where we might just want to document the substrate is there, it sort of works, but we've seen some issues which are expected to be unrelated | 15:24 |
| * dtantsur is a bit worried by the amount of "sort of" :) | 15:25 | |
| TheJulia | Well, it loads shim, shim chains to grub | 15:25 |
| TheJulia | and then grub sort of falls down on step 5 of packet processing semi-randomly | 15:25 |
| TheJulia | which leverages the http logic handler from UEFI | 15:25 |
| TheJulia | so...... | 15:25 |
| TheJulia | ¯\_(ツ)_/¯ | 15:26 |
| dtantsur | fun | 15:26 |
| TheJulia | Yeah, it doesn't block us from merging an interface, but it blocks us from having verbose "it works!" docs | 15:26 |
| rpittau | "it sort of works" :) | 15:27 |
| TheJulia | the underlying code is all identical though, so high confidence if whatever ubuntu's grub is doing can be sorted out | 15:27 |
| dtantsur | make a good sacrifice to the bootloader gods | 15:27 |
| TheJulia | "It sort of works, go complain to your vendor if it doesn't" | 15:27 |
| rpittau | lol | 15:27 |
| dtantsur | TheJulia: can we try with another distro? | 15:27 |
| TheJulia | I believe the bootloader gods are the elder gods | 15:27 |
| TheJulia | dtantsur: I can, the CI changes themselves are way down the series of changes | 15:28 |
| TheJulia | I'm also tempted to write a new "exercise all available boot interfaces" tempest job so we can do it in one-shot versus scenario test after scenario test | 15:28 |
| dtantsur | What I"m curious about is whether this a fundamental grub problem or just something in the ubuntu build | 15:28 |
| TheJulia | dtantsur: I'm honestly suspecting environmental grub + environment + ed2k | 15:29 |
| TheJulia | err, edk2 | 15:29 |
| TheJulia | Next up, Cthulhu will appear to discuss grub | 15:29 |
| rpittau | non-euclidean geometry is our last chance | 15:30 |
| TheJulia | will this permit us to fold space and time, finally? | 15:31 |
| TheJulia | ... it is clear, the meeting is over, we've folded it away. | 15:31 |
| dtantsur | \o/ | 15:31 |
| TheJulia | Anything else folks before I wrap up today's meeting? | 15:31 |
| rpittau | one small thing | 15:31 |
| TheJulia | yes? | 15:32 |
| rpittau | the api for attach detach virtual media patch has got its first +2 https://review.opendev.org/c/openstack/ironic/+/894918 | 15:32 |
| TheJulia | \o. | 15:32 |
| rpittau | we're a bit in a rush with that, if anyone has a moment for a review would be great :) | 15:33 |
| rpittau | that's all, thanks! | 15:33 |
| TheJulia | Thanks | 15:35 |
| TheJulia | Hopefully we can do the next release with some httpboot stuffs :) | 15:35 |
| TheJulia | (that would be epic) | 15:35 |
| dtantsur | True | 15:35 |
| TheJulia | Well, if there is nothing else, it seems we have code review to do | 15:36 |
| TheJulia | Thanks everyone, have a wonderful week | 15:37 |
| TheJulia | Oh, one last thing | 15:37 |
| TheJulia | Anyone want to run the meeting next week? | 15:38 |
| dtantsur | won't Jay be back? | 15:38 |
| TheJulia | He will be, but doesn't mean we can't volunteer someone so he doesn't have to worry about it | 15:38 |
| rpittau | I can run it | 15:38 |
| TheJulia | ack, thanks. due to daylight savings time, it is a bit early for him too | 15:39 |
| TheJulia | Anyhow, thanks everyone! | 15:39 |
| TheJulia | #endmeeting | 15:39 |
| opendevmeet | Meeting ended Mon Nov 20 15:39:26 2023 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 15:39 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/ironic/2023/ironic.2023-11-20-15.02.html | 15:39 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/ironic/2023/ironic.2023-11-20-15.02.txt | 15:39 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/ironic/2023/ironic.2023-11-20-15.02.log.html | 15:39 |
| rpittau | dtantsur: I sent a mail for the virtualpdu launchpad space ownership | 15:46 |
| dtantsur | thanks! | 15:46 |
| *** dmellado2 is now known as dmellado | 15:58 | |
| arne_wiebalck | Good morning, Ironic! | 16:16 |
| arne_wiebalck | I am trying to get some backports wrapped up: https://review.opendev.org/q/I13617ae77515a9d34bc4bb3caf9fae73d5e4e578 | 16:16 |
| arne_wiebalck | The Yoga one fails in tempest tests, though, and I am not really clear on whether it is due to the patch or something unrelated ... if someone has a moment to push me in the right direction ... | 16:18 |
| dtantsur | TheJulia: do I recall it right that we want to deprecate WSMAN support? I was not on that session. | 16:29 |
| Sandzwerg[m] | Hi Ironic. We had an issue today when an deployment failed because of "Node b95c6b54-15a0-4cc4-a7c9-9566f2bd9fbd failed to validate deploy image info. Some parameters were missing. Missing are: ['instance_info.image_source']"}, power: {'result': True}, storage: {'result': True}" https://paste.opendev.org/show/borGja9FA4jBiqvMLSjt/ But from my understanding of that code & OpenStack | 16:31 |
| Sandzwerg[m] | https://github.com/sapcc/ironic/blob/stable/xena-m3/ironic/drivers/modules/agent.py#L466-L473 / https://opendev.org/openstack/ironic/src/branch/master/ironic/drivers/modules/agent.py#L436-L443 the image source is not an image parameter but something set by nova(?). Some time later I was able to make a successful deployment, using the exact same image as during this error. | 16:31 |
| dtantsur | rpittau: yay https://launchpad.net/virtualpdu | 16:33 |
| rpittau | \o/ | 16:33 |
| TheJulia | dtantsur: yeah, basically I think since there is not really the humans around anymore | 16:38 |
| TheJulia | dtantsur: and dell has basically chosen redfish first/only path forward | 16:39 |
| TheJulia | Sandzwerg[m]: how did you attempt to perform the deployment each time, generally yes we expect image source to be a UUID to an image and it missing seems very problematic for a deployment | 16:40 |
| TheJulia | arne_wiebalck: I suspect unrelated, based upon the error on the agent console, it looks like eventlet/ssl related :\ | 16:42 |
| TheJulia | https://www.irccloud.com/pastebin/c0lwJ9gX/ | 16:42 |
| Sandzwerg[m] | TheJulia: There is some automation that did it. When I look at the request spec of the instance in nova I see the image data including UUID, that's why I'm a bit puzzled that nova/ronic thinks that it's missing. https://paste.opendev.org/show/bANjk3pJ7yFMUdecTURz/ has the instance spec | 16:45 |
| arne_wiebalck | thanks for checking TheJulia! The same tests failing on the first attempt, so I was hesitating to do a bare recheck again. | 16:45 |
| dtantsur | TheJulia: it would be good to have some written artefact for it. There is a certain impact for us in Metal3. | 16:46 |
| Sandzwerg[m] | I also don't remember seeing this issue before. As the image itself seems to work on ironic the image itself should be okay as well. I also checked if some ironic service was restarting during that timeframe but I found nothing. Haven't checked nova yet, will do that quickly. | 16:46 |
| TheJulia | Sandzwerg[m]: the only thing I can think was there was some race condition, possibly between scheduling across multiple distinct nova computes ?! which helped create that condition. | 16:47 |
| TheJulia | dtantsur: yeah, it is on my todo list | 16:47 |
| dtantsur | nice | 16:47 |
| dtantsur | same with ibmc and ilo4, if that ever happens | 16:47 |
| TheJulia | Sandzwerg[m]: if you can pin down what exactly, that might help, but the only thing I can think of is two locks maybe came in at the same time from two distinctly different updates | 16:48 |
| TheJulia | and one either failed, or just didn't complete, or $something, dunno. | 16:48 |
| TheJulia | or one started failed and yeah | 16:48 |
| TheJulia | Sandzwerg[m]: if you have multiple nova computes, I would start looking there, espescialy if peer_list is not set | 16:49 |
| TheJulia | dtantsur: Yeah, I was sort of thinking a whole list of them | 16:49 |
| Sandzwerg[m] | TheJulia: Interesting, as we currently have a 1:1 connection between a block of nodes (a small group of nodes) and a ironic & nova conductor, so two conductors working on it in parallel should be impossible but yeah, something like a race condition sounds sensible | 16:49 |
| TheJulia | Sandzwerg[m]: well, you can have one nova conductor, but many nova-computes | 16:50 |
| TheJulia | the computes do the actual lift | 16:50 |
| masghar | TheJulia: dtantsur: So would it be safe to say that mention of the "idrac-wsman" raid interface can now become "idrac-redfish" in metal3? Or am I jumping ahead? | 16:50 |
| dtantsur | masghar: jumping ahead a bit :) | 16:50 |
| dtantsur | we'll need to make sure that the way we configure the node is compatible with Redfish | 16:50 |
| TheJulia | the first thing is for us to define *what* exactly is deprecated | 16:50 |
| masghar | Alright. And we also keep ilo5 as it is in metal3? Yeah exactly | 16:51 |
| Sandzwerg[m] | TheJulia: for us that is also 1:1 at the time so 1 block - 1 ironic conductor and 1 nova-compute-ironic | 16:51 |
| TheJulia | and then go from there along to what dtantsur is saying | 16:51 |
| dtantsur | masghar: yeah, ignore both for now | 16:51 |
| TheJulia | Sandzwerg[m]: weird.... | 16:51 |
| masghar | Alright ^ | 16:51 |
| TheJulia | Sandzwerg[m]: do you have a read replica? | 16:51 |
| TheJulia | for your database? | 16:51 |
| Sandzwerg[m] | No we have a single mariadb for a region, that will be eventually change to a galera cluster but so far hasn't been our bottle neck. I'm aware that our setup is not the usual style to do it, and were thinking about swichting to a more "default" one | 16:53 |
| TheJulia | Nah, that is fine, just trying to understand where things went sideways in the workflow | 16:54 |
| TheJulia | maybe a thread got lost?! | 16:54 |
| TheJulia | arne_wiebalck: unfortunately I'm not sure rechecking will just make it work. Might be one of those things where all we can really do is unblock the branch to merge general fixes at this point | 16:54 |
| TheJulia | dtantsur might recognize or be aware of the error, but being yoga branch I suspect we're not going to see it fixed with how it builds in CI | 16:55 |
| arne_wiebalck | TheJulia: the other option is to abandon that specific backport and we carry fwd our downstream patch instead (for this one version) | 16:56 |
| arne_wiebalck | TheJulia: no need to break stable/yoga :) | 16:56 |
| TheJulia | well, I think it is already broken :) | 16:56 |
| TheJulia | but yeah | 16:56 |
| arne_wiebalck | heh | 16:56 |
| TheJulia | so, this sort of explains I think I saw on ironic-tempest-plugin, but instead just routed around by fixing the job configs to what branches were voting | 16:57 |
| Sandzwerg[m] | TheJulia: It might be fine but we'll probably change our setup at some point none the less. Makes it more resilent if there is more than one conductor/nova-compute and probably also useses less resources. Regarding Thread lost: Dunno, I'll look at the surrounding logs to see if I missed anything else. It's not a big issue right now but I never saw it before and I thought I ask here so I don' t miss anything. Thanks for your help | 16:58 |
| Sandzwerg[m] | :) | 16:58 |
| TheJulia | dtantsur: there is no better time than now to write deprecations | 16:58 |
| TheJulia | Sandzwerg[m]: I think I've seen something like that when two computes were interacting with the same node, but yeah.... that wouldn't bein your environment, so don't know. | 16:59 |
| rpittau | good night! o/ | 17:00 |
| Sandzwerg[m] | TheJulia: I should have checked the log better before instead of focusing on error message. It seems they tried to do a boot-from-volume which obviously won't work with ironic. There is a flag in the instance spec | 17:10 |
| opendevreview | Merged openstack/ironic-python-agent stable/2023.2: Conditional creation of RAIDed ESP for UEFI Software RAID https://review.opendev.org/c/openstack/ironic-python-agent/+/899325 | 17:36 |
| opendevreview | Merged openstack/ironic-python-agent stable/zed: Conditional creation of RAIDed ESP for UEFI Software RAID https://review.opendev.org/c/openstack/ironic-python-agent/+/899860 | 18:03 |
| opendevreview | Julia Kreger proposed openstack/ironic master: Multiple driver related deprecations https://review.opendev.org/c/openstack/ironic/+/901501 | 19:05 |
| opendevreview | Julia Kreger proposed openstack/ironic master: Deprecate configuration molds https://review.opendev.org/c/openstack/ironic/+/901502 | 19:05 |
| opendevreview | Julia Kreger proposed openstack/sushy-tools master: Simplify UEFI logic and change the UefiHttp flow https://review.opendev.org/c/openstack/sushy-tools/+/901208 | 19:09 |
| TheJulia | dtantsur: revised, with error messages | 19:10 |
| TheJulia | good issue to spot | 19:10 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!