Monday, 2023-07-10

rpittaugood morning ironic! o/07:58
dtantsurThe person behind the iBMC driver is, as we expected, just a freelancer and does not know the current situation. But they promised me to forward my question.08:47
dtantsurGood morning, happy Monday!08:47
rpittauMondays will be Mondays :/08:57
*** Continuity_ is now known as Continuity08:58
qianbiaoHi, Ironic.09:46
qianbiaoAm I on right server of openstack-ironic?09:46
opendevreviewDmitry Tantsur proposed openstack/bifrost master: [PoC] Life without Inspector  https://review.opendev.org/c/openstack/bifrost/+/88793409:51
dtantsurqianbiao: you are indeed, hi again :)09:54
JayFWelcome :)09:54
* JayF also says hello to the europe shift from London, UK09:54
dtantsurJayF: so weird to see you so early, good morning :)09:54
qianbiaohi dtantsur09:56
qianbiaoHi JayF, i still remember you. :)09:57
qianbiaodtantsur i have forward your mail to iBMC team, if i got any further info, will let you know.09:57
dtantsurqianbiao: thanks much!09:58
JayF:D09:59
qianbiaofeel guilty to bring inconvenience to ironic community. Due to the aanctions against Huawei, x86 business of Huawei has mostly been sold. 10:00
qianbiaobut there are still some agent they are using the iBMC driver as I know.10:01
qianbiaojust they are running on pretty old version of openstack.10:01
JayFIt's not your fault, don't feel bad. We're all working under a constantly changing set of rules :)10:03
dtantsurTrue that.10:04
qianbiaothanks10:06
rpittauhey JayF welcome to the old world :)11:09
opendevreviewDmitry Tantsur proposed openstack/bifrost master: [PoC] Life without Inspector  https://review.opendev.org/c/openstack/bifrost/+/88793411:14
dtantsurrpittau, JayF, should we merge https://review.opendev.org/c/openstack/ironic/+/887971/ or do we have other plans to fix the jobs?11:18
rpittaudtantsur: unfortauntely if that was proposed means that we don't have clues at the moment on how to fix that on jammy11:18
rpittauso going to approve that11:18
dtantsur+111:25
fricklerare those devstack based jobs? devstack is going to drop focal support any moment11:37
dtantsurfrickler: yep, they are...11:39
dtantsurso we may need to pin devstack as well, oh my11:39
opendevreviewMerged openstack/bifrost master: Fix key-order[task] linter warnings  https://review.opendev.org/c/openstack/bifrost/+/87485411:43
rpittauyeah, we were really late with jammy :/11:44
opendevreviewPierre Riteau proposed openstack/bifrost stable/zed: Skip unnecessary SDK get_machine calls  https://review.opendev.org/c/openstack/bifrost/+/88311611:52
JayFironic-standalone job is devstack, not bifrost based?12:12
JayFI don't think I realized that12:12
opendevreviewMerged openstack/ironic master: Move standalone jobs to focal  https://review.opendev.org/c/openstack/ironic/+/88797112:31
minWihi folks, I'm trying to wrap my head around the sushy-emulator and the ability to map virtual media via libvirt... I'm trying to map an image file (either raw or qcow2) as a usb stick... but it always gets mapped as floppy... I've tried creating a bigger image file (like 10 MB image file, >1,44MB so maybe it is recognised as usb...) and also modified the SUSHY_EMULATOR_VMEDIA_DEVICES variable to remove floppy fr12:58
minWiom the mediatypes array... but still it is created as "<disk type='file' device='floppy'>". what am I doing wrong? Thanks!12:58
JayFI'm curious what kind of testing you're doing to need to do that with sushy-tools12:59
opendevreviewPierre Riteau proposed openstack/bifrost stable/yoga: Skip unnecessary SDK get_machine calls  https://review.opendev.org/c/openstack/bifrost/+/88311713:00
opendevreviewPierre Riteau proposed openstack/bifrost stable/xena: Skip unnecessary SDK get_machine calls  https://review.opendev.org/c/openstack/bifrost/+/88311813:00
JayFWe discourage, in the strongest terms, use of sushy-tools, VirtualBMC, VirtualPDU in a production environment13:01
TheJuliagood morning13:02
JayFo/13:02
minWiyeah, this is not production environment don't worry :) basically I'm trying to deploy SUSE Linux Enterprise Micro in an automated fashion... and I'm trying to use combustion to do the initial setup... but how it works is SLE Micro expects a disk with a "combustion" label, so if the device is mapped as a floppy it doesn't work... I've tried this in a real baremetal server and it gets mapped as usb and it works..13:04
minWi. but for testing purposes I prefer to use VMs for many reasons :)13:04
JayFaha, yeah that's good stuff, we just give the disclaimer13:04
JayFyou'd be surprised the crazy ideas we hear :)13:04
minWi:)13:04
JayFnow my next comment is, I didn't know sushy-tools did disk emulation at all, so I'm not sure how much I'll be helpful :D 13:04
TheJuliaI'd remove all logic for floppies at this point, tbh13:04
rpittauyes please13:04
* TheJulia attempts to wake up13:07
* TheJulia caffinates13:08
TheJuliahttps://github.com/openstack/sushy-tools/blob/master/sushy_tools/emulator/resources/systems/libvirtdriver.py#L1074 seems pretty specific, it comes down to the device it looks like13:21
dtantsurJayF: re standalone vs bifrost: we don't have tempest support in bifrost, nor do we have neutron there13:35
JayFWhat do folks think about a partially-baked Arista and Cisco (two drivers) support in NGS? 13:43
JayFCaveats are primarily that I don't think they've been tested against a wide variety of switches, and I don't think they support all use cases13:43
JayFbut obviously documenting those limitations would have to be part of getting it upstream13:43
TheJuliaokay, I think I've reproduced the standalone job failure13:58
TheJuliaat least, playing with the held CI node13:58
rpittauTheJulia: on jammy ?13:58
TheJuliayup13:59
rpittauoh yeah, of course13:59
rpittaucool13:59
TheJulialong story short, it is neutron not being ready, and ipxe changes gears, gets the same response, but not the response it wants to try the next overall capability so it just retries again until it fails13:59
TheJuliaat least, that is my *high level* guess13:59
TheJuliaand... dnsmasq disappeared on the job run causing it all to start off14:05
TheJuliastandalone, of all of our jobs, has the most discrete scenarios tested14:11
TheJuliaso all of the dhcp updates increase the odds of issues I guess14:12
TheJuliaJul 10 13:52:01 np0034614991 kernel: dnsmasq[402665]: segfault at c378 ip 00007f81458f047e sp 00007fff0db334a0 error 4 in libc.so.6[7f8145873000+195000]14:14
TheJuliaJul 10 14:14:03 np0034614991 kernel: dnsmasq[402863]: segfault at 7318 ip 00007f3940b9147e sp 00007ffc8df4f010 error 4 in libc.so.6[7f3940b14000+195000]14:14
JayFwhoa that's kinda crazy14:14
JayFupstream bug reporting time I guess14:15
TheJuliaso boom, there is our problem14:15
opendevreviewJay Faulkner proposed openstack/networking-generic-switch master: WIP: Add VLAN and Port whitelist  https://review.opendev.org/c/openstack/networking-generic-switch/+/88804714:16
dtantsurdnsmasq segfault, omg14:16
rpittauso dnsmasq issue on jammy, yay14:17
rpittaucould be also why the other job was failing14:18
dtantsurfrickler: I wonder if devstack could keep focal support until this segfault is solved14:18
TheJuliaso looks like there *is* a fix for netbooting related crashes in 2.8714:26
TheJuliaubuntu ships 2.8614:26
rpittau\o/14:27
TheJuliabut we dont' seem to invoke the config to cause it14:27
rpittaulet's install it from lunar? I'm semi-kidding14:27
TheJuliahttps://lists.thekelleys.org.uk/pipermail/dnsmasq-discuss/2022q3/016562.html14:35
TheJuliadifferent location, but likely same casue. It is happening when changes are getting loaded, I'm wondering if it still attempts it even if disabled14:35
TheJuliaso ubuntu appears to be shipping only 2.8614:38
TheJuliahttps://packages.ubuntu.com/search?keywords=dnsmasq&searchon=names&suite=kinetic&section=all14:38
TheJuliaoh, lunar has latest14:38
dtantsurit's probably possible to install an isolated package from lunar..14:41
TheJulialooks like a fast path to dependency hell14:53
TheJuliaso... build dnsmasq?!14:56
rpittauit has one more deps though14:56
opendevreviewJay Faulkner proposed openstack/networking-generic-switch master: Fix delete_network_postcommit KeyError  https://review.opendev.org/c/openstack/networking-generic-switch/+/88805114:57
rpittauit's probably doable directly with the same deps for 2.8614:57
rpittauorrrrr14:57
rpittauisntall the one from focal :D14:57
rpittauwe used to do that for another pkg in the past14:57
TheJuliasudo dpkg -i dnsmasq-base_2.89-1_amd64.deb libnftables1_1.0.6-2_amd64.deb  libgmp10_6.2.1+dfsg1-1.1ubuntu1_amd64.deb  libjansson4_2.14-2_amd64.deb libnftnl11_1.2.4-2_amd64.deb dnsmasq-utils_2.89-1_amd64.deb 14:59
TheJulia <-- finally works14:59
rpittaucool!14:59
TheJuliaJayF: you going to run the meeting since you seem semi-around?14:59
* rpittau waits before pulling the trigger15:00
TheJuliaPull it! There is always the chairs command15:00
rpittauright!15:00
rpittau#startmeeting ironic15:00
opendevmeetMeeting started Mon Jul 10 15:00:46 2023 UTC and is due to finish in 60 minutes.  The chair is rpittau. Information about MeetBot at http://wiki.debian.org/MeetBot.15:00
opendevmeetUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.15:00
opendevmeetThe meeting name has been set to 'ironic'15:00
TheJuliao/15:00
rpittauWelcome to our weekly meeting!15:00
rpittauThe meeting agenda can be found here:15:00
rpittau#link https://wiki.openstack.org/wiki/Meetings/Ironic#Agenda_for_next_meeting15:00
rpittauit's been a while I haven't done this, so be patient :)15:01
rpittau#topic Announcements/Reminder15:01
rpittaugreat news everyone!15:01
rpittauthe next PTG has been announced15:01
rpittauThe next PTG will take place virtually October 23-27, 2023! https://openinfra.dev/ptg/15:01
rpittaulet's add a note about that15:02
rpittau#note The next PTG will take place virtually October 23-27, 2023! https://openinfra.dev/ptg/15:02
dtantsuro/15:03
rpittau#note usual friendly reminder to review patches tagged #ironic-week-prio, and tag your patches for priority review15:03
rpittauone more reminder, I've added the bobcat timeline to the agenda15:04
rpittau#note we're just 6 weeks away from the Bobcat release 15:04
rpittaunope, sorry, I can't count15:05
rpittauthe next important date is August 24, which is 6 weeks away from the Bobcat relase15:05
rpittauanyone else has anything to announce / remind ?15:06
rpittaualright, moving on15:06
rpittaunot huge attendance today :)15:06
TheJuliaumm15:06
TheJuliaif we're so far from release, why did we cut the client library release last week?15:07
JayFWe just passed B-2, which is libraries released and mostly freezed15:07
rpittauyeah15:07
JayFB-2 is the TC+releases team documented cutoff for library releases15:07
rpittaunext step is non-client libs freeze15:07
JayFI thought B-2 was all libs freeze, it was just clients?15:07
* JayF is only like half here, is working downstream15:08
rpittauclient libs freese is august 3115:08
TheJuliashould that t-5 weeks per the schedule?15:08
rpittauthe latest client release was not a cut15:08
rpittauit was a milestone15:08
TheJuliaoh, why are we doing milestone cuts then?15:08
rpittauI think we're doing them since at least 2023.115:09
TheJuliaokay, I thought we cut the stable branch on python-ironicclient last week, but it seems we did not15:09
rpittaufor the cycle-with-intermediary clients and libs15:09
TheJuliawe just did a regular releae15:10
TheJuliaso there is still time \o/15:10
rpittauyep15:10
rpittau:)15:10
rpittauglad to bring good news :)15:10
rpittauoky dokey, can we move on ?15:10
TheJuliasure15:11
rpittau#topic Review previous action items15:11
rpittauI don't see any action items from last meeting15:11
rpittauso skipping it15:11
rpittauto the fun part15:11
rpittau#topic Review Ironic CI Status15:11
rpittauhow to summarize it?15:12
rpittaucontrolled chaos?15:12
rpittaumild enthropy vortex?15:12
rpittauwe have 3 jobs pinned to focal, and the support to focal in devstack is going away soon(TM)15:12
dtantsur:D15:13
TheJuliaso... well15:13
rpittaubut it looks like a potential root cause has been found by TheJulia :)15:13
TheJuliaLatest news that is on jammy, with the lunar dnsmasq packages, we still detonate15:13
rpittau\o/15:13
rpittaudo we want to try with the focal package?15:14
TheJuliaI think, in order to provide an accurate bug report, at a minimum we'll need to pull the source and try latest, or we could punt on focal dnsmasq packages15:14
rpittaufocal has version 2.8015:14
TheJuliayeah, the race in reload issue was introduced in 2.86 it looks like15:15
rpittaugreat!15:15
TheJuliabut we may be hitting something else15:15
TheJuliaanyway, it crashes when we start to reload it15:15
TheJuliaso... *shrugs*15:15
rpittauconsidering that the devstack sword is upon us, probably using focal packages while moving to jammy nodes for the time being could be viable ?15:16
TheJuliapossibly15:18
TheJuliawe've sort of long seen issues15:18
TheJuliaand this is compounded by dnsmasq attempting httpboot as well15:18
rpittaualright, we can probably  take the discussion back again outside of the meeting15:19
rpittauat least we have a possible cause of the issues15:19
TheJulia++15:19
rpittauok, moving on15:19
rpittau#topic 2023.2 Workstream15:19
rpittau#link https://etherpad.opendev.org/p/IronicWorkstreams2023.215:19
rpittauany update worth sharing?15:20
rpittauwe should probably review the firmware interface patches :/15:20
dtantsurnothing new on the inspector front: waiting for reviews15:20
rpittauthat's looking good :)15:21
TheJulia++15:21
JayFI would really, really like to get a good review pass done this week, but if you end up with anything short a +2 ping me directly15:21
JayF(reminder I'm in BST this week)15:21
TheJulialooks like we're making good progress on dpu stuff, but overall the CI issues have been really bad for us15:21
rpittausad but true :(15:22
rpittauthanks for the updates in the etherpad15:22
rpittaumoving on if we don't have other updates15:22
rpittau#topic Open Discussion15:23
rpittauanyone has anything for open discussion ?15:24
JayFIf you're an Ironic contributor and have not gotten socks, let me know.15:24
JayFSame for stickers.15:24
rpittauJayF: I miss socks and stickers :D15:24
JayFemail me at jay at gr-oss dot io and I'll get them out when I'm back home15:25
JayFthat goes for ~anyone who is a contributor who wants some15:25
rpittauawesome! thank you!15:25
JayF(if you aren'15:25
JayF(if you aren't  a contributor and want one, email me too, but contributors will get the first stock)15:25
rpittauany other topic to discuss ?15:26
rpittaualright, friendly reminder that I will be your host for the next meeting too :)15:27
rpittauthanks everyone!15:27
rpittau#endmeeting15:27
opendevmeetMeeting ended Mon Jul 10 15:27:20 2023 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)15:27
opendevmeetMinutes:        https://meetings.opendev.org/meetings/ironic/2023/ironic.2023-07-10-15.00.html15:27
opendevmeetMinutes (text): https://meetings.opendev.org/meetings/ironic/2023/ironic.2023-07-10-15.00.txt15:27
opendevmeetLog:            https://meetings.opendev.org/meetings/ironic/2023/ironic.2023-07-10-15.00.log.html15:27
TheJulianow back to our dnsmasq topic!15:27
* rpittau facepalm15:27
TheJuliathoughts/ideas? 15:27
TheJuliaheh15:27
TheJuliaI think forward looking wise, our path is most likely just to drop dnsmasq15:28
TheJuliahonestly15:28
TheJuliabut we're not entirely there... *and* I seem to be having issues on that test job15:28
rpittauthe idea of submitting a bug is not bad, I did it for the other pkg I don't remember the name of and never got an answer!15:30
rpittauBut at some point the issue was fixed15:30
rpittauand more seriously, pinning the pkg to focal would work at least for the time being, it's the same approach we followed for the other package15:30
rpittauit was the ovmf package!15:31
TheJuliawell, to what bug tracker? there is not one for dnsmasq, it is the mailing list15:33
TheJuliaubuntu doesn't appear to have it configured for dnsmasq either15:33
rpittaummm last time it was submitted to ubuntu15:34
TheJuliaretrying on focal15:35
TheJuliawe could file a bz in lp and tag dnsmasq/ubuntu directly15:37
TheJuliadunno how well that would work15:37
rpittaufor ovmf we kept the old package for a while, it got fixed eventually....15:38
TheJuliawell, I've got 3 vms in clean wait15:40
TheJuliacleaning15:40
TheJuliaso that is a good-ish sign15:40
rpittaufingers crossed15:42
TheJuliasaving commands for later https://www.irccloud.com/pastebin/pSfKpU82/15:43
TheJuliait has yet to crash in 8 minutes, with multiple updates, so far so good15:43
TheJuliakernel error log https://www.irccloud.com/pastebin/sTAR03K0/15:45
rpittauwith focal?15:46
TheJuliajammy host, focal packages seem to be good15:46
rpittauok15:46
TheJulia11 minutes and counting, multiple updates, still original process running15:47
* TheJulia tells another node to go clean, again15:48
fricklerhttps://bugs.launchpad.net/ubuntu/+source/dnsmasq would be the ubuntu bug tracker15:50
eanderssonYou can always do something fun like this.15:51
eanderssonhttps://github.com/openstack/zaqar/commit/c4502d1593ba9e417d98b018d9a74c5d3d8be92015:51
JayFWe need to get that bug fixed upstream of us15:52
JayFMan I wish I wasn't downstream this week15:52
JayFthis is nerd sniping me hard but I don't have time for it this week :( 15:52
rpittaueandersson: hey! long time no see! :) we actually did that for ovmf :D15:54
eanderssono/15:54
eanderssonI always feel dirty going that route.. but if it works it works I guess :D15:54
rpittauheh not a lot of choice here :/15:55
TheJuliadoes it look like I'm missing anything ? https://bugs.launchpad.net/dnsmasq/+bug/202675716:09
TheJuliaand tagged16:11
TheJuliachallenge I see just adding it is if anything else gets installed, we could end up with poisoned package senario16:11
TheJuliaI'd almost prefer to just directly download the packages and install them16:12
rpittauthat looks good TheJulia 16:13
rpittauTheJulia this is what we did for ovmf https://review.opendev.org/c/openstack/ironic/+/647687/5/devstack/lib/ironic 16:14
rpittauI need to split now, see you tomorrow! o/16:15
TheJuliadmesg16:31
TheJuliadoh16:31
opendevreviewJulia Kreger proposed openstack/ironic master: CI: Use focal dnsmasq  https://review.opendev.org/c/openstack/ironic/+/88805816:40
TheJuliaeandersson: a little inspiration, just removing the file when done16:40
TheJuliathanks!16:40
eanderssonNice. I like that touch16:44
eanderssonCleaner :D16:45
opendevreviewVerification of a change to openstack/ironic master failed: Fix the HTTP code of the BadRequest exception  https://review.opendev.org/c/openstack/ironic/+/88727116:55
opendevreviewJulia Kreger proposed openstack/ironic master: Add wait step  https://review.opendev.org/c/openstack/ironic/+/87906117:18
opendevreviewJulia Kreger proposed openstack/ironic master: DNM: Eventlet science  https://review.opendev.org/c/openstack/ironic/+/88799617:22
opendevreviewJulia Kreger proposed openstack/ironic master: CI: Use focal dnsmasq  https://review.opendev.org/c/openstack/ironic/+/88805817:48
opendevreviewJulia Kreger proposed openstack/ironic master: CI: Use focal dnsmasq  https://review.opendev.org/c/openstack/ironic/+/88805818:05
opendevreviewJulia Kreger proposed openstack/ironic master: CI: Use focal dnsmasq  https://review.opendev.org/c/openstack/ironic/+/88805818:22
opendevreviewJulia Kreger proposed openstack/ironic master: CI: Use focal dnsmasq  https://review.opendev.org/c/openstack/ironic/+/88805818:35
TheJuliaso I think we're going to need to pull down the dpkgs down individually, adding it and trying to rely upon debian's package manager to do the right thing just doesn't seem to work, it can't seem to find the package in the configs19:25
opendevreviewJulia Kreger proposed openstack/ironic master: CI: Use focal dnsmasq  https://review.opendev.org/c/openstack/ironic/+/88805819:57
larsksIs there a better place than here to ask questions about sushy? I have some questions about accessing actions on a redfish resource.19:58
TheJulialarsks: o/ best place, whats up?21:16
larsksTheJulia: hey. I want to access some redfish actions on our dell hardware, and I'm not seeing an api for doing that. There is support for *specific* actions (like the system_reset method on the system object), but I don't see something for arbitrary actions.21:17
larsksIs the best option calling sushy.Sushy._conn.post directly?21:17
TheJuliaEh, I'd consider calling a post directly a semi-workaround, but the reality is, afaik, we don't automatically expose internal methods. We build an appropriate wrapper as needed. System reset along the lines of reset to factory defaults?21:19
larsksNo, I was pointing that out as an Action for which there was currently explicit support.21:19
larsksI am looking into using the SecureErase option on drives in a bunch of dell servers.21:19
larsksIt looks like that means (1) get the action url from <object>.json['Actions'][<action_name>]['target'], and then (2) pass that to <Sushy>._conn(url, ...) with appropriate parameters.21:20
larsksNot sure if that action is actually supported on our idracs, though. Maybe a better option is just to enroll everything in ironic and then trigger a clean cycle...21:22
TheJuliayeah, that might be a good option :) 21:25
TheJuliaGiven the whole nature of just in case things are not BMC managed/visible21:25
TheJuliaso we did like a virtual media attachment action, and I remember correctly it took some non-zero effort to just getit to navigate/handle21:26
TheJuliafwiw, there is also a whole retirement logic flow for operators to help take care of wiping servers prior to them being removed21:28
larsks+1. I'm going to explore that option instead, I think; it looks like the SecureErase action, though advertised, isn't functional.21:29
TheJuliawheeeee :(21:30
opendevreviewJulia Kreger proposed openstack/ironic master: Enable vendor interfaces to be called as steps  https://review.opendev.org/c/openstack/ironic/+/87908921:39
TheJulialarsks: For a minute, I was worried you were going to try to do something like the ilo "reset for handoff wipe" logic, which ... does a few truly "evil" things to the bmc. I think we actually flag the machine as in maintenance afterwards if it is invoked21:39
TheJuliarpittau: so, did we figure out a root cause on the snmp job failure? or was it just the dnsmasq curse21:44
TheJuliaI've rechecked the dnsmasq fix, it appears solid and no errors in the latest job run21:47

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!