| janders | good morning Ironic o/ | 01:07 |
|---|---|---|
| janders | JayF TheJulia when you have time I wanted to touch base with you regarding https://review.opendev.org/c/openstack/ironic/+/881358 on behalf of our partner - it seems it's ready to go, has 2 +2s, would it be possible to merge it? Thank you in advance! | 01:08 |
| JayF | I didn't realize I was the second +2, I'll approve | 03:33 |
| JayF | No, I'm only the first in that patchset | 03:33 |
| JayF | The new UI makes it more difficult to tell | 03:33 |
| janders | thank you Jay! yeah I am struggling with the new GUI too | 03:36 |
| *** dmellado9 is now known as dmellado | 05:04 | |
| rpittau | JayF: thanks for checking, I saw that too but it looks like normal behavior, it's easy to verify in any other passing snmp job with focal, for example https://de19e57dd79c29d6fa65-9c39e1b31aead2b89889cdc7bed43508.ssl.cf2.rackcdn.com/882732/2/check/ironic-tempest-wholedisk-bios-snmp-pxe/959ed74/controller/logs/screen-virtualpdu.txt | 06:41 |
| rpittau | the libvirt "domain not running" error is due to the domain being powered off, probably the message can be improved | 06:42 |
| rpittau | good morning ironic! o/ | 06:43 |
| opendevreview | Riccardo Pittau proposed openstack/ironic master: Migrate to pysnmp lextudio ecosystem https://review.opendev.org/c/openstack/ironic/+/882917 | 06:53 |
| opendevreview | Sandeep Yadav proposed openstack/ironic master: [DNM] this is just a test https://review.opendev.org/c/openstack/ironic/+/882918 | 07:24 |
| opendevreview | Rodolfo Alonso proposed openstack/ironic master: Test LP#2019186 https://review.opendev.org/c/openstack/ironic/+/882936 | 09:40 |
| iurygregory | good morning Ironic | 11:04 |
| opendevreview | Mahnoor Asghar proposed openstack/ironic master: Add to Redfish hardware inventory collection https://review.opendev.org/c/openstack/ironic/+/881783 | 11:54 |
| opendevreview | Mahnoor Asghar proposed openstack/ironic master: Add to Redfish hardware inventory collection https://review.opendev.org/c/openstack/ironic/+/881783 | 12:15 |
| opendevreview | Pierre Riteau proposed openstack/bifrost master: Skip unnecessary SDK get_machine calls https://review.opendev.org/c/openstack/bifrost/+/882950 | 13:28 |
| JayF | rpittau: okay, in that case I'm back to wtf-ville with you | 13:46 |
| TheJulia | good morning | 13:47 |
| iurygregory | morning TheJulia JayF o/ | 13:48 |
| JayF | o/ | 13:49 |
| * TheJulia does her best zombie impression | 15:02 | |
| TheJulia | brrrraaaaaaiiinnns | 15:05 |
| TheJulia | Zombie movie Thursday?! | 15:05 |
| jrosser | has anyone ever seen something like this with dell r730 era nodes 1) pxeboot tries a seemingly random nic out of all available ones 2) immediately then boots from HDD without trying the rest of the PXE capable interfaces | 15:10 |
| TheJulia | jrosser: UEFI mode or bios mode? | 15:11 |
| jrosser | in this case it is bios mode | 15:12 |
| jrosser | we're having a bit of a nightmare with these actually to get the behaviour predicatable and reliable enough | 15:13 |
| TheJulia | What I've done on r730s is disable all the other network interfaces which shouldn't be setup for pxe | 15:25 |
| TheJulia | concurrently set pxe_enabled=False on the other network ports if there are mutliple configured in ironic | 15:26 |
| jrosser | is that disable them in the server setup/idrac? | 15:26 |
| TheJulia | yers | 15:26 |
| TheJulia | yes | 15:26 |
| jrosser | we've only specified one interface in the ironic config | 15:27 |
| jrosser | doing factory reset has improved things a bit | 15:27 |
| TheJulia | I think only ilo's driver knows to set/enforce that via the bmc | 15:27 |
| jrosser | but still there are "some times" "some of the r730" do this wierd boot order | 15:27 |
| TheJulia | so in idrac, you really need to do it server side config | 15:27 |
| jrosser | and then those get stuck in cleaning as they end up rebooting back into the last deployment | 15:28 |
| TheJulia | wheeeeeee | 15:28 |
| TheJulia | ... I'd look to see if UEFI mode to see if that changes since it overall changes the model/process and even running state config | 15:28 |
| jrosser | yes, we were using the idrac driver but that was worse from this boot order resepect | 15:29 |
| jrosser | it set it persistently i think then the bmc got confused and it would never change again | 15:29 |
| jrosser | today changed it all back to ipmi and things are better | 15:30 |
| dtantsur | This is a bit concerning: https://zuul.opendev.org/t/openstack/builds?job_name=ironic-tempest-bfv&project=openstack/ironic | 15:30 |
| dtantsur | The last two runs failed with a confusing error. | 15:30 |
| dtantsur | Failed to request detach for volume 640c7e76-99fb-42ae-80b3-5000f102af1f from cinder for node 82fb464a-0581-4590-b049-42a405053084: Invalid volume: Unable to detach volume. Volume status must be 'in-use' and attach_status must be 'attached' to detach. | 15:31 |
| TheJulia | ugh | 15:31 |
| TheJulia | jrosser: I suspect what is at play is the configuration jobs which get applied in the idrac driver. some of that happens automatically with ipmi, but it is much more targetted/one time | 15:33 |
| jrosser | yes, we could see those being applied on the console | 15:34 |
| jrosser | the nodes would get stuck trying to retrieve the power state | 15:34 |
| dtantsur | Hmm, the first failure is actually ConflictNovaUsingAttachment: Detach volume from instance 2cad5e60-d082-44fd-af68-f6875e62579e using the Compute API (HTTP 409) | 15:34 |
| dtantsur | Then further retries cause the error above. | 15:34 |
| jrosser | and if we reset the bmc remotely, that would unwedge things | 15:34 |
| TheJulia | jrosser: what idrac firmware is this? | 15:34 |
| jrosser | whatever is latest on r730, in desperation we upgraded everything | 15:35 |
| TheJulia | 2.70.?100?.something ? | 15:35 |
| TheJulia | dtantsur: I *bet* the CVE fix broke the job | 15:36 |
| dtantsur | which one? | 15:36 |
| dtantsur | https://opendev.org/openstack/cinder/commit/6df1839bdf288107c600b3e53dff7593a6d4c161? | 15:37 |
| dtantsur | TheJulia: do you have an idea for a fix or should I reach to cinder folks? | 15:37 |
| TheJulia | give me 5 minutes | 15:38 |
| TheJulia | I'm on a call at themoment | 15:38 |
| dtantsur | sure | 15:38 |
| dtantsur | yep, that's the issue. "Detected user call to delete in-use attachment. Call must come from the nova service and nova must be configured to send the service token." in CInder logs. | 15:40 |
| TheJulia | yeah, looks like it is masked in the client code | 15:41 |
| TheJulia | we're just calling the client and *boom* | 15:42 |
| dtantsur | I'm a bit puzzled why the error is different the 2nd and 3rd time.. but okay. | 15:42 |
| TheJulia | we need to explicitly use a service token it seems | 15:46 |
| jrosser | TheJulia: we built an update iso which had iDRAC 2.84.84.84 / IDRAC with Lifecycle Controller V.,2.40.40.40 on it | 15:47 |
| JayF | I need to look at IRC logs, I think you all are doing a fix that nova already did | 15:47 |
| JayF | but I lost context because my internet | 15:47 |
| JayF | I think they put a fix in the cinder tempest plugin | 15:48 |
| JayF | but imbw | 15:48 |
| * JayF read the backlog | 15:48 | |
| TheJulia | yeah, i think in this case it is job configuration for us | 15:48 |
| TheJulia | but we need to look at the config | 15:48 |
| opendevreview | Verification of a change to openstack/ironic master failed: [iRMC] Fix parse_driver_info bug enforcing SNMP v3 under FIPS mode https://review.opendev.org/c/openstack/ironic/+/881358 | 15:48 |
| TheJulia | lets mark the job non-voting for now, we're going to need to sort out a proper fix and make sure we're doing the needful at the right time | 15:49 |
| JayF | might not hurt to ask someone from qa team if they have the answer in their back pocket and save some research time | 15:49 |
| JayF | but I'll let you all decide that | 15:49 |
| TheJulia | I'm looking at the code, they added additional checks | 15:49 |
| dtantsur | I don't think Nova can fix it for us, the error seems to be originating from the conductor? | 15:50 |
| TheJulia | https://review.opendev.org/c/openstack/cinder/+/882835/2/cinder/volume/api.py#2543 | 15:50 |
| TheJulia | yeah, because we do our own config toggling as well | 15:50 |
| TheJulia | Nova never updates us beyond the initial bind | 15:50 |
| JayF | ack; the thing I saw was likely nova solving a similar bug then | 15:51 |
| TheJulia | jrosser: hmm, I know the base idrac version is right, the lifecycle controller version seems oldish, but the last time I waded into that I bricked a server | 15:52 |
| jrosser | it does seem old i agree, it's not completely clear whats going on there | 15:53 |
| TheJulia | yeah | 15:53 |
| * jrosser is reminded why these were pretty much the last dells i bought | 15:54 | |
| TheJulia | Anyhow, if you can give UEFI a spin, I suspect the config application for interfaces will behave differently | 15:54 |
| jrosser | ok thats interesting to know, just getting these stable will be a big win | 15:55 |
| jrosser | it's turned into a diversion from actually what we were trying to do now | 15:55 |
| TheJulia | ack, we explicitly set interfaces in the system setup to limit what can boot, which helps with idrac and ipmi drivers | 15:56 |
| TheJulia | in uefi, it should be a static order | 15:56 |
| TheJulia | if *not*, that is definitely a firmware bug | 15:57 |
| jrosser | is uefi more explicit about which pxe device should be used? | 15:58 |
| TheJulia | yes, it gets asserted based upon NVRAM setting order | 15:59 |
| TheJulia | (for the record, the r730s and their blade brethren are the most painful lab machines I get to work with) | 16:00 |
| jrosser | doh :) | 16:00 |
| TheJulia | have you ever watched Legends of Tomorrow? | 16:01 |
| jrosser | i have not | 16:01 |
| TheJulia | ahh, you wouldn't get the idea of what it is like then :) | 16:02 |
| TheJulia | At least, the one in my head | 16:02 |
| rpittau | good night! o/ | 16:10 |
| dtantsur | "It is my pleasure to invite you to the Metal3 Community's Project Team Gathering, which will take place on May 15th, 2023 from 14:00 to 17:00 UTC." | 17:18 |
| dtantsur | see metal3-dev for details | 17:18 |
| dtantsur | See you on Monday o/ | 17:19 |
| JayF | TheJulia: iurygregory: dtantsur: others: Should we cancel the Ironic meeting since it overlaps with the metal3 PTG | 17:20 |
| JayF | Our meetings are rarely well attended and neccessary, and this is a good excuse to not go through the motions for one week lol | 17:21 |
| dtantsur | I can keep one eye on it, but I'm all for canceling too | 17:21 |
| iurygregory | I'm ok with both approaches | 17:21 |
| * TheJulia really wishes we could have gotten the metal3 community to meet alongside us | 17:21 | |
| JayF | we did, we just show up and we meet alongside them | 17:22 |
| JayF | some of them came to our PTG; we could offer the same courtesy | 17:22 |
| TheJulia | ++ | 17:22 |
| dtantsur | TheJulia: half of the community is here ;) | 17:22 |
| TheJulia | I concur, we should cancel our meeting next week | 17:22 |
| TheJulia | dtantsur: true! | 17:22 |
| opendevreview | Julia Kreger proposed openstack/ironic master: DPU modeling - parent_node DB/Model/API https://review.opendev.org/c/openstack/ironic/+/880114 | 17:33 |
| opendevreview | Julia Kreger proposed openstack/ironic master: WIP: execute on child node support https://review.opendev.org/c/openstack/ironic/+/882982 | 17:33 |
| TheJulia | getting there... on the child node execution | 17:33 |
| JayF | 'child {} execution' (another reason I wish I could think of something better than parent/child) | 17:34 |
| TheJulia | yeah | 17:34 |
| TheJulia | so, I think the issue is we just don't operate cinder with a service config in devstack | 17:49 |
| TheJulia | hmm, it *should* be working | 17:56 |
| TheJulia | openstack --os-cloud devstack-system-admin role add service --user ironic --project service --user-domain Default --project-domain Default <-- le sigh | 17:56 |
| TheJulia | apparently we might need to add more stuffs | 18:08 |
| opendevreview | Julia Kreger proposed openstack/ironic master: WIP bfv service change https://review.opendev.org/c/openstack/ironic/+/882985 | 18:09 |
| TheJulia | so the tl;dr is we're not being recognized as a service via the token, we might need to investigate client invocations | 18:17 |
| TheJulia | ^ might need to change too | 18:17 |
| JayF | ack | 18:17 |
| JayF | thank you for diggin this | 18:17 |
| iurygregory | people saying that ipa was looping over all devices they had in the machine... | 18:39 |
| iurygregory | 84 disks... in a storage node | 18:41 |
| iurygregory | OMG | 18:41 |
| iurygregory | .-. | 18:41 |
| iurygregory | and what peple do because the node was still inspecting? they just kill the metal3 pod \o/ | 18:41 |
| iurygregory | another chapther for the hardware book of horror | 18:43 |
| TheJulia | yes, by design... | 18:45 |
| TheJulia | oh, inspecting only?! | 18:45 |
| TheJulia | looping over doing what I guess is the major question | 18:46 |
| TheJulia | none of the io should be paused that long and nor disk actions take long at all | 18:46 |
| TheJulia | unless say... SATA expanders are at play | 18:46 |
| iurygregory | it was looping to identify multipath devices | 18:52 |
| iurygregory | but I think if they provide a root device hints with the disk it would just avoid looping | 18:53 |
| JayF | Chance to have feedback on swag... a couple of options we're looking at (no promises it'll be any of these): https://imgur.com/a/ArPA4ma | 18:53 |
| iurygregory | https://paste.opendev.org/show/bLyoeAuT0IoORYm4b8Ta/ | 18:53 |
| JayF | have maybe 10 minutes to give feedback if you want before we order :) | 18:53 |
| iurygregory | ipa was showing this, but they decided to kill the pod, so I don't have logs from inspector to see .-. | 18:54 |
| iurygregory | ironic socks?! | 18:54 |
| iurygregory | JayF, you rock! | 18:54 |
| JayF | g-research oss rocks* | 18:54 |
| iurygregory | tks g-research =) | 18:55 |
| JayF | seriously though, those look OK? | 18:55 |
| iurygregory | you have a +2 from me | 18:55 |
| JayF | https://imgur.com/a/N5fHYIL was an alternate we were looking at | 18:55 |
| JayF | iurygregory: which one :) | 18:55 |
| iurygregory | ok, now it's complicated | 18:55 |
| iurygregory | because I like blue a lot | 18:56 |
| JayF | the first link had two, fwiw, if you didn't see it | 18:56 |
| iurygregory | woot?! | 18:56 |
| iurygregory | I was looking at only the brown one | 18:56 |
| JayF | yeah the second one is blue there too | 18:56 |
| iurygregory | AHA | 18:56 |
| iurygregory | just saw | 18:56 |
| JayF | almost like a business sock for business (with foundation ironic logo) | 18:56 |
| JayF | and a pajama sock for fun time (with the fun original Lucas logo) | 18:56 |
| iurygregory | JayF, I think you are almost in a war zone | 18:56 |
| JayF | ? | 18:56 |
| iurygregory | because is old pixieboot vs new one | 18:57 |
| iurygregory | I like the old logo a bit more | 18:57 |
| JayF | is intentional. We are both old and new :D we cna have one of each | 18:57 |
| iurygregory | XD | 18:57 |
| iurygregory | I vote for the old one | 18:57 |
| JayF | TheJulia: you have any opinions on these? https://imgur.com/a/ArPA4ma and https://imgur.com/a/N5fHYIL | 18:58 |
| iurygregory | the old mascot in all designs would look awesome fwiw | 18:58 |
| JayF | right now barring further feedback I think we'd use the pair of designs in the first link | 18:58 |
| JayF | iurygregory: I'll see what others think. It can be changed but I like the idea of having one of each tbh | 18:58 |
| iurygregory | ++ | 18:58 |
| TheJulia | the later suspiciously looks hockey-ish ;) | 18:58 |
| TheJulia | no issues :) | 18:58 |
| JayF | TheJulia: fwiw these were all designed by the Insight Softmax HR woman (the actual company on my paycheck that contracts me to GR), she did a good job and saved me from having to design socks :) | 18:59 |
| JayF | so if they were hockey it was accidental | 18:59 |
| TheJulia | \o. | 19:00 |
| TheJulia | err | 19:00 |
| JayF | TheJulia: for that first, brown and grey one, would you go old or new logo? | 19:00 |
| TheJulia | \o/ | 19:00 |
| TheJulia | I would do newer | 19:00 |
| JayF | okay, so that's 2-1, iurygregory loses by a single vote :P | 19:00 |
| TheJulia | heh | 19:00 |
| JayF | or 4-2, depending on if we're using openstack voting rules lol | 19:01 |
| JayF | we're trying to get those socks printed by the summit; I'll have them for people attending then can mail out to other contributors who are unable to attend | 19:01 |
| JayF | dtantsur: rozman: Gauntlet thrown down. We'll have pixie boots socks, now we need metal3 gauntlets or something :D | 19:03 |
| iurygregory | XD | 19:05 |
| iurygregory | metal3 gauntlets omg | 19:13 |
| opendevreview | Iury Gregory Melo Ferreira proposed openstack/ironic master: Add DB model for Firmware https://review.opendev.org/c/openstack/ironic/+/883031 | 19:14 |
| iurygregory | now time to work on the DB API | 19:16 |
| TheJulia | wheeeeeeeee | 19:18 |
| iurygregory | probably finishing this and object structure today (I hope) | 19:27 |
| TheJulia | okay | 19:36 |
| TheJulia | I need to revisit the dpu stuff, I might go get an actual lunch and then revisit | 19:36 |
| opendevreview | Harald Jensås proposed openstack/ironic-tempest-plugin master: rback - Fix vif_attach expected return values https://review.opendev.org/c/openstack/ironic-tempest-plugin/+/883032 | 19:43 |
| iurygregory | TheJulia, happy to take another look at it later today (going to the gym and physiotherapy for my knee) | 19:45 |
| TheJulia | iurygregory: sure, I'm going to get lunch and then revisit if nobody is screaming | 19:46 |
| TheJulia | I think in the mean time, we may just want to mark the bfv stuff non-voting while we work on it | 19:46 |
| iurygregory | ++ | 19:47 |
| JayF | I'd +2 a patch to make bfv non-voting, as long as we're 100% certain it's a devstack/config issue, not an incompatability with the fix for Ironic BFV functionality, if that makes sense | 20:01 |
| JayF | If devstack (or Ironic plugin for it) is broken, but the feature works: NV it. It'd scare me to NV a job that was failing legitimately on a feature we want working. | 20:02 |
| TheJulia | JayF: If it doesn't work with the change I posted, we're looking at likely having to add more code to session handling, which means not a quick fix anyhow | 20:42 |
| JayF | well here's my concern: if we have to patch Ironic to work with the cinder patch | 20:43 |
| JayF | that means any ironic user that installs the security patch and/or config will lose cinder support which is scary | 20:43 |
| TheJulia | yup | 20:43 |
| TheJulia | c'est la vie | 20:43 |
| JayF | that will likely generate another OSSN and need us to do backports+releases | 20:43 |
| TheJulia | The security fix is also breaking for folks using cinder directly | 20:44 |
| TheJulia | since now service token usage is a hard requirement | 20:44 |
| JayF | aha, got it | 20:45 |
| TheJulia | I think since they made an intentional break, it is not OSSN territory for us, just backports + releases | 20:45 |
| TheJulia | since inherently the behavior on the service has changed | 20:45 |
| TheJulia | and is so noted | 20:45 |
| TheJulia | Speaking of! Since I've had lunch! | 20:45 |
| JayF | I'd let fungi make the call if we have to go that way. OSSN or not is just a question of "will operators know how to fix this?" | 20:46 |
| JayF | also I have not had lunch so don't make it too hearty TheJulia ;) | 20:46 |
| * JayF 's wife is out of town, which means he's lost track of all time and space | 20:46 | |
| fungi | i will try to quickly catch up on this discussion and see how i can help | 20:47 |
| TheJulia | Yeah, we're going to have to patch ironic most likely to force service token usage | 20:48 |
| TheJulia | since apparently we're not being classified as a user of the service token | 20:48 |
| JayF | fungi: tl;dr: we're fairly certain at this point Ironic will require a patch to work with Cinder post-their-security-patch | 20:49 |
| fungi | yeah, i was worried the fallout from that might be wider-spread than just nova | 20:50 |
| JayF | Bluntly, looking at the nature of the change, they're going to break downstream code for a lot of people, too. | 20:50 |
| JayF | That was an incredibly disruptive patch | 20:50 |
| TheJulia | Also, the entire model of nova being the only consumer is kind of making me want to table flip | 20:50 |
| TheJulia | Incredibly frustrating :( | 20:50 |
| fungi | yes, i pushed hard repeatedly to find alternatives and was told there basically were none if we want it fixed completely. the design was bad | 20:50 |
| JayF | In the future, someone checking to see if our CI, and other projects, work against a patch like that would be nice. Or at least roping someone from Ironic n. | 20:51 |
| fungi | probably the biggest deciding factor on whether it's a security-something is if the same vulnerability can be exploited by users of ironic in similar ways to how nova could | 20:51 |
| TheJulia | sigh, true | 20:55 |
| TheJulia | I guess it also depends on enforced policy and defaults as well | 20:55 |
| JayF | It's obvious that it's an unintended side effect of the security process, but I'm not super thrilled that we might have Ironic users actively breaking their installs right now in response to an OSSA :\ | 20:56 |
| TheJulia | https://github.com/openstack/ironic/blob/master/ironic/common/policy.py#L1518 | 20:57 |
| TheJulia | a member of the system itself, or an owner/lessee admin | 20:57 |
| TheJulia | so basically humans with administrative rights could only use the feature anyhow | 20:58 |
| TheJulia | otherwise it would be requested through nova | 20:58 |
| JayF | owner/lessee admin is a project-admin though | 20:58 |
| JayF | to be clear, right? | 20:58 |
| TheJulia | which had the elevated access | 20:58 |
| TheJulia | user in project *with* admin role | 20:58 |
| TheJulia | not any user in the project | 20:58 |
| JayF | ack | 20:58 |
| TheJulia | I'd likely need to better understand the nova issue itself, but since we can still get the bind request through and the instance boots, that is a good sign | 20:59 |
| TheJulia | I just suspect all power related recovery/sync with cinder ops are broken | 21:00 |
| TheJulia | since we detach/reattach around power actions in case the endpoint config changes | 21:00 |
| TheJulia | (so we don't inadvertently break things | 21:00 |
| TheJulia | ) | 21:00 |
| * JayF AFK, need to get away from the PC and find some lunch | 21:02 | |
| TheJulia | so, yes. We error on the detach/reattach op we pull on power operations as well | 21:27 |
| fungi | sorry for the uncomfortable silence, had to step away to grab some dinner for a sec | 21:30 |
| fungi | but yes, this is on me. we have a policy that says we don't fix bugs like this. if the only way to fix a vulnerability is by breaking existing deployments then we call it a class b1 "a vulnerability that can only be fixed in master" https://security.openstack.org/vmt-process.html#report-taxonomy | 21:32 |
| fungi | i caved to pressure for backporting security fixes in this case, called it an exceptional circumstance, maybe i should have stuck to our longstanding policy on the matter | 21:33 |
| fungi | normally we'd just say, "well we made some poor design decisions and now realize that there's a vulnerability in all existing versions, if you really care then there's a patch in master or you can wait and upgrade to the next coordinated release" | 21:34 |
| TheJulia | I think the issue is we *should* be recognized as a service which short circuits the check, but It looks like we just aren't for some unknown reason. I guess I need to add some code to dump out what we're using to send on the credentials just to make sure.... | 21:34 |
| fungi | the reason nova folks were involved in the discussion is that the original reporter reported it as a bug against nova, and i should have thought to ask the developers if the breaking change was going to break anyone not already involved in the discussion, but since we don't normally even allow breaking changes like this to be backported it didn't occur to me | 21:35 |
| fungi | no excuses, sorry about that | 21:36 |
| TheJulia | c'est la vie | 21:36 |
| fungi | also we overran our self-imposed 90 day embargo limit by a week trying to do it. if i'd known early on how far-reaching and complicated this was going to be i would have insisted we just switch to working through it in public, and maybe some of this could have been avoided (at the expense of people who get upset when they don't get a heads up in advance with a fix before a | 21:42 |
| fungi | vulnerability becomes public, but those same people are likely to be just as annoyed if the fix breaks something we didn't catch because we were trying to design fixes in a locked room) | 21:42 |
| fungi | it started out looking pretty simple, but then snowballed as the tangle was unravelled revealing more and more corner cases that were still vulnerable | 21:46 |
| TheJulia | looks like we use a different method when talking to the keystoneauth helper, going to look at details now | 21:48 |
| TheJulia | same method, different name | 21:52 |
| JayF | fungi: I think this is one of the rare cases where a public RCA, or at least post to the mailing list for discussion and such, might be a good idea. At least once we get to the end of all of it :) | 22:02 |
| fungi | i absolutely agree we should discuss it at length on the ml. i'm open to additional communication avenues too of course, but that's the bare minimum in my opinion | 22:03 |
| JayF | I say RCA just because in my previous experience that's how I'd have described the process | 22:03 |
| fungi | we had multiple process failings on this one, including some i haven't even brought up | 22:04 |
| JayF | right now my scope of concern is (calling back to the TC/board sync) us violating our biggest feature of being boring :) | 22:05 |
| fungi | this was indeed an unfortunate bout of exciting | 22:06 |
| fungi | one i do not wish to repeat | 22:06 |
| TheJulia | okay, I think I get what is going on | 22:07 |
| TheJulia | we *basically* need to do https://review.opendev.org/c/openstack/nova/+/882852 | 22:07 |
| TheJulia | or at least the same basic idea | 22:08 |
| TheJulia | because we get the pass-through rights, but in this case, we *must* use elevated privs | 22:08 |
| TheJulia | such that they are cast as service actions | 22:08 |
| JayF | if the equivalent change in Ironic is similarly scoped, it's hard for me to imagine us backporting it | 22:08 |
| JayF | unless we were extremely delicate to not break if people didn't set the new configs | 22:09 |
| TheJulia | eh... it is not horrible actually | 22:09 |
| JayF | more proof of my lack of imagination ;) | 22:12 |
| TheJulia | so yeah, the tl;dr is we end up leaning towards the request context the request comes in on, but in this very specific case, to meet the change, we need to elevate our access | 22:21 |
| TheJulia | i *think* this should do it | 22:21 |
| opendevreview | Julia Kreger proposed openstack/ironic master: WIP bfv service change https://review.opendev.org/c/openstack/ironic/+/882985 | 22:22 |
| TheJulia | (well, tests and all, but I think it is the same basic idea as what nova did | 22:22 |
| TheJulia | our operating pattern is a little different there though, but not awful | 22:22 |
| JayF | if you fixed it in that small of code | 22:36 |
| JayF | good job | 22:36 |
| JayF | will users need to set that keystone value, or just in devstack? | 22:36 |
| TheJulia | the docs suggest it should be set | 22:38 |
| TheJulia | I've not run down that code path | 22:38 |
| TheJulia | yet | 22:38 |
| JayF | ack | 22:40 |
| TheJulia | I think I'm going to end up jumping in a pool soon | 22:40 |
| JayF | sounds like a good plan for the afternoon :) after my EOD I have to keep a very sad dog exercised against his will (my wife not being here, and he has sep anxiety) then hunker down for Canes/Kraken games | 22:42 |
| opendevreview | Julia Kreger proposed openstack/ironic master: DPU modeling - parent_node DB/Model/API https://review.opendev.org/c/openstack/ironic/+/880114 | 22:58 |
| TheJulia | enjoy! | 22:58 |
| JayF | I'm off to do ^ that :D | 23:01 |
| TheJulia | oh, likely need to turn off user auth as well | 23:05 |
| TheJulia | another rev, later | 23:05 |
| opendevreview | Iury Gregory Melo Ferreira proposed openstack/ironic master: Add DB model for Firmware https://review.opendev.org/c/openstack/ironic/+/883031 | 23:46 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!