Tuesday, 2023-11-28

opendevreviewMerged openstack/sushy stable/2023.2: Remove version field from iLO error  https://review.opendev.org/c/openstack/sushy/+/90193700:21
opendevreviewJay Faulkner proposed openstack/sushy stable/2023.1: Remove version field from iLO error  https://review.opendev.org/c/openstack/sushy/+/90202600:22
opendevreviewIury Gregory Melo Ferreira proposed openstack/ironic-python-agent master: [WIP] Speed Up Multipath check  https://review.opendev.org/c/openstack/ironic-python-agent/+/90201200:40
opendevreviewMerged openstack/ironic master: Ensure enable_netboot_fallback writes out pxe config on adopt.  https://review.opendev.org/c/openstack/ironic/+/81198900:51
opendevreviewJulia Kreger proposed openstack/sushy-tools master: DNM: Don't detonate on identity not being a uuid  https://review.opendev.org/c/openstack/sushy-tools/+/90201501:15
opendevreviewJulia Kreger proposed openstack/ironic master: DNM: Add redfish https CI job  https://review.opendev.org/c/openstack/ironic/+/90109001:16
TheJuliaokay, I think we have some weirdness with sushy+sushy-tools01:17
TheJuliabut... maybe it is all okay01:17
opendevreviewMerged openstack/sushy stable/2023.1: Remove version field from iLO error  https://review.opendev.org/c/openstack/sushy/+/90202602:41
opendevreviewOpenStack Proposal Bot proposed openstack/ironic master: Imported Translations from Zanata  https://review.opendev.org/c/openstack/ironic/+/90204102:52
opendevreviewJay Faulkner proposed openstack/sushy stable/zed: Remove version field from iLO error  https://review.opendev.org/c/openstack/sushy/+/90202802:58
opendevreviewSteve Baker proposed openstack/ironic-python-agent master: WIP expose EFI boot entries in inspection boot  https://review.opendev.org/c/openstack/ironic-python-agent/+/90164503:42
opendevreviewSteve Baker proposed openstack/ironic-python-agent master: WIP Call efibootmgr -v with use_standard_locale=True  https://review.opendev.org/c/openstack/ironic-python-agent/+/90204203:42
TheJuliaJayF: w/r/t your comment on https://review.opendev.org/c/openstack/ironic/+/900964, take a look at my reply :)03:54
JayFI saw it, and I thought that I should look at it rendered in context03:55
JayFand that I will do that tomorrow :)03:55
TheJuliaHave a great evening03:59
opendevreviewMerged openstack/ironic master: Imported Translations from Zanata  https://review.opendev.org/c/openstack/ironic/+/90204104:07
rpittaugood morning ironic! o/08:01
opendevreviewMahnoor Asghar proposed openstack/ironic master: Fix Redfish request collecting storage drives  https://review.opendev.org/c/openstack/ironic/+/90199409:56
iurygregorygood morning Ironic11:15
iurygregoryrpittau, I just didn't +W https://review.opendev.org/c/openstack/ironic/+/894918 because I would like a confirmation from TheJulia (regarding the policy we should be using it)11:16
opendevreviewMark Goddard proposed openstack/bifrost stable/2023.2: Fix key-order ansible errors  https://review.opendev.org/c/openstack/bifrost/+/90203911:47
opendevreviewMark Goddard proposed openstack/bifrost stable/2023.1: Fix key-order ansible errors  https://review.opendev.org/c/openstack/bifrost/+/90204011:48
opendevreviewMark Goddard proposed openstack/bifrost stable/2023.1: Fix key-order ansible errors  https://review.opendev.org/c/openstack/bifrost/+/90204011:49
opendevreviewMark Goddard proposed openstack/bifrost stable/2023.1: Fix key-order ansible errors  https://review.opendev.org/c/openstack/bifrost/+/90204011:50
opendevreviewMahnoor Asghar proposed openstack/ironic master: Fix Redfish request collecting storage drives  https://review.opendev.org/c/openstack/ironic/+/90199412:17
rpittauthanks iurygregory that's fine13:06
opendevreviewMark Goddard proposed openstack/bifrost stable/2023.1: Fix key-order ansible errors  https://review.opendev.org/c/openstack/bifrost/+/90204013:11
drannouHello ironic ! I have a host where I need to extract the USB device list (via IPA), from what I see in the source code, nothing is done on USB devices lookup right ?13:31
dtantsurdrannou: yeaah. We can collect PCI devices, but not USB.13:35
dtantsurWe should probably have an optional thing that just collects the complete output from lshw. We already depend on it anyway.13:35
TheJuliaAt this point, I'd just collect it regardless13:53
dtantsurTheJulia: we're trying to keep the inspection data lean by default - the lshw output will immediately make it unreadable14:11
opendevreviewIury Gregory Melo Ferreira proposed openstack/ironic-python-agent master: [WIP] Speed Up Multipath check  https://review.opendev.org/c/openstack/ironic-python-agent/+/90201214:25
opendevreviewJulia Kreger proposed openstack/sushy-tools master: Don't detonate on identity not being a uuid  https://review.opendev.org/c/openstack/sushy-tools/+/90201514:27
opendevreviewJulia Kreger proposed openstack/sushy-tools master: Use a UUID instead of a named identity for httpboot  https://review.opendev.org/c/openstack/sushy-tools/+/90208514:36
TheJuliafolks, please pick one of the above, or both, it really doesn't matter :)14:36
iurygregoryI will add to my list 14:39
TheJuliaThanks14:39
TheJuliaI've not CI'ed the second one yet with the httpboot change, but given we can leak and lock on to the uuid form of the URL based upon the logs (and things generally still work (stupidly impressive!) it kind of explains why some runs last week passed and a recheck this week failed14:40
TheJulias/form/from/14:41
opendevreviewJulia Kreger proposed openstack/ironic master: DNM: Add redfish https CI job  https://review.opendev.org/c/openstack/ironic/+/90109014:41
TheJuliaThere that is14:41
iurygregorythanks for the context \o/14:42
TheJuliayeah, it made no sense to me when I started looking at it around 2:30 PM yesterday, all of a sudden it was 5:30 and... uhh.. yeah. :(14:45
* dtantsur is staring at the dnsmasq DHCP provider and has a nagging feeling that it may actually solve some of his problems15:06
drannoudtantsur: Would it be a bad idea to just list all vendor:model USB ids ?15:07
dtantsurdrannou: it's not terrible, but it's a half-measure, while lshw could solve many problems at once..15:08
drannouAnd it seems to already be in the code, so yeah +1 for lshw15:11
opendevreviewMerged openstack/ironic master: Fix Redfish request collecting storage drives  https://review.opendev.org/c/openstack/ironic/+/90199415:21
opendevreviewDmitry Tantsur proposed openstack/ironic master: Improve logging in the dnsmasq DHCP provider  https://review.opendev.org/c/openstack/ironic/+/90209115:47
opendevreviewMerged openstack/ironic master: implement basic-auth support for user image download process  https://review.opendev.org/c/openstack/ironic/+/89041116:05
dtantsurTheJulia: are you familiar enough with the dnsmasq DHCP provider to run an idea by you?16:15
dtantsur(re https://github.com/dtantsur/ironic-operator/issues/3#issuecomment-1830193586)16:21
TheJuliadtantsur: my gut feeling is "maybe", but stevebaker[m] would be the one to really bounce ideas off of/chat with 16:25
dtantsuryeah, that's what I guessed. timezones are not helping here16:25
TheJuliayeah :(16:26
TheJuliasorry16:26
dtantsurno worries, I just wanted to try my luck :)16:26
JayFMailing list is a good way to cross timezone barriers for this sorta question :)16:26
dtantsuryeah, I was thinking to drop him an email and cc the ML16:26
JayFwe don't use that venue a lot as Ironic'ers and it's actually nice now that you can use it like a forum with mailman316:26
dtantsurA benefit of specs: I came to this idea while writing an Alternatives section about why I cannot use the dnsmasq DHCP provider :D16:35
JayF+++ I love our spec templates, and use them for all sorts of things even not openstacky16:35
JayFit's good for ensuring your thought process is clean and clear16:36
opendevreviewJulia Kreger proposed openstack/sushy-tools master: Use a UUID instead of a named identity for httpboot  https://review.opendev.org/c/openstack/sushy-tools/+/90208517:05
rpittaugood night! o/17:07
TheJuliaThat one seems to be more in-line than chasing fun weirdness in sushy-tools17:09
dtantsurI think the dnsmasq idea may fail simply because bloody kubernetes...17:33
TheJulia?17:33
dtantsurI'm struggling to understand how to configure disjoint DHCP ranges for a potential dnsmasq daemonset17:33
dtantsurhttps://github.com/dtantsur/ironic-operator/issues/3#issuecomment-183033026517:33
dtantsurthe only hacky solution I can come up with means that the whole Ironic pod will get redeployed every time a control plane node is repalced17:34
dtantsurmaybe it's not THAT bad, dunno..17:34
JayFdtantsur: I don't understand the problem space well, but I think I might  have ideas for solving some of it if you wanted to fill in some of the context17:37
JayFI mean to say, I have some experience with stupid DHCP tricks and uncooperative network providers :D 17:38
JayFif you wanted to sync up on this maybe tomorrow early (more in your TZ) I would be happy to help brainstorm further ideas17:39
dtantsurJayF: that would be really appreciated! I'm staying late today, but tired of thinking already17:39
JayFmy tomorrow is wide open, anywhere from like 730a-noonish pacific time17:40
JayFwhich works out to like ... 1530-2000 UTC if my math is right17:40
dtantsurJayF: 1600 UTC would work very well, 1700 is also doable17:41
JayF16 is what... 8am, that's perfect17:41
JayFI'll toss it on your calendar, forward it around if anyone else is useful to have there17:41
JayFor ping me if you're in IRC and want on17:41
dtantsurJayF++17:41
dtantsurI may grab someone from Ericsson if they end being available17:41
opendevreviewJulia Kreger proposed openstack/ironic master: Redfish UefiHttp boot support  https://review.opendev.org/c/openstack/ironic/+/90096417:48
opendevreviewDmitry Tantsur proposed openstack/ironic master: Add missing compatibility between idrac and redfish firmware  https://review.opendev.org/c/openstack/ironic/+/90210317:53
dtantsuriurygregory: not sure how we missed this ^^^17:53
dtantsurwill need to be backported to 2023.217:53
JayFHas someone tested it in the real world against DRAC servers?17:53
JayFI didn't realize it was a "miss" in review, I thought it was intentional/untested/nonworking17:54
JayFwe scoped that work to "basic redfish" and it's not clear to me what that includes all the time tbh17:54
dtantsurJayF: idrac has redfish-derived interfaces17:54
dtantsurwithout this link, the users will need to choose between working virtual media and working firwmare updates17:54
JayFSo what I'm saying is as a reviewer17:55
JayFthere's no way for me to know "working  firmware updates" is a thing that change enables17:55
dtantsurthat's why I'm tagging iurygregory who's tested the feature17:55
JayFheh, I read the "we" as an Ironic we17:55
JayFnot as a you/iury we :P 17:55
JayFbut it's clear to me now  that we should default to, if something redfish is added, drac gets it too17:56
dtantsuronce we drop wsman, idrac will be just a subclass17:56
opendevreviewDmitry Tantsur proposed openstack/ironic master: Add missing compatibility between idrac and redfish firmware  https://review.opendev.org/c/openstack/ironic/+/90210317:57
TheJuliaoh joy, yeah, that is an easy part of the construct to miss too17:57
TheJuliabut luckily low risk17:58
opendevreviewJulia Kreger proposed openstack/ironic master: Redfish UefiHttp boot support  https://review.opendev.org/c/openstack/ironic/+/90096417:58
opendevreviewJulia Kreger proposed openstack/ironic master: Add HTTP versions of network boot interfaces  https://review.opendev.org/c/openstack/ironic/+/90096517:58
opendevreviewJulia Kreger proposed openstack/ironic master: DNM: CI test for httpboot jobs  https://review.opendev.org/c/openstack/ironic/+/90118217:58
opendevreviewDmitry Tantsur proposed openstack/ironic master: Clean-up iDRAC hardware interfaces  https://review.opendev.org/c/openstack/ironic/+/90210718:18
dtantsura somewhat opinionated change ^^ (yes, I wanted to do it for a long time)18:18
iurygregorydtantsur, sorry I was at a doc appointment18:31
dtantsuroh, I hope it went well. no worries.18:32
iurygregoryyeah, just a bunch of routine exams =)18:32
* iurygregory facepalm such a simple change and I missed 18:32
dtantsurhappens to all of us :)18:33
iurygregorytks for the fix18:34
opendevreviewDmitry Tantsur proposed openstack/ironic master: First pass at the new in-band inspection docs  https://review.opendev.org/c/openstack/ironic/+/89823718:45
opendevreviewDmitry Tantsur proposed openstack/ironic master: First pass at the new in-band inspection docs  https://review.opendev.org/c/openstack/ironic/+/89823718:47
dtantsurlong overdue ^^^18:48
JayFdtantsur: proxied your requests PR to one of my IRC channels with PSF folks in it19:05
JayFdtantsur: er, derek's, the one you toot'd about19:05
dtantsurmany thanks!19:05
dtantsurTheJulia: if we do an online data migration, it will need to be backported, right?19:06
JayFan online data migration of what?19:07
TheJuliaa data migration, no, since you run the migration before starting the new install19:07
JayF*online* data migration sounds scary to me as someone who operates at scale19:07
TheJuliaa pre-flight check, would be a "good idea" to backport19:07
dtantsurJayF: this is re TheJulia's comment on my idrac deprecated patch19:08
TheJuliawell,, it wouldn't quite be so online, really19:08
TheJuliabut conductors could be online19:08
TheJulia... That is likely a failure case we never thought of19:08
dtantsurthe online data migrations happen after installations, so it's too late to provide them in the release where stuff is removed19:08
dtantsura pre-flight check is an easier win though19:08
TheJuliayeah19:09
JayFI'll note that timing wise this is probably not great for this cycle as it's mid-SLURP from an openstack standpoint19:09
TheJuliaI mean, fully online, modified stuff yeah, it is harder to do and requires extreme intentionality anyway19:09
dtantsurJayF: this stuff has been deprecated for ages19:09
TheJuliadell left alias interface names to avoid forcing people to do any explicit upgrade action19:10
JayFdtantsur: can we put those receipts in the release notes? e.g. "These names were deprecated in the Austin cycle: https://openstack.old/lol"19:10
TheJuliaAnd that is what dtantsur is seeking to clean up19:10
dtantsurJayF: please leave a comment on the review, I'll forget otherwise19:11
JayFI was typing it on the review as I was saying it :)19:11
JayFOH, I misread the commit message19:11
JayFI thoguht this was killing idrac-wsman19:11
JayFnot idrac19:11
JayFthis makes more sense when I got to file 16 of 16 lol19:11
dtantsuryeah, idrac is an ancient alias of idrac-wsman19:12
dtantsurthe wsman stuff itself is only deprecated in this release19:12
JayFI'm trying to find the case where idrac-redfish is gone now19:12
JayFoh, just deprecated19:12
JayFthe supported=False to make it complain on startup19:13
JayFOK; I think that would be less confusing as two PRs but I know why it was mixe19:13
JayF-1 for a couple of reasons; but no objection in concept19:14
opendevreviewSteve Baker proposed openstack/ironic-python-agent master: WIP expose EFI boot entries in inspection boot  https://review.opendev.org/c/openstack/ironic-python-agent/+/90164519:29
opendevreviewSteve Baker proposed openstack/ironic-python-agent master: WIP Call efibootmgr -v with use_standard_locale=True  https://review.opendev.org/c/openstack/ironic-python-agent/+/90204219:29
stevebaker[m]dtantsur: hey are you still up?19:48
opendevreviewMerged openstack/ironic master: Deprecate configuration molds  https://review.opendev.org/c/openstack/ironic/+/90150219:55
opendevreviewMerged openstack/ironic master: [trivial] add Python 3.11 mention in setup.cfg  https://review.opendev.org/c/openstack/ironic/+/90028619:55
iurygregoryWould be ok if we retry in case of 502 Error in IPA when we try to send the inspector data back? https://opendev.org/openstack/ironic-python-agent/src/branch/master/ironic_python_agent/inspector.py#L135-L147 20:47
JayFI can't think of a reason why not; but I also can't think of a scenario other than "Ironic API is down" where that could happen20:50
iurygregoryFor context, when deploying +3k virtual BM, some of them fails during inspection, journalctl shows 502 Proxy Error when this part of the code was executed20:50
JayFI am not opposed to IPA retrying; but I'd take that behavior as a sign the API itself, or the web services fronting it, could be configured better20:50
JayFLike we should try to succeed and retry, but I also think whatever specific issue you're chasing, I'd see if you could tune the fronting proxy to make it behave better under larger numbers of connections20:51
JayF(the more we put in retries like this; the more likely we are to introduce "thundering herd" type failure modes into Ironic in API/network failure cases)20:52
iurygregoryyeah, I wish a had a better idea on how the proxy is really configured20:52
iurygregoryJayF, I see 20:53
JayFYou can't troubleshoot a distributed system (well) from one side only20:53
JayFso yeah, we can retry from an IPA standpoint20:53
JayFbut with my ops hat on, if this was my ticket, I'd not stop there20:53
iurygregorythe good thing is that the amount of failures is small (like less than 2% of vm failed if I recall)20:54
iurygregoryJayF, the problem is get access to the scale system basically, since is shared by folks and they have some sort of time with the setup20:54
JayFYeah, I've been in ... similar environments before20:55
JayFwhich is why I'm fairly certain misconfigurations there can be more common than you'd think :P 20:55
opendevreviewMerged openstack/ironic master: Add missing compatibility between idrac and redfish firmware  https://review.opendev.org/c/openstack/ironic/+/90210321:30
opendevreviewJay Faulkner proposed openstack/ironic stable/2023.2: Add missing compatibility between idrac and redfish firmware  https://review.opendev.org/c/openstack/ironic/+/90207421:34
opendevreviewMerged openstack/ironic master: Trivial: fix a typo in wsgi_service  https://review.opendev.org/c/openstack/ironic/+/90186221:53
JayFhttps://review.opendev.org/c/openstack/ironic/+/901862 should have had a release note22:10
JayFand now it's a little bit in limbo; it needs to be backported but has no release note to backport with it22:10
JayFah, was wondering why I didn't see it, never got h/t as prio22:11
TheJuliaa reno can get merged in master and then just the file included on the backported change22:12
JayFI know; but for a fixed bug we should have documentation somewhere that it was fixed; a release note, a bug, something22:13
JayFand if I hadn't seen this, it wouldn't have been documented AFAICT22:13
* JayF is already doing the followup fwiw22:13
TheJuliak22:13
opendevreviewJay Faulkner proposed openstack/ironic master: Document wsgi_service fix from 16a806f  https://review.opendev.org/c/openstack/ironic/+/90211522:18
opendevreviewJay Faulkner proposed openstack/ironic stable/2023.2: Properly cleanup unix sockets in wsgi_service  https://review.opendev.org/c/openstack/ironic/+/90211622:21
JayFTheJulia: https://review.opendev.org/c/openstack/ironic-python-agent/+/895519 (your vmedia network config drive patch) has some comments from you in patchset 6 which read like todos which are incomplete, but stevebaker[m] has it +2'd, and I'm OK with it. Do you want me to land it or give you a chance to complete those items?23:01
JayFI have my +2 on it, just tell me and I'll land it, or workflow it yourself if I'm gone23:02
TheJuliaWorkflowing is fine, all those comments are just things I’ve not closed out23:03
TheJuliaIn gerrit that is23:03
JayFack, I saw a specific comment about retrying but didn't see the retry which is why I asked; but I also think it's OK23:04
JayF+A23:04
TheJuliaI saw that one too, I think failing was better path23:04
JayFthat is sorta what I was thinking too tbh23:05
JayFjust didn't wanna land something that the author wasn't ready to land :D 23:06
* TheJulia resumes exercising23:06
TheJuliaThanks!23:06
JayFstevebaker[m]: If you want to not presume, I can tell you that was added to cover for the unreliability of the specific swift install at one large ironic-running public cloud provider around juno era :D 23:35
JayFlol23:35
JayF(I put it in the change but an upgrade note is 100% sufficient)23:36
stevebaker[m]JayF: that was my hunch, lol23:39
JayFyeah and thanks for reverting those log messages23:39
opendevreviewSteve Baker proposed openstack/ironic master: Replace swiftclient usage with openstacksdk  https://review.opendev.org/c/openstack/ironic/+/89999923:40
JayFjust about every downstream I've ever talked to has something that reads out the log messages on some failures and either increments a counter or does a thing in response23:40
* JayF looks at the diff for a quickreview23:40
stevebaker[m]JayF: cinder is going to be a similar situation, there is like 3 different retry options23:42
JayFhttps://review.opendev.org/c/openstack/ironic/+/899999/5#message-d9d2601e81b79665621614cbc467ed3e1fab62f8 23:42
JayFmakes sense; as long as we put it in the release notes and wire through any config vars that sdk has that might match up, I think we've done what we need to do23:43
JayFif people complain and want retries back, we can add them in (to the SDK, I'd assume, would be the best place?)23:43
opendevreviewSteve Baker proposed openstack/ironic master: Replace swiftclient usage with openstacksdk  https://review.opendev.org/c/openstack/ironic/+/89999923:44
stevebaker[m]second attempt at fixing log messages23:44
JayFBTW; I have been talking to an Ironic user who is in UTC+223:45
JayFand invited them to IRC23:45
JayFsince that's borderline-inverse of mine; I figured I'd give you a heads up23:46
* JayF trying to find more other-hemisphere friends for Steve :D23:46
* stevebaker[m] so lonely sniff23:46
JayFyeah, you get to be alone and productive ;)23:47
stevebaker[m]mind you, not much overlap between UTC+2 and UTC+1323:47
JayFmore than there is for UTC-8, I think 23:48
stevebaker[m]JayF: oh, how do I sign up for bug triage duty?23:48
JayFHold on, how does UTC+13 make sense?23:48
JayFshouldn't that be UTC-11?23:48
JayFwait, I should know better than to try to understand time23:48
JayFstevebaker[m]: I am going to make an etherpad when the bug deputy thing lands, but it landed while I was on vaca so I haven't done it yet :D 23:49
JayFstevebaker[m]: feel free to JFDI create an etherpad with the remaining weeks in the release as signup lines, and signup for the weeks you want23:49
JayFthat way you get the weeks you want :D 23:49
JayFotherwise I'll add it to my tomorrow-list23:49
clarkbwait until you discover kiribati23:49
* JayF EOD in 10 minutes and has been wrangling a sinus headache23:49
stevebaker[m]I live in the future23:49
clarkbthree timezones and each of them is +12 or larger23:49
JayFI hope that you all take your responsibility to warn those of us in the past of hte impending apocalypse23:50
JayFyou're a good 21 hours in the future compared to me23:50
stevebaker[m]Its time for my Wednesday lunch23:50
JayFit's 3:50pm on Tuesday for me lol23:51
JayFI visited western australia once23:51
JayFI forget which direction was which, but literally one way an entire day disappeared (left on Monday, arrived that same time on Wednesday IIRC)23:51
JayFthe other direction we arrived before we left or something like that23:51
stevebaker[m]yeah travelling over the dateline is a trip, I can arrive before I leave23:52
* zaneb waves to stevebaker[m] 23:53
JayFI guess you're like "what an interesting story, I call it 'every trip to a conference in the western hemi'"23:53
stevebaker[m]O/ zaneb23:57

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!