Tuesday, 2024-09-10

dtantsurcardoe: btw, I generally give people a blank permission to update my patches for trivial things like typos, small fixes or release notes.06:44
dtantsur(will update today if nobody beats me to it)06:44
dtantsur* blanket (wake up Dmitry!)06:44
opendevreviewDmitry Tantsur proposed openstack/ironic master: Remove legacy Query support from _paginate_query  https://review.opendev.org/c/openstack/ironic/+/89150509:11
dtantsurcardoe, TheJulia, JayF ^^09:12
dtantsurmeanwhile, ironic-inspector-tempest seems quite unstable: https://zuul.opendev.org/t/openstack/builds?job_name=ironic-inspector-tempest&branch=master&skip=009:14
dtantsurI don't think we can merge much into ironic-tempest-plugin before we fix it09:14
opendevreviewMark Goddard proposed openstack/tenks master: CI: Remove CentOS Stream 8 jobs  https://review.opendev.org/c/openstack/tenks/+/92877409:30
opendevreviewMark Goddard proposed openstack/tenks stable/2.0: CI: Remove CentOS Stream 8 jobs  https://review.opendev.org/c/openstack/tenks/+/92877509:33
opendevreviewDmitry Tantsur proposed openstack/ironic-lib master: Remove code already migrated to Ironic and IPA  https://review.opendev.org/c/openstack/ironic-lib/+/92877609:35
opendevreviewDmitry Tantsur proposed openstack/ironic-lib master: Remove the rest of the rootwrap machinery  https://review.opendev.org/c/openstack/ironic-lib/+/92877809:43
dtantsurat least we can remove a lot of code now ^^^09:45
dtantsur(I specifically waited for 2024.2 to be branches first.. just in case)09:45
opendevreviewDmitry Tantsur proposed openstack/ironic-lib master: Remove the rest of the rootwrap machinery  https://review.opendev.org/c/openstack/ironic-lib/+/92877809:55
opendevreviewDmitry Tantsur proposed openstack/ironic-python-agent master: Migrate more trivial code from ironic-lib  https://review.opendev.org/c/openstack/ironic-python-agent/+/92877910:13
opendevreviewDmitry Tantsur proposed openstack/ironic-inspector master: Try enabling port 5050 on PUBLIC_BRIDGE  https://review.opendev.org/c/openstack/ironic-inspector/+/92878311:29
TheJuliadtantsur: did you mean to https://review.opendev.org/c/openstack/ironic-python-agent/+/928779 while also removing some of the same files in another patch?13:04
dtantsurTheJulia: my ironic-lib and IPA patches are disjoint: the ironic-lib removes the already migrated code, while the IPA one migrates more13:05
dtantsurif it makes sense (also good morning)13:05
TheJuliaoh13:06
TheJuliafor some reason I thought it was ironic-lib13:06
TheJuliaclearly, more coffee required13:06
dtantsurcoffee++13:06
dtantsurwhich substance do I need to the CVE backports to OpenShift?13:06
TheJuliacoffee? lots of coffee?13:08
TheJuliaand a chart of versions to match up backports13:08
dtantsur4.15 was branched between stable/2023.2 and bugfix/24.0, so neither versions of patches actually apply cleanly13:08
* dtantsur head->table13:08
dtantsurwell, maybe if I just take test_image_cache from bugfix/24.0 and the rest from stable/2023.2....13:10
cardoemetal3-integration seems to have failed recently a few patchsets. the machines never became ready after inspection13:11
dtantsur2 failures in a row, suspicious but not terrible https://zuul.opendev.org/t/openstack/builds?job_name=metal3-integration13:13
dtantsurha, mixing versions works!13:13
* dtantsur committing git sins13:13
TheJuliadtantsur: this must be a t-shirt13:14
dtantsurtotally13:14
cardoeI'd buy it13:14
TheJulia(of course, git's 3-way merge logic also commits sins of it's own)13:15
cardoesemi-linear with up to date branches is the one true way13:15
cardoeExcept GitHub disagrees with me so I'm a heretic.13:15
TheJuliaif you ever go read about the merge logic, you'll want to table flip13:16
dtantsur:D13:16
TheJuliaBeing a heretic is good.13:16
dtantsurmy life has become much better since conflictStyle = diff313:16
dtantsurbut still13:16
TheJuliaYou get awesome shirts13:16
cardoeIt'd basically Linus's quilt behavior tweaks.13:16
cardoeLinus and Hanno Bock I think.13:16
dtantsurOn a serious note, wtf is going on in the inspector CI?13:19
opendevreviewDmitry Tantsur proposed openstack/ironic-inspector master: DNM: get a clean CI run  https://review.opendev.org/c/openstack/ironic-inspector/+/92879313:20
TheJuliahas it gone completely sideways?13:24
TheJuliaas the british say, has it gone pear shaped?13:31
dtantsur:D13:32
dtantsurI've just created a test patch. What I got in https://review.opendev.org/c/openstack/ironic-inspector/+/928783 is not exactly inspiring13:33
TheJuliaIf it fails, do we throw a party?13:33
dtantsuror the table out of the window?13:35
dtantsurso many options!13:35
TheJulia.... we have a saying about windows in the states these days13:37
TheJuliaParties are likely better, everyone can mosh pit by the pool and inspector can be forgotten about... right?13:37
* dtantsur is very ready to forget inspector13:46
TheJuliawhat is the outstanding item again?14:11
JayFI know CID is working on inspection rules14:15
JayFOr at least was helping with it in some way14:15
cardoeYou folks good with https://review.opendev.org/c/openstack/ironic/+/919779 ?14:17
cardoeWhat if I made a new hashtag instead of "ironic-weekly-prio" that's like "ironic-workflow-ready"?14:18
cidJayF: I am! Now waiting to get a little feedback on the current shape. 14:18
cardoeAnd I'll track more repos under the Ironic umbrella and tag anything with a core +2 and verified +1?14:18
TheJuliaLGTM, approving14:18
TheJuliawell, the doc change14:18
cardoehttps://review.opendev.org/c/openstack/ironic/+/928106 that's another one14:19
JayFcardoe: I've considered that before, we should discuss it at the ptg though because the biggest issue would changing it will just be evangelizing the new name14:20
cardoeI didn't understand why https://review.opendev.org/c/openstack/ironic-python-agent/+/927984 didn't merge.14:20
JayFcardoe: I will say if we changed it, changing it to something longer seems silly14:20
cardoeWell I can just tag +2/+1 stuff with ironic-weekly-prio?14:20
cardoeI'm just trying to be helpful while learning the project more.14:22
JayFIf you see a patch that needs a review and is ready and is not otherwise hashtagged with that please do so14:22
JayFcardoe: for that IPA patch, if you look at the box titled submitted together, you'll see that it is stacked on top of a patch that is not merging14:24
JayFcardoe: this is how you stack dependent patches in the same project, multiple commits on a single branch and one git review pushes them all up14:25
cardoeah. been trying to figure more gerrit bits out. thanks.14:27
TheJuliaGerrit is... different, yet exceptionally powerful14:31
JayFHonestly I don't want to admit how many minutes I've wasted WTF ing at why a patch hasn't merged only to see that it was fat14:31
JayF**stacked14:31
opendevreviewMerged openstack/ironic master: Deduplicate and remove invalid information for steps  https://review.opendev.org/c/openstack/ironic/+/91977914:31
TheJuliazuul adds a different layer of power!14:31
derekhIf anybody has some spare time (if thats a thing) I'd appreciate another review on https://review.opendev.org/c/openstack/ironic/+/927828 14:33
TheJuliawhat is time?!14:33
TheJuliaderekh: I feel like I'm missing a key piece, what is causing the agent attempt coming out of prepare?14:35
derekhTheJulia, the agent is finishing inspection, moving the node out of "inspecting", our client (in my case the openshift baremetal operator) then immediately attempts to clean the node, but fails if heart beating has yet to start, in which case we need it to retry , I've some more context in the openshift bug https://issues.redhat.com/browse/OCPBUGS-33255?focusedId=25134975&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comme14:40
derekhnt-25134975 14:40
TheJuliaokay, so an external call then14:40
TheJuliathat actually makes a lot of sense then14:40
TheJuliaThanks!14:40
derekhno prob14:41
TheJuliaapproved14:41
derekhthanks14:41
TheJuliaThanks!14:43
opendevreviewcid proposed openstack/virtualpdu master: Upgrade pysnmp-lextudio  https://review.opendev.org/c/openstack/virtualpdu/+/92882315:28
opendevreviewcid proposed openstack/virtualpdu master: Upgrade pysnmp-lextudio  https://review.opendev.org/c/openstack/virtualpdu/+/92882315:29
TheJuliaJayF: fyi https://review.opendev.org/c/openstack/ironic/+/92870915:37
JayFI am spinning up a xena devstack today to test that15:37
TheJuliaack, looks like the py39 test failure was transient15:37
JayFnot saying I'm making that a requirement to merge, just I don't want my downstream to get more egg on face15:37
JayFbut if you wanna wait I can get you feedback from that15:38
TheJuliaI was able to identify the delta, so I'm just pretty sure that is it15:39
TheJuliabut, by all means test15:39
TheJuliathat disable force_raw was sort of unexpected for me15:39
JayFyeah, it matches what they were seeing 15:39
JayFso I have confidence in it too, just would rather pay 2-3 hours of my own time than more of other folks' time15:40
TheJuliahttps://www.socallinuxexpo.org/scale/22x/events/open-infra-days <-- start brewing your talk ideas!15:47
JayFoh yaay! at scale15:47
JayFWhen is listed as > Sunday, August 18, 2024 - 08:00 <15:47
JayFwhich seems wrong?15:48
clarkbJayF: I called that out and expect it to get fixed soon15:48
JayFscale itself is in March 2025, I assume that's more correct15:48
* JayF makes plans15:48
TheJuliaplans++15:51
JayFI have a talk brewing I want to title something like "OpenStack is the bearer of bad news" or something along those lines, I'm going to get it ready for this15:58
JayFbasically talking about the phenomenon where if you're using OpenStack to run your infra, a large % of problems are blamed on it simply because it is effectively the monitoring system15:58
cardoeShould the inspector hooks only trigger on the agent inspector?16:02
TheJuliaJayF: I think the counter balance to monitoring is supplying garbage16:04
TheJuliaJayF: one of the huge complaints I hear from a large operator is their users doing things which are fundimentally just don't work and they like to blame the infra instead of just fess up that they did something breaking like... decide to run a dhcp server, or supply garbage disk images16:05
TheJuliaugh, the air quality in Southern California is awful today16:20
TheJuliacardoe: Historically, I think they have only on that. But that raises an interesting question!16:22
JayFTheJulia: I've thought of comparing OpenStack to a monitoring system as part of this point; it touches almost all parts of the infrastructure so it's often the canary for failure16:32
TheJuliaoh, absolutely16:33
TheJuliawe automate things which are painful16:33
TheJuliayou can wire it all manually, but do you really want to do that?!16:34
TheJuliaand is that really the right thing to do16:34
JayFI think it's more that people compare on-prem BM cloud to e.g. AWS16:34
JayFwhich is not apples to apples in terms of complexity or cost16:34
TheJuliaindeed16:34
TheJuliaI'm not comparing though, I'm more thinking people also don't undestand the size/scale/scope of the underlying interaction16:35
JayFyeah I have an idea of a graph that looks like a spider16:35
TheJuliaSure, you can have a VM. But if you want fancy networking... well that is a little more work. Oh you want this and that too, that is more work16:35
JayFshowing openstack crossing every network boundary, physical location boundary, etc etc16:35
TheJuliaand at some point, self-inventing in painful manual ways is not the right thing to the next person who will come along16:35
JayFand also talking about data/control plane misconceptions: an openstack "outage" is about control, not existing devices16:36
JayFTheJulia: I think part of the perception is that openstack /is the painful manual way/ due to how much hands-on you need to get a working (particularly Ironic with all the bits and trimmings) installation16:36
TheJuliaFrom a purely operational point of view, agree16:37
JayFTheJulia: I remember two grumpy folks giving a "kids these days don't know how hard things are" two scales ago that pointed in this direction ;)16:37
TheJuliaI'm thinking from a user point of view, they don't know how the sausage is made nor know how complex the kitchenaid gears are unless they start disassembling the kitchenaid16:37
TheJuliasoon they will learn they are likely never going to get it back together16:37
TheJuliawithout an expert, that is16:37
TheJulia:)16:37
TheJuliaOh, this worm gear goes in the other direction, and magically it all begins to work again16:38
JayF"why does this kitchen aid make a horrible grinding noise? I've only been using it underwater for 10 years nonstop without any grease" ;) 16:38
jssfrwhat16:39
JayFjssfr: we're just like, 5 layers deep in a hypothetical metaphor :D 16:40
JayFwhat's up?16:40
jssfrokay, I just read the last sentence and it was so weird that I wanted to hear a story about it16:41
jssfrbut I didn't catch the metaphor part, carry on please16:41
TheJuliaSpeaking of, at some point I need to rebuild my kitchenaid at the house16:42
TheJulia.... it has the metal gears16:42
TheJulia(yes, it is very, very old.16:42
JayFjssfr: oh, just a metaphor for neglected, broken, sad clouds16:42
JayFjssfr: e.g. them: "openstack is bad" me: what release? them: train!16:42
TheJuliame: *faceplam*16:43
jssfrhaha16:43
jssfrI can relate to train16:43
TheJuliachooooooo chooooooo16:53
TheJulia.... In a weird sense, I miss that hotel.16:55
JayF? that hotel17:05
clarkbThe Denver Stapleton Renaissance Hotel. The hotel that gave Train its name17:08
TheJuliahttps://maps.app.goo.gl/A4bvqX83F1isSNWM617:14
TheJuliaif you look to the north you'll see train tracks17:14
TheJuliaThe city is no longer Stapleton, fwiw. It is now called Central Park17:15
JayFoh, that's cool as hell17:15
TheJuliaBack in ?2021? they voted to rename the city.17:15
JayFreminds me of the disney hotels17:15
TheJuliaEar Plugs were provided upon check-in.17:15
clarkbTIL they renamed the city17:15
TheJuliaI only knew they renamed it because I was looking for a physical location to host a meeting17:17
TheJuliawow, the fire that is causing my air quality to be awful today must be getting worse. They just called for 40 engines, 4 dozers to clear fire lines, and both another VLAT and LAT ([very] large air tanker)17:19
* TheJulia guesses this means it is "fire season"17:22
TheJuliahttps://zuul.opendev.org/t/openstack/build/5ea8b917c45e4834adeb9e0982a0e0e4 ugh17:24
clarkbTheJulia: that is caused by having a too old version of `packaging` for your version of pip (or maybe setuptools). One or both of setuptools/pip prefer system libs to their vendored libs which can lead to this (and this is a relatively new change)17:27
clarkbpersonally I don't understand why they would vendor anything at all then prefer to use other verions that might be broken when they know the vendored versions work...17:27
TheJuliaclarkb: like, since Saturday?17:28
clarkbya17:28
TheJuliaso, hmmm17:28
clarkbI think it was setuptools updating on september 417:28
clarkbso since a week ago ish17:29
TheJuliachanging requirements on a stable branch is really not a great idea17:29
TheJuliaeek17:29
TheJuliawell, "unmaintained" but same difference17:29
* TheJulia attempts local reproduction17:31
clarkbre bumping deps on stable/unmaintained branches I feel like keeping the pypa related stuff functional is a reasonable exception to that. It shouldn't affect runtime but does ensure you can keep buildingthe software.17:34
TheJuliayeah, i guess it is a mix of factors since  tests seem to run locally17:39
TheJuliaon python 3.9 from that branch17:39
TheJuliaeww, looks like I might be getting separate errors17:40
* TheJulia waits17:40
TheJuliahmmm17:40
TheJuliarm -r -f .tox and retry17:40
TheJuliahmm17:46
TheJulialooks like we were getting fairly recent versions in CI17:55
TheJuliatempted to try and match up to reproduce, just don't understand a different error I'm now getting on 3.917:56
JayFwrong tox version?17:58
TheJuliasomehow the py39 run had a different version of ironic17:59
TheJuliainstalled in it's environment17:59
TheJuliajust.. bizzare18:00
TheJuliaso yeah, upstream ran with latest setuptools, slightly older packaging. lets see what chaos it gets into18:01
TheJuliaseems to run just fine...18:03
TheJuliahmmm18:03
TheJuliayup, looks like it18:07
TheJuliainstalling an older tox locally to try and reproduce18:07
TheJuliaI had tox 4-something locally and it was happy18:08
TheJuliawe have 3.28.0 in CI right now18:08
TheJuliaOkay, I'm feeling stumped18:17
TheJuliaclarkb: could the base OS and python libraries be an interaction regardless of versions installed?18:19
TheJulia(as part of the execution of tox...)18:19
clarkbif site packages are enabled in the virtualenv or perhaps if older virtualenv bundles packaging into the venv maybe?18:20
TheJuliaDoesn't look like it from my point of view, just seems weird I can't reproduce18:22
clarkbin the example from earlier today starglingx was using contraints to pin `packaging` back to an incompatble version18:22
TheJuliado you know what version that was?18:23
TheJuliaCollecting packaging>=20.4 (from oslo.utils>=4.5.0->-r /home/zuul/src/opendev.org/openstack/ironic/requirements.txt (line 28)) <-- upstream from the failed job18:24
clarkbTheJulia: https://opendev.org/starlingx/root/raw/branch/master/build-tools/requirements/debian/upper-constraints.txt says packaging===20.918:24
TheJuliafor science, y'all18:28
TheJuliahttps://review.opendev.org/c/openstack/ironic/+/92885118:28
cardoeHave we had a successful metal3-integration happen on ironic master today?18:38
cardoeDon't want to wastefully retry https://review.opendev.org/c/openstack/ironic/+/92828318:39
JayFlook at the failure, see what's up with it18:40
JayFif it's been the same, we should -nv it and notify dmitry/riccardo18:41
TheJulia++18:42
JayFit looks like a systemmatic failure on k8s side18:42
JayFline starting with 2024-09-09 22:07:27.904220  in https://d932237a159a818f5df3-c96838ca99c022c3211112e269bccbbb.ssl.cf5.rackcdn.com/928283/1/check/metal3-integration/1bb4f7b/job-output.txt18:45
JayFI would +2A a patch to mark it nonvoting if we haven't seen a pass today18:45
clarkbdo you report the results of those jobs back to metal3 too?18:46
clarkbjust wondering if getting the signal going in both directions might aid in awareness when things are broken and what needs fixing18:46
JayFwe have a massive overlap in cores18:47
JayFIury, Dmitry, Riccardo are all on the metal3 project18:47
JayFand I believe there are some cross-voting jobs on the metal3 side18:47
JayFprobably a little less attention being paid to it than usual due to vacations 18:47
cardoeWell Dmitry gave me this link this morning... https://zuul.opendev.org/t/openstack/builds?job_name=metal3-integration18:49
cardoeSo it's not run more than the 2 I've tried so far.18:49
JayFso either something failed between last night and today18:49
cardoeBut it's just very long when it fails18:49
JayFor we're unlucky18:49
JayFit failing with such a long tail is not something I've seen before18:51
JayFand kinda maps with the log I pointed at up there18:51
JayFI suspects it's systemmatic and should be marked nonvoting, but a sample size of two is small.18:53
JayFMarking it non-voting is a small change for a big benefit, I'd do it18:53
opendevreviewVerification of a change to openstack/ironic master failed: Set node "alive" when inspection finished  https://review.opendev.org/c/openstack/ironic/+/92782819:04
JayF> metal3-integration https://zuul.opendev.org/t/openstack/build/031713c8f81b49f7bc186184e4c5f39b : TIMED_OUT in 4h 15m 36s19:05
JayFholy yikes19:05
JayFI am -nv'ing that19:05
opendevreviewJay Faulkner proposed openstack/ironic master: [ci] Temporarily mark metal3-integration non-voting  https://review.opendev.org/c/openstack/ironic/+/92885719:07
JayFI'm slightly tempted to *completely remove* the job given it's current timeout behavior will cause all jobs to take that long, even if not voting19:07
clarkbyou can drop the timeout to a lower value at the same time19:08
clarkbbut then you may get less useful signal19:08
TheJuliait *typically* takes like 1.5 hours19:08
TheJuliaat least, that is my instant recall impression19:08
TheJuliaclarkb: packaging 21.3 == same error, trying 22.019:09
TheJuliaand of course, blows up requirements-check19:09
JayFclarkb: do you know the key for that right off the top of your head?19:09
clarkbJayF: its just timeout then a number of seconds19:10
JayFI'm happy to cap it at like, 90 minutes. We've never seen a passing job that takes over an hour.19:10
JayFack19:10
JayFsame spot as voting:?19:10
clarkbtimeout: 7200 or whatever19:10
JayF10-4 on it19:10
clarkbyes same level as name/description/voting/vars/etc19:10
opendevreviewJay Faulkner proposed openstack/ironic master: [ci] Temporarily mark metal3-integration non-voting  https://review.opendev.org/c/openstack/ironic/+/92885719:12
JayFI'm going to create release PRs for stable branches since CVE fix landed, assming they aren't already19:14
TheJuliaJayF: for what?19:14
JayFIronic and IPA19:14
JayFstable/*19:14
TheJuliafor what specifcially?19:14
JayFto cover the CVE fix19:15
JayFwe merged a fix to master, I'd like us to have it released19:15
JayFs/master/HEAD/19:15
TheJuliaoh, release changes19:15
TheJuliaokay19:15
TheJuliayeah, sounds good19:15
TheJuliaI think you got me at "PRs"19:15
TheJuliaso packaging 22.0 is incompatible19:17
TheJulia... could we just remove python 3.9 testing?19:20
TheJuliafrom yoga?19:20
TheJuliajust bizzare19:20
JayFunmaintained/ branches are "kill it with fire" 19:22
TheJuliasort of yeah19:22
JayFFYI > https://review.opendev.org/c/openstack/releases/+/928862 [ironic] Releases for OSSA-2024-00319:24
TheJuliaI guess the base fundamental issue is openstack-python3-yoga-jobs is template with the base config19:28
TheJuliaxena seems to be okay19:29

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!