Thursday, 2023-10-05

opendevreviewSteve Baker proposed openstack/ironic master: Introduce conductor touch while offline  https://review.opendev.org/c/openstack/ironic/+/89737502:26
rpittaugood morning ironic! o/06:33
opendevreviewDmitry Tantsur proposed openstack/ironic master: Bump workers_pool_size to 300 and remove queueing of tasks  https://review.opendev.org/c/openstack/ironic/+/89707106:52
opendevreviewDmitry Tantsur proposed openstack/ironic-python-agent master: [WIP] Delay heartbeat after a failure and handle Unavailable  https://review.opendev.org/c/openstack/ironic-python-agent/+/89705507:23
opendevreviewDerek Higgins proposed openstack/sushy-tools master: Add new ironic driver  https://review.opendev.org/c/openstack/sushy-tools/+/88704608:28
opendevreviewDerek Higgins proposed openstack/sushy-tools master: Add new ironic driver  https://review.opendev.org/c/openstack/sushy-tools/+/88704610:46
iurygregorygood morning 11:53
zigoHi there!13:24
zigoI can't build Ironic in Bookworm backports, with many (39 times) failures of this type:13:24
zigohttps://paste.opendev.org/show/b5I08UffMzG6BKpJrJtK/13:24
zigoAny idea?13:24
zigoTheJulia JayF ^13:24
zigoFYI, the Ironic backport is the last thing I need to do to complete the Bobcat release ! :)13:24
TheJuliaDriving like 350 miles atm13:25
TheJuliaI’ll look when I can stop13:25
rpittauzigo: hi! let me see if I can find anything13:26
zigoThanks.13:27
zigorpittau: It built fine in Unstable, so probably the issue is in a dependency that I didn't update ?13:28
rpittaupy3.11 in CI is actually passing13:28
rpittauyeah13:28
rpittauI was thinking the same13:28
rpittauI'm running locally now13:28
rpittauzigo: btw the CI runs on bookworm for 3.11 :)13:29
zigoNice ! :)13:30
rpittauyou may want to have a look here https://8c14ddfe56056e4e35ff-0ca364809dbe3b9250075209fe309a8e.ssl.cf5.rackcdn.com/897071/2/check/openstack-tox-py311/3d3a415/13:30
zigorpittau: What should I look for?!?13:38
rpittauinstalled packages, but I think I can provide you directly the list, jsut gimme a sec13:39
zigoIf the result isn't obvious to you, then I can probably get the "pip freeze" in Sid myself and do a diff, but I'm not sure I'll manage to double-guess like this either...13:40
zigoI'll try anyways ! :)13:40
rpittauzigo: there you go https://paste.opendev.org/show/baMGjNph16u16i6tQsWF/13:41
zigoCheers.13:41
masgharTheJulia: Dont IRC and drive :)13:43
dtantsur^^ this ^^13:43
zigoHere's the diff: https://paste.opendev.org/show/bTtJKI1xoY6UNmqHlcYs/13:45
zigomasghar: Don't ping someone that's driving ! :)13:46
masgharDidnt think about that uh oh13:46
rpittauzigo: maybe urllib ? teh validate_link function makes use of it and there were some fixes from 1.26.12 and 1.26.1513:53
JayFOpenInfra Live starting on youtube in about 5 minutes; going over highlights from the Bobcat release :)13:55
JayFzigo: if rpittau doesn't get you taken care of, please file a bug with the details and lets see.13:55
zigorpittau: I can easily try upgrading urllib, ok !13:56
zigoWill try in a few minutes ...13:56
TheJuliazigo: :)14:13
TheJuliazigo: good news?14:15
zigoNot yet, in a never-ending meeting ...14:15
TheJuliaJoy, sounds like your in good hands14:20
TheJuliaI’m going to get a cup of coffee and get back on the road for a little while14:20
zigoI'm going away, will try tomorrow...14:26
iurygregorytks JayF =)14:28
JayFThese updates are the easiest 30 minutes of my job every year :D 14:29
JayFeven if I'm not PTL next cycle just ask and I can play spokesmodel :P14:29
iurygregorywhy go is so complicated to debug? D: 14:31
iurygregoryjesus ...14:31
JayFI find python 10000x more difficult to debug, personally14:31
JayFin terms of, when I have to plug it into a debugger, I mean14:32
iurygregorynormally the errors are more clear for me in python 14:32
JayFI will say most of my golang work was ... simpler problem sets14:32
iurygregoryI see the error in Go, I go to that file an check all the calls but I don't see where I should change things XD14:32
JayFI did a little work in Armada but not a lot of heavy lifting14:32
JayFif it's OSS,  wanna share the error and a link to the repo?14:33
JayFmaybe fresh eyes help even if I'm not a golang-y?14:33
JayFs/a/as/14:33
iurygregoryhttps://github.com/gophercloud/gophercloud/pull/2795 :D14:33
dtantsurWell, Python rarely has segfaults14:33
JayFlol14:33
JayFdtantsur: you didn't run our big horrible agent patch on top of Juno :D 14:34
JayFdtantsur: python segfaults were the #1 cause of ir-conductor losses in the Rax onmetal env14:34
dtantsuriurygregory: there is at least one typo14:35
dtantsurLOL14:36
rpittauL408 :)14:36
dtantsurSpeaking of badges: "Python segfaulter" goes to JayF14:36
iurygregoryO.o14:36
JayFdtantsur: probably should award that one to jroll more than me :D 14:36
JayFdtantsur: I'm not sure I had a lick of code in Ironic at that point; I was the IPA dude :D 14:36
iurygregoryrpittau, ohhhhhhhhhhh14:37
iurygregoryjesus, I need more coffee!14:37
JayFhey, at least it's not like14:37
JayFsome actual problem14:37
JayFtypos are my favorite kind of bugs in WIP code; because it means my mental model of the system isn't broken14:38
iurygregoryok 3/4 done with L408 :D14:38
JayF>     convenience.go:253: Failure in requests_test.go, line 195: at .Firmware.Reason expected "", but got "Driver ipmi does not support firmware (disabled or no implemented)." 14:39
JayFthe remaining one, I assume?14:39
iurygregoryfound :D14:39
iurygregoryyeah14:39
iurygregoryI forgot to update the node to provide this message14:39
JayF\o/14:39
iurygregorynow time to test things on the bifrost setup :D14:43
clarkba long time ago (I seem to recall debugging this from my HP office in Seattle) a python version (an early 3.x iirc) had a segfault in the garbage collector that you could hit if you constructed references between objects correctly. We had to convince ubuntu that this was a severe enough bug to do a fix against on an LTS14:47
JayFIDK if we ran 2.7 or 3.x for those14:49
JayFbut the timing is right for it to have been 3.x14:49
opendevreviewMerged openstack/ironic master: Bump workers_pool_size to 300 and remove queueing of tasks  https://review.opendev.org/c/openstack/ironic/+/89707115:02
jrollJayF: I had long forgotten about that, thanks for the flashbacks :D15:35
rpittaugood night! o/15:35
JayFjroll: gonna be honest, I assumed your nick in here was a zombie bouncer lol15:40
jrollit was until it wasn't! :)15:41
JayFjroll: in any event; I hope things are still going well at stripe(?)15:41
jrollyep! good overall, hope things are good over here for everyone as well15:43
JayFBobcat (2023.2) released yesterday15:43
dtantsurgradually conquering the world, as always15:43
JayFand we're somewhere in the ballpark of Ironic's 10th birthday15:43
dtantsuroh wow15:43
JayFeverytime I say that I think another hair in my beard greys15:43
dtantsurthat's why I avoid having a beard :D15:43
JayFif I do that, my head looks like Kirby as an infant :P15:44
dtantsurROFL15:46
stevebaker[m]good morning19:22
TheJuliaGood morning19:23
TheJuliaAlthough afternoon now and was driving since like 4:30 AM19:23
zigoI still have all of these failures, even after backporting urllib3 and requests as suggsted rpittau: https://paste.opendev.org/show/bOJn14KImMDHgEdeYP2z/19:49
zigoSo, the issue isn't urllib3 and reuqests ...19:49
* zigo tries without tmpfs ...19:50
JayFzigo: those errors, I think, are angry about stdout?20:06
JayFzigo: are you putting these in some kind of limited environment with no stdout/err attached, perhaps? Or redirecting in a weird way?20:07
zigoHum...20:07
zigoI'm building in Jenkins ...20:07
JayFbingo20:07
JayFtry it locally, I bet it works20:07
zigoIt worked on my laptop yeah...20:07
JayFI'm not saying it's not a bug, but it'd be nice to confirm it's environmental20:07
zigoI mean, I didn't try with the same env.20:07
zigoWith same repo and all, on bookworm.20:07
JayFI think this is literally print() erroring 20:08
zigoBut I could build on my laptop with sbuild.20:08
zigoOk.20:08
clarkbthat exception is raised if a non blocking socket would block for the requested operation20:08
zigoAny clue how I could fix? :)20:08
JayFI'm getting there :D20:08
zigo2>&1 ? :)20:08
clarkbprobably need to check why stdout or whatever print() is writing to is set non blocking20:08
JayFSo first of all, please bug this behavior20:08
JayFbecause we sohuld not fail tests in this scenario IMO20:09
zigoI'll try building on my laptop with the same env (sbuild, bookworm, and the jenkins repos).20:09
JayFsecondly, https://opendev.org/openstack/ironic/src/branch/master/ironic/tests/unit/api/base.py#L115 is the code that is angry I think20:09
zigoThen I'll tell you if I see the same behavior, ok ?20:09
JayFzigo: worst case; I think if you patch out all the print() from that file, you'd get a test pass20:10
zigoOk, and since that's a debug thingy, in the tests, then I wont care, right?20:10
JayFexactly20:10
zigoThanks. Will try all of that.20:10
zigoFirst, build on my laptop ... :)20:11
* JayF wonders if the easiest path to this is env var for "print api requests" that defaults to true20:11
zigoIt wont take long.20:11
JayFzigo: there are 14 prints in Ironic unit tests; not all are in thoe files20:11
JayFand many should likely be removed20:12
JayFbut anything of that nature would, definitionally, be unbackportable as a CI-only fix, I think?20:12
JayFokay I need to get lunch before I get trapped down this rabbithole lol20:12
JayFfixing zigo's unit tests way more interesting than finishing my talk slides but the talk has a solid deadline :P20:13
clarkbnormally you don't print() to something that is non blocking though20:14
* zigo is building ...20:15
zigoJayF: Same issue on my laptop, let me try removing the print.20:18
zigoJayF: No print, no bug, you naied it !!! :)20:23
zigoThanks.20:23
clarkbironic doesn't do the stdout/stderr capturing in the test suite as far as I can tell so that isn't swapping out fds20:23
zigoNow, I still don't understand why building under Unstable doesn't show the issue ...20:23
zigofds as in file descriptors?20:24
zigoAnyways, I'm *VERY* happy, that makes it a Debian release of OpenStack the 2nd day of the upstream release for me! :)20:25
zigo(this was my very last issue building packages...)20:25
clarkbzigo: yes file desciptors. Some projects swap out stdout and stderr with buffers to capture prints and attach them to the subunit stream. Otherwise the prints will be all interleaved between tests can be very messy20:26
clarkbbut I don't see ironic doing that. It is weird that if this is actually stdout that it would be non blocking for some reason20:26
zigoOh, looks like I also had the issue in unstable, at least it shows in the buildd: https://buildd.debian.org/status/fetch.php?pkg=ironic&arch=all&ver=1%3A23.0.0-1&stamp=1696497516&file=log20:36
* zigo fixes it... :)20:37
zigoFun fact: the number of failures differ on each build ...20:38
JayFclarkb: zigo: Yeah honestly I have no idea how this could be triggered tbh; but it's clearly a thing20:53
opendevreviewSteve Baker proposed openstack/ironic master: Introduce conductor touch while offline  https://review.opendev.org/c/openstack/ironic/+/89737521:44
opendevreviewSteve Baker proposed openstack/ironic master: Implement drain shutdown support  https://review.opendev.org/c/openstack/ironic/+/89749621:44

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!