| opendevreview | Merged openstack/diskimage-builder master: Provide an ability to disable serial console injection https://review.opendev.org/c/openstack/diskimage-builder/+/922441 | 01:10 |
|---|---|---|
| opendevreview | Merged openstack/diskimage-builder master: remove console entries when console is disabled https://review.opendev.org/c/openstack/diskimage-builder/+/922442 | 01:32 |
| opendevreview | Merged zuul/zuul-jobs master: Support .python-version files in ensure-python https://review.opendev.org/c/zuul/zuul-jobs/+/922515 | 02:53 |
| *** liuxie is now known as liushy | 06:13 | |
| opendevreview | Tony Breeds proposed opendev/system-config master: [DNM] Run ansible-devel under python-3.11 https://review.opendev.org/c/opendev/system-config/+/922704 | 07:52 |
| *** mgoddard- is now known as mgoddard | 10:12 | |
| *** tosky_ is now known as tosky | 12:37 | |
| Clark[m] | fungi: there have been a few openstack-discuss emails since we updated server configs. I wonder if logs show the services were happier? | 14:19 |
| Clark[m] | Specifically I wonder if the blog message indicating a possible off by one queue size problem went away. And I guess maybe we have timestamps that might indicate how quickly things got out | 14:19 |
| Clark[m] | Heh log messages not blog messages | 14:20 |
| fungi | i was looking at them | 14:30 |
| fungi | message 1sMSii-00ACu0-Fa was accepted from mailman by exim at 13:29:56 and then queued for delivery to my address as 1sMSok-0007pr-Gj arriving at my mailserver at 13:36:11 (6m15s delay) | 14:32 |
| fungi | now to compare to exim's logs | 14:32 |
| fungi | looks like the exim config tuning didn't take: "2024-06-26 13:29:56 1sMSii-00ACu0-Fa no immediate delivery: more than 10 messages received in one connection" | 14:33 |
| Clark[m] | Maybe we don't properly reload exim when those files update? | 14:34 |
| Clark[m] | Or the var doesn't make it into a template ? | 14:34 |
| fungi | yeah, checking into those | 14:36 |
| fungi | /etc/exim4/exim4.conf:smtp_accept_max_per_host = 50 | 14:37 |
| fungi | /etc/exim4/exim4.conf was updated at 20:54 and /var/run/exim4/exim.pid at 20:55 | 14:39 |
| fungi | so maybe it's not the right config option? | 14:39 |
| fungi | in good news, mailman did what we wanted and batched them up 45 addresses to a message | 14:45 |
| Clark[m] | Could mailman be using something other than SMTP? Or ya just need a different var set | 14:45 |
| fungi | seems to not be smtp_accept_max at least because that's 100 already | 14:45 |
| fungi | okay, looks like we should have used smtp_accept_queue_per_connection for that, not smtp_accept_max_per_host | 14:53 |
| fungi | "This option limits the number of delivery processes that Exim starts automatically when receiving messages via SMTP..." | 14:53 |
| fungi | i'll get that patch going in a sec | 14:53 |
| opendevreview | Jeremy Stanley proposed opendev/system-config master: Correct the Exim queue threshold for Mailman https://review.opendev.org/c/opendev/system-config/+/922836 | 15:04 |
| fungi | clarkb: ^ | 15:04 |
| fungi | oh, need to fix a comment i copied | 15:08 |
| opendevreview | Jeremy Stanley proposed opendev/system-config master: Correct the Exim queue threshold for Mailman https://review.opendev.org/c/opendev/system-config/+/922836 | 15:10 |
| clarkb | fungi: that looks good to me. I did have one minor thing if there is reason to spin a new patchset | 15:30 |
| fungi | i guess i'm unclear on what the default specification is for. the default value in exim if unset is 10 | 15:31 |
| fungi | as indicated in the config documentation | 15:31 |
| fungi | setting it to 0 is not the same as not setting it | 15:32 |
| clarkb | fungi: the default at line 57 here: https://review.opendev.org/c/opendev/system-config/+/922836/2/playbooks/roles/base/exim/README.rst ? | 15:32 |
| clarkb | I would probably say null to match the entries above | 15:32 |
| fungi | "smtp_accept_queue_per_connection Use: main Type: integer Default: 10" is what https://www.exim.org/exim-html-current/doc/html/spec_html/ch-main_configuration.html says though | 15:36 |
| clarkb | right, but the readme in our role is for the ansible stuff. I think I would say default is null on line 57 then where I've commented about the grammar thing rewrite it to say "If unset the default used by exim is 10" | 15:36 |
| clarkb | then we've captured the default for the ansible role and the default exim will use in that situation | 15:37 |
| fungi | i figured indicating ":default: 10" in the role was less confusing than claiming the default is null and then writing an additional sentence saying that if it's unset exim will enforce a limit of 10 | 15:37 |
| clarkb | well the default for the role is null | 15:37 |
| fungi | but i can do it that way instead | 15:37 |
| fungi | well, the default value passed by the role is null, but the default value used by the software it's configuring is 10 | 15:37 |
| clarkb | right I think we can capture both things with the suggestion above. | 15:38 |
| fungi | makes more sense to me to document the end result of setting or not setting the option | 15:38 |
| clarkb | the options documented above do the opposite thing though | 15:38 |
| clarkb | so we'd be more consistent with the rest of the role doing it that way | 15:38 |
| opendevreview | Jeremy Stanley proposed opendev/system-config master: Correct the Exim queue threshold for Mailman https://review.opendev.org/c/opendev/system-config/+/922836 | 15:39 |
| clarkb | but I don't feel too strongly about it. I just wanted to fix the minor grammar thing to make it more readable | 15:40 |
| fungi | like that^ ? | 15:40 |
| clarkb | ya that seems to match what we've done with the options above. Document the default ansible role values and then specify how that impacts exim behavior | 15:40 |
| opendevreview | Tatiana Ovchinnikova proposed opendev/irc-meetings master: Update horizon meeting info https://review.opendev.org/c/opendev/irc-meetings/+/922841 | 15:46 |
| opendevreview | Merged opendev/irc-meetings master: Update horizon meeting info https://review.opendev.org/c/opendev/irc-meetings/+/922841 | 15:53 |
| mordred | does mailman use lmtp? | 16:22 |
| mordred | (I can't remember what uses that - maybe that was exim->cyrus- still early morning brain) | 16:22 |
| fungi | mordred: mailman can be configured to do lmtp: https://docs.mailman3.org/projects/mailman/en/latest/src/mailman/runners/docs/lmtp.html | 16:25 |
| fungi | and is how we configure ours | 16:27 |
| mordred | neat! I'm not yet hallucinating | 16:27 |
| fungi | mordred: for reference https://opendev.org/opendev/system-config/src/branch/master/docker/mailman/core/docker-entrypoint.sh#L138-L142 | 16:28 |
| fungi | so basically we do lmtp incoming from exim to mailman, but then smtp outgoing from mailman to exim | 16:29 |
| fungi | if you look at the headers on a message you received from one of our mailing lists, you'll notice there's a received header for when exim accepted the incoming post from the internet, and then another for when exim received the outbound copy from localhost, but no intermediate received hop for mailman itself | 16:32 |
| fungi | popping out to a dental checkup, but should be back online in an hour or so | 16:36 |
| clarkb | I'm around though sitting no hold seems likely in my near future so not sure how useful I'll be. However if 922836 goes in I can keep an eye on it | 16:38 |
| opendevreview | Merged opendev/system-config master: Correct the Exim queue threshold for Mailman https://review.opendev.org/c/opendev/system-config/+/922836 | 17:10 |
| clarkb | lenovo wants me to boot their original image to confirm the problem. I erasonable request I guess. However, I cannot download the image directly from them I have to download an exe that does it for me... this is a problem since I don't have a windows machine to run the exe. The solution? They'll mail an image to me | 17:37 |
| clarkb | something something web browsers have existed for decades... | 17:37 |
| clarkb | this is how laptops become home servers | 17:38 |
| frickler | exe via mail, not spooky at all ;) | 17:42 |
| clarkb | heh yes but no different than getting other install media delivered | 17:47 |
| clarkb | (including when the device was originally shipped) | 17:47 |
| clarkb | there is unforunately a relatively high level of trust involved here, but somewhat unavoidable as someone has to build hte hardware, not possible to 3d print a computer yet | 17:48 |
| clarkb | or maybe it is but it would be a very slow computer | 17:48 |
| corvus | is it a cd; do you have a cd reader? :) | 17:51 |
| JayF | clarkb: or you could ask one of a myriad of your OSS buddies who has a windows install lying around to proxy it for you :D | 17:53 |
| clarkb | you may be surprised to learn I own a usb dvd drive and a usb 3.5" floppy drive | 17:53 |
| JayF | clarkb: if it gets you online faster, I can run the exe, even if it needs to go directly on usb media, then take the image | 17:53 |
| * JayF owns a USB -> 3.5" floppy ribbon cable board | 17:53 | |
| clarkb | JayF: nah I have my desktop running fine so I can be patient | 17:53 |
| JayF | those usb floppies are low quality stuff, being able to put a vintage 3.5" floppy on a USB bus means I can read a LOT of disks that the usb readers can't | 17:54 |
| clarkb | says something about zip disks that no one has usb zip drives | 17:54 |
| JayF | all useful data that was stored on zip drives were clicked away | 17:54 |
| JayF | and there is one industry that used them until the very last possible second: journalists | 17:54 |
| clarkb | and the only reason I don't have bluray drives is that all the bluray media is encrypted and its annoying to deal with so I never bothered | 17:55 |
| clarkb | and the laptop actually works using the fallback vesa driver but only at 1920x1080 on the 2560x1440 display and without brightness control. Its also completely unaccelerated that way so streaming video might suffer (I haven't tried) and gaming is probably also out of the question (but I don't bother) | 17:56 |
| clarkb | its only when booted with amdgpu that things go weird. Stuff skews and I get tearing and black triangles | 17:57 |
| clarkb | memtest checked out ok whcih is honestly disappointing because that is a much easier issue to proove | 17:57 |
| clarkb | anyway not I get ot decide if anything on that machine is worth backing up (probably not) before reinstalling it in a couple days and trying to reproduce the problems under windows | 17:58 |
| fungi | i had a usb zip drive, not sure if i still do | 17:59 |
| clarkb | nice | 17:59 |
| fungi | also remember jazz drives | 17:59 |
| JayF | I *have* seen that kinda issue be caused by bugs in amdgpu driver | 17:59 |
| fungi | but yes i definitely still have a usb 3.5" diskette drive | 17:59 |
| JayF | so if you haven't tried on a different kernel (I assume you have), you might wanna give it a go | 17:59 |
| fungi | also usb dvd-rw burner | 18:00 |
| clarkb | JayF: yes it has occurred over about 8 months of tumbleweed kernels (6.1 to 6.9 now I think?) as well as ubuntu jammy and noble. And I used the same live ubuntu jammy disk on my brothers laptop which is almost identical (I think only memory differs?) and cannot reproduce on his | 18:01 |
| clarkb | oh and I reproduced with wayland and X | 18:01 |
| JayF | troubleshooting-wise, you went above and beyond and kept going LOL | 18:01 |
| fungi | i also have several drive docks that i can plug esata, mmc and a few other common storage interfaces (including a couple of the ones that were ~unique to macbooks). they come in handy for systems recovery | 18:02 |
| clarkb | if I had to guess what the problem is my hunch is something like firmware corruption since the gpu is embedded on the cpu and things generally work otherwise | 18:02 |
| clarkb | but I've also had several firmware updates so not sure if its something stickier than a normal firmware update | 18:02 |
| JayF | the thing that's weird is that failure mode implies video memory issues | 18:04 |
| JayF | but there is no video memory that memtest can't touch | 18:04 |
| clarkb | there are registers though | 18:05 |
| clarkb | and probably caches | 18:05 |
| clarkb | and memtest would go through the cpu versions of those not the gpu versiosn so wouldn't see problems if the problems where there | 18:05 |
| JayF | I'll also note there's a version of memtest floating around that doesn't do a great job | 18:06 |
| JayF | and they are all called variations of 'memtest' iirc | 18:06 |
| clarkb | you just install via your package manager these days | 18:06 |
| JayF | https://www.memtest86.com/ is what I use | 18:06 |
| fungi | deploy for the mailman config change failed infra-prod-base, i'll check the logs on bridge | 18:07 |
| JayF | my package manager has both `memtest86+` and `memtest86-bin`; and the `memtest86+` version I've personally seen miss memory issues on my machine (which is sad since it's the oss one) | 18:07 |
| JayF | memtest86+ is the one that is on many installers iirc | 18:08 |
| JayF | but this is old anecdata and memtest86+ seems to have been improved since then | 18:08 |
| clarkb | ya I've never had problems with it. I had an older desktop with memory that was unstable at stock clocks and underclocking made it stable and memtest was able to sort through all that | 18:09 |
| clarkb | memtest86+ I mean | 18:09 |
| clarkb | fungi: if its ansible rc -13 its the low chance of the race between ansible task starting and control persistence process stopping | 18:09 |
| clarkb | fungi: the other issue we sometimes get in base is apt-get afiling for network issues iirc | 18:10 |
| fungi | mirror01.regionone.osuosl.opendev.org Failed to update apt cache: E:Failed to fetch http://ddebs.ubuntu.com/dists/focal-updates/main/binary-arm64/Packages.xz File has unexpected size (282980 != 282708). Mirror sync in progress? [IP: 91.189.91.49 80] | 18:11 |
| fungi | clarkb wins | 18:11 |
| fungi | i'll try a manual apt update and make sure it's not persistent | 18:11 |
| fungi | Reading package lists... Done | 18:12 |
| fungi | seems fine now | 18:12 |
| clarkb | you may have gotten a different mirror behind ddebs.ubuntu.com too and or they were finishing up a sync | 18:13 |
| fungi | yeah, good point | 18:13 |
| fungi | anyway, seems transient, nothing for us to fix | 18:13 |
| clarkb | agreed, it happens periodically and the only "fix" is probably to choose an explicit mirror that updates more safely but that presents other potential issues | 18:14 |
| fungi | guess we'll just wait for the next successful infra-prod-service-lists3 run | 18:14 |
| clarkb | JayF: also I think the last time I had memory issues it surfaced as corrupt git repos | 18:14 |
| clarkb | JayF: I haven't run into problems like that yet on this machine. Though its possible the problem is only in the regions used by the gpu and I wouldn't get fs/repo corruption as a result | 18:15 |
| clarkb | fungi: depending on what the merge of that change enqueued you may be able to just reenqueue the buildset? | 18:15 |
| fungi | into deploy? | 18:16 |
| clarkb | ya. Looking at the list of jobs I think this is safe as long as we don't merge any new project additions (manage-projects is in the job set) | 18:16 |
| fungi | doing | 18:16 |
| JayF | clarkb: My memory issues surface when doing updates (compiling) ... I also have had 2 sticks fail outta the 4 in my desktop over the last 2 years (I'm fairly certain at this point there's a design flaw in this model of ram, but they're honoring the warranty)... I'm just getting too good at finding memory issues and resolving them from too much practice :D | 18:16 |
| fungi | i unfondly remember the days of maintaining badram skiplists in my kernel command-line | 18:18 |
| fungi | but i used to have a lot of antique and/or dumpster-dived systems | 18:18 |
| JayF | these are some g-skill gaming desktop sticks with leds (which were hand-me-downed, with the system, into my dev hypervisor) and I suspect something about the LEDs is causing heat issues | 18:19 |
| clarkb | back in my Intel days we would make periodic dumpster diving trips around the labs and fabs to supplement our cube hardware | 18:19 |
| clarkb | you could find some good stuff doing that and as long as it stayed in the office and didn't go home I was told it was probably fine :) | 18:19 |
| fungi | oh, yeah at my first hosting company job, the new employee hazing was that the owner was too cheap to buy new computers and we had piles of old hardware in the back storeroom, so you were told to go put together a working system of the spares/scraps and that would be your desktop | 18:22 |
| clarkb | fungi: just double checking nothing has merged in system-config or project-config since the exim queue fixup change so ya reenqueing that should be fine | 18:22 |
| fungi | i scored a dual-head sun u2 workstation because nobody else knew how to get a working linux install on ultrasparc | 18:23 |
| clarkb | and looks like it did get reenqueued | 18:23 |
| clarkb | fungi: you didn't want to run CDE as your desktop? | 18:23 |
| fungi | clarkb: yep, i too checked that we hadn't merged anything else first | 18:23 |
| clarkb | we actually ran fluxbox on a lot of those old sun boxes | 18:23 |
| fungi | oh no, i was plenty experienced managing solaris already, and despised cde | 18:24 |
| fungi | anyway, there were so many scrapped sun systems nobody wanted, i was able to max out the memory, drive capacity, add extra processors... it was, from a contemporary perspective, probably the most powerful work computer i've ever had | 18:25 |
| clarkb | then eventually I raelized that I could xforward firefox and thunderbird off of a linux machine and avoid 90% of the slowness that way | 18:25 |
| fungi | also it was awesome having a 64-bit system when most of my colleagues were still on pentiums | 18:26 |
| clarkb | fungi: those machines were good at running make -jBIGNUMBER and not much else. Also it was annoying when you'd run into weird gcc toolchain behavior differences between linux and solaris | 18:26 |
| fungi | thinking back, it was an untra60 creator3d, not u2e | 18:28 |
| clarkb | I seem to recall c++ iostream had some not well documented platform differences around (re)setting the seek pointer | 18:28 |
| fungi | s/untra/ultra/ | 18:28 |
| opendevreview | Jay Faulkner proposed openstack/project-config master: Ironic no longer uses storyboard https://review.opendev.org/c/openstack/project-config/+/922864 | 18:38 |
| clarkb | fungi: the deploy should be done now | 18:42 |
| mordred | clarkb: there were 2 useful things about having the -j128 jobs on the crazy sun box in the drizzle build farm: a) find ALL of the places where makefile deps weren't fully expressed properly b) find the places accidentally assuming an x86-ism | 18:43 |
| fungi | deploy of the mailman config update succeeded the second time around | 18:57 |
| fungi | however, it looks like exim didn't actually get restarted | 18:57 |
| fungi | last modified time on /var/run/exim4/exim.pid is 17:30 | 18:58 |
| frickler | clarkb: oh, that was mail as in physically send it. I was assuming e-mail | 18:58 |
| fungi | /etc/exim4/exim4.conf is updated though as of 17:29 | 18:59 |
| fungi | weird. i guess an hourly run took care of it? | 18:59 |
| fungi | also seems to probably be working based on a cursory glance at recent post headers for openstack-discuss | 19:03 |
| fungi | one message took 18 seconds to get from mailman to my mailserver | 19:03 |
| fungi | another took 14 seconds | 19:04 |
| fungi | so this seems to have done the trick | 19:04 |
| * fungi calls it done | 19:04 | |
| opendevreview | Jay Faulkner proposed openstack/project-config master: Ironic no longer uses storyboard https://review.opendev.org/c/openstack/project-config/+/922864 | 19:10 |
| Clark[m] | Sorry eating lunch now. | 19:23 |
| Clark[m] | frickler: ya physical media. Crazy | 19:23 |
| Clark[m] | fungi: woot | 19:23 |
| fungi | as things seem to be at a lull for the moment, i'm going to pop out and grab early dinner shortly, but don't expect to be gone long (should be under an hour) | 19:26 |
| Clark[m] | fungi: when you get back maybe you can look at topic:drop-centos-8-stream? It's got a good number of changes that should be safe to land | 19:28 |
| fungi | sure thing | 19:29 |
| fungi | Clark[m]: i left questions on a couple of them, approved the others | 19:36 |
| fungi | heading out now, will be back in an hour-ish | 19:36 |
| opendevreview | Merged opendev/system-config master: Drop CentOS 8 Stream jobs https://review.opendev.org/c/opendev/system-config/+/922641 | 19:39 |
| clarkb | fungi: re the zuul-jobs change I can try a patchset that just flips the nodeset and see what happens | 19:50 |
| clarkb | if that doesn't work I'm inlcined to remove the old job and let someone else fix the fips testing on enwer platforms though | 19:50 |
| opendevreview | Clark Boylan proposed zuul/zuul-jobs master: Switch fips fole testing to CentOS 9 Stream https://review.opendev.org/c/zuul/zuul-jobs/+/922643 | 19:53 |
| clarkb | fungi: JayF in 922864 we keep the groups specification for projects when they use an lp project that is different than the repo name? Is that correct? | 19:58 |
| clarkb | and if so projects like sushy-oem-idrac don't have lp entries. Maybe we need to update groups for more projects? | 19:59 |
| clarkb | I left some comments about the two things I found based on that assumption | 20:06 |
| opendevreview | Merged opendev/system-config master: Delete centos 8-stream mirror content https://review.opendev.org/c/opendev/system-config/+/922749 | 20:10 |
| opendevreview | Merged openstack/diskimage-builder master: minor ci: quick cleanup and remove centos8 https://review.opendev.org/c/openstack/diskimage-builder/+/922763 | 20:18 |
| opendevreview | Merged openstack/diskimage-builder master: Fix setting apt mirror for noble https://review.opendev.org/c/openstack/diskimage-builder/+/920466 | 20:20 |
| opendevreview | Merged openstack/diskimage-builder master: Fix uninstall gentoo packages https://review.opendev.org/c/openstack/diskimage-builder/+/921769 | 20:25 |
| fungi | clarkb: JayF: yeah, my unchecked assumption was that any project not specifying a groups entry either exists on lp or someone is going to make it in short order | 20:53 |
| *** dmellado0755 is now known as dmellado075 | 21:06 | |
| tonyb | infra-root: Can I get some reviews on: https://review.opendev.org/q/topic:noble-mirror | 21:57 |
| opendevreview | Merged openstack/diskimage-builder master: growvols, enforce pvcreate and overwrite old VG signature https://review.opendev.org/c/openstack/diskimage-builder/+/922059 | 22:00 |
| opendevreview | Merged opendev/system-config master: Remove centos rsync mirroring tooling https://review.opendev.org/c/opendev/system-config/+/922750 | 22:01 |
| opendevreview | Clark Boylan proposed zuul/zuul-jobs master: Switch fips fole testing to CentOS 9 Stream https://review.opendev.org/c/zuul/zuul-jobs/+/922643 | 22:07 |
| opendevreview | Clark Boylan proposed zuul/zuul-jobs master: Be more cautious initing mimetypes https://review.opendev.org/c/zuul/zuul-jobs/+/922893 | 22:07 |
| tonyb | Also thoughts on: 918482: Build a stow of python 3.13. | https://review.opendev.org/c/openstack/project-config/+/918482 are appreciated. | 22:08 |
| tonyb | Given the challenges with 3.12 and the noted issues with eventlet and 3.13, It'd be good to have some kind of python-3.13 available soon(ish) | 22:09 |
| fungi | beta3 was supposed to be tagged yesterday, but ended up delayed | 22:10 |
| clarkb | tonyb: fwiw building a non optimized python3.12 only took a few minutes. That said the stowed version may perform better depending on how it gets compiled | 22:10 |
| fungi | optimized takes 10-15 minutes for me on good hardware | 22:12 |
| fungi | most of the time is spent on make test and recompiling | 22:12 |
| tonyb | I suspect that the default stow/pyenv isn't optimized | 22:13 |
| fungi | basically optimized builds compile once, run an abbreviated `make test` with the initial build and profiling enabled, then compile again optimized for the profiling results | 22:13 |
| clarkb | tonyb: I noticed that one of the role test jobs is missing from that stack so I -1'd the ppa change to add it. I didn't -1 the very first change since that one doesn't seem to run those test jobs (but the ppa change did run those test jobs so we should ensure we've got a noble one) | 22:13 |
| clarkb | tonyb: ya pyenv only enables some minor optimizations | 22:14 |
| tonyb | clarkb: Good point. | 22:16 |
| tonyb | I'll let the first two merge and then rebase and fix the 3rd and 4th | 22:17 |
| clarkb | sounds good | 22:17 |
| tonyb | Taking a Tangent on python 3.13 .... Does Ubuntu have a published policy about which python versions get packaged for releases? | 22:20 |
| clarkb | tonyb: I think they just grab whatever is current when they freeze the distro | 22:20 |
| clarkb | and then sometimes they will backport a newer version like they did with 3.11 on jammy | 22:20 |
| tonyb | but then they add the next one? | 22:20 |
| fungi | for most things that they take directly from debian, they basically just pick a date and snapshot the state of sid/unstable, then start backporting fixes | 22:21 |
| tonyb | and similar with focal | 22:21 |
| clarkb | they have added the next one to jammy and focal. I don't think they did so with bionic or xenial | 22:21 |
| fungi | though they may handle python specially | 22:21 |
| clarkb | the next lts is likely to be 3.14 based on current ubuntu and python release cadences I think | 22:22 |
| clarkb | since they did 3.8 then 3.10 then 3.12 | 22:22 |
| tonyb | Okay so we're looking at past releases and trying to guess what current/future releases will do | 22:22 |
| fungi | yes, these days python releases yearly and ubuntu lts every 2 years | 22:22 |
| fungi | so would skip a python minor release for each lts | 22:22 |
| tonyb | as the default but with possible backports of the odd/missing releases | 22:23 |
| clarkb | simiarly debian is offset by a year and has done 3.9 and then 3.11 and presumably 3.13 will be next | 22:23 |
| fungi | though "minor" is a bit of a misnomer. there's a pre-pep discussion underway to switch python to date-based versioning, which seems to have increasing support | 22:23 |
| fungi | the suggestion being to skip from 3.13.x to 3.25.x | 22:24 |
| fungi | for the 2025 release | 22:24 |
| tonyb | I see | 22:25 |
| fungi | apparently there's a lot of angst among the python maintainers that a lot of people these days assume 3-segment version numbers mean the project follows semver conventions, which the python community is decidedly hostile toward the idea of, considering concepts like "backward compatibility" to be a folly or utter fallacy | 22:26 |
| fungi | so the thinking is that by making the middle segment of the version equivalent to the last two digits of the release year, people will no longer assume such nonsense | 22:26 |
| fungi | because semver is clearly of the debbil | 22:28 |
| clarkb | except if you release annually you haven't changed the way the numbers increment so its all the same to end users | 22:28 |
| tonyb | I'm not sure I agree with that conclusion. It'd still look like a semver | 22:28 |
| clarkb | 3.13 -> 3.14 is no different than 3.25 -> 3.26 | 22:28 |
| fungi | yes, precisely | 22:28 |
| fungi | this has been repeatedly pointed out to them. seems to be one of those inconvenient facts best ignored | 22:28 |
| tonyb | The best kind of facts | 22:29 |
| fungi | for the brave, it's https://discuss.python.org/t/pep-2026-calendar-versioning-for-python/55782 | 22:30 |
| fungi | but i don't recommend following that link without a full bottle of something in hand | 22:30 |
| * tonyb is not brave this morning | 22:31 | |
| tonyb | ... and it's too early for booze | 22:31 |
| fungi | at first i thought someone had misread the calendar and posted it a couple of months late for april 1 | 22:31 |
| tonyb | LOL | 22:31 |
| JayF | clarkb: fungi: I removed the groups parameter only from the repos that I verified were listed as having an LP in the ironic bug dashboard configuration. I'll double check my change to make sure that none of the ones I just removed storyboard from would be in a weird spot | 22:32 |
| JayF | Fwiw, sushy-oem-drac is likely going away soon so it wouldn't surprise me if it's in a weird spot | 22:32 |
| JayF | I think we're just going to integrate those quarks directly into sushy | 22:32 |
| fungi | all 6 quarks? | 22:33 |
| JayF | I'd like to buy an "i" | 22:33 |
| fungi | strange but charming | 22:34 |
| JayF | I almost accidentally let the secret slip about the CERN research to speed up ironic /s | 22:34 |
| clarkb | tonyb: I'm looking at this dib element for python stow and it might be the most obtuse bash due to it being 90% arrays | 22:35 |
| tonyb | clarkb: Yes. It takes some time to digest | 22:36 |
| clarkb | tonyb: my main concern right now is it isn't claer to me if we'll install every minor version of python 3.13 | 22:37 |
| clarkb | the way the element loops over arrays and does string matching I think that if python-build lists 3.13.0 and 3.13.1 and so on we'll get a separate build for each in a seprate dir and we don't want that. We just want 3.13.latest in one dir | 22:37 |
| clarkb | and I'm trying to decipther that and having a hard time | 22:37 |
| clarkb | I think maybe DIB_PYTHON_VERSIONS_ARRAY[${version%.*}]+="${version##*.} is trying to avoid this problem | 22:38 |
| tonyb | clarkb: I'm 99% it's the last release for any tuple | 22:38 |
| clarkb | tonyb: ya I think the code is trying to accomplish that but the python-build readme shows three tuples not two tuples | 22:40 |
| tonyb | https://opendev.org/openstack/diskimage-builder/src/branch/master/diskimage_builder/elements/python-stow-versions/install.d/70-python-build#L36 | 22:40 |
| tonyb | so we tell DIB we want 3.13 | 22:41 |
| tonyb | DIB grabs pyenv and builds an array for all the python versions supported by python-build and adds then to a has based on the tuple | 22:42 |
| clarkb | ya but we actually build based on what python-build --definitions outputs | 22:43 |
| clarkb | not what we put in then match against it here: https://opendev.org/openstack/diskimage-builder/src/branch/master/diskimage_builder/elements/python-stow-versions/install.d/70-python-build#L32 | 22:43 |
| tonyb | Yeah | 22:43 |
| clarkb | basically I'm just finding this code impossible to undersatnd and there are zero comments to help so I'm somewhat inclined to either add comments or rewrite it to be a bit more decipherable so that later if the builds break we have a chance of debugging | 22:43 |
| opendevreview | Merged opendev/system-config master: Add noble repo files https://review.opendev.org/c/opendev/system-config/+/921770 | 22:44 |
| clarkb | actually there is one comment but it comments the one thing in the whole script that is deipherable on its own :) | 22:44 |
| opendevreview | Merged opendev/system-config master: Remove *zuul-role-integration-centos-8-stream* jobs https://review.opendev.org/c/opendev/system-config/+/922360 | 22:44 |
| tonyb | LOL | 22:44 |
| clarkb | also this will silenty fail successfully if it cannot build a python version you ahve requested | 22:45 |
| clarkb | at the very least I think we should fix ^ | 22:46 |
| clarkb | because it is better for the image builds to fail than to have jobs fail beacuse the python they need randomly disappears | 22:46 |
| clarkb | (that case can happen if python-build --definitions doesn't include the requested version, we'll just loop over the list and never build it) | 22:46 |
| tonyb | Yes, if we asked for 3.99 which clearly doesn't exist it'd silently do nothing | 22:46 |
| clarkb | tonyb: I'm more concerned about asking it for 3.13 today which works but then after 3.13 is EOL'd python-build might drop it and then we'd mysteriously stop including it but yes | 22:47 |
| tonyb | Ah true | 22:47 |
| tonyb | python-build doesn't seem to drop releases and I was assuming that we'd update the build to more or less always be $next release | 22:48 |
| tonyb | so once 3.13 is in a distro we'd use that and stow build 3.14 | 22:49 |
| tonyb | If python-build only takes a few mins then I may just drop the idea for stow and use pyenv in each job | 22:50 |
| clarkb | ya but we'd still keep building the now older version of that distro and we can't drop the stowed python without going through and updating all the jobs (which is a huge pain just to delete a broken distro release even) | 22:50 |
| clarkb | basically we should plan on keeping these python installs for the entire time we build the image otherwise its a lot of cleanup | 22:50 |
| clarkb | tonyb: ya zuul and some others just did pyenv in the jobs and it was like 3 minutes total I thinK? | 22:50 |
| clarkb | it was very fast | 22:50 |
| tonyb | Okay. | 22:51 |
| tonyb | That's probably a better approach as it's per job rather than per image and is more "dynamic" | 22:52 |
| opendevreview | Merged zuul/zuul-jobs master: Be more cautious initing mimetypes https://review.opendev.org/c/zuul/zuul-jobs/+/922893 | 22:56 |
| opendevreview | Merged zuul/zuul-jobs master: Switch fips fole testing to CentOS 9 Stream https://review.opendev.org/c/zuul/zuul-jobs/+/922643 | 22:56 |
| clarkb | in any case if we do go the route of using stow on the images I think we should ensure the element fails if a requested version isn't buildable and also add comments to the script indicating what is going on so that if the element itself breaks in the future after we're depending on it we can debug it more easily | 23:01 |
| clarkb | because building a version of python that people rely on then then mysteriously not building it later is going to create a painful debugging experience with the current code | 23:01 |
| tonyb | Noted. | 23:01 |
| opendevreview | Tony Breeds proposed opendev/system-config master: Don't use the openstack-ci-core/openafs PPA on noble and later https://review.opendev.org/c/opendev/system-config/+/921786 | 23:16 |
| opendevreview | Tony Breeds proposed opendev/system-config master: Test mirror services on noble https://review.opendev.org/c/opendev/system-config/+/921771 | 23:16 |
| tonyb | With respect to ^^ would anyone object to me upgrading *all* the cloud.region mirror nodes to noble (even the ones I just upgraded to jammy), what way that whole "subsystem" would be on noble | 23:18 |
| * tonyb is full of questions this morning (sorry) | 23:18 | |
| fungi | tonyb: it's slow, boring work, but i don't object. maybe start with the ones on the oldest ubuntu versions first though in case you don't have time to do the newer ones after all | 23:31 |
| Clark[m] | Ya no objections from me. | 23:32 |
| tonyb | Yup that's fair, do the 5-6 on focal first and then the 5-6 on jammy | 23:32 |
| tonyb | then we'll have 10-12 on noble | 23:33 |
| opendevreview | Aurelio Jargas proposed zuul/zuul-jobs master: Add ensure-poetry role https://review.opendev.org/c/zuul/zuul-jobs/+/922286 | 23:33 |
| fungi | tonyb: mainly, pay attention to local filesystem/volume partitioning choices and cinder attachments, since it varies by provider capabilities and flavors | 23:45 |
| fungi | that's a lot of the tedium which prevents them from being totally cookie-cutter | 23:45 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!