Tuesday, 2023-07-11

clarkbfungi: thank you for getting the meeting agenda sorted out. I'll be there (6am tomorrow relative to my local time)01:27
clarkbtonyb: I think the two tasks I have in mind are either A) continuing the server upgrade process (probably looking at mirrors next) or B) beginning the process of adding bookworm python container base images then converting services to those images02:10
clarkbB) may be a bit more interesting since you've gotten some of the server upgrade stuff already02:10
tonybYup, I'll try B as the primary task but still look at A as well.02:11
tonybSo we need to build and publish a bookworm-3.11 container and then use that instead of the bullseye-3.1102:12
tonyband I guess also update the 3.9 containers to 3.1102:12
clarkbyup exactly. In the past we've done a 1:1 for the python versions so that we can drop the old debian stuff independent of the old python stuff02:12
tonybcontainers in opendev.org/zuul/* should pull from quay.io but others should pull from docker.io 02:13
tonyband all from lines should be fully qualified02:13
clarkbhttps://opendev.org/opendev/system-config/src/branch/master/zuul.d/docker-images has three files that currnetly define all thos ejobs. I think we can copy them over to three new bookwork files then add the jobs in https://opendev.org/opendev/system-config/src/branch/master/zuul.d/project.yaml#L152-L16002:13
clarkbya that sound sright02:13
clarkboh and here https://opendev.org/opendev/system-config/src/branch/master/zuul.d/project.yaml#L302-L31002:14
clarkbIn the past when I've done this I started out by seeing if I can drop the oldest python version. It may be worthwhile checking if anything is using 3.9 anymore02:14
* clarkb looks at codesearch02:14
tonybthere are a few, ptgbot for example02:15
clarkbhttps://codesearch.opendev.org/?q=3.9-bullseye&i=nope&literal=nope&files=&excludeFiles=&repos= ya looks like gerrit too02:15
clarkbso ya maybe we decouple the two tasks02:15
clarkbThe real fun is in converting the services to those base images though. Since that will expose you to all the things consuming them02:17
clarkbMost things using containers have decent CI testing though so that should be less adventure and more straightforwatd I hope02:17
clarkbBut the first step is the base image update and then we can use speculative testing from there. It is a great benefit when doing this stuff02:18
tonybYeah.  The whole system design is pretty darn confidence inspiring :)02:21
clarkbtonyb: it is worth calling out how the base images work02:22
clarkbgive me a minute to page that in so that I don't say the wrong then then will try to summarize02:22
tonybfor users in the zuul project are there any reasons to *NOT* update to python 3.11 (assuming testing passes02:23
clarkbI don't think so02:24
clarkbok so the builder image has all the scripts in it. The `assemble script` basically installs python projects and tracks all of the necessary dependencies and record sthem. It also captures all of the wheels for the dependencies. Then you copy that info to the python-base image and install things using the recorded deps list and wheels. What this does is keeps all build deps on the02:26
clarkbthrowaway builder image and reduces the size of the final image based on the base image02:26
clarkbThe idea is that the two stage process keeps things minimal while still being accurate/complete02:27
tonybThat makes semse, and matches what I've seen in other places02:29
opendevreviewTony Breeds proposed opendev/system-config master: Add Debian bookworm based python images  https://review.opendev.org/c/opendev/system-config/+/88810202:34
tonybFWIW, I've started taking notes in https://etherpad.opendev.org/p/opendev-python-updates  It's very very raw right now but hopefully it will be helpful next time02:37
clarkbsound sgood02:39
clarkbtonyb: I left a comment about one missed thing but I would wait on test results first02:39
clarkbit won't impact the check jobs02:39
clarkblooks like the jobs are passing so you can probably push up the fixed version03:11
clarkbuwsgi depends on the other images so will go last if we want to wait03:11
clarkbdoing code review in the heat is fun03:21
opendevreviewTony Breeds proposed opendev/system-config master: Add Debian bookworm based python images  https://review.opendev.org/c/opendev/system-config/+/88810203:37
opendevreviewTony Breeds proposed opendev/system-config master: Update accessbot to use the bookworm container  https://review.opendev.org/c/opendev/system-config/+/88810603:37
tonybI have fixed the uwsgi issue also03:37
tonyband 888106 is my best guess as to how to do the accessbot service03:37
tonybI should read the infra-manual to try and come up with an internal model for which services can/should be grouped together03:38
clarkbtonyb: the other thing that may be helpful for that grouping is the playbooks in system-config because we tend to do roles per service then aggregate them on hosts03:38
clarkb*aggregate them on hosts using playbooks03:38
clarkbthe accessbot change lgtm03:41
tonybWith my understanding of the whole speculative container testing, I'm assuming that even though the bookworm cotainers aren't promoted they'll be available to the later acessbot change.03:41
clarkbyes, they should be via the requires here https://review.opendev.org/c/opendev/system-config/+/888106/1/zuul.d/docker-images/accessbot.yaml I think03:42
tonybbased on: https://opendev.org/opendev/system-config/src/branch/master/zuul.d/infra-prod.yaml#L492-L494 I should do accesbot, ircbot and matrix-eavesdrop as one change03:48
clarkbthough you can still split them up if you prefer03:50
clarkbI don't think this will have a big or any impact on us but good to be aware of https://letsencrypt.org/2023/07/10/cross-sign-expiration.html04:25
opendevreviewTony Breeds proposed opendev/system-config master: Add Debian bookworm based python images  https://review.opendev.org/c/opendev/system-config/+/88810205:05
opendevreviewTony Breeds proposed opendev/system-config master: Update accessbot to use the bookworm container  https://review.opendev.org/c/opendev/system-config/+/88810605:05
tonybclarkb: 888106 is failing with "manifest for opendevorg/python-base:3.11-bookworm not found: manifest unknown: manifest unknown" https://zuul.opendev.org/t/openstack/build/ab79e98cdd0242649cbc50593e87dae1/log/job-output.txt#72305:27
tonybI thought the idea was that it'd pull that from the buildset registry transparently05:29
tonybis there something missing from the jobs to cause that or have I misuderstood? and we need 888102 to land before the new images are available at all05:30
clarkbhrm I expected that would work. But maybe we are missing something05:36
clarkbprovides/requires does the cross buildset plumbing and then the job dependencies does the ordering within a buildset and I thought we had both05:37
clarkbI wonder if we need an explicit depends on for zuul to catch the relationship? except these are a normal git relationship which should be fine05:39
opendevreviewTony Breeds proposed opendev/system-config master: Update accessbot,ircboot and matrix-eavesdrop to bookworm container  https://review.opendev.org/c/opendev/system-config/+/88810605:46
opendevreviewTony Breeds proposed opendev/system-config master: Update to latest tag for Limnoria  https://review.opendev.org/c/opendev/system-config/+/88811005:46
opendevreviewTony Breeds proposed opendev/system-config master: Update ircbot and matrix-eavesdrop to python-3.11  https://review.opendev.org/c/opendev/system-config/+/88811105:46
tonybOkay I'll poke around and see if I can figure out what, if anything, is missing05:47
tonybOnce the testing passes that's kinda how I see things going for those services.05:48
tonybGenerally what else needs to happen?05:48
* tonyb goes for a short walk05:48
clarkbI think its basically iterating through and sorting out getting the various services to work on the new platform05:48
clarkbthe irc bots are likely to be straightforward so are a good place to start. But something like gerrit might be more complicated05:49
tonybyeah I figured so too.06:07
tonybthanks clarkb 06:07
Clark[m]And thank you!06:13
opendevreviewMoritz Haase proposed zuul/zuul-jobs master: roles/ensure-python: Fix 'python_use_stow' option  https://review.opendev.org/c/zuul/zuul-jobs/+/87182206:17
opendevreviewMerged opendev/infra-manual master: Trivial: fix image sizing in creators guide  https://review.opendev.org/c/opendev/infra-manual/+/87755413:13
opendevreviewMerged opendev/infra-manual master: PyPI: clarify case where owner can't be removed  https://review.opendev.org/c/opendev/infra-manual/+/87824013:13
opendevreviewMerged opendev/zone-opendev.org master: Replace ze04-ze06  https://review.opendev.org/c/opendev/zone-opendev.org/+/88551414:17
corvusremoved the fact cache for ze04-6 and approved the system-config change to replace them14:43
opendevreviewMerged opendev/system-config master: Replace ze04-ze06  https://review.opendev.org/c/opendev/system-config/+/88550914:54
slittle1Looking for a little help on starlingx-SDO-rv-service-core group.  I'm a member, but for this one group I don't seem to have the power to add others.   Not sure why.16:54
fungislittle1: somehow the group owner got set to a specific user rather than to the group itself16:56
fungilooks like it was done by user "Poornima Y N"16:56
fungi(who is one of the group's members)16:57
fungii'll fix it back but it will take a moment16:57
slittle1ok, well we need to fix that.  Intel is no longer contributing to StarlingX in this area.  I need to assign a few windriver guys to maintain this git in their absence.16:58
fungislittle1: refresh and you should have control of it again16:59
fungilooks like it was probably done just after the group was created back in late 2020 and has just been that way ever since17:00
corvus#status log started zuul on replacement ze04-ze06 servers17:38
opendevstatuscorvus: finished logging17:38
corvusso far they appear to be running normally17:38
corvusthe first of my spot check builds completed successfully17:52
corvusfungi: i'm getting a lot of NDRs from lists; you seeing those?17:55
fungii saw some earlier i think, someone spoofing abuse@lists.o.o to send messages to other addresses at lists.o.o (mainly mailing lists)17:59
corvusthis is different; errors from django17:59
fungioh, i'll check again in a sec18:00
corvusdjango's emailing root@localhost which is bouncing, so an ndr goes to root@lists.opendev.org which goes out to us18:00
fungicorvus: sorry, was trying to join a conference call but found it. one of the downsides on filtering those into a separate maildir is that i forget to check it regularly18:08
corvusi think this is a new behavior starting yesterday18:09
fungilooks like django doesn't have a 429 error page template, hence the traceback. the cause looks like someone trying to brute-force account signups18:11
clarkbfungi: corvus the address mm3 is configured to send those emails to is configurable if we need ot change that. But ya the underlyng error is the lack of a 429 template to be served18:48
fungiclarkb: oh, while the tc call is still going on, what are our resource and maintenance concerns with having an ever growing number of branches left open on projects? zuul scanning them for configuration (and them possibly containing broken configs), but what else?18:51
fungialso still having periodic jobs running (and perpetually failing) for all those branches18:54
fungialso i guess we need to be able to delete resources18:54
fungiimages for distro versions contemporary with the original releases those branches are based on18:55
clarkbya test resources that a contemporary is a big one18:57
clarkbpeople pushing changes to the branches and being completely ignored18:57
clarkbincreased zuul configuration resolution times18:57
clarkbperiodic jobs18:58
clarkbI think you captured the bulk of it18:58
fungilooks like i received 53 of those django traceback bounces between 08:51:24 and 18:52:58 so it's been an hour since the last one, but given that's a 10-hour span i don't know whether it's subsided or just bursty19:51
corvustonyb clarkb it does look like there are no zuul artifacts listed in the inventory vars: https://zuul.opendev.org/t/openstack/build/ab79e98cdd0242649cbc50593e87dae1/log/zuul-info/inventory.yaml19:57
clarkbcorvus: the change is a git child to the parent that adds the new images and has requires/provides set. IIRC that is what manipulates the artifacts across buildsets?20:00
clarkbbut these are new images20:01
corvusyeah still pulling on threads20:01
clarkback thanks20:01
corvusclarkb: tonyb commented on the problem in https://review.opendev.org/88810220:04
corvusbasically looks ilke a copypasta typo20:04
corvuson the provides20:04
fungiwow, i would not have spotted that20:07
tonybAhh okay, and it's also missing from the existing bullseye ones so I missed it there20:08
* tonyb will fix both20:08
corvusyeah, can be tough to spot those once an image gets uploaded.  we should make a tool.20:08

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!