| clarkb | Anyone else here for the meeting? Apologies for getting the agenda out late today | 19:00 |
|---|---|---|
| fungi | yup | 19:00 |
| clarkb | #startmeeting infra | 19:01 |
| opendevmeet | Meeting started Tue Aug 31 19:01:25 2021 UTC and is due to finish in 60 minutes. The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot. | 19:01 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 19:01 |
| opendevmeet | The meeting name has been set to 'infra' | 19:01 |
| corvus | o/ | 19:01 |
| clarkb | #link http://lists.opendev.org/pipermail/service-discuss/2021-August/000279.html Our Agenda | 19:01 |
| clarkb | #topic Announcements | 19:02 |
| clarkb | The openstack feature freeze has begun. Keep that in mind as we make changes. In the past we've described it as a good time to be "slushy" eg we dno't have to freeze but keep in mind potential impacts and try to avoid impacting the release process if possible | 19:02 |
| ianw | o/ | 19:03 |
| clarkb | Looking at zuul queues they aren't quite as deep as I expected. but I expect that to grow through the week as last minute changes get merged | 19:03 |
| clarkb | #topic Actions from last meeting | 19:04 |
| clarkb | #link http://eavesdrop.openstack.org/meetings/infra/2021/infra.2021-08-24-19.01.txt minutes from last meeting | 19:04 |
| clarkb | ianw had an action to look into logo hosting. That has happened and I've added a separate agenda topic just for that to catch up | 19:04 |
| clarkb | #topic Specs | 19:04 |
| clarkb | #link https://review.opendev.org/c/opendev/infra-specs/+/804122 Prometheus Cacti replacement | 19:04 |
| clarkb | frickler and tristanC have left some thoughts on ^ if others can take a look and let me know what you think that would be helpful | 19:05 |
| clarkb | adding a service like prometheus seems like the sort of safe slushy activity we could do over the next few weeks while openstack puts its release together | 19:05 |
| clarkb | #topic Topics | 19:06 |
| clarkb | At this point I'm not sure that topic will ever go away :) | 19:06 |
| clarkb | #topic Mailman server upgrades | 19:06 |
| clarkb | Fungi ran through the lists.katacontainers.io upgrade to Focal last week. It appears to be working as expected | 19:07 |
| clarkb | I've cleaned up some of the testing nodes we had held/created for that process today. | 19:07 |
| fungi | yeah, upgrade went smoothly for the most part | 19:07 |
| clarkb | fungi: you had talked about measuring resource consumption on the upgraded server to try and infer if it would be a problem with the vhosts on lists.o.o | 19:07 |
| fungi | i did not find time to restart services on the servers over the weekend | 19:08 |
| fungi | but maybe some night this week when things are slow | 19:08 |
| clarkb | ok, I think that may be the last thing we are waiting on before we do an upgrade of lists.o.o? Do we want to schedule that upgrade assuming the metrics gathering happens and doesn't show any problems and postpone otherwise? | 19:09 |
| fungi | yeah, i think that would be fine | 19:10 |
| fungi | the sooner we can get it off xenial and ua the better | 19:10 |
| clarkb | fungi: you mentioned doing it on a weekend which works for me as well. Maybe September 12? | 19:10 |
| clarkb | That is the Sunday before openstack's RCs go out | 19:11 |
| fungi | yeah, the 11th or 12th would work fine for me | 19:11 |
| clarkb | do you think that would be enough time for prep? Also how concerned are you about impacts post upgrade? You did the first server so probably have the best grasp of that | 19:12 |
| clarkb | fungi: ^ | 19:13 |
| fungi | not especially concerned, the sysvinit scripts are the only real difference | 19:13 |
| fungi | and associated disk layout | 19:13 |
| clarkb | ok in that case lets say start at 1500UTC Sunday? | 19:13 |
| fungi | also the age of the server, size of the rootfs, et cetera | 19:13 |
| clarkb | er 1500UTC Sunday September 12 | 19:14 |
| fungi | sure, that seems good | 19:14 |
| clarkb | fungi: do you want to draft an announcement of that or should I? | 19:14 |
| fungi | still early enough it probably won't impact apac monday | 19:14 |
| fungi | i can write up an announcement later today | 19:14 |
| clarkb | thanks! | 19:14 |
| clarkb | Anything else on the subject of list servers? | 19:15 |
| fungi | not this week i don't think | 19:15 |
| clarkb | #topic Improve OpenDev's CD Throughput | 19:15 |
| clarkb | This should be quick as I didn't have time last week to do the mapping. Sorry about that. I ended up being very distracted with some family stuff that poppped up. Things appear far more normal this week thankfully | 19:16 |
| clarkb | Basically everything after tuesday last week is getting transplanted to after tuesday this week. I plan to get started on that in the next day or two | 19:16 |
| clarkb | That said I wanted to ask if the small easy improvements we have already made are helping | 19:17 |
| clarkb | From what I can see it seems to have helped quite a bit, but also wasn't paying too much attention last week after the inventory file matchers got updated | 19:17 |
| fungi | yeah, stuff is deploying faster i think, anecdotally observed anyway | 19:18 |
| clarkb | great and no concerns of jobs not running when we expected them? | 19:19 |
| fungi | none that i've noticed, but that could easily go unspotted for weeks/months | 19:19 |
| clarkb | ya something to watch out for, but not an immediate issue it seems | 19:20 |
| clarkb | #topic Gerrit Account Cleanups | 19:21 |
| clarkb | Last Tuesday I said I'd do the external id deletion for the last set of retired accounts tomorrow but that iddn't happen becuase I wasn't able to sit at a computer for reliably long enough periods of time | 19:21 |
| clarkb | I intend on actually doing it tomorrow and really hope nothing comes up this week :) | 19:21 |
| clarkb | Just a heads up | 19:21 |
| clarkb | #topic OpenDev Logo Hosting | 19:22 |
| clarkb | ianw dug into this after the meeting last week and came up with a few different ideas. Ultimately it seems like maybe we were going to use docker images like a packaging format? | 19:22 |
| ianw | yes, i think this works | 19:22 |
| fungi | yeah, that seemed like a great solution | 19:22 |
| clarkb | ianw: are there changes we should be reviewing for that? | 19:23 |
| fungi | we basically build a docker image of our logos and then that can be used in building or deploying our containerized services | 19:23 |
| ianw | #link https://review.opendev.org/c/opendev/system-config/+/805932 | 19:23 |
| ianw | this builds a small container that has nothing but images/logos/whatever at a defined location | 19:23 |
| ianw | #link https://review.opendev.org/c/opendev/system-config/+/805933 | 19:23 |
| ianw | then tries to use it | 19:24 |
| ianw | although 805933 failed because we didn't build the container, i'm not sure how to get around that | 19:24 |
| clarkb | cool I'll review those after lunch. Are there related changes to update the paste and gerrit themes to pull from gitea hosting ? or maybe copy them similarly to what you have there for gitea? | 19:24 |
| ianw | it doesn't touch anything that triggers the container build in the gate, and there isn't one available for download | 19:25 |
| clarkb | ianw: you need the requires and provides in the jobs? | 19:25 |
| clarkb | I think if you do that it may work? | 19:25 |
| ianw | hrm, will that *trigger* the job, or just order it correct? | 19:26 |
| clarkb | I think its meant to say that job must run (or if it is a soft requirement then it will only run if otherwise triggered) | 19:26 |
| clarkb | but I could be wrong about how that interacts with a brand new image. | 19:26 |
| ianw | anyway we can discuss in #opendev, the only other thing was it uses buildkit to achieve this | 19:27 |
| clarkb | corvus: do you know if we've written this down anywhere for opendev specifically? That might be another good output of the job mapping exercise and reorg we can be more consistent about those thinsg and write down how to use them properly for our jobs | 19:27 |
| ianw | iirc mordred switched zuul to this anyway, and it seems like it will one day be the docker default | 19:27 |
| clarkb | ya there was some buildkit feature around mounts that I think zuul wanted to use? seems fine to use buildkey for richer image creation | 19:28 |
| ianw | yes, it's the same feature; being able to copy things in from another container | 19:29 |
| fungi | ahh, as opposed to having to use the container image as a "layer" i guess | 19:30 |
| ianw | #link https://review.opendev.org/c/zuul/zuul/+/712717 | 19:30 |
| ianw | is the zuul one, it's not merged | 19:30 |
| corvus | provides/requires won't cause a job to run, just allow artifacts to pass between queue items | 19:31 |
| corvus | which includes docker images | 19:32 |
| clarkb | got it so the first change could provide the image to the second change | 19:32 |
| clarkb | and that would work even without an image published to docker hub because we have the zuul registry | 19:32 |
| corvus | so if change A runs a job that provides docker image foo, and change B runs a job that requires docker image foo, then change B's job will run with foo built for change A | 19:33 |
| corvus | clarkb: exactly | 19:33 |
| clarkb | ianw: thank you for looking into this. | 19:34 |
| corvus | after a really quick look, i think that since 805933 has a "RUN" command that references the assets image, then the gitea building jobs should require the assets image | 19:34 |
| ianw | right, but in this case, system-config-build-image-assets has file matchers that mean it only runs if one of the asset files are updated | 19:34 |
| ianw | maybe the matchers for that job need to include the Dockerfile of images that use it | 19:35 |
| corvus | ianw: but 805932 builds the image, so 933 would get it | 19:35 |
| ianw | (that job being ...build-image-assets) | 19:35 |
| clarkb | ianw: no, the chagne that adds the assets image would build and create the image and put it in the zuul registry. Then requiring that image would allow the gitea job to get it from zuul registry | 19:35 |
| clarkb | you don't need the assets job to run when gitea updates | 19:35 |
| corvus | provides/requires is specifically for bridging across changes (it doesn't do anything for jobs within a single change). to make both work, you need provides/requires and job dependencies | 19:36 |
| fungi | and later updates to the gitea image would just pull it from dockerhub if not provided by a change queued ahead of it? | 19:37 |
| corvus | so in this case we do want both things. | 19:37 |
| ianw | oh right, ok the jobs ran in parallel and the image wasn't pushed into the bulidset registry | 19:37 |
| corvus | fungi: correct | 19:37 |
| ianw | zuul didn't wait to start the gitea job because it didn't have the requires/provides. ok, can fix that | 19:37 |
| fungi | makes sense | 19:38 |
| corvus | 3 cases: 1) no change to an image: pull from hub. 2) change to assets image in same change as gitea image change: job dependencies cause assets image build to run before gitea image build. 3) change to assets image in one change, change to gitea in second: provides/requires causes assets build to run before gitea image build. | 19:38 |
| corvus | and provides/requires works whether or not the items are in the queue at the same time. | 19:39 |
| corvus | (if change B runs long after change A, it uses the database) | 19:39 |
| clarkb | Thanks | 19:41 |
| clarkb | That was all I had on the agenda I sent out. | 19:41 |
| clarkb | #topic Open Discussion | 19:41 |
| clarkb | Is there anything else to bring up? | 19:41 |
| ianw | corvus: speaking of all this, https://review.opendev.org/c/zuul/zuul-jobs/+/798969 could probably do with your review. that adds some more info on container builds | 19:42 |
| fungi | oh, openstackid is in the process of moving to foundation webdev management | 19:42 |
| fungi | i updated dns for it during the meeting and am working on the system-config cleanup change now | 19:42 |
| fungi | but expect some failures for, e.g., cert updates until we get things cleared out of system-config | 19:43 |
| ianw | i've been working on updating nodepool images to bullseye | 19:43 |
| ianw | #link https://review.opendev.org/c/openstack/diskimage-builder/+/806318 | 19:44 |
| ianw | seems to be the major thing required in a dib release, if someone wants to double-check | 19:44 |
| clarkb | ianw: how are you handling arm64 wheels when bumpingto bullseye? Do we have bullseye wheels built for openstack? | 19:44 |
| ianw | yes we should, the buildx phase has been ok | 19:45 |
| ianw | #link https://review.opendev.org/c/zuul/nodepool/+/806312 | 19:45 |
| clarkb | cool, that would be my only concern since that can be slow without wheels pre built | 19:46 |
| ianw | is the change that updates thing. i'll put some more documentation in. the release job there will fail until we release a fixed dib | 19:46 |
| clarkb | I'll add those to the list of reviews to do this afternoon | 19:46 |
| ianw | are we now in a process of updating the other images on a case-by-case basis? | 19:47 |
| clarkb | I think so, but I'm not sure if anyone has started yet. Doing the matrix eavesdrop bot should be easy since that one was done previously? | 19:47 |
| ianw | (updating them to bullseye) | 19:47 |
| ianw | ok, i can look at a few too. while there's no rush, it also seems worth getting the pain over when there isn't a pressing need | 19:48 |
| clarkb | thanks! | 19:48 |
| clarkb | Sounds like this may be it. Thank you everyone. We'll see you here next week. Feel free to reach out in #opendev or on the service-discuss mailing list | 19:49 |
| fungi | thanks clarkb! | 19:50 |
| clarkb | #endmeeting | 19:50 |
| opendevmeet | Meeting ended Tue Aug 31 19:50:05 2021 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 19:50 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/infra/2021/infra.2021-08-31-19.01.html | 19:50 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/infra/2021/infra.2021-08-31-19.01.txt | 19:50 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/infra/2021/infra.2021-08-31-19.01.log.html | 19:50 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!