| -@gerrit:opendev.org- Clark Boylan proposed: [opendev/git-review] 984279: Add py313 testing against Gerrit 3.12 https://review.opendev.org/c/opendev/git-review/+/984279 | 00:00 | |
| -@gerrit:opendev.org- Steve Baker proposed: [openstack/diskimage-builder] 984486: Skip local loop device creation for no-final-image builds https://review.opendev.org/c/openstack/diskimage-builder/+/984486 | 03:40 | |
| -@gerrit:opendev.org- Zuul merged on behalf of Clark Boylan: [opendev/git-review] 984470: Drop global statement for unassigned var https://review.opendev.org/c/opendev/git-review/+/984470 | 05:06 | |
| -@gerrit:opendev.org- Zuul merged on behalf of Clark Boylan: [opendev/git-review] 984279: Add py313 testing against Gerrit 3.12 https://review.opendev.org/c/opendev/git-review/+/984279 | 05:14 | |
| -@gerrit:opendev.org- Michal Nasiadka proposed wip: [opendev/zone-opendev.org] 984393: Revert^2 "Promote mirror04.gra1.ovh" https://review.opendev.org/c/opendev/zone-opendev.org/+/984393 | 05:31 | |
| -@gerrit:opendev.org- Michal Nasiadka proposed wip: [opendev/zone-opendev.org] 984393: Revert^2 "Promote mirror04.gra1.ovh" https://review.opendev.org/c/opendev/zone-opendev.org/+/984393 | 05:32 | |
| -@gerrit:opendev.org- Michal Nasiadka proposed wip: [opendev/zone-opendev.org] 984393: Revert^2 "Promote mirror04.gra1.ovh" https://review.opendev.org/c/opendev/zone-opendev.org/+/984393 | 05:32 | |
| -@gerrit:opendev.org- Michal Nasiadka proposed wip: [opendev/zone-opendev.org] 984393: Revert^2 "Promote mirror04.gra1.ovh" https://review.opendev.org/c/opendev/zone-opendev.org/+/984393 | 05:33 | |
| -@gerrit:opendev.org- Michal Nasiadka proposed wip: [opendev/zone-opendev.org] 984393: Revert^2 "Promote mirror04.gra1.ovh" https://review.opendev.org/c/opendev/zone-opendev.org/+/984393 | 05:33 | |
| @mnasiadka:matrix.org | Gerrit UI editor gets lost with these tabs and spaces :-) | 05:33 |
|---|---|---|
| -@gerrit:opendev.org- Michal Nasiadka proposed wip on behalf of Tony Breeds: [opendev/system-config] 963802: Add mediawiki to the opendevmirror on Quay.io https://review.opendev.org/c/opendev/system-config/+/963802 | 05:58 | |
| -@gerrit:opendev.org- Michal Nasiadka marked as active: [opendev/zone-opendev.org] 984393: Revert^2 "Promote mirror04.gra1.ovh" https://review.opendev.org/c/opendev/zone-opendev.org/+/984393 | 06:00 | |
| @noonedeadpunk:matrix.org | hey folks! with merge of https://review.opendev.org/c/openstack/project-config/+/981924 I think somebody should add me manually to the group in gerrit (https://review.opendev.org/admin/groups/1ddb2bc8c910a8de88d8d110338002b6895c0899,members) so I could further manage it? | 06:33 |
| @mnasiadka:matrix.org | Let me try | 07:35 |
| @mnasiadka:matrix.org | Dmitriy Rabotyagov: Members added to group openstack-ansible-power-reviewers: noonedeadpunk@gmail.com | 07:43 |
| @noonedeadpunk:matrix.org | mnasiadka: nice, thanks! | 07:43 |
| @mnasiadka:matrix.org | np | 07:45 |
| -@gerrit:opendev.org- Michal Nasiadka proposed: [openstack/project-config] 978566: propose-updates: Add pcu target https://review.opendev.org/c/openstack/project-config/+/978566 | 08:10 | |
| -@gerrit:opendev.org- Zuul merged on behalf of Jeremy Stanley https://matrix.to/#/@fungicide:matrix.org: [opendev/system-config] 984123: Refresh the Puppetlabs Secure APT repository key https://review.opendev.org/c/opendev/system-config/+/984123 | 14:23 | |
| -@gerrit:opendev.org- Clark Boylan proposed: [opendev/system-config] 983930: Cleanup Apache UA filters https://review.opendev.org/c/opendev/system-config/+/983930 | 14:47 | |
| @clarkb:matrix.org | infra-root I would like to proceed with ^ today. We managed to get access to google domain reporting for docs.openstack.org and it is complaining about 403s for a subset of pages. I suspect that those pages are being crawled by the google bots currently listed in ^ | 14:48 |
| @clarkb:matrix.org | I think worst case we revert and put rules back but I suspect that anubis is doing a lot of work now to push back on the crawl flood so this should largely be safe | 14:50 |
| @clarkb:matrix.org | mnasiadka: looks like https://mirror03.ord.rax.opendev.org is working now. Also re adding a group member in Gerrit we must've documented creation of the admin Gerrit account? Looks like you were able to create one anyway | 14:58 |
| @clarkb:matrix.org | (I'm always happy when we have documented things we should be documenting) | 14:58 |
| @fungicide:matrix.org | both static03 and static04 seem unstressed. load averages on both are in the 0.1-0.3 range, with essentially nothing paged out to swap. they're both using all available memory for buffers/cache, but that's to be expected | 15:07 |
| @fungicide:matrix.org | i propose to update dns (in cloudflare) to move docs.openstack.org back to the normal static.opendev.org cname which will shift its traffic to static03 | 15:08 |
| @fungicide:matrix.org | this will allow us to free up the additional 30gb server we dedicated to that site back when we were trying to get mod_security waf working for it | 15:09 |
| @fungicide:matrix.org | static03 is a 15gb server, so still roughly twice as powerful as the static02 server we started with | 15:11 |
| @clarkb:matrix.org | fungi: yup sounds good. I think we should do that alongside the UA filter cleanup I've proposed but don't think they need to happen in any specific strict order. Just that they are related and shuold happen closely together so we can continue to monitor | 15:11 |
| @fungicide:matrix.org | i can go ahead and do the dns change for docs.openstack.org at any time, i'm around all today to keep an eye on the server. ttl for the record is 5 minutes so switching back and forth should happen relatively quickly | 15:11 |
| @clarkb:matrix.org | I think sooner is better | 15:13 |
| @fungicide:matrix.org | done | 15:14 |
| @fungicide:matrix.org | #status log Updated DNS for docs.openstack.org to shift requests back to our shared static content server used by other sites | 15:15 |
| @status:opendev.org | @fungicide:matrix.org: finished logging | 15:15 |
| @fungicide:matrix.org | i'm keeping an eye on static03 load/memory and staying logged into the cloudflare dashboard in case we need to switch back quickly | 15:16 |
| @mnasiadka:matrix.org | Clark: I managed with some bits of documentation :) | 15:16 |
| @mnasiadka:matrix.org | Clark: It would be good to merge the revert^2 - https://review.opendev.org/c/opendev/zone-opendev.org/+/984393 - and then we can promote as well mirror03.ord.rax - if we're up for doing that today | 15:17 |
| @clarkb:matrix.org | mnasiadka: yup https://mirror04.gra1.ovh.opendev.org/ seems to be working for me. I've approved it | 15:19 |
| -@gerrit:opendev.org- Zuul merged on behalf of Michal Nasiadka: [opendev/zone-opendev.org] 984393: Revert^2 "Promote mirror04.gra1.ovh" https://review.opendev.org/c/opendev/zone-opendev.org/+/984393 | 15:22 | |
| @mnasiadka:matrix.org | DNS points to mirror04 now, seems fine | 15:37 |
| @mnasiadka:matrix.org | let me raise the patch to promote mirror03.ord.rax | 15:37 |
| -@gerrit:opendev.org- Michal Nasiadka proposed: [opendev/zone-opendev.org] 984590: Promote mirror03.ord.rax https://review.opendev.org/c/opendev/zone-opendev.org/+/984590 | 15:39 | |
| @fungicide:matrix.org | cpu load and memory on static03 still seems reasonable. 5-minute load average is now around 0.5 | 15:40 |
| @fungicide:matrix.org | clients claiming to be baidu, yisou and sogou spiders are still hitting static04, probably they don't re-resolve frequently | 15:41 |
| @clarkb:matrix.org | running on the jvm I guess :) | 15:43 |
| @fungicide:matrix.org | yandex and bing too | 16:10 |
| @clarkb:matrix.org | don't forget https://review.opendev.org/c/opendev/system-config/+/983930 :) | 16:10 |
| @fungicide:matrix.org | yeah, lgtm, i just wanted to give activity on static03 to some time to stabilize so we have a baseline for what the load looks like | 16:13 |
| @clarkb:matrix.org | ack | 16:17 |
| @fungicide:matrix.org | once that deploys, if load jumps up we can revisit some of the rules | 16:18 |
| @clarkb:matrix.org | right. I think most of those rules were put in place to address gitea struggles. So hopefulyl docs is largely unaffected. if that isn't the case we can reevaluate from there. But I wanted to make sure we cleaned thngs up and started over if necessary due to the false positives we hit | 16:23 |
| @fungicide:matrix.org | the apt-puppetlabs change didn't fix things, still the same expired key error, i'm suspecting that ansible doesn't replace the key if the master key id hasn't changed | 16:37 |
| @fungicide:matrix.org | though `/etc/reprepro-gpg-keys/puppetlabs.asc` was updated on the server | 16:38 |
| @fungicide:matrix.org | aha | 16:44 |
| @fungicide:matrix.org | it does indeed do that | 16:44 |
| @fungicide:matrix.org | https://opendev.org/opendev/system-config/src/commit/5e9af5c/playbooks/roles/import-gpg-key/tasks/main.yaml#L5-L12 | 16:45 |
| @fungicide:matrix.org | i wonder if there's a good way to trigger it when the exported key file content also changes? | 16:45 |
| @fungicide:matrix.org | as written it will never update a key that's already been imported | 16:46 |
| @fungicide:matrix.org | for now i reimported it on the server: https://paste.opendev.org/show/byhVkboHrkEjENxM8vTc/ | 16:48 |
| @fungicide:matrix.org | i guess it doesn't come up often enough to worry about | 16:49 |
| @fungicide:matrix.org | looks like they essentially removed the expiration date in the new version of that key | 16:49 |
| @fungicide:matrix.org | anyway, it'll trigger again in ~1.5 hours so i'll keep an eye on it | 16:51 |
| @clarkb:matrix.org | fungi: did the key id not change? | 16:53 |
| @clarkb:matrix.org | I guess in my head it seems like relying on the key id changing when the key changes is a valid approach | 16:53 |
| @fungicide:matrix.org | no, they merely refreshed (actually removed) the expiration for the selfsig | 16:53 |
| @clarkb:matrix.org | got it | 16:53 |
| @clarkb:matrix.org | so ya maybe we consider this an exceptional case and what you did is good enough (tm) | 16:54 |
| @fungicide:matrix.org | for a similar example, i've had the same personal gpg key for 13 years (ever since i replaced my old 2048-bit rsa), and i keep a short (~1 year) expiration on the selfsig but refresh it once a month or so to bump it out. the key id doesn't change, only signatures | 16:55 |
| @fungicide:matrix.org | i also add and remove uids on it from time to time, like when i get a new e-mail address i'm also going to sign messages for | 16:56 |
| @fungicide:matrix.org | er, i guess i've actually had it for 16 years, but didn't revoke the old key for a few years while transitioning | 16:57 |
| @clarkb:matrix.org | I guess if we wanted to solve this automatically we could also write out the pubkeys to a directly readable location (rather than the gpg db) and then if the file itself changes rerun the gpg import | 16:58 |
| @clarkb:matrix.org | beacuse you did update the pubkey file iirc | 16:59 |
| @fungicide:matrix.org | yes | 17:00 |
| @fungicide:matrix.org | i suppose we'd do that by moving the logic into a handler | 17:01 |
| @clarkb:matrix.org | it doesn't have to be a handler. You can still do a when on a regular task | 17:01 |
| @fungicide:matrix.org | and then trigger it if either the file changes or the key id doesn't exist | 17:01 |
| @clarkb:matrix.org | really handlers are mostly useful when you need things to happen after everything else has occurred as they run in a different phase than tasks | 17:01 |
| @clarkb:matrix.org | so like batch up a bunch of work that you only want to restart the web server once for multiple things can trigger the hanlder but then you only run the handler once at the end | 17:02 |
| @fungicide:matrix.org | i'm probably missing how to easily indicate from the reprepro role whether the gpg-import role should rerun those tasks | 17:03 |
| @clarkb:matrix.org | hrm ya I think that small little role may not have sufficient info since it only takes a key id and path as inputs? If we provided the full pub key content to it then let it manage the on disk file it would know if the file changed or not | 17:04 |
| @clarkb:matrix.org | fungi: you could stat the file and if the timestamp is within say 10 minutes then reload | 17:07 |
| @clarkb:matrix.org | https://opendev.org/opendev/system-config/src/commit/5e9af5c/playbooks/roles/import-gpg-key/tasks/main.yaml#L17 stat this particular path | 17:07 |
| @fungicide:matrix.org | seems a little hacky, but ought to work | 17:10 |
| -@gerrit:opendev.org- Zuul merged on behalf of Clark Boylan: [opendev/system-config] 983930: Cleanup Apache UA filters https://review.opendev.org/c/opendev/system-config/+/983930 | 17:43 | |
| @fungicide:matrix.org | watching to see what happens once that rolls out | 17:44 |
| @fungicide:matrix.org | infra-prod-service-static is running and the 5-minute load average on static03 has climbed to 0.8, though that may be more due to apache restarts than any decrease in requests being rejected | 17:47 |
| @fungicide:matrix.org | it's already falling again | 17:48 |
| @clarkb:matrix.org | ya we probably want to look for a baseline after config management has settled out | 17:48 |
| @fungicide:matrix.org | the job succeeded, and load average is back under 0.5 again | 17:50 |
| @fungicide:matrix.org | so no obvious change in load on the server for the moment | 17:50 |
| @clarkb:matrix.org | https://docs.opendev.org/opendev/system-config/latest/ is still reachable too so we didn't break connectivity unexpectedly either | 17:52 |
| @fungicide:matrix.org | yeah, everything seems peachy for now | 17:52 |
| @fungicide:matrix.org | apache server-status scorecard indicates it's mostly idle too | 17:53 |
| @fungicide:matrix.org | lots of unused worker slots | 17:53 |
| @fungicide:matrix.org | i saw static03 load average jump up a little over 1.0 but that didn't persist for long | 18:49 |
| @fungicide:matrix.org | now it's back down to ~0.2 | 18:50 |
| @mnasiadka:matrix.org | So we’re back to boring infrastructure? | 18:52 |
| @fungicide:matrix.org | we're back to hopping from the top of one iceberg to the next while successfully ignoring what's lurking just below the water's surface | 18:53 |
| @fungicide:matrix.org | (all the noise we're successfully filtering/absorbing without user-facing impact) | 18:54 |
| @fungicide:matrix.org | that could, and almost certainly will, shift again at some point with little or no warning | 18:54 |
| @fungicide:matrix.org | apt-puppetlabs reprepro did end up working after i manually imported the pgp key on the server, so that was it after all | 19:08 |
| @clarkb:matrix.org | https://mirror03.ord.rax.opendev.org/ is up and running from what I can see. Mount table looks correct. I'm going to approve https://review.opendev.org/c/opendev/zone-opendev.org/+/984590 to put it in the rotation then eat lunch | 20:08 |
| @clarkb:matrix.org | mnasiadka: ^ you're good with that right? | 20:08 |
| @mnasiadka:matrix.org | Clark: regarding docker 29 - have anybody tried podman compose and ditching docker cli? I know it has been bad in the past but maybe it’s an option now? I know it might require some tweaks to compose config files - but it might be that this incompatibility wilk grow | 20:08 |
| @mnasiadka:matrix.org | Clark: fine for me, time to finish juggling with mirrors | 20:09 |
| @clarkb:matrix.org | mnasiadka: ya we looked at that and it is more of a toy project. It seems like they want you to use podman's support for k8s resources instead | 20:09 |
| @clarkb:matrix.org | and I think docker-compose working with podman is more of an expected thing than docker working with podman | 20:10 |
| @clarkb:matrix.org | it was just a nice to have that docker did and I assumed that using distro docker would avoid this problem entirely. But I guess not | 20:10 |
| @clarkb:matrix.org | and at the time the primary goal was running containers that could fetch speculative container image builds from not docker hub (so that we could move to quay and keep our speculative testing) with minimal changes to the configuration management | 20:11 |
| @mnasiadka:matrix.org | Ah right, and standalone docker compose is legacy only, ugh | 20:11 |
| @clarkb:matrix.org | I think now that services are on Noble its maybe less of a problem to convert them to be podman specific. But that doesn't solve the compose problem | 20:11 |
| @jim:acmegating.com | i'm not sure we should expect the incompatability to grow? like, podman should get updated too | 20:12 |
| -@gerrit:opendev.org- Zuul merged on behalf of Michal Nasiadka: [opendev/zone-opendev.org] 984590: Promote mirror03.ord.rax https://review.opendev.org/c/opendev/zone-opendev.org/+/984590 | 20:12 | |
| @fungicide:matrix.org | likely it just lags behind docker/moby features | 20:13 |
| @clarkb:matrix.org | corvus: yes I suppose if ubuntu also updates podman then the issue would resolve itself assuming podman keeps up with the api updates (I think they do) | 20:13 |
| @jim:acmegating.com | (yeah, not to say it won't get worse, just saying that's not the only option, and we have to be cynical to expect it (but we are cynical)) | 20:14 |
| @mnasiadka:matrix.org | Well, to be frank podman 4.9.3 on Noble is not fresh | 20:15 |
| @mnasiadka:matrix.org | Maybe there’s some external repo that would allow installing podman>5 | 20:15 |
| @clarkb:matrix.org | no but it can fetch imges from mirrors of quay. Docker cannot do that | 20:15 |
| @clarkb:matrix.org | I have added meetpad02.opendev.org and jvb02.opendev.org to the emergency.yaml file | 20:37 |
| @fungicide:matrix.org | thanks! | 20:53 |
| -@gerrit:opendev.org- Jeremy Stanley https://matrix.to/#/@fungicide:matrix.org proposed: [zuul/zuul-jobs] 984652: bindep: correct bindep_profile documentation https://review.opendev.org/c/zuul/zuul-jobs/+/984652 | 21:11 | |
| @clarkb:matrix.org | fungi: do we want to proceed with the lists anubis domains chaneg now? | 21:12 |
| @clarkb:matrix.org | I know its later in your day, but that is probalby the most important change on my outstanding changes to get landed list | 21:13 |
| @fungicide:matrix.org | approved | 21:15 |
| @clarkb:matrix.org | cool I'll be around this afternoon to verify it is happy and fix it if anything goes wrong | 21:16 |
| -@gerrit:opendev.org- Jeremy Stanley https://matrix.to/#/@fungicide:matrix.org proposed: [opendev/bindep] 984653: Document support for multiple profiles https://review.opendev.org/c/opendev/bindep/+/984653 | 21:21 | |
| -@gerrit:opendev.org- Zuul merged on behalf of Jeremy Stanley https://matrix.to/#/@fungicide:matrix.org: [zuul/zuul-jobs] 984652: bindep: correct bindep_profile documentation https://review.opendev.org/c/zuul/zuul-jobs/+/984652 | 21:46 | |
| -@gerrit:opendev.org- Zuul merged on behalf of Jeremy Stanley https://matrix.to/#/@fungicide:matrix.org: [opendev/system-config] 983802: Set REDIRECT_DOMAINS for Anubis with Mailman https://review.opendev.org/c/opendev/system-config/+/983802 | 21:56 | |
| @clarkb:matrix.org | fungi: it occurs to me that we may not automatically restart the anubis service as part of deploying ^ I guess we will find out shortly | 21:57 |
| @fungicide:matrix.org | infra-prod-service-lists3 is already running in deploy now | 21:58 |
| @fungicide:matrix.org | the anubis daemon on lists01 has been running since april 1 | 21:58 |
| @fungicide:matrix.org | it did restart all the containers | 22:00 |
| @fungicide:matrix.org | including anubis | 22:00 |
| @clarkb:matrix.org | ok good I think that is what we wanted in order to pick upthe env var update | 22:00 |
| @fungicide:matrix.org | it's returning "server unavailable" at the moment, i think because some of those services take time to start up fully | 22:02 |
| @clarkb:matrix.org | yes I believe it takes a minute to restart | 22:02 |
| @fungicide:matrix.org | or several | 22:02 |
| @fungicide:matrix.org | postorius is loading again for me | 22:04 |
| @clarkb:matrix.org | lists.opendev.org loads for me. I was trying to pull up logs as I think part of the slowness ahs to do with locale/translation stuff maybe | 22:04 |
| @fungicide:matrix.org | so the new setting seems to not have broken anything | 22:04 |
| @clarkb:matrix.org | its possible we could speed this up by caching/memoizing some of that stuff | 22:04 |
| @clarkb:matrix.org | fungi: and if you update /etc/hosts to point at lists.opendev.org with a name like listsfoo.opendev.org you should get an error from anubis | 22:04 |
| @clarkb:matrix.org | (if that idea made sense) | 22:05 |
| @fungicide:matrix.org | probably, though i haven't tried | 22:05 |
| @fungicide:matrix.org | i suppose we could add a testinfra test for that fairly easily too | 22:06 |
| @clarkb:matrix.org | its also possible that apache rejects it first | 22:06 |
| @clarkb:matrix.org | firefox first gives me the ssl cert is bad warning (expected) then anubis calculates. Then it redirects me to a page that says "Redirect domain not allowed." so I think this is working as expected | 22:07 |
| @fungicide:matrix.org | sounds correct, yep | 22:08 |
| @clarkb:matrix.org | I also looked at the output comma separated list and it lgtm as well | 22:09 |
| @clarkb:matrix.org | I think we can consider this done? | 22:09 |
| @fungicide:matrix.org | yep, seems fine | 22:10 |
| @fungicide:matrix.org | deploy jobs reported success too | 22:10 |
| @clarkb:matrix.org | fungi: https://review.opendev.org/c/opendev/system-config/+/983929 is a semi related change if we want to pin anubis. I'm not quite sure how important that is yet | 22:11 |
| @clarkb:matrix.org | oh cool the git-review changes have landed | 22:12 |
| @clarkb:matrix.org | fungi: cool you approved that one too. FWIW https://github.com/techarohq/anubis/pkgs/container/anubis/693285228?tag=v1.25.0 shows that latest and v1.25.0 are the same if you look on the right hand side it says "Other tags on this version" | 22:42 |
| @clarkb:matrix.org | both latest and v1.25.0 are listed there | 22:42 |
| -@gerrit:opendev.org- Steve Baker proposed: [openstack/diskimage-builder] 984486: Skip local loop device creation for no-final-image builds https://review.opendev.org/c/openstack/diskimage-builder/+/984486 | 23:31 | |
| -@gerrit:opendev.org- Zuul merged on behalf of Clark Boylan: [opendev/system-config] 983929: Pin anubis container image to v1.25.0 https://review.opendev.org/c/opendev/system-config/+/983929 | 23:35 | |
| @clarkb:matrix.org | Looks like that did restart containers on lists again as well as gitea. I can reach lists.opendev.org so it seems to be working there. I'll test opendev.org once the giteas are done updating that way I don't have to set up a socks proxy | 23:45 |
| @clarkb:matrix.org | giteas are done and I can still access opendev.org so I think that went well as expected | 23:49 |
Generated by irclog2html.py 4.1.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!