| @clarkb:matrix.org | corvus: I rechecked https://review.opendev.org/c/opendev/system-config/+/1010020/3 as ^ should've caused things to be mirrored in the periodic builds overnight | 14:46 |
|---|---|---|
| @jim:acmegating.com | Clark: thanks! | 14:46 |
| -@gerrit:opendev.org- Clark Boylan proposed: [opendev/system-config] 1010199: Upgrade etherpad to 3.3.8 https://review.opendev.org/c/opendev/system-config/+/1010199 | 14:55 | |
| -@gerrit:opendev.org- Clark Boylan proposed: [opendev/system-config] 840972: DNM force etherpad failure to hold node https://review.opendev.org/c/opendev/system-config/+/840972 | 15:03 | |
| @mordred:waterwanders.com | Clark: a while ago we landed https://review.opendev.org/c/opendev/system-config/+/1007077 to add pgsty/silo to our mirror, but it's not in the mirror. | 16:50 |
| @clarkb:matrix.org | mordred: oh I know why | 16:50 |
| @clarkb:matrix.org | we didn't add the new job to the pipeline jobs list | 16:51 |
| @mordred:waterwanders.com | in fact, looking at it - I dont see any opendev-mirror jobs in the build history | 16:51 |
| @mordred:waterwanders.com | oh - hahahaha | 16:51 |
| -@gerrit:opendev.org- Monty Taylor https://matrix.to/#/@mordred:inaugust.com proposed: [opendev/system-config] 1010234: Actually run the silo mirror job https://review.opendev.org/c/opendev/system-config/+/1010234 | 16:54 | |
| @mordred:waterwanders.com | Clark, corvus ^^ whoopsie | 16:54 |
| -@gerrit:opendev.org- Zuul merged on behalf of Monty Taylor https://matrix.to/#/@mordred:inaugust.com: [opendev/system-config] 1010234: Actually run the silo mirror job https://review.opendev.org/c/opendev/system-config/+/1010234 | 17:08 | |
| -@gerrit:opendev.org- Armin M proposed: [openstack/diskimage-builder] 1006200: Oracle Linux support for diskimage-builder. https://review.opendev.org/c/openstack/diskimage-builder/+/1006200 | 18:10 | |
| @clarkb:matrix.org | Reminder our normal meeting this week is cancelled due to the ptg. I will hold the office hour later today though if there is anything to catch up on | 18:52 |
| @jim:acmegating.com | the zuul mariadb upgrade change passes now: https://review.opendev.org/1010020 | 18:55 |
| @jim:acmegating.com | should i go ahead and make that happen? or do we want to talk about that any more first? | 18:55 |
| @jim:acmegating.com | i kind of figured we can let zuul be the canary here and if there are no surprises, we could upgrade the other mariadb instances later (gitea, etc) | 18:56 |
| @jim:acmegating.com | (locally, i have not had to do anything other than restart in order to effect the upgrade) | 18:56 |
| @clarkb:matrix.org | Yup we have to upgrade eventually. I'm happy to let zuul lead the way | 19:00 |
| -@gerrit:opendev.org- Eric Ball proposed: | 19:05 | |
| - [opendev/system-config] 1000664: Add nftables firewall backend alongside iptables https://review.opendev.org/c/opendev/system-config/+/1000664 | ||
| - [opendev/system-config] 1003360: Expand testing of nftables https://review.opendev.org/c/opendev/system-config/+/1003360 | ||
| - [opendev/system-config] 1010267: Test firewall rules w/ both iptables and nftables https://review.opendev.org/c/opendev/system-config/+/1010267 | ||
| -@gerrit:opendev.org- Zuul merged on behalf of Clark Boylan: [opendev/system-config] 1010199: Upgrade etherpad to 3.3.8 https://review.opendev.org/c/opendev/system-config/+/1010199 | 19:48 | |
| @clarkb:matrix.org | fungi: I think taht should deploy shortly if you wanted to send your notice | 19:50 |
| @fungicide:matrix.org | #status notice The Etherpad service at etherpad.opendev.org (also used by meetpad.opendev.org) is being restarted now to apply a security update; any open pads will display an error message until refreshed, and author colors may get reset. | 19:50 |
| @status:opendev.org | @fungicide:matrix.org: sending notice | 19:50 |
| @fungicide:matrix.org | yeah, i was just waiting for the deploy job to start | 19:51 |
| @clarkb:matrix.org | it is up now for about half a minute and reports being healthy | 19:53 |
| @clarkb:matrix.org | https://etherpad.opendev.org/p/gerrit-upgrade-3.14 loads for me and appears to ahve the expected content | 19:53 |
| -@status:opendev.org- NOTICE: The Etherpad service at etherpad.opendev.org (also used by meetpad.opendev.org) is being restarted now to apply a security update; any open pads will display an error message until refreshed, and author colors may get reset. | 19:53 | |
| @status:opendev.org | @fungicide:matrix.org: finished sending notice | 19:53 |
| @clarkb:matrix.org | so that looks good to me | 19:53 |
| @fungicide:matrix.org | and the deploy has already finished | 19:54 |
| @fungicide:matrix.org | i should have left a pad open, but yeah working fine and my author color didn't even get reset | 19:55 |
| @clarkb:matrix.org | corvus: probably good to proceed with zuul DB stuff whenever you're ready. Will we pause queues for that to avoid missing any DB updates? | 20:01 |
| @jim:acmegating.com | Clark: it's in the gate now | 20:09 |
| @jim:acmegating.com | do we expect it to automatically restart? | 20:09 |
| @jim:acmegating.com | i think maybe "yes" -- i'm pretty sure we're going to run a docker-compose up -d | 20:10 |
| @jim:acmegating.com | zuul has database write retries now, so as long as the restart takes <5m it's probably fine to do nothing | 20:11 |
| @jim:acmegating.com | it's safer to pause of course | 20:12 |
| @jim:acmegating.com | but i'm kind of inclined to just see what happens without intervention | 20:12 |
| @clarkb:matrix.org | yes looks like we run docker-compose up -d. We don't do an explicit pull, but I think up -d will do a pull since the image changes | 20:13 |
| @clarkb:matrix.org | ok I'm happy to let it go and see how it handles the upgrade | 20:13 |
| @jim:acmegating.com | it looks like the docker log file may be corrupt from when the host ran out of space... we might want to do a full down/up just to resolve that. | 20:15 |
| @jim:acmegating.com | i wonder if we should do that now, so that we have good/easy access to logs during the upgrade | 20:16 |
| @jim:acmegating.com | i think i would like to do that. any objection? | 20:16 |
| @clarkb:matrix.org | That's fine with me | 20:19 |
| @clarkb:matrix.org | Just need to do it before zuul merges the change I guess | 20:19 |
| @jim:acmegating.com | #status log restarted zuul mariadb server to reset container logging (to resolve previous corruption after disk full) | 20:23 |
| @status:opendev.org | @jim:acmegating.com: finished logging | 20:23 |
| @jim:acmegating.com | that's done, was nearly instantaneous | 20:23 |
| @jim:acmegating.com | logs work now | 20:23 |
| -@gerrit:opendev.org- Zuul merged on behalf of James E. Blair https://matrix.to/#/@jim:acmegating.com: [opendev/system-config] 1010020: Upgrade zuul mariadb https://review.opendev.org/c/opendev/system-config/+/1010020 | 20:26 | |
| @clarkb:matrix.org | corvus: hrm the deploy completed for that but we didn't update. Maybe docker compose up -d does not do an implicit pull | 20:32 |
| @jim:acmegating.com | Clark: apparently we guessed wrong: | 20:32 |
| "false_condition": "mariadb_run_compose_up | bool", | ||
| "skip_reason": "Conditional result was False" | ||
| @jim:acmegating.com | i thought that was set to true for that host... but i guess not in this particular code path? | 20:32 |
| @clarkb:matrix.org | oh! its guarded by a config flag | 20:32 |
| @jim:acmegating.com | anyway, i'll just do a manual pull and up now | 20:33 |
| @clarkb:matrix.org | the default is false and we set it to true in testing but not in the prod vars | 20:33 |
| @jim:acmegating.com | okay, there was a 10 second outage on that; 5 seconds because apparently "up -d" didn't work due to some docker-compose thing (`KeyError: 'ContainerConfig'`), so it took me 5 seconds to do a down/up pair. then 5 more seconds for mariadb to upgrade the tables. | 20:36 |
| @jim:acmegating.com | one of the schedulers did hit a write error, but it did retry and was successful, so that worked as expected. | 20:36 |
| @jim:acmegating.com | and the query planner is using the new index | 20:37 |
| @clarkb:matrix.org | corvus: that host is not noble so its not doing the podman weirdness which was my first hunch as to what broke it | 20:37 |
| @jim:acmegating.com | full error from the failed compose up: https://paste.opendev.org/show/bKGOgQ7nJUzLvUPW5s4b/ | 20:38 |
| @clarkb:matrix.org | corvus: ok I think maybe the "problem" (not sure if actually a problem yet) is that `docker compose` is present on the server but so is docker-compose | 20:38 |
| @clarkb:matrix.org | is it possible you converted it from docker-compose to docker compose? | 20:38 |
| @clarkb:matrix.org | or the other way around? this is weird | 20:39 |
| @jim:acmegating.com | my previous down/up to fix the logs, was also docker-compose: `docker-compose down; docker-compose up -d` is from history | 20:39 |
| @clarkb:matrix.org | ok then its unlikely you did anything like that | 20:39 |
| @jim:acmegating.com | /usr/local/bin/docker-compose is not our script | 20:40 |
| @jim:acmegating.com | it looks like it's the actual docker-compose entry point | 20:40 |
| @clarkb:matrix.org | yes because this system isn't using podman | 20:40 |
| @clarkb:matrix.org | yup | 20:40 |
| @jim:acmegating.com | okay, so then that's all as expected | 20:40 |
| @clarkb:matrix.org | I think the surprising thing here is that `docker compose` also seems to be valid so I thought maybe it got mixed up and migrated from one to another via the manual process creating that error | 20:40 |
| @clarkb:matrix.org | but I don't think that happened based on your notes above. So the issue is probably unrelated | 20:41 |
| @clarkb:matrix.org | docker-compose-plugin <- this package brings in the `docker compose` support | 20:41 |
| @jim:acmegating.com | i guess regardless the solution is just "upgrade" | 20:41 |
| @clarkb:matrix.org | ya I think that will sort things out and for now we're good | 20:42 |
| @clarkb:matrix.org | then separately we're interested in seeing if this helps performance right? | 20:42 |
| @jim:acmegating.com | yep -- mind you, we only noticed performance problems under some conditions. but there's a metric we can check on after a day or so to see if it's improved (regardless of load) | 20:43 |
| @jim:acmegating.com | the other thing is whether there are any changes (regressions or improvements) to build/buildset queries. (though, note that queries for jobs that never ran are still expected to time out). | 20:45 |
| @mordred:waterwanders.com | corvus: that last thing - you mean like filtering the builds dashboard by job name? | 20:49 |
| @jim:acmegating.com | mordred: yes | 20:49 |
| @mordred:waterwanders.com | nod. | 20:50 |
| -@gerrit:opendev.org- Zuul merged on behalf of Clark Boylan: [opendev/infra-specs] 1008300: Mark Matrix for OpenDev as Implemented https://review.opendev.org/c/opendev/infra-specs/+/1008300 | 22:23 | |
Generated by irclog2html.py 4.1.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!