| opendevreview | Tony Breeds proposed opendev/system-config master: DNM: Initial dump or mediawiki role and config https://review.opendev.org/c/opendev/system-config/+/921322 | 00:39 |
|---|---|---|
| opendevreview | Tony Breeds proposed opendev/system-config master: Add golang based docker compose tool. https://review.opendev.org/c/opendev/system-config/+/920760 | 01:28 |
| opendevreview | Tony Breeds proposed opendev/system-config master: Add an opendev specific build of mediawiki https://review.opendev.org/c/opendev/system-config/+/921321 | 01:28 |
| opendevreview | Tony Breeds proposed opendev/system-config master: DNM: Initial dump or mediawiki role and config https://review.opendev.org/c/opendev/system-config/+/921322 | 01:28 |
| tonyb | frickler: If you're still around can you take another look at: https://review.opendev.org/c/opendev/system-config/+/920760 | 01:29 |
| opendevreview | Tony Breeds proposed opendev/system-config master: DNM: Initial dump or mediawiki role and config https://review.opendev.org/c/opendev/system-config/+/921322 | 03:37 |
| *** mordred1 is now known as Guest10115 | 04:52 | |
| opendevreview | Aurelio Jargas proposed zuul/zuul-jobs master: Add ensure-poetry role https://review.opendev.org/c/zuul/zuul-jobs/+/922286 | 07:15 |
| opendevreview | Aurelio Jargas proposed zuul/zuul-jobs master: Add ensure-poetry role https://review.opendev.org/c/zuul/zuul-jobs/+/922286 | 08:38 |
| opendevreview | Jens Harbott proposed openstack/project-config master: nodepool: revert dedicated diskimages list for rax-iad https://review.opendev.org/c/openstack/project-config/+/922295 | 09:30 |
| opendevreview | Jens Harbott proposed openstack/project-config master: nodepool: reorder providers to list rax-dfw last https://review.opendev.org/c/openstack/project-config/+/922296 | 09:30 |
| opendevreview | Merged openstack/project-config master: nodepool: revert dedicated diskimages list for rax-iad https://review.opendev.org/c/openstack/project-config/+/922295 | 10:05 |
| opendevreview | Merged openstack/project-config master: nodepool: reorder providers to list rax-dfw last https://review.opendev.org/c/openstack/project-config/+/922296 | 10:06 |
| frickler | hmm, no immediate improvement with ^^, just more errors from rax. waiting now to see what happens when the next image rebuild finishes | 10:38 |
| opendevreview | Merged opendev/system-config master: Report Zuul events for WanderTracks repos https://review.opendev.org/c/opendev/system-config/+/921776 | 11:05 |
| opendevreview | Jens Harbott proposed openstack/project-config master: Drop wheel publishing for centos-8-stream https://review.opendev.org/c/openstack/project-config/+/922313 | 12:24 |
| opendevreview | Merged opendev/system-config master: Add golang based docker compose tool. https://review.opendev.org/c/opendev/system-config/+/920760 | 13:08 |
| frickler | so ubuntu-jammy is now available on openmetal-iad3, time to boot some instances? or just add into the normal pool at once and see what happens? | 14:04 |
| SvenKieske | what do I need to do, to get this merged? https://review.opendev.org/c/openstack/project-config/+/920472 (I'm not even 100% sure this is the correct channel to ask? project-config has no real documentation wrt to code owners etc that I could deduce from the readme) | 14:22 |
| fungi | frickler: i guess the main concern is that it boots but networking is set up incorrectly such that it can't reach the mirror... though even then builds would fail early enough that they'll get retried i think? | 14:24 |
| fungi | SvenKieske: the config-core reviewers are the code owners for openstack/project-config | 14:24 |
| fungi | i'll take a look | 14:24 |
| SvenKieske | ty, so I guess this is the correct channel to ask for reviews? | 14:26 |
| fungi | SvenKieske: sure, or #openstack-infra since it's an openstack-specific request (that's the channel for the tact sig, who are openstack's liaisons to the opendev collaboratory) | 14:33 |
| fungi | ultimately it's mostly the same people wearing different hats | 14:34 |
| * fungi has a vast collection of funny-looking hats | 14:34 | |
| SvenKieske | ah I always forget that channel, put it into autojoin, ty! | 14:35 |
| fungi | ugh, did a new filelock release just break us again? | 14:36 |
| JayF | https://github.com/tox-dev/filelock/issues/343 it was just yanked | 14:38 |
| fungi | Successfully installed cachetools-5.3.3 chardet-5.2.0 colorama-0.4.6 distlib-0.3.8 filelock-3.15.2 packaging-24.1 platformdirs-4.2.2 pluggy-1.5.0 pyproject-api-1.6.1 tomli-2.0.1 tox-4.15.1 virtualenv-20.26.2 | 14:39 |
| fungi | yeah, looks like that was it. thanks JayF! | 14:39 |
| fungi | guess you were already looking into it too ;) | 14:39 |
| fungi | recheck time | 14:40 |
| opendevreview | Merged openstack/project-config master: restrict enforced backport voting to master branch https://review.opendev.org/c/openstack/project-config/+/920472 | 14:56 |
| fungi | SvenKieske: that's ^ successfully deployed to gerrit two minutes ago if you want to test at your convenience | 15:02 |
| SvenKieske | fungi: ty! | 15:11 |
| *** dtantsur_ is now known as dtantsur | 15:12 | |
| SvenKieske | seems to work :) | 15:13 |
| fungi | perfect | 15:19 |
| JayF | fungi: saw your message, knew the github repo from the other day and did a quick look | 15:19 |
| JayF | happy to get bonus credit but I don't deserve it :D | 15:20 |
| frickler | fungi: does it make sense to start with max_servers=1 to limit the possible impact? the etherpad simply says to set it to 50 next, which would be the full size of the cloud | 16:01 |
| fungi | yeah, that seems like a good way of testing the waters without the added effort of manually booting and inspecting a node. almost all the ways i can think of it failing would result in build retries not failures | 16:08 |
| opendevreview | Jens Harbott proposed openstack/project-config master: Enable openmetal cloud for nodepool https://review.opendev.org/c/openstack/project-config/+/922338 | 16:27 |
| opendevreview | Merged openstack/project-config master: Enable openmetal cloud for nodepool https://review.opendev.org/c/openstack/project-config/+/922338 | 16:45 |
| frickler | this doesn't look good: 2024-06-19 16:54:06,884 ERROR nodepool.StateMachineNodeLauncher.openmetal-iad3: [e: f237626ad7db4a0cb8323bfbc206d7af] [node_request: 900-0024530346] [node: 0037763929] Error in creating the server. Compute service reports fault: No valid host was found. | 16:57 |
| frickler | although another node is in use, hmm | 16:59 |
| fungi | wonder if we have something set wrong in placement | 17:03 |
| fungi | related to the availability zones we created? is nodepool using the correct az? | 17:04 |
| fungi | if no az is specified then nova will just try to boot in one at random, yeah? | 17:04 |
| fungi | if i remember the discussion correctly we created two host aggregates mapped to different availability zones, and one presumably only allows our control plane project to create server instances in it | 17:07 |
| frickler | not sure, there is one node running and I can log into it, not sure how to best identify which build it is associated with | 17:07 |
| fungi | i think it must have already been deleted | 17:10 |
| fungi | because i'm logged into a node with a sub-minute uptime | 17:10 |
| *** dasm is now known as Guest10177 | 17:13 | |
| frickler | o.k. found one build and it looks fine https://zuul.opendev.org/t/openstack/build/fd247729c365494f9db27b05fa40ffd3 | 17:13 |
| frickler | also no failing or retrying builds found that would match | 17:14 |
| fungi | 0037764030 seems to be running a devstack job | 17:14 |
| fungi | so yeah, current guess is that we're round-robining server create requests between the two availability zones and half of them will die with "no valid host" | 17:15 |
| frickler | I'd still like to investigate nova logs a bit for the error above, but that'll have to wait until tomorrow unless someone else is faster | 17:16 |
| fungi | the half that don't fail appear to work | 17:16 |
| frickler | hmm, that might be possible, yes | 17:16 |
| fungi | when tonyb wakes up, he may be more familiar with the details, but if that really is the problem then just specifying the right az in nodepool's clouds.yaml ought to solve it | 17:16 |
| fungi | `openstack availability zone list` returns 4 entries named "nova" and one named "reserved" | 17:20 |
| fungi | https://docs.openstack.org/openstacksdk/latest/user/config/configuration.html doesn't seem to mention how to specify a default az | 17:22 |
| fungi | maybe it has to go in nodepool.yaml | 17:23 |
| frickler | if you add "--long" you see more details, two of the "nova" zones are actually neutron zones and one cinder I think | 17:23 |
| fungi | aha, yep https://zuul-ci.org/docs/nodepool/latest/openstack.html | 17:23 |
| frickler | "If this setting is omitted, nodepool will fetch the list of all availability zones from nova." | 17:24 |
| frickler | so we want to override to just "nova" | 17:25 |
| fungi | yeah, about to push that up now | 17:25 |
| frickler | because "reserved" is meant to not be used by nodepool | 17:25 |
| opendevreview | Jeremy Stanley proposed openstack/project-config master: Use the "nova" AZ for Nodepool in openmetal-iad3 https://review.opendev.org/c/openstack/project-config/+/922343 | 17:28 |
| corvus | that should work -- though there must be some way of restricting the azs per tenant at the cloud level, right? | 17:30 |
| corvus | (it's fine if we decide it's not worth our time; but... it's super weird that we're being told about an az that we can't use :) | 17:30 |
| fungi | agreed, there probably is. the difference is i know how to configure nodepool but i'm less confident in my ability to configure... nova? keystone? what filters the az list? | 17:41 |
| fungi | when more people are around who have familiarity with how those availability zones were configured, we can clean that up if there's a way to solve it on the cloud end | 17:42 |
| corvus | probably involves policy.json :) | 17:53 |
| opendevreview | Merged openstack/project-config master: Use the "nova" AZ for Nodepool in openmetal-iad3 https://review.opendev.org/c/openstack/project-config/+/922343 | 18:18 |
| fungi | deploy of that ^ completed at 18:24:55 utc, so if we continue to see the same errors on nl02 then we'll know the guess was incorrect | 18:30 |
| fungi | there's a debian-bookworm node in use there as of 3 minutes ago, so probably booted after the config update | 18:31 |
| fungi | oh, though do launcher containers need a restart to reread their nodepool.yaml? | 18:31 |
| fungi | i never can remember. guess i'll see if the playbook does a sighup or something | 18:32 |
| corvus | nope, they reread it themselves | 18:35 |
| corvus | so within a few seconds of it landing on disk it should take effect for new launches | 18:35 |
| fungi | okay, cool | 18:35 |
| fungi | so the currently in-use node 0037764223 got a launcher thread at 18:27:19 utc, ~2.5 minutes after the deploy job reported back successful | 18:37 |
| fungi | last update time on nodepool.yaml was actually at 18:22:49 utc, so nearly 5 minutes | 18:38 |
| fungi | most recent "No valid host was found" so far was 18:03:26 utc | 18:40 |
| tonyb | the availability zones were created for us but the openmetal folks. so I don't have any real insight | 20:50 |
| tonyb | I can look into the details more | 20:52 |
| tonyb | corvus: could you look at the open mediawiki patches. I'm getting an invalid reference trying to pull the new container. | 20:54 |
| tonyb | looking at the inventory the mediawiki job knows it needs the artifact from the previous job. | 20:55 |
| tonyb | the container exists in the insecure-registry | 20:55 |
| tonyb | I thought, and this may be the root issue, that trying to pull the image with a full reference would "magically" use the buildset registry as a pull through mirror for docker.io | 20:57 |
| tonyb | saying this now I wonder if switching to compose V2 changes they way that needs to be configured | 20:58 |
| *** iurygregory_ is now known as iurygregory | 21:28 | |
| opendevreview | Curt Moore proposed openstack/diskimage-builder master: Adapt to upstream CentOS Stream mirror changes https://review.opendev.org/c/openstack/diskimage-builder/+/922352 | 21:50 |
| corvus | tonyb: do you happen to have a noble node available? | 22:17 |
| corvus | tonyb: but i agree with your supposition; if we're going to use go-docker-compose then we might need to jump to the end state of also using podman... | 22:19 |
| opendevreview | Steve Baker proposed openstack/diskimage-builder master: Change DIB_BOOTLOADER_DEFAULT_CMDLINE empty string behaviour https://review.opendev.org/c/openstack/diskimage-builder/+/884644 | 22:25 |
| tonyb | corvus: I don't have one handy but I can do. | 22:27 |
| corvus | i'm still digging through the logs | 22:28 |
| tonyb | I'll switch to python compose and see if it works to confirm that suspicion | 22:28 |
| corvus | ++ | 22:28 |
| corvus | tonyb: yeah, i don't see anything obviously wrong; so my guess is same as yours: the registry mirror config may not work for go-docker-compose. your test should confirm that. if it does, we can either work to fix that (involves some trial and error on a noble host) or skip that step and go all the way to using podman+go-docker-compose, or just use python-docker-compose for this and treat go-docker-compose as a next step. | 22:36 |
| Guest10115 | so exciting that docker-compose was re-written | 22:37 |
| corvus | i've done some preliminary work on go-docker-compose with podman that proves it out; i can provide that if we want to go that direction. | 22:37 |
| corvus | mordred: yes it's very docker. | 22:38 |
| Guest10115 | corvus: your face is very docker? | 22:38 |
| corvus | (having said that, i am glad that it works with podman which gives us a route to finally speculatively test quay.io images mixed with docker.io images) | 22:39 |
| Guest10115 | can I just say that it's weird that go-docker-compose works with podman while python-docker-compose doesn't? | 22:39 |
| corvus | it must have been an accident :) | 22:40 |
| Guest10115 | definitely | 22:40 |
| tonyb | corvus: thank you for looking. I'll verify it is a golang compose issue and then make a plan. | 22:48 |
| opendevreview | Curt Moore proposed openstack/diskimage-builder master: Add support for Fedora 40 https://review.opendev.org/c/openstack/diskimage-builder/+/922353 | 22:48 |
| tonyb | I admit that, while I'm wary of "do it later", I'm not sure I have the cycles for integrating podman right now. maybe next quarter. | 22:51 |
| tonyb | Guest10115, mordred: are you using a matrix-IRC bridge? the name I see on "regular" IRC changes from time to time | 23:09 |
| Guest10115 | Yeah ... I'm only using a matrix client currently. I should probably check and make sure it's tied in properly with nickserv stuff | 23:12 |
| tonyb | Guest10115: that would be helpful:) | 23:18 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!