| opendevreview | OpenStack Proposal Bot proposed openstack/project-config master: Normalize projects.yaml https://review.opendev.org/c/openstack/project-config/+/957995 | 02:32 |
|---|---|---|
| *** liuxie is now known as liushy | 03:29 | |
| fungi | Clark[m]: yeah, i had arrived at a similar conclusion. i wonder if the problem is the newish cloud-init release not detecting it's booting in openstack due to xen not passing in the dmi data, so defaulting to not trying the network metadata source? | 13:13 |
| frickler | on dfw02 cloud-init is disabled and there is a config drive, though I haven't found yet which service might be consuming it | 13:36 |
| fungi | it might have its networking statically configured via ifupdown | 13:47 |
| frickler | oh, right, it is that old ... https://paste.opendev.org/show/bY9TXQBrI5CDIi2XzGSx/ ... so maybe this just stopped working on noble? | 13:54 |
| opendevreview | Nicolas Hicher proposed zuul/zuul-jobs master: Refactor: multi-node-bridge to use linux bridge https://review.opendev.org/c/zuul/zuul-jobs/+/959393 | 14:13 |
| opendevreview | Nicolas Hicher proposed zuul/zuul-jobs master: Refactor: multi-node-bridge to use linux bridge https://review.opendev.org/c/zuul/zuul-jobs/+/959393 | 14:17 |
| frickler | Clark[m]: fungi: with the linelength set to infinite, the normalization now enforces every description to be on a single line, no matter how long it may become. I'm not convinced that this is really better than before, wdyt? https://review.opendev.org/c/openstack/project-config/+/957995 | 14:17 |
| fungi | frickler: i concur, i can work on a replacement using pyyaml like i pasted in here last week so we can drop ruamel.yaml, but it will probably be a few days before i get to it | 14:19 |
| frickler | yes, that sounds fine, I don't think it is urgent either | 14:22 |
| clarkb | fungi: frickler we disable cloud init on our nodes so I don't think cloud-init is the problem unless the upgrade reenables it | 14:44 |
| fungi | clarkb: yeah, more likely ubuntu's upgrade process simply doesn't take ifupdown configuration into account | 14:45 |
| clarkb | ya I suspect that it needs netplan today and isn't translating | 14:45 |
| clarkb | the other thing I realized is that we also need to be wary of the single vcpu problem with noble | 14:45 |
| clarkb | that seems to correlate to specific older versions of xen so we might want to check all of these in place upgrades for their xen version and possibly ask for migrations to new hypervisors if they are on the version we suspect is a problem | 14:46 |
| clarkb | but lets worry about that once this particular node is happy again | 14:46 |
| clarkb | frickler: huh I was expecting it to line wrap lines that were longer than the limit but leave shorter ones alone. That is quite the decision in formatting there. I guess because its losing the original format and enforcing a new one | 14:47 |
| clarkb | fungi: I guess keep us in the loop on what you find out. And I'm around if I can help | 14:50 |
| stephenfin | clarkb: fungi: apologies if this is well known and I missed something, but python < 3.8 jobs are failing on at least pbr since 19-Aug https://zuul.opendev.org/t/openstack/builds?job_name=openstack-tox-py27&project=openstack/pbr | 15:09 |
| clarkb | stephenfin: yes, it is known. OpenDev updated the default ansible version used by zuul to Ansible 11 and that doesn't work with python <3.8. WHich means all the platforms (bionic and older) running those jobs don't work by default | 15:10 |
| clarkb | stephenfin: you can pin the jobs back to ansible 9 or remove them. For PBR starting with the pin might make the most sense for now | 15:10 |
| stephenfin | Ah, I had actually seen something (on openstack-discuss?) about that | 15:11 |
| clarkb | (I've been trying to delete the jobs elsewhere as there is minimal value to them unless you actually need to support old things) | 15:11 |
| fungi | we'll probably have to drop those jobs pretty soon in pbr too | 15:11 |
| fungi | though 2.7 should still be testable on newer platforms | 15:11 |
| clarkb | ya I did run into a problem with 2.7 not being supported by tox on the newer platforms | 15:12 |
| stephenfin | likely because it's tox 4. That doesn't support 2.7 (understandably) | 15:12 |
| clarkb | I think the issue there is there must be python-requires settings for tox on 3.6 that pin it back to a version autoamtically that wokrs with 2.7. I tried updating the 2.7 jobs to jammy and it failed due to not being able to virtualenv properly iirc. So we may need to pin tox on those platforms | 15:12 |
| clarkb | stephenfin: ya that is probably it | 15:12 |
| clarkb | so yes 2.7 is still possible on jammy just have to sort out how to pin the tox version | 15:13 |
| clarkb | `ensure_tox_version: '<4'` in the job vars may do it | 15:13 |
| clarkb | so ya I think delete 3.6 and 3.7 jobs and update 2.7 to jammy with `ensure_tox_version: '<4'` is a potential long term option. Maybe start by pinning the ansible version then followup with something like ^ | 15:14 |
| fungi | or could test it on focal instead | 15:14 |
| stephenfin | I assume I've going to need to define new pbr-tox-* jobs if I want to override anything? | 15:14 |
| fungi | nodesets can be set in variants | 15:15 |
| clarkb | stephenfin: no you can do it in the pipeline config | 15:15 |
| clarkb | jobs: check: - tox-py27: nodeset: ubuntu-jammy vars: ensure_tox_version: '<4' | 15:15 |
| clarkb | that needs newlines in there sorry that ended up not being as readable as I had hoped | 15:15 |
| clarkb | fungi: I think focal may have the same issue fwiw since it has to do with the python versions supported by tox 4 | 15:16 |
| stephenfin | like https://review.opendev.org/c/openstack/pbr/+/960133? | 15:18 |
| stephenfin | (I'm just pinning the ansible version for now) | 15:18 |
| opendevreview | Nicolas Hicher proposed zuul/zuul-jobs master: Refactor: multi-node-bridge to use linux bridge https://review.opendev.org/c/zuul/zuul-jobs/+/959393 | 15:19 |
| clarkb | stephenfin: yes, you'll want to do that for the gate job definitions below too | 15:19 |
| clarkb | zuul considers that a local anonymous job variant so you don't have to go out of your way to define something new and named | 15:19 |
| clarkb | stephenfin: you might need to quote the 9 as '9' due to ansible typing reasons | 15:19 |
| stephenfin | ack, done | 15:20 |
| opendevreview | Nicolas Hicher proposed zuul/zuul-jobs master: Refactor: multi-node-bridge to use linux bridge https://review.opendev.org/c/zuul/zuul-jobs/+/959393 | 15:23 |
| clarkb | I've asked for the "docker image builds are broken" reported from last week to c onfirm the fix I wrote addressed their problems. Once that is done I suspect we can start proceeding with the python base image move back to quay | 16:05 |
| clarkb | fungi: looking at `grep ansible_bios_version /var/cache/ansible/facts/*` on bridge I think all of the afs nodes and all kerberos nodes except for afs02.dfw.openstack.org and kdc04.openstack.org are on the old xen version | 16:11 |
| clarkb | fungi: I suspect this means they will boot seeing only a single vcpu. Probably not the end of the world but we might see if that is somethign we can live/cold migrate off of? | 16:12 |
| clarkb | I wish I had throught to check that before the noble in place stuff. oh well we'll figure it out from here | 16:12 |
| fungi | yeah, maybe | 16:12 |
| corvus | i see we're talking about replacing ruamel.yaml for the projects.yaml normalization? note that we chose ruamel because it (mostly) preserves comments and i don't think anything else did at the time | 16:15 |
| fungi | corvus: yeah, i brought that up last week, we don't seem to use any inline comments in that file | 16:16 |
| corvus | are we okay saying that we never will? | 16:17 |
| corvus | i don't know that i love this idea | 16:17 |
| clarkb | the trailing space thing or lines that don't wrap are not the end of the world if we stick with ruamel | 16:18 |
| fungi | i still don't know of anything else that preserves comments in yaml files, i guess it's a trade-off between giving up the possibility of using comments in the future vs accepting the recently terrible string wrapping choices the library has made | 16:18 |
| clarkb | annoying but workable | 16:18 |
| corvus | for me: non-wrapping lines > trailing spaces > no comments | 16:18 |
| corvus | it's been a while since i dug into ruamel; with pyyaml there is the possibility of customizing output. i wonder if ruamel does something like that. | 16:19 |
| corvus | or -- what if we just ran rstrip on every line of output? :) | 16:19 |
| fungi | i don't feel super strongly either way, though in other situations where i went with pyyaml i simply declared a "comment" string in the schema for the data i was using | 16:19 |
| clarkb | corvus: I dug into the ruamel code and I don't think we can customize the trailing space. There is a todo comment in the code indicating they should clean it up but haven't yet | 16:20 |
| fungi | postprocessing it would be fine by me | 16:20 |
| corvus | clarkb: nothing like pyyaml's output formatters then | 16:20 |
| corvus | we actually already postprocess | 16:21 |
| clarkb | corvus: ya I think you can customize some things. But in this case the line wrapping allows you to set a width to wrap at and that is it. Then you hit the bug if you wrap | 16:21 |
| corvus | i'll write a change | 16:21 |
| fungi | yeah, for pyyaml i usually just subclass the emitter and dumper to override otherwise non-configurable choices | 16:21 |
| opendevreview | James E. Blair proposed openstack/project-config master: Line-wrap projects.yaml but strip trailing spaces https://review.opendev.org/c/openstack/project-config/+/960144 | 16:28 |
| corvus | clarkb: fungi frickler ^ results included in that for easy review | 16:28 |
| clarkb | corvus: any idea what line[2:] is doing for us? | 16:30 |
| corvus | well maybe not "easy" depending on browser | 16:30 |
| corvus | clarkb: i think it counteracts the "offset" argument in yaml.indent to pull everything back to the right level. | 16:32 |
| fungi | and apparently lines starting with # don't get the offset, hence the conditional | 16:33 |
| corvus | beyond that, we'd have to go to the git history to tell us how we ended up with that exact set of arguments to get everything in perfect balance. :) | 16:33 |
| clarkb | ack thanks | 16:35 |
| clarkb | frickler: if you have a chance can you check 960144? The rest of us are happy with it and I think it addressed the concerns you raised previously | 16:40 |
| clarkb | I'm going to start working on a meeting agenda for tomorrow. There is the openafs server situation (any update on that fungi?), the updates to our container image building, anything else? | 20:06 |
| fungi | no update yet, been busy with other stuff today so far and am in a meeting now | 20:07 |
| clarkb | ack | 20:12 |
| clarkb | ok my agenda updates are in. Let me know if there are any edits to make from here and I'm happy to do that before I send it out later today | 20:51 |
| clarkb | agenda is sent. | 22:47 |
Generated by irclog2html.py 4.0.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!