| opendevreview | Tony Breeds proposed openstack/diskimage-builder master: Add new openstack/devstack based functional testing https://review.opendev.org/c/openstack/diskimage-builder/+/949942 | 00:10 |
|---|---|---|
| opendevreview | tianyutong proposed openstack/project-config master: Add release branch creation permission https://review.opendev.org/c/openstack/project-config/+/952643 | 00:56 |
| opendevreview | OpenStack Proposal Bot proposed openstack/project-config master: Normalize projects.yaml https://review.opendev.org/c/openstack/project-config/+/952006 | 02:20 |
| tonyb | infra-root, gerrit is being very slow for me. Investigating .... | 09:45 |
| *** tosky_ is now known as tosky | 09:49 | |
| tonyb | Something is definitely up ... the load average is increasing and java is using 1000% CPU ? | 09:54 |
| frickler | hmm, response times look normal to me, did you already do something? | 10:14 |
| *** liuxie is now known as liushy | 11:43 | |
| *** elodilles is now known as elodilles_pto | 12:20 | |
| fungi | i think that cpu utilization may be typical? watching top right now the java process is bouncing around between 1500%-2000% cpu but load average is hovering around 10 on a 32 cpu server | 12:55 |
| fungi | infra-root: i'll send out tomorrow's meeting agenda later today, make sure you update the wiki with whatever you want covered | 15:01 |
| clarkb | fungi: frickler tonyb: I think load is more typically 2-4 most of the time | 15:25 |
| clarkb | is it still high? | 15:25 |
| clarkb | I like to look at `gerrit show-queue -w` via the ssh api as a starting point to determine what might be busy | 15:25 |
| clarkb | the web logs (either apache or the built in http log) can also be telling if there is a poorly behaved scraper | 15:26 |
| frickler | fungi: can't edit the wiki right now but maybe add https://review.opendev.org/c/openstack/project-config/+/952006 as topic? | 15:27 |
| clarkb | wow thats a monster diff | 15:28 |
| frickler | gerrit show-queue looks empty | 15:30 |
| clarkb | then my hunch is load (if still there) is likely related to web api requests | 15:32 |
| clarkb | system load is hovering around 5 right now which is more what I'd expect | 15:37 |
| clarkb | fungi: what was the dco changes review topic again? I want to amke sure I'm reviewing those that I can | 15:55 |
| fungi | hashtag:dco-signed-off-by | 15:55 |
| clarkb | thanks! After breakfast I'm going to catch up on the dib devstack testing and review those changes | 15:55 |
| fungi | it's a hashtag not a topic, in the spirit of not continuing to misuse review topics | 15:56 |
| clarkb | ++ | 15:56 |
| corvus | i'm going to create zl02 so that we have two zuul-launchers available. after that, i'll propose we start using zuul-launcher in the openstack tenant. | 16:10 |
| fungi | sounds great! | 16:13 |
| corvus | the zuul servers are still jammy, including zl01.... i could launch it as noble, but it seems like it'd be better to do that after we switch the jobs, etc.... so i'm inclined to launch this as jammy even though we'll probably replace it very soon... | 16:19 |
| corvus | does that sound good, or should we draw the line here and start switching at the same time? | 16:19 |
| opendevreview | James E. Blair proposed opendev/system-config master: Test zl servers on noble https://review.opendev.org/c/opendev/system-config/+/952693 | 16:20 |
| opendevreview | James E. Blair proposed opendev/system-config master: Switch zuul-mergers to noble https://review.opendev.org/c/opendev/system-config/+/952696 | 16:22 |
| opendevreview | James E. Blair proposed opendev/system-config master: Switch zuul-schedulers/web to noble https://review.opendev.org/c/opendev/system-config/+/952697 | 16:22 |
| opendevreview | James E. Blair proposed opendev/system-config master: Switch zuul-executors to noble https://review.opendev.org/c/opendev/system-config/+/952698 | 16:22 |
| corvus | here's what i have ready if we choose jammy: /usr/launcher-venv/bin/launch-node --cloud="openstackci-rax" --region="DFW" --flavor="8 GB Performance" --image="Ubuntu 22.04 LTS (Jammy Jellyfish) (Cloud)" zl02.opendev.org | 16:23 |
| fungi | corvus: tempting to use it as a push for switching the remaining servers to noble, but i'd rather that didn't hold things up as these are fairly separate activities we'd be conflating | 16:23 |
| corvus | how about this: if 952693 comes back clean (suggesting it should be easy), we launch as noble, otherwise, jammy? | 16:24 |
| fungi | also since we use the zuul upstream container images we don't have as much incentive to switch those servers to noble with podman in order to preserve speculative testing for the container images we're (re)building specifically for opendev's servers | 16:25 |
| fungi | so i think the zuul server deployments don't make it onto the list of things that still hit dockerhub | 16:25 |
| corvus | yep | 16:25 |
| fungi | but sure, if running mixed noble+jammy for various zuul servers works out okay, then i have no objection to doing the new launcher on noble... one server fewer to replace later | 16:27 |
| fungi | and it gets us a (slight) head start on doing the rest of those | 16:27 |
| opendevreview | Clark Boylan proposed openstack/diskimage-builder master: Add new openstack/devstack based functional testing https://review.opendev.org/c/openstack/diskimage-builder/+/949942 | 16:47 |
| clarkb | tonyb: ^ thats just a small update to run the element that allows us to use the unsigned reprepro mirrors during image builds. I'm hoping that fixes the ubuntu noble image build | 16:47 |
| corvus | the zuul noble changes passed, so i'll launch zl02 as noble | 18:15 |
| corvus | how's this: /usr/launcher-venv/bin/launch-node --cloud="openstackci-rax" --region="DFW" --flavor="8 GB Performance" --image="Ubuntu Noble OpenDev 20250110" zl02.opendev.org | 18:16 |
| clarkb | corvus: you need the --cloud-init flag too | 18:16 |
| clarkb | sorry --config-drive | 18:16 |
| clarkb | but that lgtm otherwise (the image we have for noble only undersatnds config drive in that cloud region) | 18:17 |
| clarkb | we don't need config drive in vexxhost or new rax iirc | 18:17 |
| corvus | like this? /usr/launcher-venv/bin/launch-node --cloud="openstackci-rax" --region="DFW" --flavor="8 GB Performance" --image="Ubuntu Noble OpenDev 20250110" --cloud-init zl02.opendev.org | 18:18 |
| clarkb | corvus: ya but you have to use --config-drive (sorry about my earlier typo confusing things) | 18:19 |
| clarkb | its a boolean flag so setting it tells launch node to boot with one. Not setting it means no config drive | 18:19 |
| corvus | oops: /usr/launcher-venv/bin/launch-node --cloud="openstackci-rax" --region="DFW" --flavor="8 GB Performance" --image="Ubuntu Noble OpenDev 20250110" --config-drive zl02.opendev.org | 18:19 |
| clarkb | ya that looks great | 18:19 |
| corvus | cool thanks | 18:20 |
| fungi | lgtm agreed (sorry, was afk for a few) | 18:30 |
| opendevreview | James E. Blair proposed opendev/zone-opendev.org master: Add zl02 https://review.opendev.org/c/opendev/zone-opendev.org/+/952704 | 18:38 |
| opendevreview | James E. Blair proposed opendev/system-config master: Add zl02 to ansible inventory https://review.opendev.org/c/opendev/system-config/+/952705 | 18:39 |
| opendevreview | James E. Blair proposed opendev/system-config master: Add zl02 to cacti https://review.opendev.org/c/opendev/system-config/+/952706 | 18:39 |
| opendevreview | Merged opendev/zone-opendev.org master: Add zl02 https://review.opendev.org/c/opendev/zone-opendev.org/+/952704 | 18:43 |
| clarkb | fungi: frickler: re wiki access I notice there was an alert from rax indicating there may have been a disk/fs issue with the database server? | 19:11 |
| clarkb | wondering if we need to do anything to finish up any recovery process | 19:11 |
| fungi | it seems to be working fine today | 19:24 |
| corvus | clarkb: re your question about zl and noble: https://review.opendev.org/952693 (i waited for that to go green before launching zl02) | 19:57 |
| opendevreview | James E. Blair proposed openstack/project-config master: Switch all Zuul tenants to use niz nodesets https://review.opendev.org/c/openstack/project-config/+/952712 | 20:13 |
| opendevreview | Merged opendev/system-config master: Add zl02 to ansible inventory https://review.opendev.org/c/opendev/system-config/+/952705 | 20:17 |
| opendevreview | James E. Blair proposed opendev/zuul-providers master: Switch centos-9-stream from nodepool to niz https://review.opendev.org/c/opendev/zuul-providers/+/952715 | 20:36 |
| opendevreview | James E. Blair proposed opendev/zuul-providers master: Switch debian-bookworm to niz https://review.opendev.org/c/opendev/zuul-providers/+/952716 | 20:36 |
| opendevreview | James E. Blair proposed opendev/zuul-providers master: Switch debian-bullseye-arm64 nodesets to niz https://review.opendev.org/c/opendev/zuul-providers/+/952717 | 20:36 |
| opendevreview | James E. Blair proposed opendev/zuul-providers master: Switch ubuntu-jammy-arm64 labels to niz https://review.opendev.org/c/opendev/zuul-providers/+/952718 | 20:36 |
| opendevreview | James E. Blair proposed opendev/zuul-providers master: Switch to using niz for ubuntu-noble-arm64 https://review.opendev.org/c/opendev/zuul-providers/+/952719 | 20:36 |
| opendevreview | James E. Blair proposed opendev/zuul-providers master: Switch to using niz labels for rockylinux-8 https://review.opendev.org/c/opendev/zuul-providers/+/952720 | 20:36 |
| opendevreview | James E. Blair proposed opendev/zuul-providers master: Switch to using niz for rockylinux-9 https://review.opendev.org/c/opendev/zuul-providers/+/952721 | 20:36 |
| corvus | the only images/labels in nodepool but not niz are: gentoo-17, ubuntu-bionic, ubuntu-focal, ubuntu-xenial | 20:36 |
| opendevreview | Merged opendev/system-config master: Add zl02 to cacti https://review.opendev.org/c/opendev/system-config/+/952706 | 20:47 |
| fungi | luckily codesearch doesn't see any projects referencing the gentoo-17 label or nodeset on master branches, so we could probably just drop it and nobody would notice | 20:47 |
| corvus | on it | 20:48 |
| fungi | ubuntu-focal and maybe ubuntu-bionic are likely to still be in active use on stable and unmaintained openstack branches | 20:48 |
| opendevreview | James E. Blair proposed opendev/zuul-providers master: Remove gentoo-17 nodeset https://review.opendev.org/c/opendev/zuul-providers/+/952723 | 20:49 |
| fungi | i think we (opendev) were the only remaining users of ubuntu-xenial, can't recall if we stopped running any jobs on it | 20:49 |
| corvus | ack, we may want to send out one more announcement to see if anyone wants to add those missing ubuntu images | 20:50 |
| fungi | looks like focal may only be used on unmaintained openstack branches, last pti it was listed for was zed, 2023.1 used jammy | 20:51 |
| corvus | zl02 is up and running | 21:23 |
| corvus | #status log created and started zl02 | 21:23 |
| opendevstatus | corvus: finished logging | 21:24 |
| clarkb | corvus: fungi: I think my change to remove xenial jobs from system-config did eventually and at one point. I don't see them anymore | 21:46 |
| clarkb | but we do still haev focal jobs | 21:46 |
| opendevreview | James E. Blair proposed opendev/zuul-providers master: Remove ubuntu-xenial nodeset https://review.opendev.org/c/opendev/zuul-providers/+/952726 | 21:47 |
| corvus | cool, then it's just bionic and focal left | 21:47 |
| corvus | and that's a maybe on bionic | 21:47 |
| clarkb | corvus: another thought I'm having as I look at the switch jobs is that I think on arm64 we may only have 8gb and 16gb flavors in the cloud and not 4gb? | 21:49 |
| clarkb | corvus: ya opendev.large (8gb) and opendev.xxxlarge (16gb) flavors | 21:50 |
| clarkb | there is m1.medium (not opendev specific) which has a smaller root disk too (which is probably my main concern with just using it) | 21:50 |
| clarkb | corvus: ya looking in zuul-providers/zuul.d/labels.yaml we have no labels for arm64-4GB of any image type. I suspect that means the stack starting at https://review.opendev.org/c/opendev/zuul-providers/+/952715 needs updates to remove the assumption of those -4gb labels existing. I will update my reviews | 21:53 |
| clarkb | corvus: also should that be something that zuul config verification validates? | 21:53 |
| corvus | clarkb: we can't have label verification until nodepool is fully deprecated (since any unknown label is assumed to be a nodepool label) | 21:54 |
| corvus | but we will be able to add that eventually | 21:54 |
| clarkb | got it. That makes sense | 21:55 |
| corvus | i'll remove the arm64-4 | 21:55 |
| opendevreview | James E. Blair proposed opendev/zuul-providers master: Switch centos-9-stream from nodepool to niz https://review.opendev.org/c/opendev/zuul-providers/+/952715 | 21:58 |
| opendevreview | James E. Blair proposed opendev/zuul-providers master: Switch debian-bookworm to niz https://review.opendev.org/c/opendev/zuul-providers/+/952716 | 21:58 |
| opendevreview | James E. Blair proposed opendev/zuul-providers master: Switch debian-bullseye-arm64 nodesets to niz https://review.opendev.org/c/opendev/zuul-providers/+/952717 | 21:58 |
| opendevreview | James E. Blair proposed opendev/zuul-providers master: Switch ubuntu-jammy-arm64 labels to niz https://review.opendev.org/c/opendev/zuul-providers/+/952718 | 21:58 |
| opendevreview | James E. Blair proposed opendev/zuul-providers master: Switch to using niz for ubuntu-noble-arm64 https://review.opendev.org/c/opendev/zuul-providers/+/952719 | 21:58 |
| opendevreview | James E. Blair proposed opendev/zuul-providers master: Switch to using niz labels for rockylinux-8 https://review.opendev.org/c/opendev/zuul-providers/+/952720 | 21:58 |
| opendevreview | James E. Blair proposed opendev/zuul-providers master: Switch to using niz for rockylinux-9 https://review.opendev.org/c/opendev/zuul-providers/+/952721 | 21:58 |
| opendevreview | James E. Blair proposed opendev/zuul-providers master: Remove gentoo-17 nodeset https://review.opendev.org/c/opendev/zuul-providers/+/952723 | 21:58 |
| opendevreview | James E. Blair proposed opendev/zuul-providers master: Remove ubuntu-xenial nodeset https://review.opendev.org/c/opendev/zuul-providers/+/952726 | 21:58 |
| clarkb | thanks the whole stack should have my +2 now | 22:01 |
| corvus | thank you! | 22:03 |
| corvus | since zl02 is running on noble now, i think we should approve the test change for it: https://review.opendev.org/952693 | 22:04 |
| clarkb | oh yes I meant to review that one too | 22:08 |
| clarkb | I've approved it as it is only a test change seemed safe | 22:08 |
| corvus | \o/ | 22:10 |
| opendevreview | Merged opendev/system-config master: Test zl servers on noble https://review.opendev.org/c/opendev/system-config/+/952693 | 23:06 |
Generated by irclog2html.py 4.0.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!