| opendevreview | Merged openstack/diskimage-builder master: Append detailed printing information when exec_sudo fails https://review.opendev.org/c/openstack/diskimage-builder/+/886908 | 04:38 |
|---|---|---|
| opendevreview | Merged openstack/diskimage-builder master: Don't remove packages that are requested to be installed https://review.opendev.org/c/openstack/diskimage-builder/+/747220 | 06:09 |
| *** ralonsoh_ooo is now known as ralonsoh | 07:18 | |
| frickler | rax responded to the ticket: "The IAD image imports are expected to take longer than the same workload for the DFW/ORD regions. There are key configuration differences between the named regions resulting in your observed behavior. | 07:43 |
| frickler | " so all working as expected and we should look into making the nodepool thread limit work per provider maybe? | 07:44 |
| frickler | though with the current configuration things seem to be working mostly fine for iad. just need to watch how much backlog we might build up, though I don't have a good idea yet how to actually measure that except watching image age | 07:46 |
| fungi | and checking for errors in the builder logs and maybe new leaked images | 11:28 |
| frickler | fungi: I may have leaked some images in my testing earlier this week, best generate a new baseline. also can you share how you exactly run this check? | 11:35 |
| fungi | frickler: on bridge i get an image list from the provider/region of interest and extract the uuids to a file. then on nl01 i do a nodepool image-list and filter it down by the provider/region of interest, then extract those uuids to another file. then for each uuid in list from bridge if the uuid does not appear in the list from nodepool i consider it leaked | 11:38 |
| frickler | fungi: ok, thx | 11:42 |
| fungi | frickler: `openstack image list --private` is useful for generating the first list, so that it won't have global images from the provider | 11:42 |
| frickler | fungi: do you need to apply further filtering? otherwise adding "-f value -c ID" makes the "extract the uuids" part pretty simple | 11:44 |
| fungi | frickler: one other thing worth looking at, see if the metadata on leaked images differs from the metadata for successful ones. corvus was also interested in looking at a sample to see if there might be a way for nodepool to detect and add them to its cleanup pass | 11:45 |
| frickler | right, I'll leave that to corvus for now, pretty busy today | 11:48 |
| opendevreview | Alex Kavanagh proposed openstack/project-config master: Add charms-purestorage group for purestorage charms https://review.opendev.org/c/openstack/project-config/+/893377 | 14:07 |
| opendevreview | Merged openstack/diskimage-builder master: Deprecate legacy deployment elements https://review.opendev.org/c/openstack/diskimage-builder/+/893077 | 14:23 |
| *** dviroel_ is now known as dviroel | 14:59 | |
| clarkb | I just pushed a 3.31.0 tag for dib which is on pypi now | 15:27 |
| clarkb | more a heads up than anything else | 15:27 |
| clarkb | fungi: I should probably assume that your internet and or electricity connectivity is at risk today? as well as generally being distracted by potential badness? Probably shouldn't approve a gitea bookworm switch or gerrit bookworm switch then | 16:14 |
| fungi | clarkb: i think the worst of it will probably be past us in the next couple of hours, so your afternoon will likely be fine. if i haven't lost power/internet by then i'm unlikely to do so | 16:16 |
| clarkb | ok fingers crossed then | 16:17 |
| fungi | the path the eye took resulted in favorable wind conditions for us, so it's really just utility stability (and possibly minor structural damage) from sustained winds we have to worry about | 16:19 |
| fungi | i'm mostly worried about our pumpkin plants, since their containers are on the windward side of our deck and couldn't really be relocated. i told them to hold on tight and hope for the best | 16:21 |
| SvenKieske | ah, your located in the southeast of the US where the storm is currently still going on? all the best! | 16:28 |
| SvenKieske | you're* | 16:28 |
| fungi | thanks! and yes, i think my neighborhood got lucky with this one, others didn't fare so well | 16:29 |
| clarkb | frickler: fungi: It is interesting that they haven't acknowledged that there was a change in upload time. But I guess if it is intentional/execpted the best we can do is work around it | 16:40 |
| fungi | i did ask them whether changing image type would help. like maybe they switched iad from xen to qemu or something | 16:41 |
| fungi | and all that extra time is just glance importing the vhd and converting it to qcow or whatever | 16:41 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/diskimage-builder master: Change default value of DIB_DEBIAN_ALT_INIT_PACKAGE https://review.opendev.org/c/openstack/diskimage-builder/+/891299 | 17:20 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/diskimage-builder master: Stop creating default user for cloud-init https://review.opendev.org/c/openstack/diskimage-builder/+/891322 | 17:20 |
| fungi | prying any details about the glance delays in iad out of fanatical support is proving impossible. it sounds like we should continue uploading vhd format images, try to serialize our uploads in iad as much as possible, and expect longer delays there before the images we've uploaded become available | 18:06 |
| fungi | related, i've wondered if we're updating our images too frequently. with as many as we have and as large as they are, that's a lot of bandwidth used to upload new copies of them every single day, possibly more than we save in bandwidth used by jobs downloading fewer packages and git commits. we might consider scaling back to updating them every few days or even just once a week? | 18:08 |
| fungi | or maybe it makes sense to upload frequently-used images more often but infrequently-used images less often? | 18:09 |
| clarkb | the ideal would be some sort of signal that would bump up priority to replcae images where there are known things t orebuild for | 18:19 |
| clarkb | technically we can do that by hand today | 18:19 |
| clarkb | fungi: maybe you want to start by reviewing https://review.opendev.org/c/opendev/system-config/+/893073 and we can decide if that seems safe neough for today | 18:38 |
| clarkb | I do have to do a school run this afternoon but it is raining so it will be straight there and back without playground distraction so shouldn't be too long | 18:38 |
| slittle | Sigh, StarlingX want's to 'rename' 12 git repos going forward. I don't suppose you have a way for a single git repo to exist under two names? | 18:44 |
| fungi | slittle: the way we rename them, redirects get set up so the old clone urls continue to work and browsing urls get redirected | 18:50 |
| clarkb | slittle: we can rename repos. To do so requires a gerrit downtime so it has to be scheduled during a quieter time (probably after the openstack release at this point?). When we do so the gitea servers will have redirects from old name to ne wname but that is the only 2 name setup you can have | 18:50 |
| clarkb | within gerrit there is only a single name | 18:50 |
| fungi | right, so they don't technically "exist" under both names, but people don't have to immediately update git remotes or urls in documentation, that will all continue to work with the previous names | 18:52 |
| fungi | also we do move open reviews and settings from the old project name to the new one in gerrit, so you don't have to resubmit existing changes or anything | 18:54 |
| clarkb | and if you use github replication renaming things there is up to you (if yo uuse their rename tool they also put redirects in place) | 18:54 |
| slittle | ooo, thanks for the reminder re github replication! | 18:58 |
| slittle | sounds like it's doable. What's the process to get changes like this scheduled? | 18:59 |
| fungi | slittle: basically, propose a change for the openstack/project-config repo that renames the repositories in the gerrit/projects.yaml file and zuul/main.yaml, moves any relevant acl file names, updates irc bot configs if relevant | 19:00 |
| clarkb | slittle: https://docs.opendev.org/opendev/infra-manual/latest/creators.html#project-renames is the documentation for the process ^ there | 19:00 |
| slittle | Do .gitreview files need to be updated on old release branches ? | 19:00 |
| clarkb | yes they need updating everywhere | 19:01 |
| clarkb | because gerrit doesn't do redirects | 19:01 |
| fungi | .gitreview files should be updated after the rename is completed though | 19:01 |
| clarkb | you'll also potentially need to update zuul configs but again after rename is complete | 19:02 |
| fungi | link the proposed openstack/project-config change in the opendev sysadmins meeting agenda here for so we can discuss scheduling: https://wiki.openstack.org/wiki/Meetings/InfraTeamMeeting#Upcoming_Project_Renames | 19:02 |
| fungi | and yeah, as for zuul configs, if you have jobs that define required-projects then those could need editing to reflect the new project names | 19:03 |
| fungi | codesearch.opendev.org can also be a quick way to check default branches for such references (but be aware that it doesn't index non-default branches for performance reasons) | 19:03 |
| fungi | obviously we recommend project renames be taken lightly, because it's a lot of work both for us and for the maintainers of the projects being renamed | 19:05 |
| fungi | er, not be taken lightly | 19:05 |
| fungi | but we're happy to help do our part if they're deemed necessary | 19:05 |
| slittle | and where is github forwarding configured? I know that some of our gits do forwarding, but didn't participate in it's setup. | 19:06 |
| fungi | it's done by a zuul job for each project | 19:06 |
| fungi | digging now to get you a reference point | 19:07 |
| slittle | I think I see one in starlingx/integ: .zuul.yaml | 19:11 |
| fungi | yeah, that's this one: https://zuul.opendev.org/t/openstack/job/stx-integ-upload-git-mirror | 19:12 |
| fungi | so it sets the remote repository name here: https://opendev.org/starlingx/integ/src/branch/master/.zuul.yaml#L181 | 19:13 |
| slittle | roughly how long from the request to getting it implemented ? | 19:17 |
| slittle | Is there a regularly scheduled downtime I can work toward? | 19:18 |
| fungi | since it involves an outage for gerrit to perform offline database manipulation, clarkb was suggesting we hold off scheduling it until early october, though maybe we can discuss something sooner. is there urgency? | 19:18 |
| fungi | we don't have any planned maintenance windows for gerrit at present | 19:19 |
| slittle | ha! If it was up to me, I'd leave it. Just info gathering for the higher ups at this stage. | 19:20 |
| clarkb | some openstack releases things get crazy and others it seems like we barely notice | 19:22 |
| clarkb | in general though we typically try to avoid major changes during their release time. In particular renaming projects in gerrit is not supported and its a hack we do. It seems to work fine though, but ya | 19:23 |
| clarkb | that said it is unlikely starlingx project renames would impact openstack at all even if something unexpected happened | 19:24 |
| fungi | also we have scripted the process to the point where the outages are usually no more than a few minutes these days | 19:33 |
| TheJulia | Hey guys, any chance I can get another hold for ironic-tempest-ipa-partition-uefi-pxe-grub2 on openstack/ironic? I need to look again since it is still failing with changing the knobs I know I can change and need to validate they are the doing the expected... or further identify the root issue | 20:29 |
| clarkb | ya I can reissue the request | 20:30 |
| clarkb | done | 20:31 |
| TheJulia | Thanks! | 20:31 |
| clarkb | you're welcome | 20:32 |
| clarkb | reiserfs will be removed from the linux kernel in 2025 | 20:46 |
| clarkb | thats quite the deprecation period | 20:47 |
| fungi | i guess they wanted to get it done before the 20th anniversary of his death | 21:16 |
| fungi | er, conviction i mean | 21:17 |
| clarkb | found this small change I wrote related to renames https://review.opendev.org/c/opendev/system-config/+/880692 totally not required but nice to be explicit there | 22:25 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!