Thursday, 2023-08-31

opendevreviewMerged openstack/diskimage-builder master: Append detailed printing information when exec_sudo fails
opendevreviewMerged openstack/diskimage-builder master: Don't remove packages that are requested to be installed
*** ralonsoh_ooo is now known as ralonsoh07:18
fricklerrax responded to the ticket: "The IAD image imports are expected to take longer than the same workload for the DFW/ORD regions. There are key configuration differences between the named regions resulting in your observed behavior.07:43
frickler" so all working as expected and we should look into making the nodepool thread limit work per provider maybe?07:44
fricklerthough with the current configuration things seem to be working mostly fine for iad. just need to watch how much backlog we might build up, though I don't have a good idea yet how to actually measure that except watching image age07:46
fungiand checking for errors in the builder logs and maybe new leaked images11:28
fricklerfungi: I may have leaked some images in my testing earlier this week, best generate a new baseline. also can you share how you exactly run this check?11:35
fungifrickler: on bridge i get an image list from the provider/region of interest and extract the uuids to a file. then on nl01 i do a nodepool image-list and filter it down by the provider/region of interest, then extract those uuids to another file. then for each uuid in list from bridge if the uuid does not appear in the list from nodepool i consider it leaked11:38
fricklerfungi: ok, thx11:42
fungifrickler: `openstack image list --private` is useful for generating the first list, so that it won't have global images from the provider11:42
fricklerfungi: do you need to apply further filtering? otherwise adding "-f value -c ID" makes the "extract the uuids" part pretty simple11:44
fungifrickler: one other thing worth looking at, see if the metadata on leaked images differs from the metadata for successful ones. corvus was also interested in looking at a sample to see if there might be a way for nodepool to detect and add them to its cleanup pass11:45
fricklerright, I'll leave that to corvus for now, pretty busy today11:48
opendevreviewAlex Kavanagh proposed openstack/project-config master: Add charms-purestorage group for purestorage charms
opendevreviewMerged openstack/diskimage-builder master: Deprecate legacy deployment elements
*** dviroel_ is now known as dviroel14:59
clarkbI just pushed a 3.31.0 tag for dib which is on pypi now15:27
clarkbmore a heads up than anything else15:27
clarkbfungi: I should probably assume that your internet and or electricity connectivity is at risk today? as well as generally being distracted by potential badness? Probably shouldn't approve a gitea bookworm switch or gerrit bookworm switch then16:14
fungiclarkb: i think the worst of it will probably be past us in the next couple of hours, so your afternoon will likely be fine. if i haven't lost power/internet by then i'm unlikely to do so16:16
clarkbok fingers crossed then16:17
fungithe path the eye took resulted in favorable wind conditions for us, so it's really just utility stability (and possibly minor structural damage) from sustained winds we have to worry about16:19
fungii'm mostly worried about our pumpkin plants, since their containers are on the windward side of our deck and couldn't really be relocated. i told them to hold on tight and hope for the best16:21
SvenKieskeah, your located in the southeast of the US where the storm is currently still going on? all the best!16:28
fungithanks! and yes, i think my neighborhood got lucky with this one, others didn't fare so well16:29
clarkbfrickler: fungi: It is interesting that they haven't acknowledged that there was a change in upload time. But I guess if it is intentional/execpted the best we can do is work around it16:40
fungii did ask them whether changing image type would help. like maybe they switched iad from xen to qemu or something16:41
fungiand all that extra time is just glance importing the vhd and converting it to qcow or whatever16:41
opendevreviewDmitriy Rabotyagov proposed openstack/diskimage-builder master: Change default value of DIB_DEBIAN_ALT_INIT_PACKAGE
opendevreviewDmitriy Rabotyagov proposed openstack/diskimage-builder master: Stop creating default user for cloud-init
fungiprying any details about the glance delays in iad out of fanatical support is proving impossible. it sounds like we should continue uploading vhd format images, try to serialize our uploads in iad as much as possible, and expect longer delays there before the images we've uploaded become available18:06
fungirelated, i've wondered if we're updating our images too frequently. with as many as we have and as large as they are, that's a lot of bandwidth used to upload new copies of them every single day, possibly more than we save in bandwidth used by jobs downloading fewer packages and git commits. we might consider scaling back to updating them every few days or even just once a week?18:08
fungior maybe it makes sense to upload frequently-used images more often but infrequently-used images less often?18:09
clarkbthe ideal would be some sort of signal that would bump up priority to replcae images where there are known things t orebuild for18:19
clarkbtechnically we can do that by hand today18:19
clarkbfungi: maybe you want to start by reviewing and we can decide if that seems safe neough for today18:38
clarkbI do have to do a school run this afternoon but it is raining so it will be straight there and back without playground distraction so shouldn't be too long18:38
slittleSigh, StarlingX want's to 'rename' 12 git repos going forward.  I don't suppose you have a way for a single git repo to exist under two names? 18:44
fungislittle: the way we rename them, redirects get set up so the old clone urls continue to work and browsing urls get redirected18:50
clarkbslittle: we can rename repos. To do so requires a gerrit downtime so it has to be scheduled during a quieter time (probably after the openstack release at this point?). When we do so the gitea servers will have redirects from old name to ne wname but that is the only 2 name setup you can have18:50
clarkbwithin gerrit there is only a single name18:50
fungiright, so they don't technically "exist" under both names, but people don't have to immediately update git remotes or urls in documentation, that will all continue to work with the previous names18:52
fungialso we do move open reviews and settings from the old project name to the new one in gerrit, so you don't have to resubmit existing changes or anything18:54
clarkband if you use github replication renaming things there is up to you (if yo uuse their rename tool they also put redirects in place)18:54
slittleooo, thanks for the reminder re github replication!18:58
slittlesounds like it's doable.   What's the process to get changes like this scheduled?18:59
fungislittle: basically, propose a change for the openstack/project-config repo that renames the repositories in the gerrit/projects.yaml file and zuul/main.yaml, moves any relevant acl file names, updates irc bot configs if relevant19:00
clarkbslittle: is the documentation for the process ^ there19:00
slittleDo .gitreview files need to be updated on old release branches ?  19:00
clarkbyes they need updating everywhere19:01
clarkbbecause gerrit doesn't do redirects19:01
fungi.gitreview files should be updated after the rename is completed though19:01
clarkbyou'll also potentially need to update zuul configs but again after rename is complete19:02
fungilink the proposed openstack/project-config change in the opendev sysadmins meeting agenda here for so we can discuss scheduling:
fungiand yeah, as for zuul configs, if you have jobs that define required-projects then those could need editing to reflect the new project names19:03 can also be a quick way to check default branches for such references (but be aware that it doesn't index non-default branches for performance reasons)19:03
fungiobviously we recommend project renames be taken lightly, because it's a lot of work both for us and for the maintainers of the projects being renamed19:05
fungier, not be taken lightly19:05
fungibut we're happy to help do our part if they're deemed necessary19:05
slittleand where is github forwarding configured?  I know that some of our gits do forwarding, but didn't participate in it's setup.19:06
fungiit's done by a zuul job for each project19:06
fungidigging now to get you a reference point19:07
slittleI think I see one in starlingx/integ: .zuul.yaml19:11
fungiyeah, that's this one:
fungiso it sets the remote repository name here:
slittleroughly how long from the request to getting it implemented ?19:17
slittleIs there a regularly scheduled downtime I can work toward?19:18
fungisince it involves an outage for gerrit to perform offline database manipulation, clarkb was suggesting we hold off scheduling it until early october, though maybe we can discuss something sooner. is there urgency?19:18
fungiwe don't have any planned maintenance windows for gerrit at present19:19
slittleha!  If it was up to me, I'd leave it.  Just info gathering for the higher ups at this stage.19:20
clarkbsome openstack releases things get crazy and others it seems like we barely notice19:22
clarkbin general though we typically try to avoid major changes during their release time. In particular renaming projects in gerrit is not supported and its a hack we do. It seems to work fine though, but ya19:23
clarkbthat said it is unlikely starlingx project renames would impact openstack at all even if something unexpected happened19:24
fungialso we have scripted the process to the point where the outages are usually no more than a few minutes these days19:33
TheJuliaHey guys, any chance I can get another hold for ironic-tempest-ipa-partition-uefi-pxe-grub2 on openstack/ironic? I need to look again since it is still failing with changing the knobs I know I can change and need to validate they are the doing the expected... or further identify the root issue20:29
clarkbya I can reissue the request20:30
clarkbyou're welcome20:32
clarkbreiserfs will be removed from the linux kernel in 202520:46
clarkbthats quite the deprecation period20:47
fungii guess they wanted to get it done before the 20th anniversary of his death21:16
fungier, conviction i mean21:17
clarkbfound this small change I wrote related to renames totally not required but nice to be explicit there22:25

Generated by 2.17.3 by Marius Gedminas - find it at!