Wednesday, 2022-04-20

ianw'\N{warning sign}\N{VARIATION SELECTOR-15}' doesn't do what you'd think it does00:09
ianwahh, it's 1600:12
ianw'\N{warning sign}\N{VARIATION SELECTOR-16}'00:12
fungihuh, i never knew about \N syntax. what python version did that start appearing in?00:15
opendevreviewIan Wienand proposed opendev/statusbot master: Don't inline code emojis
*** dviroel|out is now known as dviroel00:20
ianwfungi: yeah, me either -- i just found it in
*** dviroel is now known as dviroel|out00:32
ianwinteresting, the gate failed on '\N{wood}\N{VARIATION SELECTOR-16}' but it displays for me00:36
ianw🚩 New in 202000:38
opendevreviewIan Wienand proposed opendev/statusbot master: Don't inline code emojis
ianwwell this fairly pointless change has taught me a fair bit about unicode encoding, so i'll take that as a win01:01
opendevreviewMerged opendev/statusbot master: Don't inline code emojis
*** pojadhav|out is now known as pojadhav|ruck02:08
opendevreviewOpenStack Proposal Bot proposed openstack/project-config master: Normalize projects.yaml
ianw#status log test02:27
opendevstatusianw: finished logging02:27
*** pojadhav is now known as pojadhav|ruck04:36
*** ysandeep|out is now known as ysandeep05:10
opendevreviewMerged openstack/diskimage-builder master: Add interpolation note for dynamic-login password
*** pojadhav|ruck is now known as pojadhav|brb06:07
*** pojadhav|brb is now known as pojadhav|ruck06:25
*** pojadhav|ruck is now known as pojadhav|dr_appt06:30
opendevreviewMerged opendev/lodgeit master: new paste: add maxlength to input box
opendevreviewMerged opendev/lodgeit master: new paste: check input length
*** jpena|off is now known as jpena07:01
*** pojadhav|dr_appt is now known as pojadhav|ruck07:02
*** ysandeep is now known as ysandeep|lunch08:23
*** marios is now known as marios|afk08:26
*** pojadhav|ruck is now known as pojadhav|lunch08:37
*** marios|afk is now known as marios08:53
*** ysandeep|lunch is now known as ysandeep08:55
*** pojadhav|lunch is now known as pojadhav|ruck09:49
*** ralonsoh_ is now known as ralonsoh10:15
*** dviroel|out is now known as dviroel11:23
opendevreviewMerged openstack/project-config master: Normalize projects.yaml
hrwwe need to add centos/8-stream/cloud/*/openstack-yoga/ to mirror ;(12:03
fungidoes it exist finally?12:20
hrwyes, it does12:26
hrw finally landed12:26
fungihrw: i see it at and that's where we update from, we're not intentionally excluding it that i can see and we've refreshed as recently as two hours ago:
hrwuf, so it will land automagically.12:27
hrwuf. one thing less to worry ;D12:27
fungideleting cloud/x86_64/openstack-yoga/12:28
fungilooks like we *did* mirror it, and then it disappeared from the remote the next time we pulled12:28
fungii wonder if our rsync connections are being round-robin'd to more than one system at facebook and not all are updated12:29
hrwnext week/two will bring 'do we have ubuntu 22.04' but this part I plan to ignore - let someone else look12:30
fungilooks like it appeared on our mirrors as of 2022-04-05T20:44:34 judging from our logs, and then got removed 2022-04-17T14:57:1612:33
fungier, no, removed 2022-04-20T10:58:1212:35
fungiso the run a couple hours ago12:35
fungithe log i found from the 17th was just some packages being replaced in it12:35
fungiso it looks like we had it mirrored for over two weeks, but suddenly a couple hours ago it disappeared from the rsync server at facebook12:36
hrwyeah, rdo team tagged some additional packages on my request12:36
*** ysandeep is now known as ysandeep|afk12:45
*** ysandeep|afk is now known as ysandeep13:08
Clark[m]hrw: frickler has been looking at jammy support in dib which is the first step. We also need to set up mirrors which may require more disk or trimming of current afs content. I expect this will pick up speed once the release is done (is that this week?)14:03
hrwClark[m]: by someone else I meant someone else will do it in Kolla :)14:08
opendevreviewGhanshyam proposed opendev/irc-meetings master: Update policy popup meeting time
*** ysandeep is now known as ysandeep|out14:30
fungigmann: see my comment and the conflict test failure on that change14:41
fungioptionally, if the multi-arch sig has stopped meeting, we can remove them in order to solve the conflict14:42
fungi2021-05-25 was their last meeting, so i'm guessing they've been defunct for nearly a year14:43
fungiricolin: ^ do you know if the sig is going to resume using that meeting slot?14:45
ricolinWe can remove it from meeting slot, there still 3 people in the group, just no much works the SIG is plan to do.14:49
ricolinfungi: ^^^14:49
gmannfungi: as we already moved meeting things from meeting channel, we need to update the irc-meeting checks if that allow meeting on non irc. I can remove that and add meetpad link and see what fail and fix that14:51
gmannok, it need 'irc' keyword which needs to be fixed. I will fix after my breakfast14:55
fungiyes, that's where you indicate which irc channel is used14:59
opendevreviewMerged openstack/diskimage-builder master: Move grub-install to the end, and skip for partition images
*** artom__ is now known as artom15:19
clarkbfungi: the gear chagne to address python testr has me thinking we should go ahead and land to switch bindep to stestr15:22
clarkbI've also just approved to remove our buster container images15:29
clarkbI couldn't find anything using them at this point15:29
*** dviroel is now known as dviroel|lunch15:31
opendevreviewMerged opendev/bindep master: Update test tool to use stestr
opendevreviewClark Boylan proposed opendev/system-config master: Add Bullseye Python 3.10 base images
clarkbI'm guessing the centos jobs that are hitting retry limits are related to the problem that hrw foudn with our upstream mirror removing the yoga pacakges15:43
fungilooking at the log, seems like there's some churn on the mirrors which is resulting in the rsync process getting killed at the timeout15:47
fungirsync error: received SIGINT, SIGTERM, or SIGHUP (code 20) at rsync.c(644) [generator=3.1.3]15:48
fungilooks like it started the centos stream 8 rsync at 14:43z and was killed partway through at 15:13z (30 minutes later)15:49
fungithat corresponds to the TIMEOUT="timeout -k 2m 30m" we set in the script15:51
fungiwe're due for another one to start in ~50 minutes from now15:52
fungii can take the flock and run it with no timeout in a screen session15:52
fungiokay, it's in progress15:53
fungilooks like maybe they were able to figure it out?15:55
opendevreviewClark Boylan proposed opendev/system-config master: Use the build tool in assemble instead of
clarkbfungi: oh cool I had missed that15:56
clarkbcorvus: ^ fyi you inspired that update to assemble :)15:57
clarkbfungi: I had completely forgotten about that with ptg and other git problems and family visiting15:59
clarkbfungi: we'll just have to remember to toggle the partial clone flag back again when we update to the next gitea release. We have testing for that now so we'll get direct feedback on whether or not it seems to work16:00
corvusclarkb: cool now i can copy pasta the newest stuff :)16:02
fungiclarkb: 838347 failed to log into dockerhub in the gate, i think?16:03
fungimay want to keep an eye on image upload jobs in general, in case that's the start of some new problem16:03
clarkbfungi: ya. Errors like that are part of my motivation for deleting all these unneeded images16:04
clarkbthe more we have the harder it is to land updates16:04
fungimakes sense16:04
clarkb838750 may be difficult to land even after the cleanup :/16:17
clarkbfungi: I wonder if stream mirrors need a longer tiemout as rolling releases might be more likely to experience churn?16:20
fungientirely possible16:21
fungihrw: /afs/ is back again16:22
fungi#status log Manually completed a CentOS Stream 8 mirror rsync into AFS in order to bypass the safety timeout and work around a large amount of package churn16:24
opendevstatusfungi: finished logging16:24
clarkban ubuntu release mirror is on average 220GB for us if I read our afs grafana graphs correctly16:24
fungiif we increase the timeout, we probably also need to decrease the update frequency16:24
clarkbwe can in theory fit that on afs01.dfw but it will be super tight. Might be better to fur so clean up the ELK stack and free up all those volumes then reinvest 1TB of that into afs?16:25
clarkbfungi: maybe, though if it on average takes 20 minutes we can still keep the frequency at the cost of an iteration every once in a while when churn is high16:25
clarkbI think that is reasonable. basically when chrun is high we release less often. Otherwise we release more often16:26
clarkbit looks like we still have stretch content mirrored? Can that be deleted? We don't have stretch test nodes anymore16:26
fungiwe should be able to remove the stretch mirrors, yes16:27
fungiwe probably just missed cleaning them up16:27
clarkblooks like there is stetch content in debian/ and debian-security/ I think that can be cleaned up16:28
clarkbalso debian-openstack appears to only be jessie and maybe all of that can go away too16:28
hrwfungi: Thanks!16:29
clarkbI think it may still be a agood idea to reinvest a small portion of the ELK volume space into afs, but cleanup is always a good thing too16:29
clarkbthinking out loud: Maybe we remove xenial too and force jobs to pull from upstream mirrors since xenial isn't very common anymore16:30
hrwDebian buster/bullseye, Ubuntu bionic/focal/jammy, CentOS Stream 8/9?16:31
*** dviroel|lunch is now known as dviroel16:31
clarkbhrw: and centos 7, fedora 35, opensuse leap 15.x, openeuler :/16:32
hrwclarkb: I listed those I know in use ;D16:32
clarkblooks like we've got epel centos 7 source packages that can go away too16:33
hrwclarkb: will not be surprised if someone requests rockylinux 9 once released16:33
clarkbhrw: we have rocky 8 already but are experimenting with not mirroring it for the less commonly used distros16:33
clarkbI think we could probably do similar with openeuoler16:34
hrwclarkb: good point. iirc we cleaned source dirs for centos/epel 8/9 recently16:34
fungiwell, there's the list of platforms for which we build images, and then the list of platforms for which we mirror packages16:34
clarkbthe bulk of our jobs run on ubuntu and centos so mirroring those makes sense16:34
clarkbfungi: yes exactly16:34
hrwopeneuler can be problematic as they have small amount of mirrors so jobs timeout 16:34
clarkbI think we should continue to mirror for stuff that execute the bulk of our jobs to keep network traffic down and reliability up16:34
clarkbbut then things like opensuse leap, openeuler, rocky, etc that haven't gotten a ton of adoption can probably use upstream mirrors16:35
fungiand yes, i would say that if we can somehow estimate the amount of use for those various platforms, we could probably safely identify which ones aren't heavily in need of package mirrors16:35
clarkbfungi: grafana shows you usage by label16:35
clarkbso we can take the integral of that or hand wave it16:35
fungihand wave is plenty good enough16:36
clarkbanyone know what epel/8/Modular is? can we delete all of that or just the source and arch packages for arches we don't have?16:39
clarkbhrm looks like maybe we need to keep Modular. I'll filter it by source and arch then16:39
opendevreviewMerged opendev/system-config master: Remove our buster python images
clarkbwe can remove all aarch64 centos 7 too16:42
hrwyes, c7 is not supported on aarch64 anymore16:43
clarkb(I'm working on a change for the epel and centos 7 stuff fwiw. rsync easier togrok than reprepro :) if someone else wants to look at stretch removal that woudl be great or I can take a look)16:43
opendevreviewClark Boylan proposed opendev/system-config master: Cleanup CentOS mirroring
clarkbEpel is ~110GB and Centos ~281GB. Once ^ lands and syncing happens we should be able to napkin math the savings pretty easily16:49
opendevreviewClark Boylan proposed opendev/system-config master: Cleanup OpenSUSE mirroring
opendevreviewClark Boylan proposed opendev/system-config master: Remove isos and other images from CentOS/Fedora mirroring
*** jpena is now known as jpena|off17:13
opendevreviewClark Boylan proposed opendev/system-config master: Remove isos and other images from rpm mirroring
opendevreviewClark Boylan proposed opendev/system-config master: Remove unneeded arches from opensuse updates mirror
clarkbI'm beginning to think one big upside to the way deb mirroring works is you are a lot more specific about what you actually want17:19
clarkbI think the opensuse mirror is an area we can investigate more for cleanups once ^ those chagnes land since it is much larger per distro release than even ubuntu17:19
clarkbit wouldn't surprise me if that stack frees enough disk space to add jammy in without deleting anything else, but it is really hard to estimate without running an actual accounting of those contents17:21
opendevreviewClark Boylan proposed opendev/system-config master: Cleanup debian jessie and stretch keys
clarkbfungi: ^ I have no idea if that is correct. But also we don't seem to configure reprepro to mirror stretch anymore. I suspect we just need to delete the stretch content? I'm not sure what thebest way to do that is17:31
clarkbI'm trying to sort out if the pool/ contains stretch packages or if we just have stale repo data17:32
clarkb only has the buster and bullseye package versions in it. SO I think the stretch stuff is largely just the repo indexes and not packages themselves.17:36
fungiright, debian package repositories try to avoid duplication by using a common package pool for multiple distro releases17:40
fungiso it's likely we just never cleaned up the old indices when we stopped mirroring them17:41
clarkband reprepro's normal package removal process for things that fall out of the indexes would've purged the pool?17:42
clarkbshould be as simple as rm'ing the stretch files/dirs in debian/dists and debian/lists ?17:46
clarkbLooks like there is some jessie stuff in debiab/lists too17:46
fungiyeah, i think all of that can go. we should be able to tell from the last modified dates on things whether reprepro is still replacing them17:48
clarkbdoesn't look like it in lists/17:48
clarkband likely not dists either (more recent there from november)17:49
clarkbfungi: also I think we can remove ubunut-ports xenial content17:52
clarkbwe don't have a xenial image on's config that I see17:52
clarkbmaybe we can batch up removal of xenial from ubuntu-ports with stretch and jessie cleanups in debian/17:53
clarkboh yup its in a similar situation to stretch and jessie already I think. Just stale indexes and similar content. Not actual packages17:54
fungiright, i think removing archives from the reprepro config simply orphans the indices for them17:59
hrwnb03 is aarch64, right? iirc we have buster and bionic as oldest images for that arch18:02
clarkbhrw: yes18:02
hrwit can be kind of 'drop x, y, z and wait to see who complain' action18:03
clarkbya in this case I think they have all already been dropped (we don't ahve images for xenial or centos 7 on aarch64) so now we're catching up more completely on our mirrors to free up space18:04
clarkband that stack of changes I've pushed foudn some other stuff to clean up too18:04
hrwthe good part of cleanups18:05
hrwI dropped one functionality from kolla in Zed. about -3k lines and there is still space for cutting more18:05
hrwwill leave it for friday as I will travel by train18:06
clarkbya we've been trying to trim things where it makes sense to reduce the amount of stuff we have to juggle as the team is quite small now18:07
clarkbwe've definitely been making progress. The reduced problem set + better testing of our services have really helped us continue to maintain the remaining services18:07
hrwbtw - where grafana.opendev dashboard are stored? there is one for kolla18:08
clarkbonce I've got the ELK and subunit stuff cleaned up (hopefully thursday/friday) I'll try to context switch back to the gerrit 3.5 upgrade18:09
hrwuf. zuul gave +1. we can move further with yoga release in kolla ;D18:09
hrwclarkb: thanks18:10
clarkbyou're welcome18:11
hrwwe either update or drop it - depends on is there anyone using it18:12
clarkbfungi: re I copied them from and so I guess we had them backwards there too? I'll double check the keys and18:14
clarkbif they are backwards I should update both locations?18:14
clarkbfungi: but I agree they are backwards. I'll update to flip them around in the other files too18:15
corvusimma gonna restart zuul; thinking slow rolling restart of everything18:20
clarkbslow restart sounds good18:21
corvusmerger/executor restart is in progress18:23
opendevreviewClark Boylan proposed opendev/system-config master: Cleanup debian jessie and stretch keys
clarkbfungi: ^ hopefully that makes more sense now18:26
opendevreviewClark Boylan proposed opendev/lodgeit master: DNM testing the update to assemble script
opendevreviewClark Boylan proposed opendev/system-config master: DNM testing assemble with refstack image
clarkbcorvus: ^ unlikely to be as complicated as zuul but I figure if there are big issues that should shake them out. ThenI can test zuul once those are happy18:34
clarkbcorvus: also I've been thinkg about the insecure zuul registry not being pruned and wonder how terrible would it be to have a flag day and swap over the backend container? Then we can delete the old container?18:38
clarkbI think jobs will fail if they look for images that aren't present. So we'd need to tell people to rebuild any dependent images?18:38
clarkbor will we just fallback to docker hub and mostly work but maybe not test what we itnend?18:39
corvusi think the pulls will fail, so more like the first thing18:39
corvus(the pull-from-intermediate-registry role)18:40
clarkbbut that should only affect jobs for changes like the two DNM chagnes I just pushed right?18:40
clarkbsince they know there is some artifact to be looking for? And we can recheck the depends-on to populate?18:40
clarkbMostly just wondering if this might be a reasonable way to deal with it on the cloud side so that we aren't terrible tenants18:41
corvusyep, should only be depends-on18:41
clarkbya so maybe this is somethign we should plan to due during a quiet time? I'll have to think on it a bit more. The other thing is I can't remmber if it shards containers or not. I guess if it does as long as we can set a prefix that would work too18:42
corvusmight be easier to fix the pruning?18:42
clarkbmaybe? My concern there is every time I look at the docker registry api my brain melts18:43
corvusthere's a good set of data now to test with.  also, if you run the prune and it fails, the worst case scenario is already what you're proposing.  :)18:43
clarkbthe protocol is quite crazy. That is a good point re worst case scenario though18:43
corvusyou could just not fix the pruning, run it anyway, and possibly be no worse off :)18:43
clarkbso ya maybe best to look at fixing it first and then fallback to the idea above18:43
clarkbcorvus: the issue is that it deletes too much stuff ya?18:44
clarkbif I remember correctly the issue is we potentially delete blobs we think aren't used anymore but are used by another manifest18:46
clarkbsince the protocol dedups18:46
clarkbhrm no the code takes each blob from every kept manifest and adds it to a set. Then only deletes blobs that are not in that set.18:51
clarkbI'm not sure why this wouldn't work. Which isn't surprising as their protocol really does make my brain melt everytime I have to look into the details of it18:51
*** artom__ is now known as artom19:07
clarkbok cool no major issue swith assemble for refstack (lodgeit didn't actually run the service :( ) I guess now I need to look at testing with zuul locally19:10
*** hrww is now known as hrw19:10
clarkblunch first though. ianw  when your day starts can you look at and child? feel free to approve as I should be around today. Also, The stack starting at affects centos mirroring to trim what we don't need19:11
mnaserhrw: fyi if you're around at soem point - :)19:14
*** hrww is now known as hrw19:21
clarkbcorvus: I'm going to remove my WIP from as a local build of zuul using the artifact images for that change as python-builder and python-base results in what I believe is the static js content and the index.html etc under the web install. I also confirmed in the logs that python -m build was used20:49
clarkbcorvus: the one oddity I noticed was that we install zuul_base extras when we get to the base side after the builder and that causes a couple of packages to be downloaded as they aren't part of requirements. This is unexecpted but I'm fairly certain existnig behavior since we don't install the extras on builder first20:50
clarkbI think if we want to fix that we need to have the builder install the extras too so that the wheels for the extras deps are cached20:50
clarkbspecifically packaging<22,>=21 and pyparsing!=3.0.5,>=2.0.2 end up getting downloaded20:51
clarkbhrm actually I see that we actually do install the extras under the builder and that says "Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /tmp/venv/lib/python3.8/site-packages (from packaging<22,>=21->limits->python-logstash-async) (3.0.8)"20:54
corvusclarkb: sgtm20:56
corvusadded a +1 to my +220:57
corvuser, +120:57
clarkboh pyparsing and packaging are deps of pip or wheel. I think they may not get cached normally due to that20:57
clarkbAnd so when we copy the cache from one image to another they don't go along. But in the builder we use the cached version Then on base it isn't until we go to install a wheel with pip that it pulls them in20:57
clarkbsomething like that20:57
clarkbbut ya pretty sure that is unrelated to the swap here as it isn't happening in the step udpated20:58
opendevreviewSteve Baker proposed openstack/diskimage-builder master: Set machine-id to uninitialized to trigger first boot
ianwclarkb: sure; i also have a change out there to trim the fedora mirrors22:06
clarkbianw: huh that somehow managed to not conflict with theo ne I pushed too22:09
clarkboh also would be a good one to land to get off of the test package for liberasurecoding22:09
ianwok will review in ~ 10 mins22:12
opendevreviewSteve Baker proposed openstack/diskimage-builder master: Move reset-bls-entries to post-install
clarkbinfra-root is an idea I've been throwing around with some foundation people since yesterday around how we can make OpenDev's presence a bit more explicit. One thing that happened at the ptg that concerned me was that feedback I thought should go to us went ot the openstack tc in the tc project leadership feedback session.22:27
*** dviroel is now known as dviroel|out22:27
clarkbThe idea here is if I/we/someone write a monthlyish update email (or maybe a blog post if we grow a blog) that would remind people that we are here and can be contacted directly. But also try to cover that we are getting work done and trying to help them get work done22:27
clarkbif that doesn't seem like a terrible idea reviewing the draft I've written there would be great. I think it is in a shape that can be sent out ~Friday22:28
ianw++ lgtm22:28
opendevreviewMerged opendev/system-config master: Remove python3.7-bullseye docker images
opendevreviewMerged opendev/system-config master: Add Bullseye Python 3.10 base images
opendevreviewMerged opendev/system-config master: Use the build tool in assemble instead of
ianwclarkb: for ... should we be deleting the .asc files to cleanup too?23:29
clarkb++ let me spin up a new patchset23:30
clarkbheh my ssh keys have unloaded. WIll be a minute :)23:31
corvusfyi, this kolla job may have a post-timeout that is too generous:
corvusit's been sitting there for 2.5 hours.23:32
ianwi've approved 838759 based on hrw's looking too; will let it merge and monitor for issues23:32
corvusi manually killed ansible in that build23:35
opendevreviewClark Boylan proposed opendev/system-config master: Cleanup debian jessie and stretch keys
clarkbianw: ^ now with asc files removed23:37
corvusrestarting web/scheduler on zuul0123:37
clarkbcorvus: looks like it spent 2.5 hours attempting to collect logs?23:37
clarkb is the task23:38
clarkbI bet it is the `docker system df` that was slow23:39
clarkbI'm not sure why they need to run df then df -v afterwards23:40
clarkbthat definitely seems like overkill but I suspect we're getting stuck either in docker info or docker system df23:40
corvusi think it was info23:40
fungiguessing it's the script23:40
fungivery end of that play23:41
clarkbfungi: if you look at the console output it doesn't seem to get that far but maybe it isn't flushing to stdout?23:41
corvusanyway, there's a "post-timeout" job attr that can be set... maybe opendev should set that to a lower value on the base job?23:41
clarkbcorvus: ya we probably need to do a bit of data gathering before we do that but a good idea23:41
corvusoh, we set it to 30m already23:42
clarkbI think tripleo log processing can also be slow. Its a bit frustrating/unfortunate that this is why the whole ELK processing pipeline happens separately. Its slow and you don't want to hold up CI23:42
clarkbcorvus: interesting a bug then ?23:42
corvusclarkb: well, the frozen job says post-timeout 10800, so likely overidden.23:46
corvusmight be worth a chat with the kolla folks to find out why they think they need that23:46
clarkboh I guess those are overrideable23:48
clarkbwe have to set the max value in zuul.conf?23:48
corvusthey also overrode attempts.  i think theoretically, that job could take 30 hours to fail.23:48
opendevreviewClark Boylan proposed opendev/system-config master: Upgrade Gitea to 1.16.6
clarkbfungi: ^ fyi gitea just made a release including that fix23:49
clarkb3 hours for run, 3 hours for post-run ~= 6 hours then 5 attempts? ya that math checks out23:50
corvus(pre-run in the retry case, but i think we use the run timeout for that, so yes)23:50
corvusrestarting zuul0223:51
clarkb and are the only two toher cases indexed by codesearch where attempts is increased from the default of 323:51
clarkb(some lower it)23:51
corvus#status log rolling restarted all of zuul on d8011793f94f82452338ee3e0b193928f80a4a4623:52
opendevstatuscorvus: finished logging23:52

Generated by 2.17.3 by Marius Gedminas - find it at!