| ianw | '\N{warning sign}\N{VARIATION SELECTOR-15}' doesn't do what you'd think it does | 00:09 |
|---|---|---|
| ianw | ahh, it's 16 | 00:12 |
| ianw | '\N{warning sign}\N{VARIATION SELECTOR-16}' | 00:12 |
| fungi | huh, i never knew about \N syntax. what python version did that start appearing in? | 00:15 |
| opendevreview | Ian Wienand proposed opendev/statusbot master: Don't inline code emojis https://review.opendev.org/c/opendev/statusbot/+/838613 | 00:19 |
| *** dviroel|out is now known as dviroel | 00:20 | |
| ianw | fungi: yeah, me either -- i just found it in https://docs.python.org/3/howto/unicode.html | 00:20 |
| *** dviroel is now known as dviroel|out | 00:32 | |
| ianw | interesting, the gate failed on '\N{wood}\N{VARIATION SELECTOR-16}' but it displays for me | 00:36 |
| ianw | 🚩 New in 2020 | 00:38 |
| opendevreview | Ian Wienand proposed opendev/statusbot master: Don't inline code emojis https://review.opendev.org/c/opendev/statusbot/+/838613 | 01:00 |
| ianw | well this fairly pointless change has taught me a fair bit about unicode encoding, so i'll take that as a win | 01:01 |
| opendevreview | Merged opendev/statusbot master: Don't inline code emojis https://review.opendev.org/c/opendev/statusbot/+/838613 | 01:31 |
| *** pojadhav|out is now known as pojadhav|ruck | 02:08 | |
| opendevreview | OpenStack Proposal Bot proposed openstack/project-config master: Normalize projects.yaml https://review.opendev.org/c/openstack/project-config/+/838620 | 02:27 |
| ianw | #status log test | 02:27 |
| opendevstatus | ianw: finished logging | 02:27 |
| *** pojadhav is now known as pojadhav|ruck | 04:36 | |
| *** ysandeep|out is now known as ysandeep | 05:10 | |
| opendevreview | Merged openstack/diskimage-builder master: Add interpolation note for dynamic-login password https://review.opendev.org/c/openstack/diskimage-builder/+/835206 | 06:04 |
| *** pojadhav|ruck is now known as pojadhav|brb | 06:07 | |
| *** pojadhav|brb is now known as pojadhav|ruck | 06:25 | |
| *** pojadhav|ruck is now known as pojadhav|dr_appt | 06:30 | |
| opendevreview | Merged opendev/lodgeit master: new paste: add maxlength to input box https://review.opendev.org/c/opendev/lodgeit/+/838418 | 06:34 |
| opendevreview | Merged opendev/lodgeit master: new paste: check input length https://review.opendev.org/c/opendev/lodgeit/+/838421 | 06:34 |
| *** jpena|off is now known as jpena | 07:01 | |
| *** pojadhav|dr_appt is now known as pojadhav|ruck | 07:02 | |
| *** ysandeep is now known as ysandeep|lunch | 08:23 | |
| *** marios is now known as marios|afk | 08:26 | |
| *** pojadhav|ruck is now known as pojadhav|lunch | 08:37 | |
| *** marios|afk is now known as marios | 08:53 | |
| *** ysandeep|lunch is now known as ysandeep | 08:55 | |
| *** pojadhav|lunch is now known as pojadhav|ruck | 09:49 | |
| *** ralonsoh_ is now known as ralonsoh | 10:15 | |
| *** dviroel|out is now known as dviroel | 11:23 | |
| opendevreview | Merged openstack/project-config master: Normalize projects.yaml https://review.opendev.org/c/openstack/project-config/+/838620 | 11:59 |
| hrw | morning | 11:59 |
| hrw | we need to add centos/8-stream/cloud/*/openstack-yoga/ to mirror ;( | 12:03 |
| fungi | does it exist finally? | 12:20 |
| hrw | yes, it does | 12:26 |
| hrw | http://mirror.centos.org/centos/8-stream/cloud/x86_64/openstack-yoga/ finally landed | 12:26 |
| fungi | hrw: i see it at https://mirror.facebook.net/centos/8-stream/cloud/x86_64/ and that's where we update from, we're not intentionally excluding it that i can see and we've refreshed as recently as two hours ago: https://static.opendev.org/mirror/centos/timestamp.txt | 12:26 |
| hrw | uf, so it will land automagically. | 12:27 |
| hrw | uf. one thing less to worry ;D | 12:27 |
| fungi | deleting cloud/x86_64/openstack-yoga/ | 12:28 |
| fungi | looks like we *did* mirror it, and then it disappeared from the remote the next time we pulled | 12:28 |
| hrw | ;) | 12:29 |
| hrw | happens | 12:29 |
| fungi | i wonder if our rsync connections are being round-robin'd to more than one system at facebook and not all are updated | 12:29 |
| hrw | next week/two will bring 'do we have ubuntu 22.04' but this part I plan to ignore - let someone else look | 12:30 |
| fungi | looks like it appeared on our mirrors as of 2022-04-05T20:44:34 judging from our logs, and then got removed 2022-04-17T14:57:16 | 12:33 |
| fungi | er, no, removed 2022-04-20T10:58:12 | 12:35 |
| fungi | so the run a couple hours ago | 12:35 |
| fungi | the log i found from the 17th was just some packages being replaced in it | 12:35 |
| fungi | so it looks like we had it mirrored for over two weeks, but suddenly a couple hours ago it disappeared from the rsync server at facebook | 12:36 |
| hrw | yeah, rdo team tagged some additional packages on my request | 12:36 |
| *** ysandeep is now known as ysandeep|afk | 12:45 | |
| *** ysandeep|afk is now known as ysandeep | 13:08 | |
| Clark[m] | hrw: frickler has been looking at jammy support in dib which is the first step. We also need to set up mirrors which may require more disk or trimming of current afs content. I expect this will pick up speed once the release is done (is that this week?) | 14:03 |
| hrw | Clark[m]: by someone else I meant someone else will do it in Kolla :) | 14:08 |
| opendevreview | Ghanshyam proposed opendev/irc-meetings master: Update policy popup meeting time https://review.opendev.org/c/opendev/irc-meetings/+/838727 | 14:18 |
| *** ysandeep is now known as ysandeep|out | 14:30 | |
| fungi | gmann: see my comment and the conflict test failure on that change | 14:41 |
| fungi | optionally, if the multi-arch sig has stopped meeting, we can remove them in order to solve the conflict | 14:42 |
| fungi | 2021-05-25 was their last meeting, so i'm guessing they've been defunct for nearly a year | 14:43 |
| fungi | ricolin: ^ do you know if the sig is going to resume using that meeting slot? | 14:45 |
| ricolin | We can remove it from meeting slot, there still 3 people in the group, just no much works the SIG is plan to do. | 14:49 |
| ricolin | fungi: ^^^ | 14:49 |
| gmann | fungi: as we already moved meeting things from meeting channel, we need to update the irc-meeting checks if that allow meeting on non irc. I can remove that and add meetpad link and see what fail and fix that | 14:51 |
| gmann | ok, it need 'irc' keyword which needs to be fixed. I will fix after my breakfast | 14:55 |
| fungi | yes, that's where you indicate which irc channel is used | 14:59 |
| opendevreview | Merged openstack/diskimage-builder master: Move grub-install to the end, and skip for partition images https://review.opendev.org/c/openstack/diskimage-builder/+/826976 | 15:14 |
| *** artom__ is now known as artom | 15:19 | |
| clarkb | fungi: the gear chagne to address python setup.py testr has me thinking we should go ahead and land https://review.opendev.org/c/opendev/bindep/+/818435 to switch bindep to stestr | 15:22 |
| fungi | sure | 15:26 |
| clarkb | I've also just approved https://review.opendev.org/c/opendev/system-config/+/838347 to remove our buster container images | 15:29 |
| clarkb | I couldn't find anything using them at this point | 15:29 |
| *** dviroel is now known as dviroel|lunch | 15:31 | |
| opendevreview | Merged opendev/bindep master: Update test tool to use stestr https://review.opendev.org/c/opendev/bindep/+/818435 | 15:37 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Add Bullseye Python 3.10 base images https://review.opendev.org/c/opendev/system-config/+/838750 | 15:41 |
| clarkb | I'm guessing the centos jobs that are hitting retry limits are related to the problem that hrw foudn with our upstream mirror removing the yoga pacakges | 15:43 |
| fungi | looking at the log, seems like there's some churn on the mirrors which is resulting in the rsync process getting killed at the timeout | 15:47 |
| fungi | rsync error: received SIGINT, SIGTERM, or SIGHUP (code 20) at rsync.c(644) [generator=3.1.3] | 15:48 |
| fungi | looks like it started the centos stream 8 rsync at 14:43z and was killed partway through at 15:13z (30 minutes later) | 15:49 |
| fungi | that corresponds to the TIMEOUT="timeout -k 2m 30m" we set in the script | 15:51 |
| fungi | we're due for another one to start in ~50 minutes from now | 15:52 |
| fungi | i can take the flock and run it with no timeout in a screen session | 15:52 |
| fungi | okay, it's in progress | 15:53 |
| fungi | clarkb: https://github.com/go-gitea/gitea/pull/19430 | 15:55 |
| fungi | looks like maybe they were able to figure it out? | 15:55 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Use the build tool in assemble instead of setup.py https://review.opendev.org/c/opendev/system-config/+/838752 | 15:56 |
| clarkb | fungi: oh cool I had missed that | 15:56 |
| clarkb | corvus: ^ fyi you inspired that update to assemble :) | 15:57 |
| clarkb | fungi: I had completely forgotten about that with ptg and other git problems and family visiting | 15:59 |
| clarkb | fungi: we'll just have to remember to toggle the partial clone flag back again when we update to the next gitea release. We have testing for that now so we'll get direct feedback on whether or not it seems to work | 16:00 |
| fungi | yep | 16:00 |
| corvus | clarkb: cool now i can copy pasta the newest stuff :) | 16:02 |
| fungi | clarkb: 838347 failed to log into dockerhub in the gate, i think? | 16:03 |
| fungi | may want to keep an eye on image upload jobs in general, in case that's the start of some new problem | 16:03 |
| clarkb | fungi: ya. Errors like that are part of my motivation for deleting all these unneeded images | 16:04 |
| clarkb | the more we have the harder it is to land updates | 16:04 |
| fungi | makes sense | 16:04 |
| clarkb | 838750 may be difficult to land even after the cleanup :/ | 16:17 |
| clarkb | fungi: I wonder if stream mirrors need a longer tiemout as rolling releases might be more likely to experience churn? | 16:20 |
| fungi | entirely possible | 16:21 |
| fungi | hrw: /afs/openstack.org/mirror/centos/8-stream/cloud/x86_64/openstack-yoga is back again | 16:22 |
| fungi | #status log Manually completed a CentOS Stream 8 mirror rsync into AFS in order to bypass the safety timeout and work around a large amount of package churn | 16:24 |
| opendevstatus | fungi: finished logging | 16:24 |
| clarkb | an ubuntu release mirror is on average 220GB for us if I read our afs grafana graphs correctly | 16:24 |
| fungi | if we increase the timeout, we probably also need to decrease the update frequency | 16:24 |
| clarkb | we can in theory fit that on afs01.dfw but it will be super tight. Might be better to fur so clean up the ELK stack and free up all those volumes then reinvest 1TB of that into afs? | 16:25 |
| clarkb | fungi: maybe, though if it on average takes 20 minutes we can still keep the frequency at the cost of an iteration every once in a while when churn is high | 16:25 |
| clarkb | I think that is reasonable. basically when chrun is high we release less often. Otherwise we release more often | 16:26 |
| clarkb | it looks like we still have stretch content mirrored? Can that be deleted? We don't have stretch test nodes anymore | 16:26 |
| fungi | we should be able to remove the stretch mirrors, yes | 16:27 |
| fungi | we probably just missed cleaning them up | 16:27 |
| clarkb | looks like there is stetch content in debian/ and debian-security/ I think that can be cleaned up | 16:28 |
| clarkb | also debian-openstack appears to only be jessie and maybe all of that can go away too | 16:28 |
| hrw | fungi: Thanks! | 16:29 |
| clarkb | I think it may still be a agood idea to reinvest a small portion of the ELK volume space into afs, but cleanup is always a good thing too | 16:29 |
| clarkb | thinking out loud: Maybe we remove xenial too and force jobs to pull from upstream mirrors since xenial isn't very common anymore | 16:30 |
| hrw | Debian buster/bullseye, Ubuntu bionic/focal/jammy, CentOS Stream 8/9? | 16:31 |
| *** dviroel|lunch is now known as dviroel | 16:31 | |
| clarkb | hrw: and centos 7, fedora 35, opensuse leap 15.x, openeuler :/ | 16:32 |
| hrw | clarkb: I listed those I know in use ;D | 16:32 |
| clarkb | looks like we've got epel centos 7 source packages that can go away too | 16:33 |
| hrw | clarkb: will not be surprised if someone requests rockylinux 9 once released | 16:33 |
| clarkb | hrw: we have rocky 8 already but are experimenting with not mirroring it for the less commonly used distros | 16:33 |
| clarkb | I think we could probably do similar with openeuoler | 16:34 |
| hrw | clarkb: good point. iirc we cleaned source dirs for centos/epel 8/9 recently | 16:34 |
| fungi | well, there's the list of platforms for which we build images, and then the list of platforms for which we mirror packages | 16:34 |
| clarkb | the bulk of our jobs run on ubuntu and centos so mirroring those makes sense | 16:34 |
| clarkb | fungi: yes exactly | 16:34 |
| hrw | openeuler can be problematic as they have small amount of mirrors so jobs timeout | 16:34 |
| clarkb | I think we should continue to mirror for stuff that execute the bulk of our jobs to keep network traffic down and reliability up | 16:34 |
| clarkb | but then things like opensuse leap, openeuler, rocky, etc that haven't gotten a ton of adoption can probably use upstream mirrors | 16:35 |
| fungi | and yes, i would say that if we can somehow estimate the amount of use for those various platforms, we could probably safely identify which ones aren't heavily in need of package mirrors | 16:35 |
| clarkb | fungi: grafana shows you usage by label | 16:35 |
| clarkb | so we can take the integral of that or hand wave it | 16:35 |
| fungi | hand wave is plenty good enough | 16:36 |
| clarkb | anyone know what epel/8/Modular is? can we delete all of that or just the source and arch packages for arches we don't have? | 16:39 |
| clarkb | hrm looks like maybe we need to keep Modular. I'll filter it by source and arch then | 16:39 |
| opendevreview | Merged opendev/system-config master: Remove our buster python images https://review.opendev.org/c/opendev/system-config/+/838347 | 16:42 |
| clarkb | we can remove all aarch64 centos 7 too | 16:42 |
| hrw | yes, c7 is not supported on aarch64 anymore | 16:43 |
| clarkb | (I'm working on a change for the epel and centos 7 stuff fwiw. rsync easier togrok than reprepro :) if someone else wants to look at stretch removal that woudl be great or I can take a look) | 16:43 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Cleanup CentOS mirroring https://review.opendev.org/c/opendev/system-config/+/838759 | 16:49 |
| clarkb | Epel is ~110GB and Centos ~281GB. Once ^ lands and syncing happens we should be able to napkin math the savings pretty easily | 16:49 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Cleanup OpenSUSE mirroring https://review.opendev.org/c/opendev/system-config/+/838761 | 16:53 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Remove isos and other images from CentOS/Fedora mirroring https://review.opendev.org/c/opendev/system-config/+/838764 | 17:11 |
| *** jpena is now known as jpena|off | 17:13 | |
| opendevreview | Clark Boylan proposed opendev/system-config master: Remove isos and other images from rpm mirroring https://review.opendev.org/c/opendev/system-config/+/838764 | 17:18 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Remove unneeded arches from opensuse updates mirror https://review.opendev.org/c/opendev/system-config/+/838765 | 17:18 |
| clarkb | I'm beginning to think one big upside to the way deb mirroring works is you are a lot more specific about what you actually want | 17:19 |
| clarkb | I think the opensuse mirror is an area we can investigate more for cleanups once ^ those chagnes land since it is much larger per distro release than even ubuntu | 17:19 |
| clarkb | it wouldn't surprise me if that stack frees enough disk space to add jammy in without deleting anything else, but it is really hard to estimate without running an actual accounting of those contents | 17:21 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Cleanup debian jessie and stretch keys https://review.opendev.org/c/opendev/system-config/+/838766 | 17:31 |
| clarkb | fungi: ^ I have no idea if that is correct. But also we don't seem to configure reprepro to mirror stretch anymore. I suspect we just need to delete the stretch content? I'm not sure what thebest way to do that is | 17:31 |
| clarkb | I'm trying to sort out if the pool/ contains stretch packages or if we just have stale repo data | 17:32 |
| clarkb | https://mirror.bhs1.ovh.opendev.org/debian/pool/main/0/0xffff/ only has the buster and bullseye package versions in it. SO I think the stretch stuff is largely just the repo indexes and not packages themselves. | 17:36 |
| fungi | right, debian package repositories try to avoid duplication by using a common package pool for multiple distro releases | 17:40 |
| fungi | so it's likely we just never cleaned up the old indices when we stopped mirroring them | 17:41 |
| clarkb | and reprepro's normal package removal process for things that fall out of the indexes would've purged the pool? | 17:42 |
| fungi | right | 17:44 |
| clarkb | should be as simple as rm'ing the stretch files/dirs in debian/dists and debian/lists ? | 17:46 |
| clarkb | Looks like there is some jessie stuff in debiab/lists too | 17:46 |
| fungi | yeah, i think all of that can go. we should be able to tell from the last modified dates on things whether reprepro is still replacing them | 17:48 |
| clarkb | doesn't look like it in lists/ | 17:48 |
| clarkb | and likely not dists either (more recent there from november) | 17:49 |
| clarkb | fungi: also I think we can remove ubunut-ports xenial content | 17:52 |
| clarkb | we don't have a xenial image on nb03.opendev.org's config that I see | 17:52 |
| clarkb | maybe we can batch up removal of xenial from ubuntu-ports with stretch and jessie cleanups in debian/ | 17:53 |
| clarkb | oh yup its in a similar situation to stretch and jessie already I think. Just stale indexes and similar content. Not actual packages | 17:54 |
| fungi | right, i think removing archives from the reprepro config simply orphans the indices for them | 17:59 |
| hrw | nb03 is aarch64, right? iirc we have buster and bionic as oldest images for that arch | 18:02 |
| clarkb | hrw: yes | 18:02 |
| hrw | it can be kind of 'drop x, y, z and wait to see who complain' action | 18:03 |
| clarkb | ya in this case I think they have all already been dropped (we don't ahve images for xenial or centos 7 on aarch64) so now we're catching up more completely on our mirrors to free up space | 18:04 |
| hrw | yep | 18:04 |
| clarkb | and that stack of changes I've pushed foudn some other stuff to clean up too | 18:04 |
| hrw | :) | 18:04 |
| hrw | the good part of cleanups | 18:05 |
| hrw | I dropped one functionality from kolla in Zed. about -3k lines and there is still space for cutting more | 18:05 |
| hrw | will leave it for friday as I will travel by train | 18:06 |
| clarkb | ya we've been trying to trim things where it makes sense to reduce the amount of stuff we have to juggle as the team is quite small now | 18:07 |
| clarkb | we've definitely been making progress. The reduced problem set + better testing of our services have really helped us continue to maintain the remaining services | 18:07 |
| hrw | btw - where grafana.opendev dashboard are stored? there is one for kolla | 18:08 |
| clarkb | once I've got the ELK and subunit stuff cleaned up (hopefully thursday/friday) I'll try to context switch back to the gerrit 3.5 upgrade | 18:09 |
| clarkb | hrw: https://opendev.org/openstack/project-config/src/branch/master/grafana/kolla.yaml | 18:09 |
| hrw | uf. zuul gave +1. we can move further with yoga release in kolla ;D | 18:09 |
| hrw | clarkb: thanks | 18:10 |
| clarkb | you're welcome | 18:11 |
| hrw | we either update or drop it - depends on is there anyone using it | 18:12 |
| clarkb | fungi: re https://review.opendev.org/c/opendev/system-config/+/838766 I copied them from https://review.opendev.org/c/opendev/system-config/+/838766/1/playbooks/roles/reprepro/tasks/debian.yaml and https://review.opendev.org/c/opendev/system-config/+/838766/1/playbooks/roles/reprepro/tasks/debian-security.yaml so I guess we had them backwards there too? I'll double check the keys and | 18:14 |
| clarkb | if they are backwards I should update both locations? | 18:14 |
| clarkb | fungi: but I agree they are backwards. I'll update to flip them around in the other files too | 18:15 |
| corvus | imma gonna restart zuul; thinking slow rolling restart of everything | 18:20 |
| clarkb | slow restart sounds good | 18:21 |
| corvus | merger/executor restart is in progress | 18:23 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Cleanup debian jessie and stretch keys https://review.opendev.org/c/opendev/system-config/+/838766 | 18:26 |
| clarkb | fungi: ^ hopefully that makes more sense now | 18:26 |
| opendevreview | Clark Boylan proposed opendev/lodgeit master: DNM testing the update to assemble script https://review.opendev.org/c/opendev/lodgeit/+/838771 | 18:33 |
| opendevreview | Clark Boylan proposed opendev/system-config master: DNM testing assemble with refstack image https://review.opendev.org/c/opendev/system-config/+/838772 | 18:33 |
| clarkb | corvus: ^ unlikely to be as complicated as zuul but I figure if there are big issues that should shake them out. ThenI can test zuul once those are happy | 18:34 |
| clarkb | corvus: also I've been thinkg about the insecure zuul registry not being pruned and wonder how terrible would it be to have a flag day and swap over the backend container? Then we can delete the old container? | 18:38 |
| clarkb | I think jobs will fail if they look for images that aren't present. So we'd need to tell people to rebuild any dependent images? | 18:38 |
| clarkb | or will we just fallback to docker hub and mostly work but maybe not test what we itnend? | 18:39 |
| corvus | i think the pulls will fail, so more like the first thing | 18:39 |
| corvus | (the pull-from-intermediate-registry role) | 18:40 |
| clarkb | but that should only affect jobs for changes like the two DNM chagnes I just pushed right? | 18:40 |
| clarkb | since they know there is some artifact to be looking for? And we can recheck the depends-on to populate? | 18:40 |
| clarkb | Mostly just wondering if this might be a reasonable way to deal with it on the cloud side so that we aren't terrible tenants | 18:41 |
| corvus | yep, should only be depends-on | 18:41 |
| clarkb | ya so maybe this is somethign we should plan to due during a quiet time? I'll have to think on it a bit more. The other thing is I can't remmber if it shards containers or not. I guess if it does as long as we can set a prefix that would work too | 18:42 |
| corvus | might be easier to fix the pruning? | 18:42 |
| clarkb | maybe? My concern there is every time I look at the docker registry api my brain melts | 18:43 |
| corvus | there's a good set of data now to test with. also, if you run the prune and it fails, the worst case scenario is already what you're proposing. :) | 18:43 |
| clarkb | the protocol is quite crazy. That is a good point re worst case scenario though | 18:43 |
| corvus | you could just not fix the pruning, run it anyway, and possibly be no worse off :) | 18:43 |
| clarkb | so ya maybe best to look at fixing it first and then fallback to the idea above | 18:43 |
| clarkb | corvus: the issue is that it deletes too much stuff ya? | 18:44 |
| clarkb | if I remember correctly the issue is we potentially delete blobs we think aren't used anymore but are used by another manifest | 18:46 |
| clarkb | since the protocol dedups | 18:46 |
| clarkb | hrm no the code takes each blob from every kept manifest and adds it to a set. Then only deletes blobs that are not in that set. | 18:51 |
| clarkb | I'm not sure why this wouldn't work. Which isn't surprising as their protocol really does make my brain melt everytime I have to look into the details of it | 18:51 |
| *** artom__ is now known as artom | 19:07 | |
| clarkb | ok cool no major issue swith assemble for refstack (lodgeit didn't actually run the service :( ) I guess now I need to look at testing with zuul locally | 19:10 |
| *** hrww is now known as hrw | 19:10 | |
| clarkb | lunch first though. ianw when your day starts can you look at https://review.opendev.org/c/opendev/system-config/+/838348 and child? feel free to approve as I should be around today. Also, The stack starting at https://review.opendev.org/c/opendev/system-config/+/838759/ affects centos mirroring to trim what we don't need | 19:11 |
| mnaser | hrw: fyi if you're around at soem point - https://review.opendev.org/c/openstack/devstack/+/838775 :) | 19:14 |
| *** hrww is now known as hrw | 19:21 | |
| clarkb | corvus: I'm going to remove my WIP from https://review.opendev.org/c/opendev/system-config/+/838752 as a local build of zuul using the artifact images for that change as python-builder and python-base results in what I believe is the static js content and the index.html etc under the web install. I also confirmed in the logs that python -m build was used | 20:49 |
| clarkb | corvus: the one oddity I noticed was that we install zuul_base extras when we get to the base side after the builder and that causes a couple of packages to be downloaded as they aren't part of requirements. This is unexecpted but I'm fairly certain existnig behavior since we don't install the extras on builder first | 20:50 |
| clarkb | I think if we want to fix that we need to have the builder install the extras too so that the wheels for the extras deps are cached | 20:50 |
| clarkb | specifically packaging<22,>=21 and pyparsing!=3.0.5,>=2.0.2 end up getting downloaded | 20:51 |
| clarkb | hrm actually I see that we actually do install the extras under the builder and that says "Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /tmp/venv/lib/python3.8/site-packages (from packaging<22,>=21->limits->python-logstash-async) (3.0.8)" | 20:54 |
| corvus | clarkb: sgtm | 20:56 |
| corvus | added a +1 to my +2 | 20:57 |
| corvus | er, +1 | 20:57 |
| clarkb | oh pyparsing and packaging are deps of pip or wheel. I think they may not get cached normally due to that | 20:57 |
| clarkb | And so when we copy the cache from one image to another they don't go along. But in the builder we use the cached version Then on base it isn't until we go to install a wheel with pip that it pulls them in | 20:57 |
| clarkb | something like that | 20:57 |
| clarkb | but ya pretty sure that is unrelated to the swap here as it isn't happening in the step udpated | 20:58 |
| opendevreview | Steve Baker proposed openstack/diskimage-builder master: Set machine-id to uninitialized to trigger first boot https://review.opendev.org/c/openstack/diskimage-builder/+/837251 | 21:41 |
| ianw | clarkb: sure; i also have a change out there to trim the fedora mirrors | 22:06 |
| ianw | https://review.opendev.org/c/opendev/system-config/+/837637 | 22:06 |
| clarkb | ianw: huh that somehow managed to not conflict with theo ne I pushed too | 22:09 |
| clarkb | oh also https://review.opendev.org/c/openstack/openstack-zuul-jobs/+/838763 would be a good one to land to get off of the test package for liberasurecoding | 22:09 |
| ianw | ok will review in ~ 10 mins | 22:12 |
| opendevreview | Steve Baker proposed openstack/diskimage-builder master: Move reset-bls-entries to post-install https://review.opendev.org/c/openstack/diskimage-builder/+/838792 | 22:22 |
| clarkb | infra-root https://etherpad.opendev.org/p/X5Kb03ErV7kJkrr5B2Iu is an idea I've been throwing around with some foundation people since yesterday around how we can make OpenDev's presence a bit more explicit. One thing that happened at the ptg that concerned me was that feedback I thought should go to us went ot the openstack tc in the tc project leadership feedback session. | 22:27 |
| *** dviroel is now known as dviroel|out | 22:27 | |
| clarkb | The idea here is if I/we/someone write a monthlyish update email (or maybe a blog post if we grow a blog) that would remind people that we are here and can be contacted directly. But also try to cover that we are getting work done and trying to help them get work done | 22:27 |
| clarkb | if that doesn't seem like a terrible idea reviewing the draft I've written there would be great. I think it is in a shape that can be sent out ~Friday | 22:28 |
| ianw | ++ lgtm | 22:28 |
| opendevreview | Merged opendev/system-config master: Remove python3.7-bullseye docker images https://review.opendev.org/c/opendev/system-config/+/838348 | 22:33 |
| opendevreview | Merged opendev/system-config master: Add Bullseye Python 3.10 base images https://review.opendev.org/c/opendev/system-config/+/838750 | 23:21 |
| opendevreview | Merged opendev/system-config master: Use the build tool in assemble instead of setup.py https://review.opendev.org/c/opendev/system-config/+/838752 | 23:21 |
| ianw | clarkb: for https://review.opendev.org/c/opendev/system-config/+/838766 ... should we be deleting the .asc files to cleanup too? | 23:29 |
| clarkb | ++ let me spin up a new patchset | 23:30 |
| clarkb | heh my ssh keys have unloaded. WIll be a minute :) | 23:31 |
| corvus | fyi, this kolla job may have a post-timeout that is too generous: https://zuul.opendev.org/t/openstack/stream/e7f38567737843e89e05dbb88b6f7e63?logfile=console.log | 23:31 |
| corvus | it's been sitting there for 2.5 hours. | 23:32 |
| ianw | i've approved 838759 based on hrw's looking too; will let it merge and monitor for issues | 23:32 |
| corvus | i manually killed ansible in that build | 23:35 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Cleanup debian jessie and stretch keys https://review.opendev.org/c/opendev/system-config/+/838766 | 23:36 |
| clarkb | ianw: ^ now with asc files removed | 23:37 |
| corvus | restarting web/scheduler on zuul01 | 23:37 |
| clarkb | corvus: https://zuul.opendev.org/t/openstack/build/e7f38567737843e89e05dbb88b6f7e63/log/job-output.txt#19298-19320 looks like it spent 2.5 hours attempting to collect logs? | 23:37 |
| clarkb | https://opendev.org/openstack/kolla/src/branch/master/tests/playbooks/post.yml#L9 is the task | 23:38 |
| clarkb | I bet it is the `docker system df` that was slow | 23:39 |
| clarkb | I'm not sure why they need to run df then df -v afterwards | 23:40 |
| clarkb | that definitely seems like overkill but I suspect we're getting stuck either in docker info or docker system df | 23:40 |
| corvus | i think it was info | 23:40 |
| fungi | guessing it's the process_build_logs.py script | 23:40 |
| fungi | very end of that play | 23:41 |
| clarkb | fungi: if you look at the console output it doesn't seem to get that far but maybe it isn't flushing to stdout? | 23:41 |
| corvus | anyway, there's a "post-timeout" job attr that can be set... maybe opendev should set that to a lower value on the base job? | 23:41 |
| clarkb | corvus: ya we probably need to do a bit of data gathering before we do that but a good idea | 23:41 |
| corvus | oh, we set it to 30m already | 23:42 |
| clarkb | I think tripleo log processing can also be slow. Its a bit frustrating/unfortunate that this is why the whole ELK processing pipeline happens separately. Its slow and you don't want to hold up CI | 23:42 |
| clarkb | corvus: interesting a bug then ? | 23:42 |
| corvus | clarkb: well, the frozen job says post-timeout 10800, so likely overidden. | 23:46 |
| corvus | clarkb: https://opendev.org/openstack/kolla/src/branch/master/.zuul.d/base.yaml#L94 | 23:46 |
| corvus | might be worth a chat with the kolla folks to find out why they think they need that | 23:46 |
| clarkb | oh I guess those are overrideable | 23:48 |
| clarkb | we have to set the max value in zuul.conf? | 23:48 |
| corvus | they also overrode attempts. i think theoretically, that job could take 30 hours to fail. | 23:48 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Upgrade Gitea to 1.16.6 https://review.opendev.org/c/opendev/system-config/+/838799 | 23:49 |
| clarkb | fungi: ^ fyi gitea just made a release including that fix | 23:49 |
| clarkb | 3 hours for run, 3 hours for post-run ~= 6 hours then 5 attempts? ya that math checks out | 23:50 |
| corvus | (pre-run in the retry case, but i think we use the run timeout for that, so yes) | 23:50 |
| corvus | restarting zuul02 | 23:51 |
| clarkb | https://opendev.org/openstack/kayobe/src/branch/master/zuul.d/jobs.yaml#L58 and https://opendev.org/openstack/kolla-ansible/src/branch/master/zuul.d/base.yaml#L7 are the only two toher cases indexed by codesearch where attempts is increased from the default of 3 | 23:51 |
| clarkb | (some lower it) | 23:51 |
| corvus | #status log rolling restarted all of zuul on d8011793f94f82452338ee3e0b193928f80a4a46 | 23:52 |
| opendevstatus | corvus: finished logging | 23:52 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!