Thursday, 2023-09-14

opendevreviewJeremy Stanley proposed opendev/system-config master: Move Airship and Kata lists to Mailman 3
fungithat's ^ for the 15:30 maintenance, and i've warmed preliminary rsyncs of both sites to the new server in preparation (will run again just before the outage)02:55
fungittls on the existing dns records are already short, so no need to adjust them in advance02:58
hasharfungi: hello, I have been upgrading Wikimedia to tox version 4 and encountered an issue with skipsdist + usedevelop not working anymore04:16
hasharyou commented on the upstream task at  and I think I might have found a solution (`package=editable` instead)04:17
hasharI found a few more oddities which I have collected at , but most are listed on the upstream upgrade guide
hasharI am unlikely to follow up here, but happy to chat about it if it can helps you migrate :]04:18
hashar(also the current Debian comes with tox 3.28.0 :\ )04:18
*** osmanlicilegi is now known as Guest004:33
opendevreviewMayank Patel proposed openstack/project-config master: Add node-interface-metrics-exporter app to StarlingX
fungihashar: i think all of our projects have migrated to tox v4 or switched to nox by now anyway, so it's not a concern at this point, but thanks for the additional info!11:43
fungiwhat most projects ended up doing is just removing skipsdist from tox.ini11:46
fungiif we end up needing to skip installing for performance reasons, the package=editable setting does look useful11:53
hasharfungi: yeah looks like one can use `package=editable` and for a venv that still requires a full install optin by using `package=sdist`11:54
hasharI should like at nox one day :]11:55
fungiwe switched a lot of opendev's projects over to nox, and also zuul's projects are using nox now. most of openstack stayed on tox though12:00
opendevreviewJeremy Stanley proposed opendev/system-config master: Move Airship and Kata lists to Mailman 3
fungimissed cleaning up the airship domain from the old listserver tests12:09
*** amoralej is now known as amoralej|lunch12:12
opendevreviewMerged openstack/project-config master: Add node-interface-metrics-exporter app to StarlingX
fungii've refreshed the airship and kata rsyncs to the new server, will do it one more time right before i make dns changes12:34
fungicloudnull: just heard your news! now can you fix image processing times in iad for me? ;)12:47
fungi(also congrats)12:47
*** amoralej|lunch is now known as amoralej13:06
fungiinfra-root: is passing tests, we'll want to merge it early enough so it deploys prior to the start of the 15:30 utc maintenance window13:07
* fungi remembered right before bed that we didn't have a change for that yet13:07
Clark[m]fungi: change lgtm but I'm on a phone until after school drop off so can't really +2 in gerrit13:43
fungithere's no rush13:44
fungii'm fleshing out details for the maintenance steps near the bottom of starting at line 13013:44
fungifor the previous migrations we temporarily pointed their dns names to the addresses for review02 (just so that message delivery attempts would get a tcp connection refused rather than a socket timeout), but since that actually serves http(s) and has redirects which browsers might cache, i think this time i'll point traffic to insecure-ci-registry as it has neither smtp nor http(s) listening13:47
*** tosky_ is now known as tosky14:06
fungiokay, i think i have all the steps outlined with cut-n-paste commands/values, and also an explicit post-import testing checklist14:10
fungiwe're at the one hour mark (slightly past) so i'll approve 895125 and refresh the import content one more time14:33
fungii've staged the list completion notices as postponed draft e-mails so i can send them quickly when we're at the end14:44
fungii might should have approved that change earlier. zuul is still estimating another hour for it to finish in the gate pipeline (though that's probably a skewed estimate due to the paused registry job)14:51
fungithough maybe not. it thinks system-config-run-gitea is going to take nearly that long on its own14:53
*** dviroel__ is now known as dviroel14:59
*** diablo_rojo_phone is now known as Guest9615:00
clarkbyes the gitea job is not fast15:07
fungieverything else has succeeded now, just waiting for the gitea job15:09
clarkbI'm here properly now fwiw15:12
fungicool, we're almost certainly looking at a delayed start at this point anyway, barring miracles15:12
fungii guess the lengthy tasks in the gitea job involve cloning git repos?15:13
fungithe stream log says it completed the apache2 service restart 10 minutes ago, and since then it's been silent15:14
clarkbfungi: the main time sink is that we create all the gitea projects and then we do a second pass to ensure it is idempotent15:14
clarkbebcause we've had problems with that in the past15:15
fungiaha, yeah15:15
fungizuul-stream keeps timing out the socket because these tasks sit silent for so long15:15
fungiat least it should merge before the next hourly batch15:21
fungiat this rate i'm going to need another cup of tea15:23
fungi(green tea, not gitea)15:23
fungifor those just tuning in, the maintenance scheduled for 15:30 utc is on a delayed start, probably won't get underway properly until 16:00 utc due to unanticipated gating job durations15:30
fungizuul estimates another 16 minutes until change 895125 merges, and then we need to wait for all the deployment jobs which will be triggered from it (or at least wait through the serialized list of them until the lists jobs complete)15:32
fungiat that point we can proceed with the steps at the bottom of (line 146 onward)15:33
fungiapparently that was indeed the create repos/orgs task that silently took about 30 minutes to complete15:36
fungiit's finally on to taking screenshots, so shouldn't be too much longer15:38
opendevreviewMerged opendev/system-config master: Move Airship and Kata lists to Mailman 3
fungiand there it is. in deploy pipeline now15:46
fungiso many deploy jobs...15:47
clarkbprobably due to modiying the inventory15:47
fungiin hindsight, i could have split that to a separate change15:47
fungidoesn't apply to future migrations except maybe the final one15:47
clarkband we can do the server cleanup independent of the migration I think15:48
fungilists and lists3 deploy jobs are about 2/3 of the way through the series15:48
clarkbre ssh host keys for gerrit we could publish them at https://opendev.org15:49
fungithis is probably going to push our start time out to more like 16:3015:49
fungiclarkb: yeah, i'm open to alternative locations to stick the host key fingerprints as long as the new user documentation links to it clearly15:50
clarkbwe mgiht also be able to edit our gerrit theme plugin to add that content somewhere on gerrit15:51
clarkbsome sort of location conditional check and then put it in the footer15:51
clarkbbut a simple addition to the gitea landing page is easy and probably effective15:51
fungiclarkb: what do you think about a feature request for the "get started" link? it seems like people are finding that first and then not noticing project-specific instructions15:53
fungior maybe our getting started guide should have a first step that says to look at any CONTRIBUTING.rst file in the repository you're planning to work on15:53
fungii suppose a highly visible admonition in our docs is easier than changing gitea's interface15:54
clarkbthe problem with that is gitea will tell you to use the wiki15:55
clarkbI highly suspect that wouldn't get far15:55
clarkbor to put it in the project readme which is rendered on the project root page15:55
clarkbthats what projects should do15:55
fungiyeah, i guess our guide could say to look at both the readme and contributing files for the project15:56
clarkbwell the readme is just as visible as a project specific getting started link would be15:56
clarkbso not sure you need to do anything extra if you just put the info in the readme. Maybe I'm old school but that is why readmes are called read me :)15:57
fungii don't know. i mean we have project-specific buttons to take you to the defect tracker or open changes, those are more visible than digging through a wall of prose15:57
fungithe tl;dr problem15:58
fungii agree people should read. doesn't mean they will15:58
clarkbI guess. But also upstream issue tracker and code review links built in. Unfortauntely "getting started" isn't a project level thing15:58
clarkbwe can file a request but I suspect the response will be "this is why we render readmes on the front page and give you a wiki"15:59
fungiright, so for people who click the get started button and end up in our generic guide, we can just remind them to read whatever project-specific instructions might be provided first15:59
fungii.e. people who should be starting from the openstack contributor guide instead of the infra manual16:00
fungiand then get confused because openstack has additional setup expectations which don't apply to all projects we host16:00
fricklerwe can make comments that cross reference both guides at the start of each of them maybe16:07
frickleror possibly also for others like starlingx from the opendev one16:08
fungiwe already do to some extent. like the section about the icla links over to the openstack guide, while the openstack guide links to ours where account setup is concerned16:08
fungiabout 5 more jobs to complete before it starts on the lists jobs16:26
fungilists jobs are starting now. once both have completed i'll proceed with the maintenance (no need to wait for the rest of the deploy jobs to finish as they're for wholly unrelated services)16:41
fungi#status notice The and sites will be offline briefly for migration to a new server16:46
opendevstatusfungi: sending notice16:46
-opendevstatus- NOTICE: The and sites will be offline briefly for migration to a new server16:46
fungiip addresses were updated at 16:4816:49
opendevstatusfungi: finished sending notice16:49
fungiat 16:53 we should be safe to proceed (all prior records had a 5-minute ttl)16:49 and no longer respond to smtp or http(s)16:53
clarkbfwiw I am following along :)16:58
clarkbjust let me know if I can do anything to be useful16:59
fungiairship list imports are in progress now16:59
fungilog is being recorded in /root/import_airship.log on the new server16:59
fungikata list imports are in progress now. log is being recorded in /root/import_kata.log on the new server17:03
fungiand that's done, fiddling with django next17:06
fungidjango has been fiddled17:10
fungiupdating dns to point to the new server now17:10
fungidns updates applied as of 17:14 so we should be safe to proceed at 17:1917:14
funginote that as of now we should expect the migration to be done, we're just waiting for dns propagation before we test to be sure everything's working as expected17:15
clarkbyup the etherpad seems to reflect that its checking stuff at this point17:15
fungionce we perform cursory webui checks, then i'll send confirmation messages to lists on both sites and make sure they come through and appear in the archive17:16
fungidns propagation should be done, we can proceed with tests17:19
clarkbthe mm3 web ui stuff for both lgtm at a quick glance17:21
fungithanks. yeah i checked all the urls in the pad and they go where i expect and contain what i expect17:22
fungii'll send the completion notifications next17:22
fungimessages sent17:24
fungiand received17:24
clarkbmm3 archives show the sender time in the thread view. Unexpected but cool17:25
fungilinks to both archived messages are in the pad now17:25
clarkbI see both in the archive and in my inbox17:26
fungiso once the config deployed, maintenance was ~40 minutes17:27
fungii'll start writing the notifications for and once i get a snack17:28
fungisent them. also drafted a maintenance plan for next week, starts at line 198 in
fungiinfra-root: ^ when you get a chance, let me know if you have any concerns but it's basically a rip-n-replace of today's maintenance with adjusted server/site/list names17:57
fungii'll get the corresponding config change put together shortly17:58
clarkbfungi: the plan for today seemed to go well other than unexecpted timing (which isn'ta big deal imo)18:19
clarkblooks like you've given yourself time for that in the new plan18:19
clarkbso ya lgtm18:19
opendevreviewJeremy Stanley proposed opendev/system-config master: Move OpenInfra and StarlingX lists to Mailman 3
fungiand there's the config change for it18:28
fungiokay, other that reviewing that change, we're as prepared as we can be for next week's maintenance, i think18:41
opendevreviewMatthias Runge proposed openstack/project-config master: Add python-observabilityclient
clarkbtobias-urdin: I think you are involved with gnocchi things? In our Zuul config gnocchixyz/gnocchi is being accessed anonymously which hits rate limits when we do a big zuul restart like we did last weekend18:48
clarkbtobias-urdin: I would like to improve this. One way to do that is to have gnocchi install the opendev zuul app in github then we get the app isntall rate limits which are much higher. Additionally if you are willing to help us test things we could use gnocchi as a test case for limiting access to gnocchi for that app as its is largely only used on the openstack side for integration18:49
clarkbI think by default that app asks for permissions sufficient for managing the github project test more completely. But that shouldn't be required. Unfortuantely I don't personally have a good github setup/project to go testing this myself18:49
clarkbbut I thin gnocchi would be a good test case since it is I think actively used in downstream integration testing18:49
opendevreviewClark Boylan proposed opendev/system-config master: Convert haproxy-statsd to bookworm base image
clarkbthat change should be very straightforward and risk is low if something uenxecpted happens20:37

Generated by 2.17.3 by Marius Gedminas - find it at!