Thursday, 2023-10-12

opendevreviewDr. Jens Harbott proposed opendev/puppet-openstack_health master: Retire opendev/puppet-openstack_health: remove repo content
opendevreviewRadosław Piliszek proposed openstack/project-config master: Add nebulous/monitoring
opendevreviewJan Marchel proposed openstack/project-config master: Add NebulOuS repos: exn and gui
opendevreviewMartin Magr proposed openstack/project-config master: Complete config for python-observabilityclient
opendevreviewMerged openstack/project-config master: Add NebulOuS repos: exn and gui
yoctozeptooooh, it seems the "c++" broke the deploy12:23
yoctozeptoI think we should change it to cpp12:23
yoctozeptofungi ^12:24
yoctozeptowhat's the procedure now?12:24
fungiyoctozepto: i'll take a look, but yeah maybe we need to limit what punctuation is allowed in repository names12:36
Clark[m]It looks like nothing has been created for it? If we can verify that (I don't trust my lookups this early on the morning) then we may just be able to update the name in a change and proceed from there. If anything got created we will need to do cleanup/renaming12:51
fungiyeah, that's my hope, i'm just trying to get dug out from morning tasks so i can take a closer look12:55
fungigitea servers rejected the create request with "422 Client Error: Unprocessable Entity for url: https://localhost:3000/api/v1/orgs/nebulous/repos"13:03
fungiso i think it bailed before trying to add anything to gerrit13:03
fungiyeah, the "Create repos on gitea servers" task is the very first thing in the manage-projects playbook, so basically nothing was done, therefore nothing to clean up13:05
fungiyoctozepto: feel free to proceed with a change to swap the project name to one without "+" symbols, i'll start looking at adding a ratchet for the projects list to limit acceptable characters13:05
fungiinfra-root: config-core: heads up that "+" in project names won't work (at least for gitea), and also please don't approve any other changes touching acls or the projects list until we get the cleanup patch landed13:07
Clark[m]fungi: and you confirmed it never ran against gerrit?13:11
Clark[m]Excellent should be straightforward to fix then13:12
fungithe playbook stopped at the failed gitea task, which is the very first task13:12
fungiwe should pass all project names through an orwell filter13:13
opendevreviewMerged opendev/system-config master: Move OpenStack lists to Mailman 3
fungiearly maintenance prep is done once that ^ deploys, so we should be all set for starting on time in a little over 2 hours from now13:16
fungireminder, maintenance plan is at starting at line 285 now13:17
opendevreviewJeremy Stanley proposed openstack/project-config master: Check proposed project names for problems
fungithat ^ should fail on the tip of master, and should succeed once rebased onto yoctozepto's cleanup change (once it exists)13:36
opendevreviewMerged opendev/puppet-openstack_health master: Retire opendev/puppet-openstack_health: remove repo content
* yoctozepto going to help fix13:52
opendevreviewRadosław Piliszek proposed openstack/project-config master: Fix c++ to cpp
opendevreviewRadosław Piliszek proposed openstack/project-config master: Check proposed project names for problems
* fungi is fixing the flake8 complaints for 89811813:59
yoctozeptoI also reviewed your change14:00
yoctozeptoyou might want to improve that minor bit there14:00
fungioh, sure14:00
fungiyoctozepto: why are the continues a problem? we already short-circuit the loop on a different error condition, i'm just adding two more possible ones14:01
yoctozeptofungi: that one was necessary because the other checks (which check the name) could not run sensibly14:01
yoctozeptothese simply prevent the other name checks from happening at the same time14:02
yoctozeptoor actually this has really affected the logging14:03
yoctozeptoas without the name it did not make sense14:03
yoctozeptononetheless you don't need the continues now14:03
yoctozeptoand you get all errors at once14:03
fungithe check for slashes could still lead to problems for other checks after it i think?14:04
fungithough looks like it does nothing with the repo_group part14:05
fungiand repo_name is only checked against one thing14:05
fungiso i agree, dropping the additional continues is reasonably safe14:06
opendevreviewJeremy Stanley proposed openstack/project-config master: Check proposed project names for problems
fungithanks again!14:08
yoctozeptoso let's merge :-)14:10
fungiyep, i just approved the fix14:14
Clark[m]We can also run system-config-run-gitea against projects.yaml updates if we want to check gitea's acceptance of names14:22
fungiyep, but i'm more worried about choices that cause less-immediate breakage14:23
fungiit's good that gitea at least rejected that at the api rather than going ahead and creating something that would have had subtle problems resulting in a future rename maintenance14:24
opendevreviewMerged openstack/project-config master: Fix c++ to cpp
fungijust under an hour until maintenance time14:32
fungii've added some approximate times for the longer steps in the maintenance plan, based on earlier test import timings and such14:40
fungithe longer steps involve ~3 hours to complete, so that leaves us only an hour of fudge time in the 4-hour window14:41
fungiyoctozepto: "Build succeeded (deploy pipeline). infra-prod-manage-projects [...] SUCCESS in 4m 03s"14:43
* clarkb is finally at the proper computer15:03's ssl cert expires in 27 days or less. I don't think we care to debug that as we are switching servers15:10
fungiindeed, chuckled at that as well15:12
clarkb is a quick change to align jeepyb's gerrit builds with the new base image for gerrit15:15
fricklerthe manage-projects failure on the lists change is harmless I think? not sure why it ran there at all 15:17
clarkbfungi: ^15:17
clarkbfrickler: the error is likely the c++ error15:18
clarkbI don't know why it ran on a lists change15:18
frickleryes, just making sure nothing important was skipped15:18
clarkbit should recreate all the things when it ran with the cpp fix. We don't do deltas and instaed operate on the whole set (in part for this raeson it is really difficult to know what the state is across ansible runs)15:19
fungicorrect, i checked it, the lists server deployment succeeded, just manage-projects failed and it's not relevant to the maintenance15:21
fungi(also it ran successfully since then)15:22
fungistarting maintenance in 1 minute15:28
fungi#status notice The site will be offline over the next few hours for migration to a new server15:30
opendevstatusfungi: sending notice15:30
-opendevstatus- NOTICE: The site will be offline over the next few hours for migration to a new server15:30
fungidns records updated as of 15:31, we can proceed with service shutdown as early as 15:3615:31
fungiwe're at line 292 now15:32
clarkbgood reminder I should open the etherpad15:32
opendevstatusfungi: finished sending notice15:32
fungithe penultimate rsync is underway in a root screen session on lists.openstack.org15:33
fungii've also got a root screen session open on for running the migration commands once we reach that step15:34 is resolving to the new addresses for me15:36
fungias soon as this penultimate rsync concludes, i'll stop and disable mailman services on the old server15:37
opendevreviewGhanshyam proposed openstack/project-config master: End gate and update acl for retiring js-openstack-lib
fungiby "new addresses" i mean the temporary addresses15:37
clarkbfungi: I see updated resolution to the temp address as well15:38
fungifor some reason the penultimate rsync is taking waaaay longer than ones i performed in the past hour15:42
fungidone in 13m7.536s15:43
fungiservices stopped/disabled and final rsync in progress now15:44
fungiso much variability in these rsyncs15:49
fungithat one was at least only 5m55.615s15:50
fungimoving on15:50
fungiimport is in progress now15:51
fungiprior tests on held nodes took nearly 2.5 hours, but this server is probably faster so it could finish sooner than that15:51
clarkbwe can hope it is faster anyway :)15:52
fungiregardless, i'll do something else while i keep an eye on the progress15:52
fungiload average on the old server is finally sub-0.115:53
fungiup 475 days since last reboot15:54
clarkbI'll be glad to not worry about kernel compression anymore15:54
fungiyes, i'd be lying if i said it kept me up nights, but it will be nice to have one less oddity to care about15:55
fungifor those following along, we're at line 301 in now15:55
fungiand will be for... a while15:55
fungitime estimate is probably spot on. at least the archive import for openstack-discuss took 19m51.119s and when i did the initial timings a year ago that step was 18m42.199s16:34
fungia little more content in the archive, a little more time to finish importing16:34
fungiso probably getting to our final steps around 18:20z16:35
fungiunless something else is way faster16:35
clarkbsomeone might press the turbo button on the front of the server :)16:52
fungii'm a little worried though, the archive reindex for openstack-discuss took 43m23.527s just now vs the 6m41.204s i saw in testing17:12
clarkbthats the biggest one though?17:13
clarkbso even if it is slower hopefully its within a manageable time frame17:13
fungimaybe database contention with the other active lists which i didn't account for in testing17:13
fungiyeah, it's by far the largest17:13
fungiclarkb: i lied (or, well, forgot)... the biggest one is openstack-stable-maint because of the previously hundreds (now merely dozens) of periodic job failure notifications it receives every day17:48
fungiit's currently 22% into importing the mbox for that now17:49
clarkbdoes it run them serially or concurrently? Hopefully serially which would mean this particular one is going faster having not run since the start?17:51
clarkbok so once this one is done we'll hopefully finish up quickly17:52
* yoctozepto looking for approvals on
fungilooks like it probably takes an hour for the import step of this list and it's maybe 20 minutes in, so probably shouldn't expect it to move onto indexing until 18:3017:52
* yoctozepto sorry for interrupting otherwise17:53
fungithough there aren't really any other lists after this one (it's going alphabetically by list address), so should still hopefully finish on time17:53
fungiyoctozepto: yep, i was going to go through a few more pending project-config updates shortly17:53
clarkbfungi: I wonder if the speed difference is in the root disk. I think the test nodes had an ephemeral local drive but this one doesn't?17:54
clarkbanyway probably not worth worrying too much about at this point. Just need patience17:54
fungiyeah, i wouldn't be surprised at i/o bottlenecks there. and different phases of this process are probably impacted in different ways17:55
clarkbyoctozepto: the repo has a .DS_Store file is that intentional?17:56
fungimacos to the rescue17:57
yoctozeptoclarkb: yeah, well, it is like it is18:01
fungithere are 3 more lists after openstack-stable-maint but they're relatively small (openstack-zh, release-announce, release-job-failures)18:05
fungiof those, release-announce is probably going to take the most time18:05
fungii don't have relative timing data for all the mailing lists though, so hard to say exactly how long to expect18:06
opendevreviewMerged opendev/system-config master: Remove ara from source install option
opendevreviewMerged openstack/project-config master: Report Vitrage changes to stable branches in the IRC
clarkbfungi: how are things looking an hour later?18:47
fungiindexing openstack-stable-maint took 61m58.525s19:02
fungier, importing i mean19:02
fungiindexing is running now19:02
clarkband -discuss the import was half the time of the indexing?19:02
fungihopefully will finish quickly? hard to say19:03
clarkbprobably still on track to get done before the day ends, but take longer than we expected? May be worth doing a #status notice (can't send email heh)19:03
fungiopenstack-discuss: import21=1m12.552s hyperkitty_import=19m51.119s update_index_one_list=43m23.527s19:05
clarkbfungi: is that 62 minute time for import the two import steps? In which case we should expect about another hour maybe?19:06
clarkboh wait thats import21 not import 21m19:06
clarkbso ya ~2x19:06
fungiopenstack-stable-maint: import21=0m5.389s hyperkitty_import=61m58.525s update_index_one_list=?(in progress)19:06
fungion a held test node, update_index_one_list took less time than hyperkitty_import, so different resource constraints19:07
fungiopenstack-stable-maint: import21=0m5.389s hyperkitty_import=61m58.525s update_index_one_list=29m24.880s19:08
fungijust finished19:08
fungiit's on to openstack-zh19:08
clarkbI guess you can't really infer from one list to another19:08
clarkbthat stable maint list is probably also fairly unique in that the data in it has a lot of overlap and consistent senders etc19:08
fungiright, apples and oranges19:10
fungiit's already on release-announce19:10
fungione more after this19:11
fungiwe may still finish roughly on time19:11
fungithe remaining steps go quickly19:11
fungii've prepped the completion e-mail which will serve as the delivery test, so should be able to send it as soon as dns updates propagate when we get to that step19:28
fungialso i'm logged into everything that needs to be logged in for the remaining steps19:32
clarkbI'm here ready to assist should that be necessary19:34
fungilooks like lists01 has not been added to cacti yet19:39
opendevreviewJeremy Stanley proposed opendev/system-config master: Remove the old mailing list server
opendevreviewJeremy Stanley proposed opendev/system-config master: Add Mailman v3 server to Cacti
opendevreviewMerged openstack/project-config master: Remove ara from Zuul config
clarkbfungi: I left a couple of notes on that old list server cleanup change19:51
opendevreviewMerged openstack/project-config master: JayF volunteering for IRC ops
clockworkyi see that on the main page it says i can host my project on here, but how would i do that?19:55
opendevreviewMerged openstack/project-config master: End gate and update acl for retiring js-openstack-lib
opendevreviewMerged openstack/project-config master: Add nebulous/monitoring
clarkbclockworky: there is a large probably too long document on the process at because it tries to cover all the details.19:56
clarkbI was just a few seconds too slow...19:56
fungispeaking of slow, archive indexing for release-announce is still going20:01
fungii think the archive import time depends on the number of messages, while the search indexing time has more to do with the volume of text inside them20:03
fungimaybe release announcements should be made shorter ;)20:05
clockwor2yclarkb: saw your reply, changed irc clients because the one i was using was a bit too slow20:06
clockwor2ythx :)20:06
clarkbclockwor2y: oh good. The short version of it is that you push a change to a specific repo with some details about your project and when that merges things get created for you20:07
clarkbclockwor2y: is an example of that20:07
clarkbin general projects need to be OSI licensed and you should probably have some understanding of how Gerrit works (it isn't a PR model but a change model)20:07
clarkbjust to make sure that Gerrit is appropriate for you20:07
clockwor2yworked with gerrit back when i was messing around with a mediawiki skin in wikimedia, i liked it pretty well, and my project is OSI licensed20:11
clarkbbeyond that we encourage people using the systems to at least have some familiarity with how they are hosted. Part of the reason we haven't tried to hard to get away from the project creation process we have. Those of us operating the systems are also its users and if we can grow that user and operator base all the better20:13
fungi898088 is an incomplete example because it's relying on an already existing acl20:13
clarkbfungi: good point20:13
fungi#status log Archive imports for are taking longer than anticipated to complete... revised maintenance conclusion estimate is 21:00 UTC20:20
opendevstatusfungi: finished logging20:20
fungithe search indexing seems to be very i/o bound, top says most of this is iowait20:23
fungiit's spread across multiple cpus, but not using more than 1 in aggregate, which is odd20:23
fungiprobably some weird behavior for linux kernel scheduling with i/o-bound single-threaded processes20:24
clarkband this is the announce list? I wouldnt' ahve expected it to have a ton of content to index20:26
fungirelease-announce, yes20:26
fungii blame doug for scripting all that amusing variation into openstack release announcement messages ;)20:26
fungialso possible the timing differences we're seeing can be chalked up to the "noisy neighbor" phenomenon20:28
clarkbfungi: does it give a completion percentage for indexing? I think you indicated imports do20:30
fungiimports do, indexing does not unfortunately and that's what we're waiting on20:31
fungiand finally, release-job-failures is importing (i guess it's a good thing openstack's release jobs fail very infrequently and the buildset result messages from zuul's smtp reporter are brief)20:43
clarkbis this the last list?20:43
fungii believe so (alphabetically, which seems to be how they're processed)20:44
fungiit's already onto building the search index for it20:44
fungimoving onto django config20:48
fungidns updates applied as of 20:5120:51
fungishould be able to start testing at 20:5620:51
* clarkb waits patiently to load https://lists.openstack.org20:51
opendevreviewMerged openstack/diskimage-builder master: Change default value of DIB_DEBIAN_ALT_INIT_PACKAGE
opendevreviewMerged openstack/diskimage-builder master: Stop creating default user for cloud-init
clarkbit is loading for me and I can see things in the service-discuss archive that I expect to be there20:56
fungiyeah, all the urls in the pad do as expected. sending completion message to test a list20:58
clarkbI have recieved your test email20:59
fungias have i20:59
clarkbit is also in the archive21:00
fungiyep, test checklist complete21:01
clarkbanyone know how to play taps for the old server when we shut it down?21:01
fungiwe'll have to send out a call for trumpet players21:02
fungiand now i'm overdue to switch to making pizza21:02
clarkbmmm pizza21:02
fungiwill check back in a while to see whether deferred messages are trickling in21:02
fungialready one from 20:36:35 utc has been delivered21:03
clarkbyup I see it too21:03
clarkbfungi: should we put lists.o.o in the emergency hosts lists to prevent anything from restarting services there?21:14
JayFlookin' good!21:18
JayFIt's very strange that it has you create a username, then you login with the email, but the prompt on the login page is ambiguous ("login", not username or email)21:20
Clark[m]I think it will accept both on login21:21
JayFthat does not match my experience, at least if usernames have consistently applied case (I signed up as JayF and logged in as JayF; I guess if it's lower() on signup but not login it could fail though)21:22
Clark[m]I mean you can use username or email to login. I don't know about case handling21:23
Clark[m]I don't remember how I login. It's however keepassXC tells me to21:23
JayFusername works but lowercase only21:23
fungiclarkb: it's already in the emergency hosts list since around 12:30 utc per the maintenance plan, and will stay there until we remove it from the inventory22:05
fungibut also i set the services to disabled after stopping them, just to be extra sure in case of reboots22:06
fungirelated, whatever the problem was with ansible scraping our xenial pip ppa earlier in the week, seems to be solved now22:44
fungiglad i put off looking any deeper into that situation22:45
clarkbthe best kind of bugfix is the one that happens magically22:47
opendevreviewMerged opendev/system-config master: Add Mailman v3 server to Cacti
clockwor2yforgot to go in command mode23:20

Generated by 2.17.3 by Marius Gedminas - find it at!