Friday, 2021-08-20

opendevreviewSteve Baker proposed openstack/diskimage-builder master: Move grubenv to EFI dir
opendevreviewSteve Baker proposed openstack/diskimage-builder master: Support grubby and the Bootloader Spec
opendevreviewSteve Baker proposed openstack/diskimage-builder master: RHEL/Centos 9 does not have package grub2-efi-x64-modules
opendevreviewSteve Baker proposed openstack/diskimage-builder master: Add policycoreutils package mappings for RHEL/Centos 9
opendevreviewSteve Baker proposed openstack/diskimage-builder master: Add reinstall flag to install-packages, use it in bootloader
opendevreviewSteve Baker proposed openstack/diskimage-builder master: Add DIB_YUM_REPO_PACKAGE as an alternative to DIB_YUM_REPO_CONF
*** ysandeep|away is now known as ysandeep05:29
*** rpittau|afk is now known as rpittau07:38
*** jpena|off is now known as jpena07:38
*** ysandeep is now known as ysandeep|lunch07:45
ykarelHi jobs are failing while installing boto3 in ci jobs08:43
ykarelERROR: Cannot install cinder== because these package versions have conflicting dependencies.08:43
ykarelexample log
ykarelon checking found that issue is when using infra mirrors08:44
ykarelpip install --index-url= -c boto308:44
ykarelwith mirror it fails, and without mirrors it works fine, can someone check how it can resolved08:44
*** ysandeep|lunch is now known as ysandeep08:53
fricklerykarel: for me this is working fine now when testing your command locally. maybe we had cached an index? seems the pkg is pretty new. I can't really spot the exact cause for the failure in the job log, either. please double-check whether you still see the issue09:27
opendevreviewSorin Sbârnea proposed zuul/zuul-jobs master: Include podman installation with molecule
ykarelfrickler, yes working fine now, but atleast some issue was there11:10
ykarelnot sure it was cache or something else11:11
ykarelwhen the issue was there, it was fetching 1.18.1 not even versions release after that and before 1.18.2411:11
*** dviroel|ruck|out is now known as dviroel|ruck11:26
fricklerykarel: I don't doubt that there was an issue, it's just that when it's gone now, it is difficult to debug it further11:30
*** bhagyashris_ is now known as bhagyashris11:31
ykarelfrickler, ack let's see if it get's reproduced11:31
fricklerykarel: sure, if you see anything similar again, please mention it here using the keyword "infra-root" to ensure we can have a look as soon as possible11:34
ykarelsure will do11:34
*** jpena is now known as jpena|lunch11:36
fungiykarel: keep in mind that pip's new dep solver will try earlier and earlier versions if it can't find a suitable wheel and then also fails to build something from sdist. what python version were you using, and do you have a log of the problem pip install attempt?11:48
fungiahh, you linked to an example log, let me try to figure out what the build result page was for that (the raw job logs are a pain to look through)11:49
fungiokay, so that was
fungiykarel: unfortunately that job doesn't collect pip's debugging logs like we do in tox-based jobs, so i can't really tell what failed to get it to that point11:55
fungiit doesn't seem to say why it refused to use the constrained version11:56
ykarelit was not found that's why?11:57
ykarelactually when it was reproduced locally, the latest version available was 1.18.111:58
fungione distinct possibility is the fastly cdn endpoint near inap-mtl was returning old copies of the pypi index, we've seen this occur in the past, though the only way we've managed to track it down is to excavate those copies from the apache cache and map them to retrieval times12:04
fungifor some reason it seems to hit most often in the montreal canada area providers12:04
fungiwhen it does happen, it's like 1 in 20 times when our proxy requests the page, the cdn returns very old instead of current content, which makes the problem seem to come and go at random12:06
fungiso at least in those cases the problem really is pypi (or their cdn anyway) but you'd only see it fail randomly if you were around that region of the world and repeatedly cleared your pip cache12:08
fungiif the problem does resurface, we can try to find proof that pypi is sometimes returning old index copies in some places, and have them try to flush those from their cdn network12:10
ykarelack will post here if see this again12:23
opendevreviewMerged opendev/elastic-recheck rdo: Make elastic recheck compatible with rdo elasticsearch
ykareli should have checked other providers at same time12:25
ykarelbut yes all the errors i seen were in inap-mtl0112:29
*** jpena|lunch is now known as jpena12:36
mordredTIL that git-review is in homebrew for the mac users: ... and the formula for it totally looks legit12:57
fungiand looks like it does work, judging from the gerritbot message in #zuul13:03
*** rpittau is now known as rpittau|afk13:55
opendevreviewJing Li proposed openstack/diskimage-builder master: Add new element rocky
*** ysandeep is now known as ysandeep|away14:31
opendevreviewSorin Sbârnea proposed zuul/zuul-jobs master: Include podman installation with molecule
opendevreviewClark Boylan proposed opendev/system-config master: Test on focal
clarkbfungi: ^ fyi it occurred to me that we also want to check the ansible is happy with focal.15:14
clarkbcurrently Non Interactive Users has proposal bot, openstack release bot, and usagestats in it15:25
clarkbgrepping for that group name in openstack/project-config shows no results so acl use would have to be in all-projects alone?15:26
clarkbpriority = batch group Non-Interactive Users says our documentation15:29
clarkbso the only extra privs they haev is using a different thread pool15:30
clarkbassuming we don't have undocumented use somewhere15:30
fungimakes sense, i think we ended up using a separate group to convey voting privileges for zuul15:47
fungiso it should be safe to include the different ci groups in non-interactive users, i guess15:48
clarkbYup and I think this is a good reminder that we can tune the thread counts to balance resources between CI systems and interactive users if necessary15:49
clarkbright now the bulk of the interaction is using the interactive pool15:50
clarkbbut we could split that (I suspect that the bots overall need more threads than interactive)15:50
*** jpena is now known as jpena|off15:51
opendevreviewSorin Sbârnea proposed zuul/zuul-jobs master: ensure-docker: enable centos-8-stream testing
opendevreviewSorin Sbârnea proposed zuul/zuul-jobs master: ensure-podman: enable testing of centos-8-stream
clarkbfungi: passed testing so no anticipated issues with the ansible on a focal listserv16:29
opendevreviewJames E. Blair proposed opendev/system-config master: Matrix-eavesdrop: handle notices
corvusclarkb, fungi, mordred, tristanC: ^ that should get us our missing feature from eavesdrop16:36
corvus(i just tested that locally)16:36
clarkbleft a note about thread safety as it came up before. tldr is file io isn't async io native so we don't risk being preempted and that should be safe16:41
corvusclarkb: i'm going to run some errands; hopefully that zuul change will land and i can restart after i get back17:04
clarkbsounds good17:05
mordredfor the gerritbox-matrix we're running - it doesn't look like we're doing nick lookup in like the one in - is there something else we need to configure to make that work?17:06
clarkbtristanC: ^17:08
clarkbmordred: is it looking up the nick of the person proposing changes?17:08
mordredyeah. if you look in the gerritbot channel, you'll see that some of the lines resolve into links to the matrix user id17:09
mordredcorvus is an excellent example case :) 17:09
corvusmordred, clarkb:
mordredcorvus: +A17:10
mordred(I note that you have created the needed token)17:11
clarkbHow is it able to do that reliably? I guess it would only have problems if there was a name collision in the room(s) gerritbot is in?17:11
mordredit uses matrix identity - so one can register and publish an email address to be associated with your matrix identity - and then the identity api can look up "who is the matrix user with this published email"17:13
clarkbah I didn't realize it allowed lookups by email like that17:13
mordredyah - but I believe only a 1:1 lookup in that direction. So - I have your email, what's your matrix id - the inverse doesn't work, I can't ask for all the emails your matrix id may have published17:14
mordredso you can opt-in to publishing your email as an identifying piece of data, or you can choose not to17:15
fungineat, though i guess it relies on you associating your preferred gerrit address with your matrix id? (you can no longer lookup secondary addresses in gerrit unless you're a full admin)17:17
corvusThe zuul matrix docs explain how to do that17:17
fungii'll make sure i do that17:18
corvusOne of the optional appendices17:19
fungithis weekend will likely be me getting around to replacing the shell server where i run weechat, since i would like to be on latest available when i try to get the matrix plugin for it working17:20
*** mgoddard- is now known as mgoddard17:43
opendevreviewMerged opendev/system-config master: Add gerritbot-matrix identity lookup configuration
opendevreviewMerged opendev/system-config master: Matrix-eavesdrop: handle notices
clarkbI just noticed that the infra-prod-service-eavesdrop job can run while the promote job for the image is running17:54
clarkbthat may result in us not actually updating the image17:54
fungioh, yep17:55
fungior, rather, updating it on a delay17:56
fungiwe won't get restarted on the new image until the subsequent infra-prod-service-eavesdrop build17:56
fungieventually consistent, but not ideal17:56
fungii guess that's input into our deployment dependencies rework17:57
opendevreviewClark Boylan proposed opendev/system-config master: Run service-eavesdrop after promoting the matrix eavesdrop bot
clarkbfungi: I think ^ will fix it now17:58
clarkbbut ya that becomes input to that whole thing17:58
clarkbone of the issues here is taht we sometimes do that in the pipeline and sometimes in the job definition. When we map this out coming up with a good set of rules for how to appraoch this would be good too17:59
tristanCclarkb: it seems like the matrix gerritbot is down, e.g. curl fails18:01
fungitristanC: it may be restarting?18:01
tristanCperhaps its related to 803396 , could i get a copy of the docker logs?18:01
fungitristanC: gerritbot-matrix: user error (Could not get hash details: MatrixError {meErrcode = "M_TERMS_NOT_SIGNED", meError = "Terms not signed", meRetryAfterMS = Nothing})18:03
clarkbwe need to accept's terms and conditions ?18:03
tristanCfungi: thanks, i missed that steps in the instruction18:04
fungilooks like it's been trying in a loop since 17:4718:04
opendevreviewTristan Cacqueray proposed opendev/system-config master: Add matrix term accept instruction
tristanCclarkb: yes, we would need to run these extra step ^  corvus has been runnning the one before from the bridge i think18:07
tristanCotherwise we can revert 803396 to restore the service18:07
clarkbpreviously we have had to accept terms and conditions for our hosted homeserver and its pretty straightforward. I expect that the tos for is similar18:10
corvusYeah I think we should run commands from  tristanC . I'm not on a terminal with access now.18:12
clarkbI can do it18:12
clarkb is the agreement which I'm skimming now18:13
clarkbreading that it almost seems like the bot itself is opting into the lookup18:15
clarkbeg to do lookups you yourself must opt into being look up able18:15
clarkbOur bots contact info should already be fairly public so I don't think that is an issue18:16
clarkbI don't see anything else on there that gives me any concern.18:16
clarkbcorvus: fungi  ^ you good with me accepting it now?18:16
clarkboh this might be the biggest issue: the lookups live in their logs for half a year and that would be regardless of whether or not a user has opted into the lookup discovery on their end18:23
clarkb is the linked issue which has been closed with an assertion that rotating logs is sufficient cleanup18:24
clarkbI'm personally ok with that, but want to amek sure others are before continuing18:24
clarkbtristanC: re ^ the bot will only do a lookup if it decides to post the notification to a channel? Will lookups be done for every event from the event stream?18:26
tristanCclarkb: only after the event match a channel18:30
clarkbthen ya I'm comfortable with it if other infra-root are18:30
tristanCclarkb: and the lookup is performed with a hash of the email18:31
tristanCthe bot is actually failing to start because of an error when getting the pepper used to salt the hashs18:33
clarkboh that is interesting the terms don't say anything about a hash but that makes sense18:35
clarkbin that case I'm really comfortable with it18:35
clarkbI'll give fungi and corvus another little bit to chime in but I'll go ahead with that if I don't hear back in like 10-15 minutes18:35
tristanCclarkb: i need to go afk for an hour, i'll check again when i come back18:36
fungiyeah, sorry, had to take a break to catch up on a couple of chores but looking now18:38
fungiclarkb: seems fine to me, i'm good with it18:39
clarkbcool corvus already mentioned accepting it above so I'll goa head18:41
clarkbok done. fungi was the container restarting in a loop? if so we should see it working again automatically?18:42
fungigerritbot-matrix_1  | 2021-08-20 18:42:40.830 [ThreadId 19]: Connecting to
fungii guess someone could push a dnm change or something18:43
fungito exercise it18:43
clarkbwe can approve and then it will report the merge18:45
clarkbI've +2'd the chagne as the commands worked for me18:45
clarkbfungi: ^ maybe double check it for bash correctness then +A?18:45
fungiyeah, done18:47
opendevreviewMerged opendev/system-config master: Add matrix term accept instruction
fungiwe got the irc report ^19:50
fungican someone monitoring matrix confirm it was echoed there by the new bot?19:50
clarkbfungi: yup we did19:50
fungiwas 805361 the change corvus was waiting to have in a new container before we restart?19:51
clarkbI think we're ready to restart as soon as corvus confirms he is ready19:52
fungicool, i didn't see anything else mergeable which was already approved anyway19:52
corvusi am back now19:59
corvusshould we go ahead and restart now?20:02
fungii'm ready20:02
fungii'll let #openstack-release know we're doing it shortl20:02
clarkbI'm here too20:02
clarkbthough I think I odn't have keys loaded on this device I can fix that quickly if necessary20:02
corvusfungi: lemme know when you're ready20:03
fungiyou're initiating it? i'm on hand to test and help troubleshoot20:04
corvusi pulled images to be sure.  and i'm restarting now20:04
abhishekkstandard zuul restart?20:14
corvus#status log restarted all of zuul on commit 919c5a36546117c4ad869ff9b580455970ecd26820:14
opendevstatuscorvus: finished logging20:14
fungiabhishekk: yep!20:15
fungijust getting new features/fixes in advance of the next release being tagged20:16
abhishekkfungi, ack, is it daily activity ?20:16
corvusabhishekk: it's as-needed20:16
abhishekkcorvus, ack, thank you20:17
corvusi believe we should see the lock cleanups after 60 minutes, so probably worth checking for log entries regarding those at 21:1520:18
fungiit's been a bit more frequent lately since there's a lot of work getting merged leading up to zuul 5.0.0 and the scale-out (redundant) scheduler capability20:18
corvus"Removing stale lock" is what we'll be looking for20:18
corvusand hopefully not "Error cleaning up locks"20:19
fungiabhishekk: good news is by zuul v5 these scheduler restarts will likely no longer be noticed (at least once we deploy a second scheduler and load balance the dashboard20:19
fungicorvus: noted, i'll check the logs20:19
corvuswe could add dedicated web servers now if we want.  that might be a good idea to get ahead20:20
abhishekkI noticed because, I just pushed couple of patches and suddenly those went out of the queue20:21
abhishekknow those are back in queue so no need to worry20:21
corvusre-enqueue complete20:22
corvusclarkb: did matrix-eavesdrop get restarted?20:22
corvus looks like yes20:23
clarkbcorvus: yup it seemed ot be in a fail loop and when the agreement was signed it startedup again20:27
corvuseavesdrop i mean20:27
corvus(not gerritbot)20:28
corvusbut it looks like they're both good, because the eavesdrop log includes a notice and the notice includes a matrix id :)20:28
clarkboh for eavesdrop I wasn't sure due to the issue that should fix20:29
clarkbthe promote and service jobs were running at the same time20:29
clarkbfungi: corvus  ^ a quick double check on that change would be good then we can land it to avoid races in the future20:35
corvusclarkb mordred tristanC fungi i'm making the room now -- i'm thinking for this question i should set "Anyone" (but the default is the second option, members since selecting this option).  i think Anyone might make it more compatible with room previews and the proposed matrix workflow of incrementally joining a room20:35
* corvus uploaded an image: (25KiB) < >20:35
corvus(i can't remember what they call that right now, but it's the idea that you can "join" a room anonymously, and then if you want to start talking, you can get a nick and make an account)20:36
corvusanyway, what do you think?20:36
clarkbdoes encryption play into that? I think in our case because we don't need the room to be encrypted the anyone option is fine?20:37
corvuscorrect, and the room will not be encrypted20:37
corvus(and yes, encryption would alter the available choices there)20:37
fungisounds like a neat feature, i'm in favor20:38
clarkbprobably the only other consideration is how do we prune history should that be necessary. With eavesdrop it is easy (we just edit it). I'm sure matrix has some tooling for this but I don't know what it is20:39
clarkband in most cases I think we would tell people to reset passwords instead of pretending they weren't disclosed20:39
clarkbbasically I can't come up with a reason to not set it to anyone for a public development focused channel20:41
fungi"pruning history" is a fiction with irc too20:42
fungiredacting things in the published logs didn't remove them from people's own client logs/scrollback20:42
corvusin #test @admin just deleted a message from @corvus20:43
corvusif you want to see what that looks like.  pretty easy to do in the web ui20:43
clarkbah ok so login as admin and select a message to delete, straightforwatd20:43
clarkbstilla good idea to tell epople to not treat that as truly deleted, but we have the ability20:43
clarkbat least as much as we do with irc20:44
fungiyep, general guidance is still "once public, always public" (can't put the cat back in the bag, beans back in the can, whatever idiom you prefer)20:46
mordredbeans in the cat?21:03
fungino "Removing stale lock" in /var/log/zuul/debug.log yet, though it's still a bit early. i need to run a quick errand but will check again when i return if nobody else gets to it first21:04
tristanCcorvus: anyone sounds good to me21:05
clarkblooks like users can delete their own messages too21:08
clarkbadmin only necessary to delete someone else's messages21:09
corvusfrom my own private element -- if i do "explore public rooms" and then "add server" and add "" i get a message saying i don't have permission to view the room list on that server.  but it works if i do "".   any idea what we're missing?21:09
corvusmordred, clarkb: ^?21:09
clarkbI don't recall anything related to room listing when setting up the homeserver. Sorry I don't know21:10
corvusi wonder if it could be related to the client well-known file21:11
clarkboh maybe?21:12
clarkbwe didn't do the client one because we need to serve it with apache to set the right cors header iirc21:13
clarkbgitea will serve it but with the wrong cors headers. Apache is there and we can serve the files with it if we choose to21:14
corvusi checked my homeserver log -- it tried to make a request to ''21:15
corvusthat seems like it might be well-known related21:16
corvusso sounds like apache may be the way to go21:16
clarkbsorry small correction we did do the client one, but gitea doesn't serve it with the correct cors header value21:17
corvuswhy is the apache proxy conditional?21:23
clarkbcorvus: we weren't sure we would need it to start but then the ddos persisted so we switched to it21:25
clarkbI think we can probably stop using it conditionally now and just assume we'll have it21:25
opendevreviewJames E. Blair proposed opendev/system-config master: Serve matrix well-known files from apache
opendevreviewJames E. Blair proposed opendev/system-config master: Assume gitea reverse proxy
opendevreviewJames E. Blair proposed opendev/system-config master: Remove matrix well-known files from gitea image
corvusclarkb, fungi, mordred: ^  ianw fyi21:34
clarkbcorvus: left a note on the second change that I think is worth double checking21:36
opendevreviewJames E. Blair proposed opendev/system-config master: Assume gitea reverse proxy
opendevreviewJames E. Blair proposed opendev/system-config master: Remove matrix well-known files from gitea image
clarkbturning off the auto preview for urls makes things more readable in element when dealing with gerrit urls21:42
clarkbanyone know what sort of protocol gerrit needs to support to do those previews properly?21:43
fungi2021-08-20 21:05:26,472 ERROR zuul.ExecutorQueue: Removing stale lock: /zuul/executor/unzoned/locks/e42339f7ad3844f499b6c4f4eeb8418f21:43
fungi5352 of those logged with different lock names21:43
opendevreviewJames E. Blair proposed openstack/project-config master: Remove gerritbot from #zuul
fungicorvus: ^ seems like it worked!21:43
corvusfungi: excellent -- now the next question is whether we will see any further ones21:44
clarkbslack supports a bunch of different "unfurl" methods21:45
opendevreviewJames E. Blair proposed opendev/system-config master: Move #zuul from OFTC to Matrix
clarkbtwitter and facebook use two different methods21:46
clarkbI suspect that Open Graph Data may be the thing though21:47
opendevreviewJames E. Blair proposed opendev/system-config master: Serve matrix well-known files from apache
opendevreviewJames E. Blair proposed opendev/system-config master: Assume gitea reverse proxy
opendevreviewJames E. Blair proposed opendev/system-config master: Remove matrix well-known files from gitea image
corvusclarkb: ^ updated based on your comment21:48
fungianother thing 805459 gives us is the option to stop using gitea itself to serve the main page. that can now be something served directly by apache21:49
clarkboh another though do we need to allow access to that doc root explicitly?21:49
clarkbfungi: corvus  ^ 21:49
corvushrm maybe21:50
corvusmaybe we can testinfra that21:50
fungigood idea21:50
corvusthis is literally what my ansiblefest talk is about so i better walk the walk21:55
clarkbfwiw it has been really helpful with testing things like the gerrit upgrade and downgrade between 3.2 and 3.3. Prior to that with getting the gitea upgrade for 1.15.0 sorted out21:56
clarkbre putting more data in the gerrit urls the content that is served is very limited and basically says preload a bunch of these api endpoints then run this script21:57
corvusif my talk has an audience you should be a plant :)21:57
opendevreviewJames E. Blair proposed opendev/system-config master: Serve matrix well-known files from apache
clarkbI wonder how difficult it would be for the gerrit server ti write out this metadata when so much of it is deferred to the client21:57
corvusclarkb, fungi: ^ how's that look?21:58
* fungi will be a thallophytic plant-like organism21:58
corvusfungi: you're fabulous21:58
opendevreviewTristan Cacqueray proposed opendev/system-config master: Update gerritbot-matrix to the latest version for on behalf annotation
fungi(fabulous fungi ii: absolutely fabulous fungi)21:59
clarkbcorvus: that latest patchset lgtm21:59
corvusi wasn't 100% sure what apache would do with the value of the cors header so i omitted the "*" from the check, assuming that presence would be sufficient21:59
corvuswe could tighten that up later if we want22:00
corvus(would the value be '""' or '' i dunno)22:00
clarkbya I think we won't get the header at all otherwise so just asserting it is there is probably sufficient22:00
corvusoh it ate my stars22:00
corvusanother weechat nit :)22:00
corvusweechat-matrix that is22:00
fungiso the matrix protocol expects *foo* to indicate some sort of emphasis, similar to _bar_?22:02
corvusfungi: no that's just matrix-weechat and doing a halfway job of dealing with it22:03
fungii expect i'll eventually get annoyed enough by that to hack some sort of escaping into the plugin22:03
mordredwe have 3x+2 on the apache change ... do I need to wait?22:04
*** dviroel|ruck is now known as dviroel|out22:04
corvusfungi: i should have typed: (would the value be "*" or * i dunno)22:04
clarkbthe testinfra change should test the bits that matter22:04
clarkbmordred: ^ that means it should be safe to approve and it will fail if it odens't work for some reason22:04
corvusyeah, i feel like with testinfra there we can bombs-away22:04
clarkbcorvus: going back to the separate zuul webs idea. Is there a reason to split them from the scheduler? or sould we plan to colocate those services when we run multiples of them?22:05
fungiagreed, i suppose there's a chance that we've all overlooked some error in the test itself which we might have caught by waiting for it to run and looking at the log, but i'm not overly concerne2d22:05
corvusfungi, clarkb, mordred: can you leave some votes on and ?  i have WIPd those and will +W them tomorrow22:06
corvusclarkb: probably depends on what we want our new scheduler size to be.  if we have an extra core or 2, then colocating them is probably a good idea for efficiency.  if we max out the cpu for the scheduler process then split hosts.  that's all i can think of.22:07
fungithough we could also consider that the web and fingergw processes are okay to leave as a spof because they're quick to restart22:08
fungias long as they're entirely decoupled from the scheduler22:09
clarkbthe scheduler is still largely only going to use a single cpu right? but do to memory needs and the way the clouds do flavors I think we should have extra cpu typically22:09
clarkbfungi: in my head every scheduler would also run a web and fingergw and then we can use haproxy or omething to point to them. But ya that is an option too I guess22:09
corvusyeah.  probably like 1.2 cpus.22:09
fungiclarkb: leaving web and fingergw as spofs also means not needing to add a lb22:10
mordredcorvus: I have +2d all the things - a few need to be rebased - want me to rebase them or want to wait?22:11
fungii don't know which i prefer, just thinking through it, but the reduced complexity might outweigh the gains of hiding occasional brief outages22:11
corvusmordred: oh i'll rebase now22:11
mordredalso - I think +A as you feel22:11
opendevreviewJames E. Blair proposed opendev/system-config master: Assume gitea reverse proxy
opendevreviewJames E. Blair proposed opendev/system-config master: Remove matrix well-known files from gitea image
corvusthe run-gitea job is running for the main change now22:12
corvusestimated time remaining 40m  oy22:12
mordredthat's a lot of m22:13
corvusso it'll be at least 1 hour 40m best case because of clean check :/22:13
corvuswhich is like 100m22:13
mordredthat's even more m22:13
corvustechnically an order of magnitude more m22:14
fungiso not exactly a 100m dash22:14
clarkbits also a bit faster now that we use the api I think22:16
clarkbbut ya it creates all the projects. We might be able to reduce that down to a representative sample now22:16
clarkbthe cost is almost entirely in create all the projects then create them again to ensure we noop22:16
opendevreviewClark Boylan proposed opendev/system-config master: Update Gerrit images to most recent releases
clarkbinfra-root ^ fyi I think landing and restarting on that can likely wait until Monday22:56
opendevreviewJames E. Blair proposed opendev/system-config master: Serve matrix well-known files from apache
corvusin about an hour, we'll find out the next typo :(22:59
opendevreviewMerged opendev/system-config master: Update gerritbot-matrix to the latest version for on behalf annotation

Generated by 2.17.2 by Marius Gedminas - find it at!