Monday, 2022-01-24

*** rlandy is now known as rlandy|out00:15
opendevreviewNeil Hanlon proposed openstack/diskimage-builder master: Add new container element - Rocky Linux
opendevreviewHarald JensÃ¥s proposed openstack/diskimage-builder master: dhcp-all-interfaces: opt let NetworkManager doit.
opendevreviewNeil Hanlon proposed openstack/diskimage-builder master: Add new container element - Rocky Linux
opendevreviewIan Wienand proposed opendev/grafyaml master: Generate and use UID for acessing dashboards
opendevreviewIan Wienand proposed opendev/system-config master: grafana: update to oss latest release
fzzf[m]<clarkb> "fzzf: you need to ensure you..." <- Hi, Is there any way to only monitor the events of the manila project?  I have added a lot projects in the zuul tenant configuration.02:03
fzzf[m]I use zuul as CI, but after submit manila, CI doesn't seem to be triggered, where to check02:05
*** melwitt is now known as Guest34402:26
*** chkumar|rover is now known as chandankumar03:03
*** bhagyashris is now known as bhagyashris|ruck03:04
opendevreviewIan Wienand proposed openstack/diskimage-builder master: Switch 9-stream testing to use opendev mirrors
opendevreviewIan Wienand proposed openstack/diskimage-builder master: Add debian-bullseye-arm64 build test
opendevreviewIan Wienand proposed openstack/diskimage-builder master: Add 9-stream ARM64 testing
opendevreviewIan Wienand proposed openstack/diskimage-builder master: debian-minimal: remove old testing targets
opendevreviewIan Wienand proposed openstack/diskimage-builder master: Switch 9-stream testing to use opendev mirrors
opendevreviewIan Wienand proposed openstack/diskimage-builder master: Add debian-bullseye-arm64 build test
opendevreviewIan Wienand proposed openstack/diskimage-builder master: Add 9-stream ARM64 testing
opendevreviewIan Wienand proposed openstack/diskimage-builder master: debian-minimal: remove old testing targets
*** chandankumar is now known as raukadah05:54
opendevreviewDr. Jens Harbott proposed opendev/system-config master: Add docs for restoring an etherpad
fricklerclarkb: ^^ I adapted that from a comment from you that I found in my irc logs, please doublecheck06:53
*** bhagyashris is now known as bhagyashris|ruck07:05
*** ysandeep is now known as ysandeep|afk07:07
*** ysandeep|afk is now known as ysandeep07:45
opendevreviewMichal Nasiadka proposed openstack/diskimage-builder master: Add rocky-container element
*** jpena|off is now known as jpena08:36
*** ysandeep is now known as ysandeep|brb08:37
*** ysandeep|brb is now known as ysandeep08:44
*** odyssey4me is now known as Guest38009:43
*** bhagyashris_ is now known as bhagyashris|ruck09:52
*** rlandy|out is now known as rlandy|ruck11:14
*** dviroel|out is now known as dviroel11:22
rlandy|ruckhello - anyone with merge rights on project-config, can you take a look at
*** marios_ is now known as marios11:37
opendevreviewEduardo Santos proposed openstack/diskimage-builder master: Fix openSUSE images and bump them to 15.3
*** ysandeep is now known as ysandeep|afk12:07
rlandy|ruckfungi: thanks for the review of
fungisure thing12:47
*** ysandeep|afk is now known as ysandeep13:13
*** amoralej is now known as amoralej|lunch13:18
*** amoralej|lunch is now known as amoralej14:08
*** rcastillo|out is now known as rcastillo|rover14:09
opendevreviewNeil Hanlon proposed openstack/diskimage-builder master: Add new container element - Rocky Linux
*** bhagyashris is now known as bhagyashris|ruck14:35
opendevreviewNeil Hanlon proposed openstack/diskimage-builder master: Add new container element - Rocky Linux
*** dviroel is now known as dviroel|lunch15:01
*** ysandeep is now known as ysandeep|out16:05
*** dviroel|lunch is now known as dviroel16:14
clarkbfrickler: lgtm thanks!16:16
clarkbanyone know if we have a change to flip the gerrit image to 3.4 in system-config yet? If not I can get that up. I'll also work on going over the etherpad, adding a reboot step and trying to combine the renames and upgrades abit more16:17
*** Guest344 is now known as melwitt16:18
clarkbinfra-root I'm working on updating the etherpad plan for today's gerrit work and one slightly awkward thing is landing the chagnes to reflect the new realities. For manage projects we land the projcet-config updates after we rename. ianw has proposed in his upgrade doc that we land the 3.4 gerrit image update change first since we don't auto restart/upgrade gerrit.16:29
clarkbThe problem is we'll have gerrit in the emergency file or we need to take the time in the middle to ensure that project-config runs successfully before proceeding with the upgrade. The doc I've got punts everything to the end but if we want to pause in the middle which may take some time we can do that instead16:30
fricklerclarkb: hmm, is it possible that the etherpad revert is no longer working? wget doesn't give any output, trying with curl I see: {"code":3,"message":"no such function","data":null} and the pad seems unchanged16:32
fricklertrying to revert to
clarkbfrickler: oh I think your url may need to specify the api version16:32
clarkbfrickler: it is likeyl that v1 doesn't have that function but a newer version does16:33
clarkbfrickler: ya api 1.2.11 adds that function. I think you can replace 1 in the url with 1.2.1116:33
*** amoralej is now known as amoralej|off16:36
fricklerclarkb: yes, that did the trick, thx. I'll update the docs patch with it16:37
opendevreviewDr. Jens Harbott proposed opendev/system-config master: Add docs for restoring an etherpad
opendevreviewMerged openstack/project-config master: Adding nested-virt pools for centos-9-stream
clarkbheads up Gerrit is asking if anyone uses/relies on the file is reviewed functionality in the web UI. Sounds like the number of queries that makes can make gerrit quite slow for some changes. I'm letting them know we didn't migrate the DB when we moved servers last and no one complained.16:57
clarkband that removing it would simplify our CI as we now have to check it works due to previous issues with the mariadb jdbc connector16:57
*** rlandy is now known as rlandy|ruck17:11
fricklerclarkb: oh, so far I think I only heard about the marks getting lost during an upgrade, not about removing the feature completely. it sure is helpful for me sometimes, a) by showing which files were unchanged from a previous PS and reviewed there, b) allowing for easier restart if I had to interrupt a review halfway through17:12
noonedeadpunkhey! We have some issue while trying to drop centos-8 jobs - might be you can suggest some solution.... So our Xena patch fails on W having jobs and V failing because of X 17:18
noonedeadpunkI'm not sure I understand what's really wrong with them17:18
noonedeadpunkah, we likely get job mentioned elsewhere....17:20
noonedeadpunkI can't find where though17:21
noonedeadpunkalso a bit weird that it reference cross-branch17:22
clarkbfrickler: I can followup with those pieces of info if you like17:23
clarkbnoonedeadpunk: usually when that happens it implies there is a stray branch: matcher somewhere17:27
clarkbnoonedeadpunk: I see that your ubuntu bionic jobs have bad branch matchers for example. But I haven't found any for centos 8 yet17:28
noonedeadpunkyou mean these experimantal ones?
* noonedeadpunk wishes codesearch searching on specific branches17:32
clarkbnoonedeadpunk: yes everye one of your stable branches define that job as applying to devel and master which means when the job runs on devel and master it is going to get config from every stable branch17:33
clarkbwhich is almost definitely not what you want and that causes problems like the one you are seeing now with the other jobs17:33
fungiyeah, branch matchers in a multi-branch repository is almost never what you want. either use branch matchers in another single-branch repository, or only put the configuration in the branches where you want it to apply17:38
clarkbI'm still not sure why the specific complaints above are happening though17:39
*** jpena is now known as jpena|off17:40
clarkbfungi: if you get a chance can you look over the gerrit downtime etherpads and if possible form an opinion on whether or not we should do the renames first and complete that with successful runs on project-manage or if we should try and combine them a bit more and do the project-manage and merging post upgrade17:40
clarkbI think that is the biggest open question for me today on the gerrit stuff17:40
noonedeadpunkwe can easily drop this job from every branch if it's the one that causing issues. I believe when implementing, branches there were aimed to reffer this extra defined required-project but I doubt that this setup ever worked or was used17:43
jrosseron W we use the centos-8 job but only define the centos-8-stream one17:45
noonedeadpunkwe've dropped that with ?17:45
clarkbjrosser: noonedeadpunk ya I am beginning to suspect that creating the variant in the project-template causes zuul to do lookup resolution for that job and it finds it in xena and uses that definition17:45
clarkbif you were to leave off the voting: false then zuul should say the job doesn't exist or just ignore it I think. But since you create a variant in the template itself the job is resolved17:46
clarkband once you remove it from xena it can no longer resolve that job17:46
jrosseri think we have tripped over this before17:47
jrosserit would be nice to be able to choose that jobs were not visible across branches and just failed hard straight away17:48
clarkbI think that is the default before for most of the zuul config. But creating a variant is like creating a new job17:48
clarkbthe implicitness of the variant is what is confusing17:48
jrosserahhh ok17:48
clarkbwhen you say voting: false there you create an entire new job config for wallaby17:48
clarkbwhich is different than saying just run this job. Because in the just run this job cause it would fail since the job doesn't exist17:49
clarkbin the make me a new job case it does its best to make the job and then it becomes runable17:49
clarkbThat said maybe the variants shouldn't be so greedy across branches17:49
clarkbcorvus: ^17:49
noonedeadpunkclarkb: thank for help and your time!17:54
*** prometheanfire is now known as Guest41417:54
clarkbyou're welmcome18:01
*** sshnaidm is now known as sshnaidm|afk18:05
opendevreviewClark Boylan proposed opendev/system-config master: Manage apt.conf.d/20auto-upgrades
opendevreviewClark Boylan proposed opendev/system-config master: Replace 10periodic with 20auto-upgrades
clarkbinfra-root ^ that is followup on a thing we discovered last week18:23
clarkbI'm not 100% sure I got the 10periodic -> 20auto-upgrades replacement in debuntu packaging correct so I split the cahgnes in two and reviewers can double check that. I'm happy to squash the cahnges if we prefer too18:23
clarkbBut the first one should be very safe to land as it reflects the state of most of our servers18:24
clarkbfrickler: gerrit has clarified that it is only the automatic setting of the reviewed flag on files that they are looking at changing. I guess you would be able to make it explicitly if you like18:29
clarkbfrickler: is the automatic flip important to you are is it sufficient to flip it manually?18:30
*** Guest414 is now known as prometheanfire18:47
opendevreviewClark Boylan proposed opendev/system-config master: Upgrade Gerrit to 3.4
opendevreviewClark Boylan proposed opendev/system-config master: Manage apt.conf.d/20auto-upgrades
opendevreviewClark Boylan proposed opendev/system-config master: Replace 10periodic with 20auto-upgrades
clarkbI missed the testinfra testing for those files. Should be better now19:17
fungiclarkb: sorry, i've been pulled in 50 different directions today. looking over the upgrade plan again now19:26
fungiwe're planning to start downtime at 2200z right?19:26
clarkbyes 2200z is what we emaile dabout19:26
clarkbI think the big open question is whether or not we're trying to combine steps for the two different actions or if we want to run the rename to full completion first then start the upgrade after19:27
clarkbMy only concern with doing them as completely separate tasks is the amount of time we might have to deal with with zuul. I suppose we can always dequeue/enqueue in order to speed things up though so maybe doing one then the other each to completion is fine19:27
fungiwhere "full completion" means bringing gerrit back online and waiting for changes to merge and deploy?19:28
clarkbthe alternative is wait for gerrit to come back online and spot check the rename was fine. Then do the upgrade. Then merge and deploy the upgrade and rename changes together19:28
clarkbBut it would also be good for people to double check the rename input file in the opendev/project-config change and review the changes to land after the rename and the upgrade image change change19:29
fungiand i guess the reason we don't want to do the rename after the upgrade is that it complicates any potential version rollback?19:31
clarkbalso 3.4 has never had renames done in production. We do test renames against 3.3 and 3.4 though19:33
fungiboth very good points19:33
clarkb(we have done one rename on 3.3)19:35
clarkbOverall I'm reasonably confident in both tasks. More just concerned about how best to combine them.19:37
clarkb is the etherpad if anyone needs it19:39
fungiso given all the above, i agree rename first and then upgrade is the way to go, we're not stopping/starting zuul so i suppose the runtime for merging the changes won't be bad, though the deploy to run manage-projects likely will be dicey19:40
fungithe manage-projects run should be a noop, so it's really the config update we care about, right?19:42
clarkbyes its mostly just a sanity check to ensure that the change lands to reflect the new state and that when we run manage-projects it noops19:42
fungimaybe we could speed it along by manually updating the projects.yaml?19:42
clarkbWe don't expect it to get any real work done, just confirming it runs as expected19:42
fungiwe could manually run manage-projects too19:43
clarkbYa I think what we could do is dequeue the hourly jobs. Then when we force merge teh change it should enqueue and run right away?19:43
fungias long as we don't have half a dozen other deploy jobs triggered by the same merge which have to run before it19:44
clarkbI don't think we will it should just be review and manage projects iirc19:44
clarkbit is triggered by project-config which has very lmited deploy actions19:44
clarkbSo ya maybe the thing to do is run the rename to completion and manually dequeue the hourly jobs if necessary, remove the hosts from emergency, force merge project-config update, ensure it runs happily, then put review back into emergency?19:45
clarkbThen we can roll through ianw's process pretty much as is19:45
fungiin theory we could simply approve the project-config change so that it can merge normally, but if you want to bypass zuul to merge it then that does save a bit of time19:47
fungiand yes, the real risk is that we end up merging some other change to projects.yaml first which triggers a manage-projects run and recreates the old project names19:48
fungiif we're careful not to approve any other changes then it's just a question of whether we want to incur the gating delay19:48
clarkbI think we do have to force merge it because zuul won't know about the new project yet (I don't know that merging the chagne will trigger zuul jobs either)19:48
clarkbfungi: well its not just that because the other option is to do it after the 3.4 upgrade19:49
clarkbwhich is msotly what I'm wondering about. Should we try to combine the post change merges for the rename and upgrade for after both are done19:49
clarkbor do them sequentially19:49
clarkbI think force merging is a given for the project-config change either way19:49
fungithe project-config change which updates projects.yaml is already mergeable19:50
fungiupdates gerrit/projects.yaml i mean19:50
fungithe zuul.d/projects.yaml change is separate, and yes zuul will need to find out about the project rename for that to work19:51
clarkboh right19:51
clarkbya its the second change that would need to wait and we should merge the second chagne normally19:51
fungiin theory we could merge both normally, it's merely a question of whether we want to wait for gating on the first one19:52
fungi(or on either one for that matter)19:52
clarkbya I think I'm coming around to lets do the rename and complete it first then do the upgrade. And if we do that we do not want to wait for gating in the middle19:53
*** melwitt is now known as Guest44019:54
fungii'll go ahead and review the changes. going to need to take a break before maintenance to make/eat dinner19:55
clarkbya I'm about to have lunch now19:55
fungiotherwise i could be eating very late if the window runs long19:55
clarkbfungi: I updated the process on the tehrpad to reflect ^ basically lets do the rename then force merge the first project-config change and ensure manage-projects runs normally as expected. Then put review02 back in the emergency file and do the upgrade once reindexing is done19:55
clarkbwe have to wait for reindexing anyway so amy as well use that time to complete the rename fully19:56
fungithankfully, reindexes are way faster than they used to be19:56
clarkbthe other nice thing about doing it this way is we can worry about one set of challenges at a time. If we run over time a bit oh well19:57
clarkbAlright lunch now. Back in a bit19:59
fungiall four changes related to the maintenance lgtm (822222, 825680, 825677, 826148)20:04
clarkbI put some #status notice drafts into the therpad20:41
fungiooh, thanks!20:42
clarkbif those looks good I'll send the first one out at 21:00 UTC and put servers in the emergency file20:42
fungiclarkb: i often tack on the url for the maintenance announcement from the ml archive just for completeness20:43
fungiianw: good morning! we're on track for the announced time20:44
fungisome adjustments have been made to the plan in order to speed up the delay between renaming and upgrading20:44
clarkbfungi: like that?20:45
fungiclarkb: yep, lgtm20:45
clarkbianw: ya basically I think the plan I've settled on is that we should try and complete the rename as completely as possible before the upgrade. This means force merging the project-config change so that we run the manage-projects job more quickly rather than waiting around for that20:45
clarkbianw: then once we're happy with the renaming and reindxing has completed we can proceed with the gerrit upgrade. One questyion for you is if you think we need to land the 3.4 update change prior to upgrading or if we can land that after? I 've proposed after in my etherpad since that will speed things up a bit but your etherpad indicates before20:46
clarkbotherwise I think we can largely follow your etherpad for the gerrit bits20:46
ianwwhat's the rename etherpad link sorry -- i closed it and can't find it now20:47
clarkbre dequeue I'm suddenly remembering that I don't know if we can dequeue the hourly jobs because they are periodic20:48
clarkbcorvus: ^ do you know?20:48
clarkbI can try to dequeue the 2100 UTC hourly jobs20:48
fungiyeah, that'll be a good test20:50
fungii was able to use zuul-client dequeue successfully yesterday from zuul0220:50
clarkbfor a check entry though right?20:50
fungithough also the webui might be able to do it20:50
clarkbI don't know how to login to the web ui20:51
fungii have not tested authenticated actions in the zuul webui yet20:51
clarkbI think I need a keycloak account which I've not made20:51
ianwclarkb: i think 826148 can land after, but we just need to hand edit in a s/3.3/3.4/ to the docker-compose then?20:52
clarkbianw: yup that is what I proposed on my etherpad20:52
fungiclarkb: ahh, yeah i was helping test the keycloak poc so have an account i can try with20:52
clarkbok cool I think we have a reasonable plan then20:53
clarkbianw: do you want me to finish migrating your steps from your etherpad to mine or should we just jump from mine to yours when ready?20:54
ianwi think we can minimise work and just jump over?  i can update to reflect hand editing20:55
*** melwitt_ is now known as melwitt20:55
clarkbianw: can you also edit it to indicate we shouldn't start upgrade until the reindexing for renaming is done?20:55
clarkbthat was the other small thing20:55
funginevermind, i think i have an account in a different test realm than the one zuul is set up for, so this is probably not the time to try to iron it out20:56
fungibest to just try with zuul-client for now20:57
ianwyeah, on that, i think we can see that in the config file. ; luca likes to drop some cryptic-ish comments which i'm trying to convert to "not a gerrit java developer" level :)20:57
clarkbya though I am fairly certain it will fail now that I think about it because there isn't a way to express the ref to remove iirc20:57
clarkbianw: you can do a gerrit task listing against the ssh api20:57
clarkbit also logs when it is done in the error log iirc20:58
*** melwitt is now known as Guest45020:59
clarkbI'm going to send the first notice now21:02
clarkbthen I'll update the emergency file. Then I'll try dequeing21:02
clarkb#status notice will have a few short outages over the next few hours (beginning 22:00 UTC) while we rename projects and then upgrade to Gerrit 3.4. See for details.21:02
opendevstatusclarkb: sending notice21:02
-opendevstatus- NOTICE: will have a few short outages over the next few hours (beginning 22:00 UTC) while we rename projects and then upgrade to Gerrit 3.4. See for details.21:02
clarkbok I've just realized one minor thing. Zuul has multiple scheduelrs now21:04
*** melwitt_ is now known as melwitt21:05
*** melwitt is now known as Guest45121:07
opendevreviewClark Boylan proposed opendev/system-config master: Run zuul project rename steps on a single scheduler
clarkbI think we'll need ^ that21:08
clarkbwe can manually apply it for this time21:08
clarkbthe dequeue worked as listed on the etherpad21:10
*** Guest451 is now known as melwitt21:10
clarkbI updated the plan to use a checkout of 82615621:12
clarkbthe etehrpad notes don't list zuul as needing to go in the emergency file. I put zuul01 and zuul02 in the emergency file anyway as I don't think it will hurt anything21:15
corvusclarkb: was your Q answered?21:22
clarkbcorvus: yes we tested against the 2100 UTC enqueue and it worked. I guess zuul-client is smarter than the gearman one21:23
clarkbianw: I've just remembered (yay) that the manage-projects.yaml job doesn't sync project-config to the remote before updating gitea servers because it uses teh checkout on bridge for that. Do you know what will update project-config on bridge? Is it just the sync for the change landing?21:24
clarkbianw: ya it looks like that is what does it:
clarkbso ya I think we are safe to force merge and have those jobs run as normal21:27
clarkbalright I'm going to take a break now and be back before 22:00 UTC to start things21:28
clarkbI did start a root screen on bridge that others can join when we are ready too21:29
ianwclarkb: yeah i think that's right; runs against the zuul version21:43
clarkbianw: where does ^A H log to?21:46
clarkbI can turn that on in the bridge screen but not sure where it will write21:47
opendevreviewJames E. Blair proposed zuul/zuul-jobs master: Add upload-logs-ibm role
fungiit will write something like a hardcopy.txt in the current working dir21:47
fungii don't remember the precise extension (if it even has one)21:48
*** dviroel is now known as dviroel|afk21:48
clarkblooks like hardcopy then a digit21:48
clarkbI'll do that now21:48
clarkboh it told me screenlog.021:48
ianwyep :)21:50
ianwi've attached to the screen21:50
clarkbwe'll start with the review02 reboot21:51
clarkbI'll docker-compose down and reboot then docker compose up after wards to make sure gerrit is in the state that the playbook expects21:52
fungii've joined the screen session, thanks21:54
clarkbfungi: ianw: and my use of a local checkout of system config to get looks good to you?21:55
clarkbI'll be on the meetpad for our etherpad just because I can be if anyone else wants to join21:57
ianw++ from me21:57
fungilgtm, yeah. i don't think i'm set up for meetpad at the moment unfortunately but happy to coordinate in here and in the etherpad(s)21:58
clarkbya if others prefer here thats fine21:58
clarkbno one has joined and I'm ok with that21:59
clarkbalright it is 22:00 now22:00
clarkbsending the second notification22:00
clarkb#status notice The maintenance work is beginning now. Expect Gerrit outages over the next couple of hours. See for details.22:00
opendevstatusclarkb: sending notice22:00
-opendevstatus- NOTICE: The maintenance work is beginning now. Expect Gerrit outages over the next couple of hours. See for details.22:00
ianwall you would hear here is me nagging kids that if they don't do their chores as requested they will not be leaving for their playdates with friends :)22:00
clarkbinfra-root I'll go ahead with the review02 docker-compose down and reboot. then start gerrit up post reboot then we can run the playbook22:01
fungisounds good, thanks22:01
clarkbit is rebooting now22:01
fungiianw: oh, is it summer break there?22:01
clarkbgerrit is starting up again22:02
ianwyes, back to school next week22:02
clarkbuname -a lgtm22:02
ianwalso back in and lgtm22:03
clarkbit says it is ready22:03
clarkbI already checked that I can ssh from bridge to the hosts involved in this playbook22:03
clarkbThat takes us to running the rename playbook. Are we ready to do that now?22:03
*** melwitt is now known as Guest45622:04
clarkbthis bit will run in the screen22:04
fungiyep, seems fine22:04
clarkbok proceeding now22:04
clarkbfungi: maybe you can prep to do the force merge? I can get the dequeue ready and ianw will probably want to monitor the reindexing22:08
fungireindexing should be underway now22:08
fungiyeah, i'll prepare for submitting the change22:08
clarkband ya the playbook seems to have run happily22:09
fungifollowing our process documented at
clarkbfungi: ya can you hold off on actually doing it until we dequeue the hourly jobs and pull hosts out of emergency?22:09
clarkbbut I know you have to add yourself to groups and stuff sof igured getting started was a good idea22:10
clarkbthe gitea redirects lgtm22:10
clarkband the projects show up in gerrit so I think sanity checking can be marked done22:10
fungiright, that's what i'm doing22:10
clarkbcool I'll proceed with dequeing hourly now22:10
ianwyeah i can see all the index jobs, keeping an eye22:10
fungiit's just 822222 we want merged right?22:11
clarkbfungi: yes since that is the one that updates projects.yaml22:12
*** Guest456 is now known as melwitt22:12
clarkbfungi: you can merge now22:12
clarkbhosts are removed from the emergency file and the hourly runs have been dequeued22:13
opendevreviewMerged openstack/project-config master: Rename skyline/skyline* project to openstack/skyline*
*** melwitt is now known as Guest45822:13
clarkbthat replicated so thats good22:14
fungispecifically, i followed our cut-n-paste cli example at
fungii'll leave my admin account in project bootstrappers for the moment in case we need to do that for any other changes related to the maintenance22:14
*** Guest458 is now known as melwitt22:15
*** dviroel|afk is now known as dviroel22:15
clarkbsounds good22:15
clarkbI figured the reindexing would be slow and it appears to not be fast :) I'm happy we didn't try to squash things together given that.22:17
clarkbbut given there is online reindexing with the 3.4 upgrade not having a stale index felt important (we'll backup a good index)22:17
ianwit did 20 indexing tasks in 27 seconds22:19
ianw1485 to go -- 20 minutes or so maybe22:19
clarkbianw: ya the cost of each tasks depends on the size of the repos themselves and the number of changes etc22:20
clarkbI think the manage projects playbook is complete22:20
clarkbthe redirects still work as expected22:20
fungiyeah, the tasks are per-repo and the duration of each seems to scale linearly by the number of changes for a repo22:20
clarkband searching gerrit for skyline only shows the new names22:20
clarkbI think that means manage-projects appropriately noop'd22:20
clarkbI skimmed the log too and didn't see anything unexpected and the project-config on review02 was updated22:21
clarkbOther than the reindexing and the zuul config update which should happen shortly I think we can consider this largely done for the renames22:21
fungithe change reindexer packs its threads with the largest (change-count-wise) repos first, in order to maximize parallel throughput22:22
fungiso in theory it should speed up a bit as it finishes the longer-running ones22:22
clarkbI think that might not be true for onlien reindexing22:22
clarkbsince nova is queued up still22:22
clarkbthere may be an optimization that upstream could make there22:22
fungioh, maybe they changed it :/22:22
clarkbanyway this seems like a hurry up and wait for reindexing step so I'm going to grab a glass of water back in a few22:22
clarkbunder 1k tasks now22:26
clarkbI've rechecked as zuul should know about the renamed projects now22:29
ianwgetting close now22:30
clarkbI wonder if we need towait for the zuul smart reconfigure to end22:31
clarkbbecause zuul pulls from gerrit22:31
clarkbcorvus: ^ is there an easy way to track when that is done?22:31
clarkbmaybe when the zuul queue length drops back to 0 it is implied22:32
clarkbI think zuul may be done as it has enqueued the change I rechcked22:34
clarkband now it has queued a job for that chagne so ya I think we're good there22:34
fungiyeah, otherwise it would have errored back immediately22:34
clarkbshould I exit the root screen on bridge? I dont think there is anything else to do there22:35
ianwfd7cfc10              22:08:46.774      Index all changes of project openstack/nova22:35
ianwthat's the long ones22:36
fungithe upgrade will be done from review.o.o directly, yeah. i'm fine closing out the session on bridge22:36
clarkbya though it seems to be making decent progress22:36
clarkbfungi: yup upgrade will be on review0222:36
clarkbok closing the bridge screen now22:36
fungii have a root screen session started on review now22:37
clarkb[2022-01-24T22:36:58.885Z] [Reindex changes v60-v60] INFO : Reindex changes to version 60 complete22:37
clarkbianw: I think you are good to proceed with your upgrade now. Let me know how I can help :)22:37
fungiwe need review back in the emergency list now, yeah?22:38
clarkbfungi: has the steps for review now22:38
fungii'll add it22:38
fungii can edit the docker-compose file too22:39
fungisee if what's in the screen session looks right22:40
fungiif so, i'll pull22:40
clarkbit does, but I was going to let ianw drive22:40
fungisure, waiting for him to be ready22:41
ianwi'm happy to drive if we like22:41
fungifeel free!22:41
clarkbI mean I'm happy to let fungi do things I just meant I didn't want to say go for the next step if ianw wasn't ready :)22:41
clarkbbut ya go for it22:41
fungiwe're up to step u422:41
clarkbfungi: I think you're on the wrong etherpad22:41
clarkbthough I don't think any of the steps you've done interfere greatly with ianw's document22:41
clarkbsorry maybe leaving my plan on the other one is confusing22:42
fungiis there another one besides and ?22:42
clarkbfungi: just those two but has the canonical steps for the gerrit upgrade22:43
clarkbafter we discussed earlier I noted on my etherpad that these steps are historical as ianw updated his more detailed steps on his pad22:43
fungifor some reason gerrit-upgrade-3.4 shows me only myself on the pad22:44
fungiokay, reloaded and now i see everyone else on it too22:44
fungii guess it was a etherpad session bug22:44
clarkbianw: if the shas don't match it may be due to the image update on friday22:44
clarkbwe did an update to pull in a logging bugfix22:44
clarkblooks like they match what you've got in the etherpad though22:45
ianwyep f64852c9fb561bea81f9e522ad6c0edb3e81168ea392348f2392674c77771bf4 is the latest, and that's what we have22:45
clarkbgerrit log says it is ready22:47 : Gerrit Code Review 3.4.3-22-gf8ae0e043a-dirty ready22:47
clarkbI get my personal dashboard22:48
ianwlooks up22:48
ianwi'm seeing grey circles next to peoples names, that is new i think22:48
clarkbI think those might show a profile pic if properly configured somehow22:49
clarkband yes I see them too after a hard refresh22:49
ianwwonder if we can plug into gravatar or something22:49
ianwthat must be where upstream got my head profile22:50
clarkbI can generally browse changes22:50
ianwok, has zuul picked something up22:50
clarkbI don't see anything since the restart22:50
fungiwe can approve something like 82567722:51
fungisee if it notices?22:51
opendevreviewClark Boylan proposed opendev/system-config master: DNM testing things
clarkbI just pushed ^ to check22:51
ianwi put in a recheck on and it is running22:51
clarkb should report back soon too22:52
ianwonly thing unconfirmed is gitea replication but i think we can keep an eye on that22:52
clarkbianw: the replication log should show us if 826190 replicated. Looking22:53
ianwhrm, i wonder why we have differences in the config files22:54
clarkbnew gerrit will add stuff to configs and it never makes sense22:54
ianwbut we should be validating that and erroring out in the job now22:55
fungireplication seems to have worked22:55
clarkbianw: oh I see22:56
fungizuul reported back +1 on 825680 now22:56
clarkbfungi: thanks for looking I was just catfhing up to that in thel ogs and I see some replication completed events. I see some not attempted log messages for the change metadata but I think that may be intentional since its a lot of data?22:56
funginot sure, i thought we were replicating change metadata previously22:57
fungiat least the notes ref22:57
clarkbianw: what are you diffing?22:57
clarkbI just did a manual diff and I think it is empty?22:57
clarkbwell I diffed the main config file. I'm assuming the delta you found is in another file22:58
ianwdiff -urN /home/gerrit2/tmp/upgrade-3.3/etc /22:58
ianwhome/gerrit2/review_site/etc > /home/gerrit2/tmp/upgrade-3.3/config.diff22:58
clarkbianw: thats the 3.3 path?22:59
clarkbI think you might be diffing against 3.2 configs22:59
clarkbwe named the dirs after the target version not the version they were generated with22:59
clarkbya `cp -r /home/gerrit2/review_site/etc /home/gerrit2/tmp/upgrade-3.4` happened pre upgrade23:00
clarkbso upgrage-3.4 has the 3.3 configs in it23:00
ianwoh doh, i've put in the wrong command in the etherpad23:00
clarkbis it clean when you use the newer path?23:00
*** dviroel is now known as dviroel|out23:00
fungilooking like i won't need bootstrappers membership for my admin account any longer, so i'll remove it again23:01
fungiand done23:02
clarkbreading the replication log more carefully I think the NOT_ATTEMPTED is logged before we attempt it. Then it completes a few seconds earlier and we actually are replicating the meta stuff23:03
ianwok, yeah it's clean.  will fix those commands before 3.5 upgrade!23:03
fungionce we merge a new change i can check whether i get an updated review notes ref23:03
clarkbcool. I'm double checking replication of the meta ref now23:04
ianwgoign to quit the screen now23:04
clarkbbut ya I think we can generally call this done and monitor as we land changes to catch up23:04
fungiready for me to approve the remaining changes, or did someone else want to?23:04
fungialso i guess we need to take the server back out of the disable list now?23:05
clarkbrefs/changes/90/826190/meta I can fetch that from and that change was created after the upgrade so we wouldn't be fetching old meta tehre23:05
clarkbI think replication is happy23:05
clarkbya I think we can remove from emergency, then approve the various changes. I'll get links in a minute.23:05
clarkbThen we can also consider dequeing the hourly jobs23:06
clarkbianw: ^ that sounds good?23:06
ianw++ i think we're ready to go23:06
fungithat technically reverses the order of steps 15 and 1623:07
fungido we care if they're done at the same time?23:07
clarkbfungi: no I think it is fine23:07
clarkbworst case we'll change the docker-compose.yaml back to 3.3 for a short time23:07
clarkbthe reindexing is still running so gerrit will probably be a little slower than usual until that is done23:08
fungiokay, i'll start approving unless someone else wants to23:08
clarkbI think you can go for it. I pushed many/most of them so should avoid approving if I can23:08
ianwhave we removed from emergency?23:08
clarkbdoesn't look like it23:09
clarkbfungi: ^ were you still going to do that?23:09
fungiwe haven't, but i can23:09
ianwok, i just did it since i had it open23:09
opendevreviewMerged opendev/project-config master: Record project renames happening January 24, 2022
fungiwfm, thanks23:09
ianwi've approved 826148 to update the docker-compose23:10
fungiconfirmed, refs/notes/review was updated for 825677 and tells me the approval vote and submitted time23:12
fungiSubmitted-at: Mon, 24 Jan 2022 23:09:34 +000023:12
clarkbWe can start looking at adding gerrit 3.5 image builds next, update the upgrade job to do 3.4 to 3.5 :) But that isn't urgent23:12
fungiand git claims it pulled refs/notes/review from not review.opendev.org23:12
fungiso pretty sure that proves it replicated to gitea23:13
clarkband I'd like to keep the 3.3 image builds around until we confident we won't revert23:13
fungiof course23:13
ianw++.  this feels routine now -- a testament to a lot of hard work put in!23:14
clarkbYes! tahnk you everyone who helped make this as easy as it has been23:15
opendevreviewMerged openstack/project-config master: Configure renamed skyline projects with openstack jobs
fungivery smooth indeed. thanks ianw and clarkb for all the upgrading and renaming planning!23:17
clarkbfungi: ianw  do we want a #status notice along the lines of "Unless any major issues show up we expect the maintenance to be complete." ?23:19
fungiwe don't usually, but up to you23:19
clarkbya I'm fine leaving it out then23:20
clarkbreindexing is complete too now23:24
clarkbAnything else I can do to be helpful? if not I'm going to take a break. Then check back in in half an hour or so. I'll send out the meeting agenda then too23:24
ianwi think we're all good, thanks!23:24
clarkbside note: we've overtaken wikimedia's gerrit now :)23:25
clarkbnot that it is a competition, but really does show how well we've managed to get ahead of where we were in the past23:25
clarkbok back in a bit23:25
opendevreviewIan Wienand proposed opendev/system-config master: gerrit: add avatar-gravatar plugin
*** rlandy|ruck is now known as rlandy|out23:39
johnsomHow do we audit gerrit access these days?  The access page is blank now,,access23:50
opendevreviewIan Wienand proposed openstack/project-config master: gerrit: add gravatar avatar plugin
opendevreviewIan Wienand proposed opendev/system-config master: gerrit: add avatar-gravatar plugin
clarkbjohnsom: through openstack/project-config/gerrit/acls23:55
clarkbjohnsom: unfortunately one of the side effects of the bug that fungi and I discovered before we upgraded to 3.2 was that to fix it they locked that down much more23:56
johnsomAh, ok, a new, maybe temporary thing.23:56
clarkbIt might be worth having a conversation with them about opening that up again just for the acl listings23:56
clarkbjohnsom: ya the problem was they were exposing too much notedb stuff previously. They locked that down and now they expose too little :/23:57
clarkbit isn't ideal23:57
johnsomYeah, it was super handy23:57
clarkbwe might be able to expose it specifcally in the leaf projects23:57
clarkbI think the main issue is in the central project All-Projects and All-Users you do not want to expose those refs23:58
clarkbfungi: ^ may recall more details on that23:58
clarkbinfra-root and child are relevant to things fungi and I discovered last week. Not all servers had properly enabled unattended-upgrades.23:58
clarkbjohnsom: but we don't allow any changes to those refs except through openstack/project-config/gerrit/acls so that should be a reasonable place to go for now23:59
opendevreviewIan Wienand proposed opendev/grafyaml master: Generate and use UID for acessing dashboards

Generated by 2.17.3 by Marius Gedminas - find it at!