Sunday, 2021-10-10

*** snbuback_ is now known as snbuback09:22
*** TheJulia_ is now known as TheJulia10:36
*** gouthamr_ is now known as gouthamr10:39
clarkbinfra-root if any of you are around today prior to the gerrit upgrade it might be good to look over as landing that and getting it applied is one of the first items on the upgrade todo list.17:45
clarkbI was thinking I could approve it around 19:30 UTC to ensure it was done by the time the window started but realized it doesn't have any reviews yet17:45
fungioh, good call. i'll go over it now, thanks!18:00
fungii was planning to be around for the 21:00-23:00 window anyway18:01
opendevreviewJeremy Stanley proposed opendev/infra-specs master: Add a specification for Mailman 3
opendevreviewJeremy Stanley proposed opendev/infra-specs master: Add backups section to template
opendevreviewJeremy Stanley proposed opendev/infra-specs master: Simplify assignees section of template
clarkbfungi: re your comment: yes, that whole stack of changes works through adding 3.4 images, moving the upgrade testing to 3.3 -> 3.4 and then removes the 3.2 images19:26
clarkbLooks like I've got the +2's now. I'll hit +A a bit closer to 20:00 so that it is ready and in place for when the window begins19:27
clarkbok I have approved the change that will udpate our docker-compose file to reference the 3.3 image instead of the 3.2 image. This shouldn't affect the running gerrit and we will have to walk through the steps in the etherpad in an hour or so to acutally do the upgrade19:56
clarkbfeel free to -W or -2 should something come up that questions any of that while it is in the gate19:56
ianwthanks for gettign that started20:35
opendevreviewMerged opendev/system-config master: Upgrade Gerrit to 3.3
fungiso we just need it to deploy in the next 15 minutes20:46
clarkbya I realized we could spend half the window just waiting for that to land and deploy figured getting it done early was a good idea20:50
clarkboh this is unexpected though it is running manage-projects20:51
clarkbI think it may have pulled the new images for us to run manage-projects20:53
clarkbsince we do a docker run not a docker exec20:53
clarkbthat should be fine and we can pull as planned to ensure we have all the images we need20:53
clarkbhrm maybe not. the manage-project script still references the 3.2 image20:54
ianwyeah i think to avoid this sort of thing we have to manually pull the latest images on gerrit20:55
clarkbI see what happened. The manage-projects task file pulls the gerrit image using the variable we set to set the version20:55
clarkbbut it doesn't update its own script file20:55
fungii guess that's preferable anyway20:55
clarkbI think the "fix" there is to have it also update its script file. But ya I don't think this did anything we didn't want so I'm happy with it20:56
fungithe pull will be a no-op now, but it didn't run with the new image either20:56
ianw#status notify Gerrit will be unavailable for maintenance for up to a few hours; see
opendevstatusianw: unknown command20:57
clarkbianw: it is notice not notify20:57
ianw#status notice Gerrit will be unavailable for maintenance for up to a few hours; see
opendevstatusianw: sending notice20:58
-opendevstatus- NOTICE: Gerrit will be unavailable for maintenance for up to a few hours; see
fungiBuild succeeded (deploy pipeline).21:00
fungiguess we're ready21:00
ianwcool; is what is being worked through21:00
clarkbhas been put in the emergency file yet?21:01
fungii can do that now21:01
clarkbthe hourly jobs ignore it now, but seems like a good thing anyway21:02
fungiand done21:02
clarkbon item 3.5 I think I convinced myself while checking on that that it isn't really necessary21:03
clarkbthe cron jobs run at times that shouldn't conflict21:03
fungii think i may have collided with someone else's edit of the emergency file21:03
clarkbfungi: wasn't me so probably ianw21:03
clarkbI'll let you two sort out things and I'll stay out of the way :)21:03
ianwfungi: heh sorry yes was doing it :)  i've started a screen 21:03
clarkbI'm in the screen on review is there one on bridge?21:04
fungii'm attached to the screen session on review now21:04
fungilooks like you commented out the cronjobs21:04
ianwno i didn't do one on bridge21:05
clarkbok I don't think we need one, but thought I'd ask21:05
clarkband ya disabling the crons is fine with me21:05
ianwyep; i think it's good to keep this like a generic upgrade template as much as possible21:05
fungiso we're up to pulling images now21:08
fungia no-op as expected21:08
clarkbwe should make a mental note to prune the images on that server21:09
clarkbmaybe in a week when we are happy with things we can do that21:09
ianwok, all looks sane21:10
clarkbyup and mariadb can take a bit of time to start21:11
clarkbah cool it logs explicitly when it is ready for connections that is nice21:11
clarkbthose java warnings are expected21:12
clarkbpeople have complained about them to the gerrit maintainers21:12
fungithat's a fun warning21:13
clarkband that looks about right, it is a simple migration to change the name of the service user group21:13
fungiillegal access? /me looks around for the cops21:13
clarkbI suspect that may be what is keeping them from goign to java 17 any time soon21:13
clarkb(java 17 is the next lts)21:14
fungii can get to the webui21:14
fungipretty snappy21:14
clarkbthe log indicates it is ready too21:15
ianw🎉 No changes need your attention 🎉 -- party time21:15
fungii'll drink to that21:15
clarkbheh the emoji are literal too21:15
clarkb(this is the attention set stuff I've talked to corvus about in the past)21:16
clarkbbasically gerrit tries to keep track of what comments need addressing by code submitters and reviewrs and then calls things to your attention so taht you can address them21:16
clarkbI think it has a lot of potential but we'll have to see how it goes.21:16
ianw has already uploaded something that is in the gate21:16
clarkbwow that was quick21:17
clarkbone thing we should double check is that recheck comments work properly. I can recheck my prometheus spec change21:17
clarkb(because the way gerrit emits those has changed but we do believe zuul should handle them properly already)21:17
ianwa recheck in is in the gate21:18
clarkbI rechecekd the spec and it is in the check queue so that lgtm21:18
ianwthere's always interesting things to find in the error log, but nothing seems fatal21:21
clarkbianw: the gc log complaints are something I wish I understood better, but also I guess at this point we could drop the gc logging since we don't have gc problems21:21
clarkbthat should make it quiet about those logs file rotations21:21
ianwgerrit.config appears to have acquired quotes21:25
clarkbianw: fungi: that diff looks pretty clean to me.21:25
ianw-       reportBugUrl = #contributing21:25
ianw+       reportBugUrl =
clarkbianw: other way around the new side is the old config21:25
ianwdoesn't look right21:25
clarkband ya I think that may be the only thing that is possibly broken21:25
ianwi think it may think that is a comment21:25
clarkbfor the email template things I don't think we should manage them unless we decide we need to change their content and we haven't had to do that so far21:26
clarkbanyway I suspect we can clean this up non urgently21:26
clarkb vs when you click report a bug is probably close enough for now21:26
fungiseems fine to me, yep21:26
clarkbI config is the link when you hover over report a bug at the buttom of a gerrit page21:27
fungiand also we could stand to replace that url anyway21:27
clarkbfungi: ya its stale as is21:27
fungianything else we want to check before we undo the emergency file and restore the cronjobs?21:29
clarkbonce we are happy with things is a good change to land. too though less urgently21:29
clarkbfungi: I'm not seeing anything. A good test is to acutally land something like 813081, but I think that can happen afterwards21:30
fungiclarkb: 813081 drops system-config-build-image-gerrit-master and doesn't mention why, though i expect that's good cleanup21:31
clarkbfungi: mostly because it was a waste and doesn't work half the time21:31
fungiyeah, that was the reason i expected21:32
clarkbI guess I should've called that out more explicitly in the commit message. I didn't want jeepyb changes being held up by flaky gerrit master builds21:32
clarkband seemed like a good time to make that edit while I was updating the file21:32
clarkbI have detached from the screen so that it can be clsoed saving the log file. I'll let ianw or fungi clean up the mergency file since you two edited it previously :)21:33
opendevreviewIan Wienand proposed opendev/system-config master: gerrit: Update quotes for 3.3
clarkbianw: I wonder if adding quotes to the bug url will produce the same result21:35
clarkbworht a try21:35
ianwwe should be able to inspect the config file collected from the job, i guess?21:36
clarkbianw: oh g ood point21:36
clarkbyes that should be the case21:37
ianwi also cleaned up a few other things in emergency that have gone now (review-test, review-dev01,
fungii've also detached from the screen session21:37
clarkbianw: do you want to review and hopefully approve as a sanity check we can land code?21:38
clarkb(that chagne needs to land to reflect the new state of the system anyway)21:39
clarkband then we can also double check replication for project-config afterwards21:39
clarkbI don't expect replication trouble fwiw, the plugin hasn't really changed between several gerrit versions21:40
ianwi does tell me it's "my turn to take action"21:41
clarkb(though sometimes being super sure of that is difficult with the way they merge up instead of down)21:41
clarkbya I think to make it work right we need to be more disciplined about marking things as addressed or not21:41
clarkbit may take some time to get the workflow down21:41
ianwon you just put a +2, which it seems to take as a comment21:42
clarkbthe default for top level comments is that they are "resolved" which I thought would prevent it from kicking back to you21:44
clarkbI guess it assumes that its on you if I've said my end is resovled21:45
ianwwhen you add your vote, down the bottom there's "bring to the attention of" which seems to autofill ... i'm not sure, at least the owner, and maybe other reviewers21:47
clarkbya its meant to figure it out based on who is commenting and interacting with the system21:48
clarkbI suspect we'll just have to work with it a bit to udnerstand what it wanst then maybe we can write some guidelines for people to use it effectively21:48
ianwfor our workflow, it seems like making positive votes with no comments should default to not updating attention sets21:49
clarkbthat seems reasonable. I don't know how tunable this is. But all stuff we can provide as feedback upstream at least21:49
clarkbre the gerrit stack of chagnes should also be safe to land, but I think we should hold off on until we are reasonable happy with 3.3 and don't want to revert (mid week maybe?)21:50
ianw"If a reviewer replies, then the change owner (and uploader) are added to the attention set"21:50
clarkbmaybe we treat that as "find a second reviewer now if you got one +2?" but we typically dno't operate that way21:51
opendevreviewMerged openstack/project-config master: Update jeepyb to trigger Gerrit 3.3 image builds
clarkb ^ replicated21:52
clarkbthat all appears happy. ianw anything else you want me to stick around to look at? I'll check back in later this evenign either way, but I think I'm running out of things to do without starting my workweek extra early :)21:53
ianwclarkb: nope, many thanks to yourself and fungi for helping out!21:53
fungithanks for driving this!21:53
fungii'll try to be on hand earlyish in my day too in case people spot any problems with 3.321:54
clarkb++ its nice to make progress on stuff like this. Assuming it remains happy I'll plan to do the gitea 1.15.4 upgrade tomorrow too21:54
fungithough i expect it'll mostly just be confusion from the new workflows21:54
clarkbI marked WIP too just to avoid it landing early21:55
clarkbfungi: ya I suspect that one thing we'll need to do is add third party CI bots to the Service Users group21:56
clarkboh and possibly zuul too?21:56
clarkbianw: fungi: I'd be ok if we want to just go ahead and start doing that. I had previously checked if the Service Users group was used for anything and from what I could tell it wasn't21:57
clarkb(well its old name anyway)21:57
ianwthat stops them adding the attention set right?21:58
clarkbMaybe double check that and if we're happy with the double checking we add Zuul and third party CI systems to it to avoid confusion with attention sets. Also worst case we can just disable attention sets.21:58
clarkbianw: ya, it also allows us to pool them into different thread pools but I don't think we're setting up special thread pools for non interactive users21:58
clarkbsshd.batchThreads is the option21:59
ianwi'll look into that22:00
clarkbya we don't set that value so everything shares the same pool so should be a noop to add those users to Service Users group if I read things correctly. The other thing to check is just that we don't give Service Users any unexpected permissions somewhere (this is what I was saying would be good to double check)22:00
clarkbianw: thanks! I'm going to pop out now, but will check back in in a few hours22:01
clarkbchange.enableAttentionSet is the option to set to False should we decide it just isn't working, but I think it is worth tryting to make work first22:02
opendevreviewMerged opendev/system-config master: Start building gerrit 3.4
ianwit looks like zuul is adding the uploading user back to the attention set23:05
ianwi'll just get a cup of tea then look into the service users group23:06
ianwcurrently has release and proposal bot, which seems right23:20
ianwi think we should add "Third-Party CI" and "Continuous Integration Tools"23:22
ianwok, that is done23:30
ianwthat seems to have fixed it.  "Starting gate jobs" in didn't modify the attention set23:38
fungigood call23:38
fungithanks for the fix23:38
fungihopefully that doesn't some with unexpected side effects23:38
ianwyeah, i'm keeping a close eye to make sure and watching zuul logs, it seems ok23:39

Generated by 2.17.2 by Marius Gedminas - find it at!