Tuesday, 2020-11-24

ianwfungi: did you modify gerritlib/gerrit.py around the _read() function on the eavesdrop host?00:02
ianwadding the "if"00:03
fungii applied https://review.opendev.org/763658 but once that was in it was filling the logs with empty strings so i added the "if l:" around the _read() to silence those00:04
fungii can push up a new patchset of 763658 with that worked in00:04
ianwahh, ok00:04
ianwi have the ssh close/retry working, i was just trying to piece together how the code on the system came to be :)00:05
ianwmaybe just leave it if you like, and i'll put up something we can compare to00:06
fungii've updated 763658 with that now00:07
fungiyou can also feel free to hack on that change if it helps00:07
ianwthe problem is it never breaks out of the read.  it will be easier to explain in diff format :)00:09
fungii guess we need to return if it's empty?00:10
fungithat's as far as i'd gotten before i was too out of it to continue yesterday00:11
*** tosky has quit IRC00:17
ianwfungi: pretty much, just beating it into shape, but i can close the connection from gerrit and have it re-start now00:25
fungiooh nice!00:28
*** brinzhang has joined #opendev00:38
*** openstackgerrit has joined #opendev00:39
openstackgerritIan Wienand proposed opendev/gerritlib master: Handle empty reads as closed connections  https://review.opendev.org/c/opendev/gerritlib/+/76389200:39
ianwfungi: ^ ... not clear to me if it didn't work before, or something else changed00:41
*** ralonsoh has quit IRC00:46
*** ralonsoh has joined #opendev00:46
fungiianw: thanks! entirely possible this is a behavior change in newer mina-sshd or something00:55
openstackgerritIan Wienand proposed opendev/gerritlib master: Handle empty reads as closed connections  https://review.opendev.org/c/opendev/gerritlib/+/76389200:59
*** mlavalle has quit IRC01:26
openstackgerritJeremy Stanley proposed zuul/zuul-jobs master: Pin keystoneauth1 and cachetools on older Python  https://review.opendev.org/c/zuul/zuul-jobs/+/76386601:29
openstackgerritJeremy Stanley proposed zuul/zuul-jobs master: Use Python 3.x with launchpadlib  https://review.opendev.org/c/zuul/zuul-jobs/+/76383401:29
*** ysandeep|away is now known as ysandeep01:44
*** brinzhang has quit IRC02:16
*** iurygregory has quit IRC02:25
*** ykarel has joined #opendev02:26
*** whoami-rajat__ has quit IRC02:31
*** ykarel has quit IRC03:19
*** auristor has quit IRC03:49
*** auristor has joined #opendev03:56
*** raukadah is now known as chandankumar04:09
*** amaron has quit IRC04:38
*** ykarel has joined #opendev04:48
*** ykarel_ has joined #opendev04:53
*** ykarel has quit IRC04:53
*** ykarel_ is now known as ykarel04:54
*** ysandeep is now known as ysandeep|afk05:22
openstackgerritIan Wienand proposed opendev/gerritbot master: Build against gerritlib master branch  https://review.opendev.org/c/opendev/gerritbot/+/76392705:50
*** ykarel has quit IRC05:55
ianwfungi: ^ i think we can incorporate changes with that, and test the container?05:57
*** marios has joined #opendev06:04
*** ykarel has joined #opendev06:08
openstackgerritOpenStack Proposal Bot proposed openstack/project-config master: Normalize projects.yaml  https://review.opendev.org/c/openstack/project-config/+/76393006:12
*** ralonsoh has quit IRC06:46
*** ysandeep|afk is now known as ysandeep06:57
*** iurygregory has joined #opendev07:02
*** marios has quit IRC07:11
*** ykarel_ has joined #opendev07:22
*** ykarel has quit IRC07:25
*** marios has joined #opendev07:29
*** DSpider has joined #opendev07:36
*** rpittau|afk is now known as rpittau07:39
*** ykarel__ has joined #opendev07:45
*** ykarel_ has quit IRC07:48
*** lpetrut has joined #opendev07:53
*** sboyron__ has joined #opendev07:58
*** slaweq has joined #opendev08:02
*** eolivare has joined #opendev08:02
*** ykarel__ is now known as ykarel08:05
*** hrw has joined #opendev08:09
hrwDid we lost 'cherrypick' button with Gerrit upgrade?08:09
hrwah. it is in menu now...08:10
*** sboyron__ is now known as sboyron08:12
*** ralonsoh has joined #opendev08:15
*** DSpider has quit IRC08:31
*** DSpider has joined #opendev08:32
*** andrewbonney has joined #opendev08:42
*** mgoddard has joined #opendev08:44
*** tosky has joined #opendev08:48
*** dtantsur|afk is now known as dtantsur09:01
openstackgerritSorin Sbârnea proposed zuul/zuul-jobs master: Add ensure-ansible role  https://review.opendev.org/c/zuul/zuul-jobs/+/74970609:04
*** hrw has left #opendev09:17
*** marios has quit IRC09:30
*** iurygregory has quit IRC09:37
*** marios has joined #opendev09:39
*** iurygregory has joined #opendev09:43
*** mgoddard has quit IRC10:11
*** fressi has quit IRC10:18
sshnaidmI wonder if there is a way to mark failed jobs by red as it was before?10:20
sshnaidmAnd if possible to see list of jobs in the main page, instead of looking for them in comments..10:21
zbrsshnaidm: you will have to wait a while, likely few weeks.10:21
zbrthe feature needs to be reimplemented and there are other stuff that are more important.10:22
zbrsshnaidm: but if you are feeling confident in your JS skills, you could try to update the gresemonkey stripts that do add these features and we can start using them without having to update gerrit itself. See https://opendev.org/x/coats10:23
*** mgoddard has joined #opendev10:24
zbrmy guess is that we only need to find the new HTML elements where to inject the table10:25
*** mgoddard has quit IRC10:35
*** fressi has joined #opendev10:48
*** mgoddard has joined #opendev10:49
*** ralonsoh has quit IRC11:20
*** ralonsoh has joined #opendev11:23
*** fressi has quit IRC11:32
*** fressi has joined #opendev11:33
openstackgerritSorin Sbârnea proposed opendev/git-review master: Fix "git-review -d" erases work directory if on the same branch as the change downloaded  https://review.opendev.org/c/opendev/git-review/+/39977911:50
*** ysandeep is now known as ysandeep|brb12:00
*** hashar has joined #opendev12:00
*** eolivare_ has joined #opendev12:09
*** eolivare has quit IRC12:12
*** dtantsur is now known as dtantsur|brb12:22
*** ykarel is now known as ykarel|away12:23
*** ysandeep|brb is now known as ysandeep12:25
*** fressi has quit IRC12:25
*** slaweq has quit IRC12:29
*** slaweq has joined #opendev12:31
*** eolivare_ has quit IRC12:32
*** ykarel|away has quit IRC12:44
*** fressi has joined #opendev12:44
*** eolivare_ has joined #opendev12:50
*** fressi has quit IRC12:59
*** fressi has joined #opendev13:01
openstackgerritwes hayutin proposed openstack/project-config master: add review-priority for tripleo-ci  https://review.opendev.org/c/openstack/project-config/+/71506913:08
fricklerinfra-root: did we change something in gerrit some time after the upgrade that would cause me to no longer receive mails about zuul results? I did get some after the upgrade, but now I notice some responses missing. I do still receive proper mails about comments and new patchsets13:15
zbrfrickler: email template changed, you need to update your filters.13:28
*** lpetrut has quit IRC13:36
*** lpetrut has joined #opendev13:38
*** dtantsur|brb is now known as dtantsur13:41
*** whoami-rajat__ has joined #opendev13:44
fricklerzbr: I filter on "From: review@openstack.org", don't think that has changed13:45
*** mgoddard has quit IRC13:46
*** mgoddard has joined #opendev13:53
*** pabelanger has left #opendev13:57
zbrfrickler: i still receive my emails, so i doubt is working only for me. but there is a section mentioning changes related to how emails are sent. also now the default is to send HTML unless you change your user config to force plain.14:00
openstackgerritTristan Cacqueray proposed opendev/system-config master: gerrit: install zuul-results plugin to display the build table  https://review.opendev.org/c/opendev/system-config/+/76389114:10
tristanCinfra-root: i've tested that zuul-results plugin in a test setup using docker.io/opendevorg/gerrit:3.2 , which seems to work, but it would be good to get it running on review-dev first. Let me know if i can help with that.14:21
openstackgerritAlbin Vass proposed zuul/zuul-jobs master: Add option to install kubernetes with kind  https://review.opendev.org/c/zuul/zuul-jobs/+/74093514:22
openstackgerritAlbin Vass proposed zuul/zuul-jobs master: Add option to install kubernetes with kind  https://review.opendev.org/c/zuul/zuul-jobs/+/74093514:25
fungifrickler: i don't believe we changed anything about e-mails after the upgrade (not yet anyway)14:35
*** gouthamr_ has quit IRC14:36
openstackgerritSorin Sbârnea proposed openstack/project-config master: add review-priority for tripleo-ci  https://review.opendev.org/c/openstack/project-config/+/71506914:39
fricklerfungi: any idea how to view gerrit logs regarding how it sends mails? the mails that I see in the exim log seem to get delivered just fine14:42
openstackgerritSorin Sbârnea proposed opendev/elastic-recheck master: WIP: Run elastic-recheck container  https://review.opendev.org/c/opendev/elastic-recheck/+/72962314:50
openstackgerritThierry Carrez proposed openstack/project-config master: Add release jobs to oslo.metrics  https://review.opendev.org/c/openstack/project-config/+/76398614:53
fungifrickler: we can add a mailrouter in exim which delivers copies of everything to a local mbox file14:56
fungior are you not looking for copies of the messages?14:57
openstackgerritMerged zuul/zuul-jobs master: Pin keystoneauth1 and cachetools on older Python  https://review.opendev.org/c/zuul/zuul-jobs/+/76386614:57
fungifrickler: if you're looking for mail sending errors from gerrit, looks like those end up in its error_log, look for SendEmail14:58
*** fressi has quit IRC14:59
*** roman_g has joined #opendev15:00
fricklerfungi: I was looking for logs for successfully sent mails, but indeed the error log has a lot of "java.lang.NullPointerException: Null email"15:06
fricklerdoes the zuul accout not have an email address assigned?15:06
fricklermaybe this is related to the fact that gerrit now tries to include every reviewer in the reply-to for a msg15:06
fricklerseems there's also a huge number of errors related to stackalytics-bot-215:09
*** ysandeep is now known as ysandeep|away15:17
fungiyeah, saw those too15:23
fricklerI only find these "Null email" msgs in error_logs after the upgrade. also except zuul there's also fungi's account without an email at least according to the mouseover in the UI15:23
fricklernot sure why some mails still get sent at all, then15:23
fungithe fungi.admin account? yeah, that may be yet another reason to assign e-mail addresses on accounts like that15:24
fricklerfungi: actually I think that's the normal account, it just says "fungi", e.g. on https://review.opendev.org/c/zuul/zuul/+/763333/15:25
fricklerand the status says "missing, presumed fed"15:25
fungioh, that's where i updated my display name field, that account still has an e-mail address set15:26
fungiif you look in preferences there are now separate full name and display name fields15:26
fungii left my full name as Jeremy Stanley but set my display name to fungi (the field was previously blank)15:27
fungii haven't changed the e-mail address on that account15:27
fricklerfungi: yeah, but the mouseover doesn't show me your email address for some reason, it does do so for the other reviewers except zuul15:27
fungii bet showing the e-mail address is its fallback when display name is null15:29
fungilikely almost nobody has set a display name yet15:29
fricklerfungi: hmm, if I set my display name, the mouseover still shows both the full name and the email for myself15:31
frickleranyway, /me needs a break. do we have a meeting about the gerrit update later or no meeting and just tackle issues?15:33
fungii assume we'll have at least a brief meeting at 19:00 for anyone around to recap, though i haven't seen a meeting agenda15:34
fricklerclarkb mentioned something about not doing an agenda yesterday, but I wasn't sure whether that would mean no meeting, too15:34
fungiahh, yeah, i'm good either way15:35
*** lpetrut has quit IRC15:35
openstackgerritJay Faulkner proposed ttygroup/gertty master: Fix auth-type in opendev example config  https://review.opendev.org/c/ttygroup/gertty/+/76389015:36
clarkbfrickler: fungi I meant no agenda so that we could recap the upgrade during our meeting15:43
clarkbso still meeting but more low key15:43
zbrclarkb: i guess that the high load after upgrade explains the long push times (20-40s) and random delays in page load times, 5-15s, sometimes even timing out.15:51
clarkbthat is my hunch15:52
clarkbprevious to the upgrade we saw this from people scanning the gerrit api for change history15:52
clarkbso this isn't necessarily an upgrade specific issue, but it likely does need further debugging15:52
zbrwhat is not clear to me yet is this is the rush after the break or if we really have a notable regression in performance.15:52
clarkbthe next step is likely for gerrit admins to check the java melody dashbaord15:54
clarkbas that gives us a bit more info on what internally is consuming resources and taking time15:55
zbrthat graph does not paint a picture regarding number of pages served or average page serve time.15:55
zbri will not be surprised if admins find that performance is caused by a bunch of bots and not humans15:56
mwhahahamy gertty is getting read timeouts on a fairly frequent basis (likely related to the load stuff)15:56
clarkbzbr: correct its system level metrics. java melody gives us finer detail, but unfortunately cannot be exposed publicly because allows you to kill threads15:56
zbrmelody is quite nice and useful15:56
fungi...and dangerous15:56
* zbr could say the same about alcohol,... hmm, that gave him an idea.15:57
clarkbhistorically the things we've seen cause problems are GC'ing being slow (I don't think this is a likely cause here because memory use is low) and people running expensive queries often16:00
clarkbmelody should give us insight to both of those or at least further hints. I have meetings and then will see if I have time to look closer16:00
openstackgerritSorin Sbârnea proposed openstack/project-config master: add review-priority for tripleo-ci  https://review.opendev.org/c/openstack/project-config/+/71506916:03
*** rosmaita has joined #opendev16:04
openstackgerritPaul Belanger proposed zuul/zuul-jobs master: DNM - testing registry jobs  https://review.opendev.org/c/zuul/zuul-jobs/+/76400616:09
clarkbthere are multiple stackalytics accounts producing errors in the gerrit error_log due to Stream is already closed connection reset by peer16:09
clarkbI wonder if gerrit may continue to process those queries after the connection closed and we pile them up (somethign to look at in melody)16:09
*** hashar is now known as hasharAway16:11
clarkbfungi: I guess the process now is to add our normal account to admins temporarily to view the java melody stuff? /me works on that now16:12
fungioh, yep that's probably necessary unless we want to create a separate acl/group for melody16:12
*** mlavalle has joined #opendev16:13
clarkbgarbage collection looks fine. Active threads is "high"16:14
clarkbrelative to what it was like yesterday16:15
clarkbmemory looks stable too16:15
clarkbsorting threads by cpu time my suspicions about ssh queries seem valid16:17
clarkbbut it is also some zuul's as well I think. Notably our zuul does not show up there. I think because we use http for those queries16:18
clarkbfungi: can you add yourself to admins as well and take a look? make sure I'm not missing anything16:19
clarkbah except I think these threads are reused16:20
clarkbso specific queries aren't necessarily bad, more that we're spending a lot of time in ssh16:20
clarkbya refreshing the list we see that the threads remain but the queries change or go away16:21
clarkbI also notice that send email comments is another cpu consumer and gerrit show-queue shows we may only have one thread sending email16:22
fungithe error log is spammed by errors related to ssh sessions for stackalytics16:22
fungiwondering if it could be related16:22
clarkbfungi: ya, I think they may be contributing to the cpu time build up as they query over and over and fail16:22
clarkbI think we should try increasing sendemail.threadPoolSize from its default value of 1 as well16:22
funginot a terrible idea if there's a backlog16:23
clarkbfungi: ya checkout gerrit show-queue16:23
fungioh indeed16:23
fungicould also be related to the bug you discovered with watchers of all-projects getting spammed16:24
clarkbI'm thinking lets start by increasing that threadPoolSize value and see if things end up being happier16:24
clarkbseparately I'll go through the list I generated of people with those watches and maybe we start disabling them?16:24
clarkbactually lets do one thing at a time16:24
clarkbto have a clearer picture of what is going on. threadPoolSize is easy so lets start there16:25
clarkbfungi: for sendemail do you think ~4 is enough or maybe should be 8 ?16:25
fungiseeing if i can make an educated guess based on queue size and times16:26
fungilooks like it's nearly half an hour behind and has over 100 tasks waiting16:27
*** roman_g has quit IRC16:27
fungispends a while (15 seconds maybe?) on each of those16:29
fungisome seems to go faster than others, presumably determined by how many recipients there are16:29
clarkbseems likely16:29
clarkboh you know what we can do is set that other flag I foudn16:30
clarkbbut may do that after threads16:30
fungithe backlog is growing rather than shrinking, so i think on average it's taking longer to process each one than the frequency at which new entries are being enqueued16:31
fungifairly slowly though16:31
fungiso yeah a small thread count increase is likely sufficient for it to catch back up16:32
fungi4 seems like plenty16:32
clarkbcool. Lets start there, if we continue to backlog lets look at change.sendNewPatchsetEmails next (set that to false) then if still sad we look at cleaning up project watches?16:32
clarkbchange incoming16:32
*** chandankumar is now known as raukadah16:32
fungii can imagine some folks like to get e-mail notification of new patchsets for existing changes16:32
sshnaidmfungi, clarkb do you know if main page in new gerrit UI is customizable? I wonder if it's possible to see jobs results in main page without looking for them in comments.16:33
fungisshnaidm: it is customizable by creating plugins16:33
fungigerrit has some excelent developer docks about writing ui plugins16:34
fungier, docs16:34
sshnaidmfungi, java?16:34
clarkbsshnaidm: java and javascript16:34
clarkbsshnaidm: tristanC has started work on that16:34
openstackgerritClark Boylan proposed opendev/system-config master: Increase gerrit sendemail thread pool size  https://review.opendev.org/c/opendev/system-config/+/76401916:34
sshnaidmclarkb, great16:34
clarkbas a note we totally told peopel about this stuff like 6 weeks ago and asked for help then too :P16:35
clarkbfungi: think we should manually apply ^ then restart then approve it?16:35
clarkbfungi: I'm leaning towards yes16:35
fungiclarkb: yes, i concur16:36
fungisshnaidm: https://review.opendev.org/Documentation/pg-plugin-dev.html is the documentation for it btw16:37
*** roman_g has joined #opendev16:37
clarkbfungi: ok I'm in the root screen going to edit the config now if you want to check me16:38
fungioh, also someone commented on the webapp route bug saying they think it's reasonable to migrate away from using /x/* for plugins... sort of not the answer i was hoping for16:38
clarkbya :/16:39
clarkbfungi: config is update if it looks good to you16:39
fungiclarkb: yep, looks like what we discussed16:39
clarkbfungi: should i down then up -d now?16:39
clarkbactually maybe send a notice first?16:40
openstackgerritPaul Belanger proposed zuul/zuul-jobs master: Refresh intermediate TLS certs for testing  https://review.opendev.org/c/zuul/zuul-jobs/+/76402316:40
fungistatus notice The Gerrit service on review.opendev.org is being restarted quickly to troubleshoot an SMTP queuing backlog, downtime should be less than 5 minutes16:40
fungiclarkb: ^ like that?16:40
* fungi was already drafting it16:40
clarkbfungi: ++16:40
fungi#status notice The Gerrit service on review.opendev.org is being restarted quickly to troubleshoot an SMTP queuing backlog, downtime should be less than 5 minutes16:40
openstackstatusfungi: sending notice16:40
clarkbfungi: good to down up -d now?16:40
-openstackstatus- NOTICE: The Gerrit service on review.opendev.org is being restarted quickly to troubleshoot an SMTP queuing backlog, downtime should be less than 5 minutes16:41
fungigo for it16:41
openstackgerritPaul Belanger proposed zuul/zuul-jobs master: Switch to container_images for push-to-intermediate-registry  https://review.opendev.org/c/zuul/zuul-jobs/+/76383616:42
clarkbfor anyone following along I think our next step is to set change.sendNewPatchsetEmails to false and restart (I think this may have increaded the amount of email we send) and if that still is sad look at cleaning up all-projects watches like what cgoncalves had16:43
clarkband basically monitor between each step so we can see if things are improving16:43
clarkbshow-queue is currently much happer but that is to be expected after a restart16:43
openstackstatusfungi: finished sending notice16:44
fungiif we disable sendNewPatchsetEmails i think we ought to consider it temporary until the all-projects watch filtering bug is fixed16:44
clarkbfungi: well I think thati s a new feature16:45
fungibecause as i said, i wouldn't be surprised if some folks rely on e-mail notification of new patchsets for changes they're reviewing16:45
clarkbbefore you only got emailed if you explicitly watched things16:45
clarkbbut I may be wrong about that16:45
fungioh, maybe i should reread what that option really does then16:45
clarkbits an implied watch system for changes you own and review16:45
clarkbI'm not 100% sure if 2.13 did that by default too16:45
clarkbbut I don't seem to remember getting those emails if it did16:45
fungii recall 2.13 sending e-mail notification of each new patchset uploaded, because i explicitly filtered those into a separate trash inbox16:46
*** hasharAway is now known as hashar16:46
cgoncalvesclarkb, I can confirm that deleting *both* notifications settings I had on at the top of the list stopped the notification influx. just to reinforce: *both* need to be deleted16:47
*** roman_g has quit IRC16:48
clarkbcgoncalves: yup, fungi and I ended up spending some time yesterday workign to reproduce it to udnerstand it better16:48
clarkbcgoncalves: I also believe it is an upstream bug16:48
clarkbwhich I intend on filing as soon as things settle enough to do so16:48
clarkbmwhahaha: zbr: do you notice a difference now? looking at metric data it seems happier16:51
mwhahahawill keep an eye on it, not currently timing out16:52
mwhahahaseems snappy, was having some hangs earlier16:52
fungigranted, it may simply have cleaned up/disconnected/reset whatever was dragging it down, and it will begin to bog down again as it settles16:54
clarkbya to be seen if this was the fix we needed. I did check there are more email threads running in melody16:54
clarkbso the config option stuck16:54
clarkbfungi: I think we should approve the change I pushed/16:55
clarkbfungi: re the /x/* bug the good news is they've confirmed the minimal usage of that path so we really shouldn't have conflicts without current setup16:56
fungii just did16:56
fungiand yeah, i followed up with another comment on that bug16:56
mwhahahahrm tried abandoning via the ui and it's just spining17:00
*** timburke has quit IRC17:00
mwhahahathat took a while17:00
clarkbhrm ya load is climing again and looks like ssh threads are busy again. So email is likely a symptom and not a cause17:01
*** rpittau is now known as rpittau|afk17:01
tristanCclarkb: sshnaidm: i think the work is completed for the zuul result table, https://review.opendev.org/c/opendev/system-config/+/763891 should be enough17:01
clarkbfungi: https://gerrit-documentation.storage.googleapis.com/Documentation/2.13.13/config-gerrit.html#change doesn't show that new setting. Could still be it was unconfigurable default17:01
clarkbtristanC: ok, we're unlikely to deploy that in the near future. We're still battling issues and its a holiday week and all that17:02
fungitotal connection count is not super high, ~12817:03
sshnaidmtristanC, can we include showing running jobs also? Like in https://opendev.org/x/coats/src/branch/master/coats/openstack_gerrit_zuul_status.user.js17:03
clarkbI wonder if it is contention over locks for resources? We get queries like `gerrit query --format json --all-approvals --comments --commit-message --current-patch-set --dependencies --files --patch-sets --submit-records 737864` from a number of CI systems all asking for the same content on th esame change17:03
clarkband then they all go away and load drops back down again17:04
clarkbsshnaidm: I think we should avoid that17:04
clarkbsshnaidm: last time we tried it it killed zuul17:04
*** marios is now known as marios|out17:04
sshnaidmclarkb, wow17:04
clarkbbasicallylets do one thing at a time and ensure things are happy17:04
sshnaidmclarkb, agree to try one by one17:04
sshnaidmclarkb, I'm fine to keep using it with greasemonkey17:05
fungisshnaidm: we'd want to test it carefully. what tends to happen is people leave fifty gerrit changes open in different browser tabs and when a few hundred users do that all those queries beating on the zuul api eat up a ton of resources17:05
sshnaidmfungi, I see17:06
clarkbmwhahaha: ya looks like we're almost idle now. So it seems bursty when CI systems are fetchign chagne info. Still not sure if the CI system sare the cause (perhaps using http like zuul is better for some reason) or a symptom of something else going on17:06
clarkbwe'll have to continue to monitor it I guess17:06
mwhahahayea not setting gertty time outs at the moment, so it might just one of those things for now17:07
clarkbcould even be contention between email event thread locks and query locks from the CI side and flushing email more quickly results in less contention. Lots of things could be going on. Hopefully melody and further observation will allow us to better characterize it17:07
clarkbbunch of CI queries for 763997 currently17:08
*** roman_g has joined #opendev17:09
clarkbif we had better coordination with CI systems it would be nice to flip them over to http and see if things change at all17:09
clarkbin particular upstream zuul's queue lengths as reported by the dashboard are basically empty17:09
fungiinterestingly, even though the system load average is relatively high, cpu is still mostly idle17:09
clarkbfungi: ya that is why I suspect locks?17:10
clarkbiowait is nil17:10
clarkb(double check me on that)17:10
fungiit is, yes17:10
clarkbalso zuul uses http and if you look at its dashboard it reports empty queues (implying to me that it isn't waiting for long periods for this info)17:10
fungiinterrupt handling is relatively minimal as well17:10
*** roman_g has quit IRC17:11
clarkbI'm beginning to suspect that those ssh queries that zuul does over ssh are not very efficient for some reason17:11
fungistill no smtp backlog, ssh connection count has climbed slightly17:11
clarkband melody report ssh workers are dominating the time again (and show-queue seems to imply the time there is spent doing those ci queries)17:12
fungisendemail threads are occasionally complaining about "Account ... has invalid filter in project watch"17:13
clarkbthe tracebacks for the ssh threads from those ci systems show they are doing external id lookups in jgit17:15
clarkbNow I had thought that this is what the account index is for17:16
fungiare ssh keys external ids now?17:16
fungigranted the username has always been considered an external id17:16
clarkbthey are in ergular account info17:16
clarkbthis could just be looking up data to go with the comments and stuff that the CI system is querying17:17
fungior at least username used to be stored in the account_external_ids table in the old db17:17
clarkbthe traceback does show it going through a cache layer though17:17
clarkbthe caches are h2 db's iirc17:17
clarkbso basically this implies to me that we aren't hitting the account index nor the cache for some reason17:18
clarkbperhaps we're just stable beacuse we restarted?17:18
clarkbor maybe there is a bug (and http doesn't have it and that is why zuul.opendev is happier?) and they aren't properly looking up in those areas from ssh?17:18
clarkbit wouldn't surprise me considering that upstream doesn't enable sshd17:18
fungithe historical system load graph in cacti indicates that load for the past two days is an order of magnitude higher than normal, and inline with when we were handling the research crawler's queries17:20
clarkbnext time we have a number of those threads I'll use melody to get a thread dump then maybe we can file a bug and see what upstream thinks17:20
fungior higher17:20
clarkbbut I do find it odd they seem to be spending a bunch of time with account stuff when that should all be indexed and cached.17:20
clarkbanother approach we can try is reach out to these ci systems and ask them to use http for those lookups17:22
clarkband see if that changes things17:22
fungiunless we need to reindex after the file lock got unacquired for a bit yesterday and it was failing to update the index?17:22
clarkbsean-k-mooney: ^ you're in the list :) maybe you're willing to be our first test of that17:22
clarkbfungi: hrm that seems like it could be possible17:23
clarkbfungi: do you want to trigger an online reindex for accounts?17:23
fungiyeah, just revisiting the syntax for it now17:24
sean-k-mooneywhat do i have to do?17:24
clarkbsean-k-mooney: one sec I'll find details17:24
fungiclarkb: "Nothing to reindex, index is already the latest version" maybe i should add --force?17:25
sean-k-mooneyis it my sean-k-mooney account or sean-mooney-ci account17:26
clarkbsean-k-mooney: the ci account, its from zuul17:27
fungii went ahead and did it again with --force17:27
clarkbfungi: k17:27
clarkbsean-k-mooney: its doing queries on changes to see change attributes and we're theorizing that this may be slow because its bypassing indexes and caches17:27
clarkbsean-k-mooney: but still trying to collect more data17:27
sean-k-mooneyah i see well i stil have the ci in debug mode because i didnt get around to not doing that17:28
sean-k-mooneywould logs help?17:28
clarkbsean-k-mooney: no I think this is normal zuul operations17:28
clarkbsean-k-mooney: but we're thinking that if you use http to query this stuff it may be happier becauase our zuul doesn't seem to exhibit these issues (it uses http)17:28
clarkbsean-k-mooney: trying to get our config to show you what we do17:29
sean-k-mooneysure i can try swaping it over17:29
fungisean-k-mooney: https://zuul-ci.org/docs/zuul/reference/drivers/gerrit.html#attr-%3Cgerrit%20connection%3E.password17:29
clarkboh thanks17:29
clarkbapparently our config is secret beacuse it has secrets17:29
clarkbthat is why I couldn't find it on gitea17:30
sean-k-mooneywe need to use basic auth or digest?17:30
fungibasic now17:30
sean-k-mooneyand just remoe the sshkey line17:30
clarkbsean-k-mooney: no, it will still use ssh for other things17:30
clarkblike stream events17:30
fungii think you still need the sshkey for the event streamm right17:31
clarkbbut it will prefer http for everything that it can do over http17:31
sean-k-mooneyah ok17:31
fungiyeah, gerrit has never added an event stream over http17:31
fungithough the checks plugin was sort of getting there17:32
clarkbfungi: looks like show queue shows the reindex is done? I wonder if the log will tell us if it was happy17:33
fungi[2020-11-24T17:32:19.832+0000] [Reindex accounts v11-v11] INFO  com.google.gerrit.server.index.OnlineReindexer : Reindex accounts to version 11 complete17:33
fungi[2020-11-24T17:32:19.832+0000] [Reindex accounts v11-v11] INFO  com.google.gerrit.server.index.OnlineReindexer : Using accounts schema version 1117:33
clarkbya just found it too17:33
fungilooks right17:33
clarkbcool, I guess we see if those tracebacks come back17:34
sean-k-mooneyfull reconfigure enough or should i do a full service restart17:34
clarkber thread states with tracebacks in melody17:34
clarkbsean-k-mooney: I think a restart for anything in the ini config17:34
clarkbreconfigure will do the yaml side17:34
sean-k-mooneycool ill do that so17:34
clarkbI'm being dragged to breakfast. I'ev got the pre reindex thread dumps in my browser and can get post redinex threaddumps if they persist17:34
sean-k-mooneyi can pick up my debug logs changes too then :)17:34
clarkbbut then maybe file the cgoncalves bug, this issue, then meeting17:35
clarkbsean-k-mooney: if you haven't restarted yet maybe wait a minute17:35
clarkbI can double check our config for any other flags that may be necessary17:36
clarkbnope looks like the http token is the only thing17:36
sean-k-mooneyi have jsut stop the scheduler17:36
clarkbno worries I think you're good17:36
sean-k-mooneyi can put my config in an etherpad if you want to edit it17:37
clarkbno I think that was all you need17:37
clarkbI just couldn't remmeber if there was another flag to say prefer http17:37
clarkbbut that config option does it17:37
sean-k-mooneycool ill start it again so . i did not see anything else that seamed related17:37
clarkband now really steping away for sustenance17:38
sean-k-mooneyshould be running let me know if you want me to do anything else17:40
sean-k-mooneyit might be just becasue its starting up but the log output seams faster as if its processing updates to the repos faster17:41
sean-k-mooneyoh its used for comment reporting and is need for the line in file reprotign like the pep8 job does cool17:45
fungilooks like system load average on review.o.o is climbing again17:49
fungiand now falling again. that was relatively brief17:49
sean-k-mooneydid ye do a restart a few minutes ago17:50
sean-k-mooneyor was that an hour ago17:50
*** marios|out has quit IRC17:50
fungisean-k-mooney: at 16:41z so yes a little over an hour ago17:51
sean-k-mooneyok was going to sugget maybe it was cis reconnecting or soemthing17:51
sean-k-mooneyzuul is quite chatty as its loading all its config initally17:52
sean-k-mooneyafter that its basically just reactive to the gerrit event stream17:52
*** iurygregory has quit IRC17:59
*** dtantsur is now known as dtantsur|afk18:01
fungiwell, we were seeing it before the restart too. one of the things we were hoping might clear up with the config change we made to increase smtp threads18:02
*** fressi has joined #opendev18:05
*** mgoddard has quit IRC18:07
tristanCsshnaidm: i think that can be added, but right now the implementation is really simple, it just display build result in a table under the commit message from the zuul comment18:09
sshnaidmtristanC, that's fine, I think we can tweak it to work with your plugin together18:09
tristanCthough we don't know where is the zuul rest api from the gerrit comment only, so to display build progress bar, we need a configuration option to indicate where zuul is running18:09
sshnaidmtristanC, anyway I use it in browser scripts18:10
tristanCi wrote that zuul-results plugin using the https://gerrit-review.googlesource.com/Documentation/js-api.html api18:10
fungiclarkb: load average has bottomed out (i saw it below 4 a few moments ago). noticing that show-queue -w does include the usernames at the ends of queries so can provide a bit of a snapshot of what's interacting at any point in time. arista-test seems to be almost always present18:13
fungilooks like it might be cloning nova?18:13
clarkbya that looks like a clone18:15
fungithat accounts had a "git-upload-pack /openstack/nova.git" for 8 minutes already according to the queue timestamps18:15
clarkbnova is >1GB now iirc. Hopeflly they're caching it18:15
clarkbfungi: I wonder then if the account index lock issue was kick us out to direct access on those queries if things are happier now18:15
fungigerrit show-queue -w|grep ')$'|sed 's/.* (\(.*\))$/\1/'|sort|uniq -c|sort -n18:15
fungiprovides an interesting view on account use at different points in time18:16
fungiokay, arista-test finished the nova clone18:16
fungior it was terminated somehow18:16
clarkbmelody does seem to indicate things are largely happy?18:17
fungino user-related tasks in the queue a moment ago when i checked18:17
clarkbI'll remove myself from admins now18:17
clarkband if this continues to be happy we can attach the thread dumps I got to the account index bug?18:17
fungiworried this is just a sample of what we'll see next week when so many people aren't vacationing18:18
clarkbfungi: based on zuul queues we're pretty active right now18:18
clarkb133 chagnes in check and 43 in gate18:18
clarkbthats pretty normal18:18
clarkbbut also I can't tell if you think things are happier now or less happy :)18:19
*** andrewbonney has quit IRC18:19
fungii try not to get my hopes up ;)18:20
clarkbif it spieks again I think the thing to do is get into melody and get the threads as a txt dump18:20
clarkbthat includes the stacks for each thread and you can see if we're spending time in jgit for externalids again18:20
clarkbbut doesn't seem like we're doing thatn ow at least so maybe that issue was resolved by the reindex18:20
fungimmm... maybe. i did observe system load climb up to almost 40 around 17:49z but only for a few minutes18:23
clarkbfungi: I put the stacks in review.o.o:~clarkb/gerrit-3.2-slow-external-id-lookups for when I caught it an hour ago or whatever it was18:25
clarkbmy hunch since things have been quieter since is that the account reindex did help18:25
*** yumiriam has joined #opendev18:26
clarkband if that holds up I think we clean up ^ that file to remove account info and then attach it to your existing bug as a related thing. If we see the issue return without index problems then we create a new bug?18:26
fungialso doesn't seem to have lost the lock on the accounts index again yet18:26
*** iurygregory has joined #opendev18:27
clarkbbecause ya I wonder if that state makred the index and cache as dirty so we went to disk for everything18:27
fungiyeah, i suppose it could be helpful on the existing bug, though it's more of a symptom than anything18:27
clarkband it was probably locking around those reads18:27
fungithough if so, it's surprising that reindex without --force said there was nothing to do18:27
clarkbfungi: I think that is only checking if the index itself says it is done18:27
openstackgerritMerged opendev/system-config master: Increase gerrit sendemail thread pool size  https://review.opendev.org/c/opendev/system-config/+/76401918:32
*** yumiriam has quit IRC18:36
clarkbhttps://bugs.chromium.org/p/gerrit/issues/detail?id=13733 has been filed upstream about cgoncalves issue18:37
clarkbfungi: re the index force thing I'm fairly certain it is just checking the schema version18:40
*** eolivare_ has quit IRC18:45
fungiahh, okay18:48
*** whoami-rajat__ has quit IRC18:53
openstackgerritAlec Hothan proposed openstack/project-config master: Enable python 3.6 job only for x/vmtp project.  https://review.opendev.org/c/openstack/project-config/+/76405419:19
openstackgerritMerged zuul/zuul-jobs master: Use Python 3.x with launchpadlib  https://review.opendev.org/c/zuul/zuul-jobs/+/76383419:23
openstackgerritMerged opendev/gerritlib master: Handle empty reads as closed connections  https://review.opendev.org/c/opendev/gerritlib/+/76389219:28
*** DSpider has quit IRC19:34
clarkbfungi: arista seems to be cloning nova again19:37
clarkbI wonder if they are doing authenticated clones to gerrit for every job or something like that19:37
fungiwouldn't surprise me19:38
ianwit's definitely these ssh threads19:40
ianwif you strace on one of the threads you can see what it's up to19:40
ianwstat("/var/gerrit/git/All-Users.git/HEAD", {st_mode=S_IFREG|0644, st_size=22, ...}) = 019:44
ianwread(372, "ref: refs/meta/config\n", 22) = 2219:44
ianwread(372, "", 22)                       = 019:44
ianwclose(372)                              = 019:44
fungiso i guess it's busy mapping user info for the change details?19:45
ianw... over and over and over19:45
clarkbianw: if you can characterize it even a little bit I bet that wouldbe worth an upstream bug19:48
clarkbianw: also call out that similar queries from CI systems over http don't seem to have this issue19:48
clarkbmaybe ssh is simply bypassing a cache that it shouldn't19:48
clarkband upstream doesn't notice because they disable ssh19:48
mnaserare we talking about potentially an issue with slow gerrit ops?19:49
clarkbmnaser: we've been talking about it all day :P19:49
clarkbmnaser: the tldr seems to be that ci systems doing change info lookups over ssh (but not http) induces slowness19:49
mnaserclarkb: ah, that would be a hard one to sort out...19:49
clarkband that seems to be related to the ssh path doing al ot more git lookups than the http one19:50
clarkbso our zuul does a lookup quickly and its fine. But third party CIs dont19:50
mnaserfor me the symptoms i notice is on what seems to be "write" ops, they seem to take a while19:50
mnaserso viewing a change is fine, but reviewing one / abandoning / etc all take time19:50
clarkbmnaser: I think locking is invovled. We see basically no iowait and low cpu use for the system load but system load ends up fairly high19:51
clarkband for reading you probably don't need locks19:51
clarkbfor writing I bet it does19:51
clarkbs/for the system load/compared to system load/19:51
mnaseris gerrit running on ssds?19:52
clarkbmnaser: it is running on an "ssd" cinder volume19:52
clarkbso as far as we are able to say: yes19:52
mnaserhaha :)19:52
fungiopenstack doesn't lie19:52
clarkbmnaser: fwiw you reviewed the cahgne that is causing all the problem right now19:52
clarkbhttps://review.opendev.org/c/openstack/magnum/+/763997/ that one is what a number of ci systems are doing lookups against19:53
clarkbunfortunately the pushing of three patchsets in quick succession there made it worse19:53
mnaseri wonder if ci systems are doing this lookup on every comment19:53
clarkbbeacuse each one has to be queries19:53
clarkbmnaser: no its the patchset created events19:53
clarkbI think, Ihaven't fully tracked it back to the zuul code19:53
mnaserwow so a whole 20 minutes after19:53
clarkbgerrit query --format json --all-approvals --comments --commit-message --current-patch-set --dependencies --files --patch-sets --submit-records19:54
clarkbmnaser: ya because its doing it for each patchset19:54
clarkband each one is slow19:54
mnaserwe probably don't really have a record of how quick this thing was before19:54
mnasercause we never used it anyways..19:54
mnaserssh lookups19:54
clarkbthey were always used before by these CI systems19:54
mnaseryou mentioned opendev's zuul relied on http looups19:54
clarkbbut the bulk of this info was coming out of the sql database19:55
clarkbso the code paths are compltely different now19:55
mnaserwouldn't both ssh and http implementations slow down though if notedb was the case?19:55
mnaser(sorry, i'm sure this conversation has already been had today)19:55
mnaser(this is for my selfish curiosity so if y'all are busy debugging, feel free to leave all my questions for later :p)19:56
fungiyes recent zuuls configured with an http password don't query change details over ssh, but those configured without one or zuul v2 or other ci systems which only used ssh seem to be what's putting heavy load on things currently19:56
clarkbmnaser: my hunch is the http path is using cache and indexes more aggressively19:56
clarkbmnaser: using that example change though our zuul started running jobs for ps4 almost immediately19:57
fungiand it looks like it's because queries via ssh are causing direct reads to notedb to get things like user details to include in the query results, while querying that over rest api seems to use caches19:57
clarkbwhereas based on the gerrit show queue output the other cis are still cranking away at it19:57
clarkbalso upstream doesn't use ssh19:57
fungithey don't even expose an ssh socket19:57
clarkbso this is one of those areas where it would likely be easy for them to miss regressions. Our best bet is to characterize it as well as we can and file a bug/start a mailing list thread19:58
fungior if they do it's very selective (maybe the upstream gerrit's zuul is still using stream-events?)19:58
mnaserperhaps pinging our friends at gerrithub if they have that option enabled :)19:58
clarkband in the meantime if people want to reach out to third party ci systems and suggest they use the http functionality that might be a good thing19:58
fungi(or is it relying entirely on queues in the checks plugin?)19:58
clarkbfungi: it is relying entirely on checks plugins19:58
clarkbfungi: gogoel doesn't allow ssh is the issue aiui19:58
fungiahh, okay, so then yes i guess upstream gerrit's gerrit has no ssh allowed at all19:59
clarkbwhen you psuh code you set up "git cookies" and that authenticates you via http19:59
mnaserso there's https://github.com/GerritCodeReview/gerrit/blob/stable-3.3/java/com/google/gerrit/server/query/change/OutputStreamQuery.java19:59
fungiout of curiosity how do they serve the commti hook to add change ids?19:59
mnaser`final QueryStatsAttribute stats = new QueryStatsAttribute();`20:00
mnaserit looks like gerrit is collecting 'query stats'20:00
mnasermaybe if we can find a way to access those20:00
clarkbfungi: http (its always been that way iirc)20:00
fungiinteresting. git-review always fetched it via scp20:00
mnaserseems like the query stats are only printed out in the response itself20:00
clarkbfungi: I used git review to get it from upstream gerrit yseterday20:01
fungimaybe i've forgotten adding http support for fetching that commit hook20:01
clarkbya git review supports http and https for it20:02
clarkb/tools/hooks/commit-msg is the path for it via http according to git-review20:02
mnaserdo we have any plugins inside our gerrit install?20:02
fungiindeed, set_hooks_commit_msg() does try via http/https first20:03
fungiand falls back to scp now20:03
clarkbmnaser: a small number, mostly official20:03
clarkbits all in the gerrit job info20:04
mnaseri see, looking at this output stream query stuff, it seems to loop every change through plugins that can 'augment' the info20:04
mnaseri guess we can compare https://github.com/GerritCodeReview/gerrit/blob/stable-3.3/java/com/google/gerrit/sshd/commands/Query.java to https://github.com/GerritCodeReview/gerrit/blob/stable-3.3/java/com/google/gerrit/server/restapi/change/QueryChanges.java20:06
clarkbI would compare stable-3.2 but ya that would be one appraoch20:06
mnaserit looks like it is a similar api20:07
mnaser`processor.query(join(query, " "));` vs `queryProcessor.query(qb.parse(queries));`20:08
mnaserso i guess maybe caching is happening at a higher layer (or the old query hits harder and is maybe not as efficent?)20:08
mnaseri dunno, i don't understand it all enough, but maybe a little bit of digging here helped :<20:09
*** sboyron has quit IRC20:09
clarkbmnaser: ya I do wonder if the caching might be happening in http sessions tate or somethign like that20:12
*** hamalq has joined #opendev20:16
*** sboyron has joined #opendev20:18
hasharmnaser: about the QueryStats / OutputStreamQuery.java , that is Gerrit measuring how long it took to process a query which is reported back to the user in the response payload as "runTimeMilliseconds"20:27
hasharfor queries over ssh there should be some timing stats added in the sshd_log20:28
clarkbhashar: oh thanks for that pointer that may be useful20:28
hashargit-upload-pack responses have finely grained details about the time it took.  A series of -1 -1 -1 .. indicates a full clone,  else that is a fetch of some sort20:28
hashardo you happen to have a prometheus collector and the metrics-reporter-prometheus plugin ?20:29
hashar(to be fair I only deployed that a few days ago for wikimedia but that has been a life changer!)20:30
hasharwe now have history of caches misses  ( https://grafana.wikimedia.org/d/vHQVaGsWk/caches-upstream?orgId=1 )20:30
hasharand even query latencies https://grafana.wikimedia.org/d/IDkFbYeWz/latency-upstream?viewPanel=6&orgId=1&refresh=1m20:30
hasharthe devil is of course to find out what is being crippled exactly :-\20:32
fricklerclarkb: fungi: fwiw I'm still missing gerrit mails with zuul responses, e.g. for https://review.opendev.org/c/openstack/project-config/+/763888 I got a mail with the recheck, but not with the zuul +120:39
fungipossible that was lost in the queue when we restarted?20:40
fungiahh, no, that's more recent than the restart20:40
fungifrickler: maybe gerrit doesn't send e-mails for bot comments?20:41
clarkbdo your project watches have a query maybe? oh thats another good theory too20:41
fungizuul specially marks its comments/votes as automated, and i wouldn't be surprised if gerrit intentionally leaves out notifications for those20:41
fungifrickler: you might try adjusting your preferences for "email notifications" to the "every comment" option20:44
openstackgerritPaul Belanger proposed zuul/zuul-jobs master: Switch to container_images for push-to-intermediate-registry  https://review.opendev.org/c/zuul/zuul-jobs/+/76383620:44
sean-k-mooneyo/ did swappign to http have any effect?20:46
fricklerfungi: I have all notifications enabled. I did get mails with zuul votes earlier, also after the upgrade, but it seems to have stopped sometime between yesterday and today20:46
clarkbsean-k-mooney: yes, and your CI system should be happier20:47
clarkbsean-k-mooney: ianw in particualr is still digging in but getting close to understanding this we think20:47
openstackgerritDaniel Blixt proposed zuul/zuul-jobs master: Use script to populate test file tree fixtures  https://review.opendev.org/c/zuul/zuul-jobs/+/76406220:48
sean-k-mooneyclarkb: updating refs in the git repos seam to be faster20:53
sean-k-mooneythat just anicdotial since i have not messgured it but the logs were updating much faster after a gerrit event20:54
*** hamalq has quit IRC20:55
*** hashar has quit IRC20:59
*** hamalq has joined #opendev21:10
*** hamalq has quit IRC21:15
*** ralonsoh has quit IRC21:17
*** sboyron has quit IRC21:34
*** sboyron has joined #opendev21:34
*** sboyron has quit IRC21:58
*** sboyron has joined #opendev21:59
fungiclarkb: *sigh* https://gerrit-review.googlesource.com/28984222:01
clarkbfungi: I mean I guess they can deprecate it and we cam keep pushing the broader issue22:02
openstackgerritAlec Hothan proposed openstack/project-config master: Enable python 3.6 job only for x/vmtp project.  https://review.opendev.org/c/openstack/project-config/+/76405422:03
*** sboyron has quit IRC22:11
*** sboyron has joined #opendev22:11
*** sboyron has quit IRC22:12
*** sboyron has joined #opendev22:13
*** jentoio has quit IRC22:14
openstackgerritPaul Belanger proposed zuul/zuul-jobs master: Switch to container_images for push-to-intermediate-registry  https://review.opendev.org/c/zuul/zuul-jobs/+/76383622:14
*** sboyron has quit IRC22:15
*** sboyron has joined #opendev22:16
*** sboyron has quit IRC22:20
*** sboyron has joined #opendev22:20
ianwclarkb: if you have a quick sec, would you mind a run over https://review.opendev.org/q/topic:dib-3.4.0 which is two little fixes; one for the kubernetes job and then bump dib dependency.  this is required to get centos builds working again so i can put nb01/02 back in operation22:24
*** sboyron has quit IRC22:25
clarkbI'll try22:26
*** sboyron has joined #opendev22:26
*** jmorgan has joined #opendev22:29
*** sboyron has quit IRC22:33
*** sboyron has joined #opendev22:34
ianwok, back to gerritbot22:38
*** sboyron has quit IRC22:40
*** sboyron has joined #opendev22:41
openstackgerritIan Wienand proposed opendev/gerritbot master: Build against gerritlib master branch  https://review.opendev.org/c/opendev/gerritbot/+/76392722:42
ianwfungi: ^ was just missing the required-projects on the upload step22:45
fungioh, heh22:46
fungithat makes sense22:46
*** sboyron has quit IRC22:48
*** sboyron has joined #opendev22:48
*** sboyron has quit IRC22:57
*** sboyron has joined #opendev23:07
*** slaweq has quit IRC23:09
openstackgerritMerged opendev/gerritbot master: Build against gerritlib master branch  https://review.opendev.org/c/opendev/gerritbot/+/76392723:11
*** slaweq has joined #opendev23:28
*** slaweq has quit IRC23:33
*** hamalq has joined #opendev23:37
*** hamalq has quit IRC23:41
*** hamalq has joined #opendev23:52
*** tosky has quit IRC23:57
ianwgoing to try the gerritbot container that should have uploaded with master gerritlib23:57
*** hamalq has quit IRC23:57
*** openstackgerrit has quit IRC23:59

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!