Friday, 2020-12-11

ianwhahaha "git gc --prune" makes the repo *bigger*00:01
corvusit's like when you empty out a box of stuff on the floor to get everything organized and then don't.00:02
corvusianw: at one point we had some really specific repack arguments we used with gerrit; unsure what the current state is00:03
fungior when you get that box out of the attic and dig through it to find something, then try to put everything else back but it no longer fits00:03
ianwcorvus: did it look like ?00:04
ianwi'm going to try an explicit repack with those opts00:04
corvusianw: about 1/4 as complex, but window/depth etc were set specifically00:05
corvuslooks like the current state is: find /home/gerrit2/review_site/git/ -type d -name "*.git" -print -exec git --git-dir="{}" gc \;00:06
corvusso i guess the last time we did this, we decided we couldn't do any better than a plain gc00:06
corvusbut, you know, things change :)00:06
ianwdefinitely repacking with 2.28.0 gave the 10x decrease00:08
ianwfungi: if you have a sec, would you mind another pass at the bup removal @  i just had to stop it trying to manage the bup server, as with no clients it hit an undefined var error00:23
openstackgerritIan Wienand proposed opendev/system-config master: centos mirror: exclude dot-files
fungican do00:40
openstackgerritMerged opendev/system-config master: bup: Remove from hosts
*** DSpider has quit IRC01:05
openstackgerritAkihiro Motoki proposed zuul/zuul-jobs master: nodejs-test-dependencies: Install xdg-utils
ianwthe git in the gitea containers is 2.20.1 ... although not 100% sure if gitea uses that or some sort of go-git-magic?01:07
ianwthis is the size of every nova repo on the gitea hosts01:21
fungiinterestingly, /home/gerrit2/review_site/git/openstack/nova.git on review.o.o is a mere 964M01:24
fungiand 1.1gb on ze12 where it just recloned a few hours ago01:24
ianwi am more confused than ever.  how does "git clone" return ~160mb repo?01:28
*** mlavalle has quit IRC01:38
fungior 250mb in my case (possibly owing to the inclusion of git notes)01:41
fungii blame newer git for being more efficient01:42
openstackgerritMerged opendev/system-config master: centos mirror: exclude dot-files
openstackgerritAndrii Ostapenko proposed opendev/system-config master: Add focal octopus mirror
*** hamalq_ has quit IRC02:05
*** smcginnis has joined #opendev02:19
*** smcginnis has quit IRC02:24
fungii suppose nowish is a good time for a gerrit service restart... any objections?02:44
fungii'll take that resounding silence as tacit assent02:52
fungi#status notice The Gerrit service on is being restarted quickly to enable support for Git protocol v2, downtime should be less than 5 minutes02:52
openstackstatusfungi: sending notice02:52
-openstackstatus- NOTICE: The Gerrit service on is being restarted quickly to enable support for Git protocol v2, downtime should be less than 5 minutes02:53
ianwheh sorry ++02:54
ianwi give up on this git packing stuff.  i don't know what's changed or where02:55
fungiit's a headscratcher for sure02:55
openstackstatusfungi: finished sending notice02:55
ianwfungi: so should we add a v2 flag to /home/zuul/.git config on an executor, or two?02:56
fungii think so02:57
ianwze01 seems like a logical place to start02:58
ianwzuul@ze01:~$ cat .gitconfig03:00
ianwversion = 203:00
fungilooks right03:00
openstackgerritMerged opendev/system-config master: Add focal octopus mirror
*** ykarel has joined #opendev05:40
openstackgerritOpenStack Proposal Bot proposed openstack/project-config master: Normalize projects.yaml
*** marios has joined #opendev06:39
*** marios is now known as marios|ruck06:40
*** adrian-a has joined #opendev06:49
*** adrian-a has quit IRC07:05
*** adrian-a has joined #opendev07:09
*** marios|ruck is now known as marios|rover07:10
fricklerianw: I did the repack test on focal. "git clone" gives me 1.1G, running "git repack -a -d -F" gets that down to 219M, similar to the clone from gitea07:20
fricklergit version 1:2.25.1-1ubuntu307:21
fricklerI'll add that to the etherpad07:21
fricklerfrom the log during cloning, it also looks like gitea may be compressing during the clone, while gerrit doesn't07:32
*** slaweq has joined #opendev07:45
*** sboyron has joined #opendev07:54
*** hamalq has joined #opendev07:56
*** rpittau|afk is now known as rpittau07:57
*** hamalq_ has joined #opendev07:57
*** eolivare has joined #opendev08:00
*** hamalq has quit IRC08:01
*** fressi has joined #opendev08:02
*** smcginnis has joined #opendev08:10
*** user_19173783170 has joined #opendev08:12
*** smcginnis has quit IRC08:15
*** andrewbonney has joined #opendev08:22
*** zbr has quit IRC08:23
*** zbr has joined #opendev08:23
*** zbr has quit IRC08:26
*** zbr has joined #opendev08:26
*** user_19173783170 has quit IRC08:40
*** tosky has joined #opendev08:47
*** hashar has joined #opendev08:51
priteauGerrit is very slow this morning. I've just seen this error:09:00
priteauError 400 (Bad Request): limit of 10 queries09:01
priteauEndpoint: /changes/09:01
*** adrian-a has quit IRC09:03
gibiseconded, gerrit timed out on me couple times in the past 20 minutes09:06
yoctozeptonow it is timeouting permanently on me09:07
slaweqI don't see timeouts but it's veeeery slow for me too :/09:09
yoctozeptogibi: I also noticed our userscript starts failing in these situations due to failed observation assumptions09:11
*** adrian-a has joined #opendev09:12
gibiyoctozepto: yeah, that use script is not bug free :)09:13
gibiyoctozepto: also if you navigate from a list of commit ot a single commit sometimes the script fails09:13
yoctozeptogibi: yes, the disclaimer is there! :D09:15
*** hamalq_ has quit IRC09:15
yoctozeptoanyhow, I am not touching it anymore; I'll try to find some time to play with polygerrit itself09:15
*** hamalq has joined #opendev09:15
gibiyoctozepto: i'm also consider that script as good-enough09:34
ianwyoctozepto: i haven't sent another mail about it but we're working on getting into gerrit's repo.  it's a polygerrit/polymer version09:44
yoctozeptoianw: oh, lovely!09:45
yoctozeptothat would make a great starting point for getting the dynamic too09:46
*** dtantsur|afk is now known as dtantsur09:58
ianwi found some problem queries against gerrit.  it should hopefully be running a bit better now10:02
*** brinzhang has quit IRC10:04
*** brinzhang has joined #opendev10:04
ianwyoctozepto: if you mean "show what's currently running by querying the zuul api" there's already a plugin that does similar to that10:05
ianwif we ever get past these fire drills with gerrit performance, i'll finally have much better testing we can put a gerrit instance on hold with a bunch of sample data, and you/anyone can play with plugin development all day long10:06
*** hamalq has quit IRC10:10
*** smcginnis has joined #opendev10:11
*** smcginnis has quit IRC10:16
*** DSpider has joined #opendev10:17
openstackgerritchandan kumar proposed openstack/diskimage-builder master: Enable dracut list installed modules
*** adrian-a has quit IRC10:21
*** adrian-a has joined #opendev10:45
yoctozeptoianw: that would be great! yes, I mean that; I only know the userscript by Michael, me and gibi10:49
*** adrian-a has quit IRC11:28
*** adrian-a has joined #opendev11:33
*** slaweq_ has joined #opendev11:46
*** dtantsur is now known as dtantsur|brb11:56
*** sshnaidm is now known as sshnaidm|off11:59
*** slaweq has quit IRC12:02
*** slaweq_ has quit IRC12:03
*** slaweq has joined #opendev12:06
*** ykarel_ has joined #opendev12:14
*** ykarel has quit IRC12:16
*** slaweq has quit IRC12:20
*** slaweq has joined #opendev12:21
*** eolivare has quit IRC12:21
*** eolivare has joined #opendev12:22
*** kopecmartin has quit IRC12:22
*** kopecmartin has joined #opendev12:26
*** dtantsur|brb is now known as dtantsur12:27
dtantsura bug report for zuul.o.o: information from a build list cannot be copied with a mouse. e.g. try copying a date/time from
*** adrian-a has quit IRC12:58
*** adrian-a has joined #opendev12:59
fungidtantsur: thanks, i believe ianw has a ui change proposed to fix that13:04
dtantsurI don't understand a thing, but looks impressive, and the commit message seems relevant :)13:06
*** smcginnis has joined #opendev13:16
*** fbo|off is now known as fbo13:18
fricklerpriteau: the query limit error is not related to slowness, this seems a general restriction of the new gerrit version. in fact the limit seems to have been the same earlier, but it seems queries are expanded in a different way now, I've seen multiple dashboards that worked earlier and now don't13:31
fricklerseems currently the only solution is to simplify your dashboard. assuming that it was one that you tried to access, but that's the only context where I've seen that error so far13:32
fungiit also couldn't hurt to look at what the query is being translated into. in watching apache access logs i've noticed some very dysfunctional looking queries from some users and am not sure what's generating them, but they seem to include long embedded sequences of ++++++++++13:40
fungiif gerrit is treating all those as additional query terms, it might hit the query term limit much more easily13:41
fricklerfungi: good idea, I'll check that for the designate dashboard13:41
*** smcginnis has quit IRC13:42
*** smcginnis has joined #opendev13:56
*** mlavalle has joined #opendev13:58
*** ykarel_ is now known as ykarel14:01
openstackgerritJan Zerebecki proposed zuul/zuul-jobs master: Switch from Debian Stretch to Buster
*** adrian-a has quit IRC14:32
*** adrian-a has joined #opendev14:34
*** klonn has joined #opendev15:01
*** klonn has quit IRC15:13
*** klonn has joined #opendev15:14
openstackgerritVishal Manchanda proposed zuul/zuul-jobs master: [DNM] Testing horizon nodejs job with Chrome browser
*** dtantsur is now known as dtantsur|brb15:15
*** rpittau is now known as rpittau|afk15:15
*** klonn has quit IRC15:18
*** klonn has joined #opendev15:39
openstackgerritJan Zerebecki proposed zuul/zuul-jobs master: Switch from Debian Stretch to Buster
*** adrian-a has quit IRC15:45
openstackgerritJan Zerebecki proposed zuul/zuul-jobs master: Better error output for
openstackgerritJan Zerebecki proposed zuul/zuul-jobs master: ensure-pip: install virtualenv, it is still used
openstackgerritAlbin Vass proposed zuul/zuul-jobs master: Add option to install kubernetes with kind
openstackgerritAlbin Vass proposed zuul/zuul-jobs master: Update ensure-helm to default to latest version
openstackgerritAlbin Vass proposed zuul/zuul-jobs master: Add helm role
openstackgerritAlbin Vass proposed zuul/zuul-jobs master: Add functional helm test jobs using kind
*** zbr3 has joined #opendev16:00
*** zbr has quit IRC16:01
*** zbr3 is now known as zbr16:01
*** klonn has quit IRC16:04
danpawlikianw: hey, could you check the Disk Image Builder Patch set: ?16:09
danpawlikianw: it would be nice also to check related patch set16:10
danpawlikianw: thanks16:10
*** adrian-a has joined #opendev16:12
clarkbfungi: corvus: is next step with git protocol v2 to set that gitconfig in the zuul images properly?16:18
clarkb(or maybe we need to gather more data first? migth be good to get that image update moving though so it can be approved when we are happy with the results?)16:19
corvuslooks like ianw set that on ze01; i'm guessing nothing blew up; but do we know if things improved?  what would we measure?16:21
*** chkumar|ruck is now known as raukadah16:21
*** klonn has joined #opendev16:22
clarkbcorvus: I believe that the sshd log at least records session times. We could compare git fetch operations for ze01 before and after the change for nova and neutron ? (two large repos)?16:22
clarkbze01 may also record that info on its side as well16:23
corvusclarkb: i'll inspect the ze01 logs16:23
clarkbthanks. I'm still sorting out breakfast and not quite able to help with that. Was just making sure I understood where we were in the process16:25
clarkbcorvus: also I think number of refs matters more than pure repo size though those two probably correlate16:26
corvusclarkb, ianw: i compared nova repo update times on ze01 yesterday (gitv1) and today (gitv2) between the hours of 14:00-16:30; yesterday's mean was 5.4s with a median of 5.9s.  today's was 1.9s with a median of 2.3s.16:45
corvusi think that's a substantial improvement and we should update zuul or its images to take advantage of v216:45
clarkbcorvus: sounds like a plan16:46
corvusi'll work on a zuul patch16:47
*** marios|rover is now known as marios|out16:47
*** brinzhang_ has joined #opendev16:50
*** brinzhang_ has quit IRC16:51
*** brinzhang_ has joined #opendev16:52
*** brinzhang_ has quit IRC16:53
*** brinzhang has quit IRC16:53
*** brinzhang_ has joined #opendev16:54
*** dtantsur|brb is now known as dtantsur16:58
*** hashar has quit IRC17:03
*** marios|out has quit IRC17:04
openstackgerritMerged openstack/project-config master: Revive os_monasca
*** hamalq has joined #opendev17:27
*** hamalq has quit IRC17:28
openstackgerritMerged openstack/project-config master: Deprecate openstack-ansible-galera_client role
*** hamalq has joined #opendev17:29
openstackgerritSorin Sbârnea proposed opendev/gear master: tox: enable extra python versions
openstackgerritSorin Sbârnea proposed opendev/gear master: tox: enable extra python versions
*** tosky has quit IRC17:46
*** dmellado has quit IRC17:48
*** dmellado has joined #opendev17:50
*** hashar has joined #opendev17:56
*** ralonsoh has quit IRC18:00
*** eolivare has quit IRC18:03
*** dtantsur is now known as dtantsur|afk18:17
*** andrewbonney has quit IRC18:24
*** adrian-a has quit IRC18:41
*** klonn has quit IRC18:51
*** hashar is now known as hasharDinner19:04
*** sboyron has quit IRC19:08
ianwi looks like the linaro arm64 api has disappeared19:12
ianwthe mirror host is still up, but us.linaro.cloud19:13
ianwis not19:13
ianwkevinz doesn't seem to be in here, i'll reach out via email19:14
corvusinfra-root: the promote jobs for the zuul change to enable v2 succeeded, so we should be able to pull and restart globally19:24
*** smcginnis has left #opendev19:25
corvuswe should probably do that, and if nothing blows up, ask all 3rd party ci's to do similar (either run latest container images or apply the same change via whatever config they do)19:25
fungiooh, excellent, i'll get ready to start on that now19:25
fungithink we should do one as a trial, or just go all the way?19:25
fungii mean, i guess it's not substantially different from what ianw already set on ze01?19:25
corvusfungi: let's do one, just to make sure there isn't some gotcha we didn't hit in testing19:26
corvusmaybe do ze0219:26
corvuswe'll want to make sure we clean up the .gitconfig on ze0119:26
fungiyup, i can pull/down/up it19:26
fungion ze0219:26
corvusfungi: thx, i'm going to fry some rice, biab.19:26
fungiit's not a fast image to pull19:28
fungithere it is19:28
fungizuul/zuul-executor   latest              4d979a29238d        About an hour ago   1.19GB19:28
fungithat seems right19:28
*** slittle1 has joined #opendev19:31
*** klonn has joined #opendev19:32
fungifungi@ze02:/etc/zuul-executor$ sudo docker-compose exec executor cat /etc/gitconfig19:33
fungi        version = 219:33
slittle1now why isn't zuul picking up
fungii like puzzles, trying my hand at that one now19:39
avassfungi: fyi if you've got a normal docker installation you could add yourself to the docker group and you won't need to run docker with sudo :)19:40
slittle1we know19:41
fungislittle1: it looks like it depends on
fungiwhich also isn't enqueuing19:41
slittle1there order got reversed in a recent update.  Perhaps it is now circular ?19:42
slittle1i.e. git review ... git rebase -i ... change order of commits... git review19:44
fungislittle1: i don't see any depends-on footers in the commit messages for either of those changes, but perhaps you have another change declaring a circular depends-on involving those?19:44
slittle1but isn't where an implicit dependency when two commits are submitted together from a common local branch19:46
fungiyes, but they would be strictly orderable in that case because of the git dag, so nothing circular19:46
fungii'm going to cheat and check the scheduler's debug log around the time of the first approval on 765828 patchset 2 (14:22z yesterday)19:47
fungioh, nevermnid, i see it19:49
fungi765828 was rebased while we were having some gerrit problems, looks like zuul was unable to report the result on that one, so it currently has no verified +1 result on its latest patchset19:50
fungiyou'll see the lack of a zuul verified vote in the little votes summary near the top-left, and it says "verified: no votes"19:51
corvusavass: i think folks like the additional safety/auditing of using sudo for docker commands19:51
fungislittle1: since the gate pipeline in that tenant is set to only enqueue a change when it has an existing verified +1, it did not act on the approval votes19:52
fungiif you leave a "recheck" comment on that change, it should get checked, and then enqueued into the gate19:52
fungi(assuming jobs succeed for that patchset)19:52
fungiit's still taking me time to find my way around the new gerrit webui, and so it's easy to overlook things like a missing verified vote, sorry took me so long to spot it19:53
slittle1so you can kick it to life for it's +1 vote ?19:55
fungislittle1: if you leave a "recheck" comment on that change, it should get checked, and then enqueued into the gate19:55
slittle1is that a button or menu selection ?   Or literally the word 'recheck' in a comment?19:56
fungiand the other change you mentioned which is parented to that one will also get enqueued when that happens, as it already has a verified +1 from zuul on its latest patchset19:56
fungislittle1: yes, we have the check pipeline in that tenant configured to enqueue changes any time someone leaves a comment starting with the word recheck19:57
fungijust a normal review comment19:57
fungiit's a convenient way to rerun check pipeline jobs for any change19:57
slittle1it's running.  Thanks19:58
fungiand once it completes the check pipeline jobs for that change, both of those approved changes should automatically enter the gate pipeline together19:58
fungiand sorry for the confusion there, the gerrit upgrade has needed a lot of performance tuning and so there are certain to be some changes which didn't get check results over the past couple of weeks19:59
fungiin executor stability news, no new git.exc.InvalidGitRepositoryError exceptions in the ze12 debug log since replacing the nova clone on it20:05
fungii think we can consider that solved20:05
*** adrian-a has joined #opendev20:06
*** hasharDinner is now known as hashar20:19
*** ykarel has quit IRC20:46
*** tosky has joined #opendev21:05
openstackgerritTatiana Ovchinnikova proposed zuul/zuul-jobs master: [DNM] Testing horizon nodejs job with Firefox
corvusfungi: ze02 doesn't seem to have blown up; we're probably good for a complete restart of the mergers and executors?21:28
*** klonn has quit IRC21:32
*** adrian-a has quit IRC21:43
*** adrian-a has joined #opendev21:43
fungithat's my take on it22:20
fungiand sorry, stepped away to grill some hamburders22:20
fungibut yeah, i'm around to help with restarts22:20
*** adrian-a has quit IRC22:55
*** hashar has quit IRC23:57

Generated by 2.17.2 by Marius Gedminas - find it at!