Wednesday, 2021-01-13

*** tosky has quit IRC00:01
openstackgerritMerged zuul/zuul-jobs master: Clarity tox_environment accepts dictionary not list
openstackgerritMerged zuul/zuul-jobs master: Document Python siblings handling for tox role
*** zbr5 has joined #opendev00:22
*** zbr has quit IRC00:22
*** zbr5 is now known as zbr00:22
*** mlavalle has quit IRC00:47
openstackgerritMerged zuul/zuul-jobs master: Add configuration to make logs public
*** kevinz has joined #opendev01:03
*** icey has quit IRC01:15
*** icey has joined #opendev01:16
openstackgerritMerged zuul/zuul-jobs master: Allow to retrieve releasenotes requirements from a dedicated place
kevinzianw: Morning!01:37
ianwkevinz: hi, happy new year01:37
kevinzianw: Thanks! Happy new year! I saw there is an issue accessing to Linaro Us right? But checking the pinging, I see both the IPv4 and IPv6 works01:38
ianwi only got back from pto today, and wasn't aware of anything.  fungi: ^ are there current issues?01:39
kevinzianw: OK, np01:39
kevinz, this can not work, but ping is fine01:39
ianwhrm, will check in a little01:40
*** tkajinam has quit IRC01:41
*** tkajinam has joined #opendev01:42
ianwok, host is up, noting interesting in dmesg01:57
ianwkevinz: do you perhaps mean ?02:02
kevinzianw: aha, after checking the IRC log, this issue for linaro-us has been fixed after reboot instance02:07
kevinzianw: thanks for helping02:07
*** tkajinam has quit IRC02:09
*** tkajinam has joined #opendev02:10
ianwkevinz: did you manage to find anything on why these nodes shutdown suddenly?02:10
ianwI don't think we have an InRelease file, just Release because we don't sign our repos02:11
ianwbut I guess per that link, apt *looks* for InRelease, and if the mirror is down will give that error02:11
kevinzreq-3a0fe54f-e97f-4ca0-b24f-6bcdabc9be27StartJan. 12, 2021, 10:21 a.m.0881516836d94a8f890a031f84c985ef-02:13
kevinzreq-1e4b24f1-13f3-4c1f-bf19-e4f1e0c8b053StopJan. 11, 2021, 2:52 p.m.--02:13
kevinzreq-2e2cc170-c6b4-491a-804c-5af5efd604d0StartDec. 19, 2020, 12:44 a.m.0881516836d94a8f890a031f84c985ef-02:13
kevinzreq-5cf099bb-011a-4e64-902d-40ab2e8795a5StopDec. 18, 2020, 9:25 p.m.--02:13
kevinzreq-556cbdab-8639-42ab-b624-30b6b4ade719StartNov. 8, 2020, 10:06 p.m.0881516836d94a8f890a031f84c985ef-02:13
kevinzreq-b04bbf39-2897-4e62-a30d-99d4722c3c70StopNov. 5, 2020, 7:18 a.m.-02:13
ianwi had a remote netconsole running and didn't get any sort of message out of the host02:14
kevinzianw:, looks it has been shutdown every month02:14
ianwit was like it was just killed02:14
kevinzianw: run out of the resources and got killed?02:15
kevinzby host02:15
ianwmaybe?  I'd expect some sort of logs in nova ...02:15
kevinzI will check the nova-log for this req number02:16
ianwi setup a console on 2020-11-0902:16
ianwdoh, i dropped the "-a" from the tee command so i stupidly have overwritten when it stopped02:20
ianwSep 14 09:09:58 <ianw>  Alex_Gaynor: thanks for pointing that out.  it seems we have a problem with the mirror node in that region.02:22
ianwreq-4c549e46-760b-4353-b92d-2503e13a96c5StartSept. 13, 2020, 10:39 p.m.0881516836d94a8f890a031f84c985ef-02:23
ianwprobably matches; are those times UTC?02:23
kevinzianw: yes, it is UTC timezone02:35
kevinzianw: checking the log from nova-compute, just get this: Will find more in nova-api and conductor02:45
ianwthat definitely seems like nova noticed the vm had already shutdown, then updated the db02:48
ianwkevinz: i'd be looking for corresponding oom/kill type messages in syslog for qemu-kvm around the same time ...02:50
kevinzianw: will check02:50
*** hamalq has quit IRC02:56
ianwkevinz: do the compute nodes have a little swap space?02:57
*** ysandeep|away is now known as ysandeep03:24
kevinzianw:,  yes, totally 4578M03:28
kevinzAnd I see some Qemu failed at Jan 11.03:28
ianwhrm, so 96gb ram, 4gb swap (approx) right?03:31
ianwalthough there's a lot of free ram now, the swap does seem used, which suggests to me it might have been under memory pressure at some other time03:32
*** sboyron has quit IRC03:41
kevinzianw: yes, look memories pressure. I will disable this node scheduling for a while, to see if there would be better03:58
kevinzianw: I see there are quite a lot of instance are scheduling and running at this node,  they are always scheduling to this node.  Looks the nova scheduler are not good decided...04:27
kevinzI disable this node scheduling and I will check what is wrong with the nova-scheduler04:28
ianwkevinz: thanks; something like that would explain the very random times it seems to stop i guess.  we can go month(s) with nothing but then a few failures in a week it feels like04:30
kevinzianw: yes, definitely.  Let's see what will happen recently.04:32
fungiianw: kevinz: the mirror and builder instances were both found in a shutdown state again, we managed to boot them though ended up needing to delete the afs cache on the mirror as it ended up seemingly corrupted to the point where afsd would just hang indefinitely04:45
ianwfungi: yeah, that seems to be a common issue when it is shutdown unsafely04:46
fungilooking at grafana we're still behind on node requests (though on track to catch up by the time daily periodics kick off), and tripleo still has a 10-hour gate backlog04:52
fungiso maybe we should postpone the scheduler restart04:52
ianwi'm heading out in ~ 30 mins, so won't be able to watch this evening04:58
ianwif tomorrow we get reviews on the zuul summary plugin, it might be worth restarting scheduler and gerrit at the same time04:58
fungigreat point05:23
*** ykarel has joined #opendev05:41
*** marios has joined #opendev06:28
openstackgerritAndreas Jaeger proposed openstack/project-config master: Move snaps ACL to x
openstackgerritMerged openstack/project-config master: Create microstack-specs project
openstackgerritMerged zuul/zuul-jobs master: Enable installing nimble siblings
*** ralonsoh has joined #opendev07:19
*** eolivare has joined #opendev07:35
*** openstackgerrit has quit IRC07:47
*** jpena|off is now known as jpena07:51
*** JayF has quit IRC07:52
*** openstackgerrit has joined #opendev07:53
openstackgerritSorin Sbârnea proposed opendev/git-review master: Bring zuul configuration in-tree
*** fressi has joined #opendev07:58
*** slaweq has joined #opendev07:59
*** diablo_rojo__ has quit IRC08:01
*** hashar has joined #opendev08:03
openstackgerritSorin Sbârnea proposed opendev/git-review master: Bring zuul configuration in-tree
*** slaweq has quit IRC08:04
openstackgerritSorin Sbârnea proposed openstack/project-config master: Move git-review zuul config in-tree
openstackgerritSorin Sbârnea proposed opendev/git-review master: Bring zuul configuration in-tree
*** slaweq has joined #opendev08:10
*** andrewbonney has joined #opendev08:13
openstackgerritSorin Sbârnea proposed openstack/project-config master: Move git-review zuul config in-tree
openstackgerritSorin Sbârnea proposed opendev/git-review master: Bring zuul configuration in-tree
*** rpittau|afk is now known as rpittau08:25
*** sboyron has joined #opendev08:27
*** tosky has joined #opendev08:39
openstackgerritSorin Sbârnea proposed opendev/git-review master: Bring zuul configuration in-tree
*** hrw has left #opendev09:22
openstackgerritSorin Sbârnea proposed opendev/git-review master: Bring zuul configuration in-tree
openstackgerritSorin Sbârnea proposed opendev/git-review master: Drop support for py27
jrosseri am seeing a number of Could not connect to ( - connect (113: No route to host)09:49
lourot^ same for us, e.g. in
frickleryet another mirror gone offline, this is getting creepy. /me tries to take a look10:09
fricklerinfra-root: ^^ console log shows a lot of CPU/rcu related issues. trying a restart via the api10:12
openstackgerritMerged openstack/project-config master: Move git-review zuul config in-tree
*** hemanth_n has joined #opendev10:28
*** hashar has quit IRC10:39
*** dtantsur|afk is now known as dtantsur10:41
*** hemanth_n has quit IRC11:01
*** ysandeep is now known as ysandeep|afk11:04
*** sshnaidm|afk is now known as sshnaidm|ruck11:19
openstackgerritSorin Sbârnea proposed opendev/git-review master: Bring zuul configuration in-tree
sshnaidm|ruckinfra-root is problem with retries in limestone known?12:08
*** DSpider has joined #opendev12:10
zbrsshnaidm|ruck: one mirror went down two hours ago.12:22
fricklersshnaidm|ruck: zbr: trying to restart the server via the api hasn't worked. doing a stop/start cycle next12:49
*** jpena is now known as jpena|lunch12:49
fricklerGPF while trying to start the AFS client ... guess we'll need to rebuild the mirror or talk to limestone about a possibly broken hypervisor. disabling that region for now12:53
fricklerhmm ... actually the node did finish booting but failed with afs. did "rm -rf /var/cache/openafs/*" and another reboot, maybe that'll be enough for now12:55
fricklero.k., that seems to have worked for now, maybe the GPF was in fact related to afs cache corruption13:03
frickler#status log stopped and restarted after it had become unresponsive. need afs cache cleanup, too.13:04
openstackstatusfrickler: finished logging13:04
fricklerjrosser: lourot: sshnaidm|ruck: zbr: ^^ please let us know if you encounter any further issues, should be safe to recheck now.13:05
sshnaidm|ruckfrickler, thanks a lot!13:05
*** brinzhang has quit IRC13:11
*** brinzhang has joined #opendev13:11
openstackgerritSorin Sbârnea proposed opendev/git-review master: Bring zuul configuration in-tree
*** ysandeep|afk is now known as ysandeep13:25
*** whoami-rajat__ has joined #opendev13:46
*** jpena|lunch is now known as jpena13:51
*** d34dh0r53 has quit IRC14:10
openstackgerritSorin Sbârnea proposed opendev/git-review master: Drop support for py27
*** d34dh0r53 has joined #opendev14:13
*** hashar has joined #opendev14:47
*** auristor has quit IRC14:47
*** fressi has quit IRC14:48
*** auristor has joined #opendev14:50
fungifrickler: did afsd also not start completely on that mirror after restarting?15:08
fungiahh, you said yes15:09
mnaserdiablo_rojo_phon: when you have a second, if you could rebase
kopecmartinhi all, I'd like to update refstack server ( about the latest changes in the refstack repo (, could anyone point me a direction on how? Thank you15:21
clarkbkopecmartin: currently it is deployed using opendev/system-config and puppet-refstack iirc. But I proposed a change a while back to instead build docker images for it and deploy that instead in opendev/sytem-config. the problems there were that a number of changes needed to be made to refstack itself to be viable and I ran out of steam on it15:34
clarkbkopecmartin: I think we should pick that back up again if we are going to try and make updates15:35
fungialso the current refstack server is running ubuntu 14.04 lts15:35
fungiand current master branch says it needs python 3.6 or newer, while that version of ubuntu only has python 3.415:35
kopecmartinoh, I see .. I'm happy to help .. also we have reformed refstack group a little, so there is enough core reviewers now if anything needs to be changed on refstack side15:39
kopecmartinin regards of the server OS update, i'm happy to help if you point me a direction15:40
clarkbkopecmartin: please feel free to take over that change in system-config it should show up if you earch for me and refstack in system-config15:40
clarkbkopecmartin: that would happen as part of the redeployment with docker. So get the docker stuff working in CI then an infra-root will work with you to do the host migration and all that15:40
kopecmartinclarkb: thanks, I'm gonna have a look15:42
*** hashar is now known as hasharAway15:43
openstackgerritMerged opendev/git-review master: Bring zuul configuration in-tree
ttxkopecmartin: thanks for picking that up! I was looking into updating instructions when I realized they were already updated but just missing a current deployment15:53
openstackgerritSorin Sbârnea proposed opendev/git-review master: Assure git-review works with py37 and py38
openstackgerritJeremy Stanley proposed opendev/engagement master: Initial commit
*** ykarel is now known as ykarel|away16:09
*** ysandeep is now known as ysandeep|out16:12
*** ykarel|away has quit IRC16:16
*** slaweq has quit IRC16:34
*** slaweq has joined #opendev16:36
*** chrome0 has quit IRC16:40
*** tosky has quit IRC16:41
*** tosky has joined #opendev16:42
*** chrome0 has joined #opendev16:45
*** eolivare has quit IRC16:52
*** jpena is now known as jpena|off17:05
openstackgerritJeremy Stanley proposed opendev/system-config master: Adjust the example Etherpad API delete command
clarkb#status log Manually deleted an etherpad at the request of dmsimard.17:12
openstackstatusclarkb: finished logging17:12
fungiinteresting, looks like some of our afs vos release runs began to consistently fail at 04:23 today17:14
fungii'll check the fileservers17:14
fungiboth are up and there are no recent restarts17:14
clarkbfungi: how is disk utilization?17:14
clarkb(I think we've been good on that side of things but could potentially explain it?)17:15
fungithe /vicepa fs has 392G available on afs01.dfw and 1.1T available on afs02.dfw17:15
fungidmesg indicates io errors talking to a cinder volume on afs02.dfw starting at 03:32:4217:17
clarkbfungi: are there stale locks? thats about the only other thing I can think of that would cause something like that17:17
clarkboh interseting I guess that could do it too17:17
fungi[Wed Jan 13 03:32:42 2021] INFO: task jbd2/dm-0-8:484 blocked for more than 120 seconds.17:17
fungi[Wed Jan 13 03:35:58 2021] blk_update_request: I/O error, dev xvdk, sector 1232817:17
fungii'll reboot the server17:18
fungimake sure it reconnects to the volumes correctly17:18
*** marios is now known as marios|out17:18
fungiokay, server's back up. i'll try to stop/fix any lingering hung vos releases or locks17:23
fungimay not be any cleanup required, looks like the errors stopped as soon as afs02.dfw was restarted17:29
clarkbI wonder if increasing that kernel timeout would help (and if it is even tunable)17:29
*** sshnaidm|ruck is now known as sshnaidm|afk17:30
fungisome volumes needed "full" releases, so it's taking a bit of time to catch up again17:31
fungi#status log rebooted afs02.dfw following hung kernel tasks and apparent disconnect from a cinder volume starting at 03:32:42, volume re-releases are underway but some may be stale for the next hour or more17:34
openstackstatusfungi: finished logging17:35
*** mlavalle has joined #opendev17:57
*** hamalq has joined #opendev18:03
*** rpittau is now known as rpittau|afk18:05
*** marios|out has quit IRC18:06
openstackgerritAndy Ladjadj proposed zuul/zuul-jobs master: [ensure-python] install python version only if not present
clarkbfungi: doesn't look like zuul is any happier with its backlog today?18:09
*** cloudnull has quit IRC18:12
*** cloudnull has joined #opendev18:12
*** andrewbonney has quit IRC18:12
funginot really no18:13
clarkblooking at grafana nothing stands out as being very broken. I guess just backlogs due to demand and potentially made worse by the mirror issue in limestone earlier today18:13
clarkbboot times seem consistent and failures are infrequent18:13
clarkbI do wonder if we have no quota in vexxhost though as that is a semi common thing due to volume leaks18:14
clarkbgrafana indicates that no we are using our quota there18:14
clarkbthe number of jobs a single neutron change runs is not small18:16
fungigerrit event volume has started to dip, so looks like the node request backlog is plateauing around 2-2.5k at least18:18
fungiwe seem to max out at roughly 600 nodes in use18:19
*** ralonsoh has quit IRC18:20
fungiaha, mystery solved on the afs02.dfw issue. ticket rackspace opened: This message is to inform you that our monitoring systems have detected a problem with the server which hosts your Cloud Block Storage device,, '9f19fd0d-a33e-4670-817c-93dd1e6c6e6f' at 2021-01-13T03:59:33.166398.18:28
fungiif i hadn't been so distracted this morning i might have read the root inbox earlier and noticed/fixed the problem sooner18:30
*** dtantsur is now known as dtantsur|afk18:43
openstackgerritAndy Ladjadj proposed zuul/zuul-jobs master: [ensure-python] install python version only if not present
openstackgerritAndy Ladjadj proposed zuul/zuul-jobs master: [ensure-python] install python version only if not present
openstackgerritAndy Ladjadj proposed zuul/zuul-jobs master: [ensure-python] install python version only if not present
*** hamalq has quit IRC19:29
*** paladox has quit IRC19:34
*** hasharAway has quit IRC19:36
*** paladox has joined #opendev19:39
*** whoami-rajat__ has quit IRC19:55
openstackgerritAndy Ladjadj proposed zuul/zuul-jobs master: [ensure-python] install python version only if not present
*** sboyron has quit IRC20:09
openstackgerritAndy Ladjadj proposed zuul/zuul-jobs master: [ensure-python] install python version only if not present
*** slaweq has quit IRC20:41
fungitarballs volume has been releasing for 4 hours now. i probably should have had the foresight to create locks preventing the mirror volumes from getting released until that was done :/21:27
fungiat this point cronjobs have started what are probably full releases which are still in progress for deb-octopus, yum-puppetlabs, debian, opensuse, debian-security, ubuntu-ports, epel, centos, and fedora21:29
mordredfungi: "fun"21:30
fungiyeah, i expect the tarballs release would have only required two hours based on the transfer volume cacti is clocking on the network interface, but roughly an hour in the package mirrors started also getting releases triggered21:31
fungiwhich likely slowed the second half of the tarballs volume transfer to a crawl21:32
fungii'm hesitant to abruptly terminate any of the mirror volume releases though over worries that will lead to even more cleanup work21:33
clarkbwe don't typically do global locks but I guess the issue now is its a full release?21:33
fungiyeah, and a full release of a bunch of different volumes at once i expect21:33
fungii guess because these tried to release at some point while the filesystem was hitting write errors21:34
ianwfungi: the tarballs one, i don't think that runs with -localauth?  the mirror ones should be running via a ssh session and not hit a timeout21:41
fungi2021-01-13 17:28:56,918 release DEBUG    Running: ssh -T -i /root/.ssh/id_vos_release -- vos release project.tarballs21:47
fungii'm not worried about auth timeouts, just that it's going to be an age before tarballs.o.o and some other sites (, et cetera) are current again21:51
*** brinzhang_ has joined #opendev23:02
*** brinzhang has quit IRC23:05
clarkbianw: for does ansible work with symlinks like that? any reason to no just keep treating the canonical srever as until we properly rename it?23:09
clarkb(mostly concerned that we'll run against prod without the necessary vars loaded in)23:09
ianwclarkb: one sec, context switching back to it :)23:10
clarkbits just seems like we're getting ahead of schedule with that one23:10
fungiyeah, elsewhere we still use in the inventory name for it23:10
clarkbyup the server is still named canonically in nova too23:11
clarkbwe just server on it too23:11
ianwso yeah, i think that started with me looking at the testinfra, which was trying to match against and thus not actually running the tests; which i think is only "is this listening"23:13
fungii have a feeling this is also not going to be a good time for a gerrit restart looking at the graphs... wonder if we should shoot for late utc friday, next week is openstack wallaby milestone 2 which likely explains the rush on the gate23:13
ianwso iirc my issue was as i expanded the testing, i didn't want to have it in a weird state of testing against review01.openstack.org23:15
clarkbianw: ya I think we should fix the test to look at Then when we switch the host over we can update that too?23:15
clarkbI don't think that is a weird state if that is reality23:16
clarkbbut maybe I'm missing something else too23:16
clarkbfungi: ya agreed we should probably wait for CI to settle before restarting services like gerrit and zuul23:16
ianwjust that it's already in a dual state, in that the vhost name is set to review.opendev.org23:17
fungiianw: well, we have two vhosts there (we could redo the vhost config to use an alias instead)23:18
ianwi guess what i mean is23:19
ianwinventory/service/host_vars/ review.opendev.org23:19
fungiright, we do that23:20
clarkbyes because the server is canonically named (that will change when it gets upgraded)23:20
clarkb(it is confusing, but I worry that changingCI will make it more confusing because CI will be different than prod)23:20
clarkbcoule of question on too but I think we can probably land that one as is then make those changes if we want to23:21
fungiit wouldn't technically be all that different if we had inventory/service/host_vars/ because the ansible inventory hostname and apache vhost canonical name are still not the same23:21
clarkbthe more I think about it hte more I'm thinking we should keep the status quo with then update inventory when we update prod. That way we don't have an unexpected delta between prod and testing and weirdness in our host vars23:23
ianwyeah, i guess that what i was doing was building extensively on the system-config tests, and found it quite confusing with the server in the testing etc.23:23
clarkbI don't think it is necessarily wrong, but it makes things different enough to be confusing23:24
fungiit'll like23:24
fungily be confusing either way ;)23:24
clarkbright but the previous was mathed production23:25
clarkbso its the confusing we have to deal with :)23:25
ianwthe other thing is, we could push for the replacement server to clear this up23:25
clarkbwhat does gerrit init --dev do?23:25
clarkbianw: we can do that as well :)23:25
clarkbwe will need to be careful turning it on to avoid having it replicate to gitea and such23:26
clarkbbut ya thats anotherthing to sort out23:26
ianwclarkb: when the auth type is set to DEVELOPMENT_BECOME_ANY_ACCOUNT *and* you run gerrit init --dev, gerrit will create the initial admin user for you23:26
clarkbah both are required23:27
ianwyes.  it's slightly different to the quickstart stuff, that uses the upstream gerrit container.  that includes a LDAP connected to it, where you have the initial admin23:27
fungianother alternative would be to ssh as "Gerrit Code Review" using the host key and create an initial admin account with the cli23:28
clarkbthis is fine I had just never seen the --dev flag before23:29
ianwfungi: i couldn't get that to work.  i couldn't get that to make the initial account23:29
ianwyou can go in with that after you have an account, and suexec, but it can't create the initial account23:30
fungioh, create-user needs to run via suexec as an existing user?23:30
fungiyeah, now i somewhat recall that23:30
ianwit's been a bit since i tried, but using the "Gerrit Code Review" was my first attempt at doing it23:31
clarkbianw: thank you for I had meant to do that but then things got crazy when we were slimming the jobs down23:32
*** DSpider has quit IRC23:32
ianwdo we know off the top of anyone's head if we have enough headroom to launch another review server in dfw?23:37
clarkbI don't. We would if we retired review-test (we can also clean up review-dev but its much smaller)23:37
ianwok, i'm happy to drive this one, i can give things a go and start on an etherpad of steps23:38
clarkbthanks! I imagine the spin up for it would look a lot like review-test with a pruned host vars setup23:38
clarkbthat way it avoids replicating and stuff until we switch and add more config to it23:38
ianwapril seems far away but it isn't :)23:39
clarkb(if you needexamples in recent history for doing the thing)23:39
clarkbianw: I'm into the bazel stuff and it looks like the pure js plugins don't get copied automagically to the war like the java plugins do?23:42
clarkbhrm we also have to specify a different bazel target for the plugin. Any idea why the other plugins don't need this?23:42
ianwclarkb: i think because they're default plugins?23:43
clarkbthat makes sense23:43
fungineed to do something like the copy i did in ansible for the pg plugin of the opendev theme?23:43
ianwdon't take anything i say about bazel as true though :)  i would love for someone who actually understands it to look at it23:43
clarkbfungi: ya and tell bazel to build the plgin explicitly23:44
clarkbianw: I bet that is it23:44
clarkband/or js vs java plugins23:44
fungioh, got it, so there's also a build step for that one23:44
clarkblike maybe it can autodiscover java things but not the js23:44
ianwit should probably grow to have a java component.  what we'd like is for the summary plugin to be able to order the results via config; but the only way to really do that is to write a java plugin that then exposes a REST endpoint23:45
clarkbre making room for new review. If we need to we can probably put review-test into cold storage and revive it again after if necessary (basically snapshot the root disk and its cinder volumes then delete the instance)23:45
clarkbthis new testing stuff also reduces the need for review-test (though testing the migration to 3.3 on review-test with its bigger data set would be nice hence the cold storage idea23:46
clarkbworst case we just rebuild review-test entirely23:46
clarkbianw: the symlink thing with bazel is a fun one23:49
ianwyeah, that's a great intersection of bazel and docker23:49
ianwyou can not convince bazel to not use the symlinks, and you can not convince docker to follow them23:50
clarkbok the rest of that stack lgtm. I did leave some nits and thoughts in a few places. You may want to double check them to make sure they are fine as is23:52
ianwthanks, i'll go through soon.23:53
clarkbneutron is running ~36 jobs per change in check and the vast majority look like expensive full integration style tets23:54
* fungi sighs23:55
clarkbthose are both failing non voting jobs23:56
clarkbI wonder too if we've got a bunch of always failing non voting jobs in there :/23:56
clarkbfungi: I wonder if we need to talk to projects about taking a critical eye to tests like that especially if we're producing a large backlog as a result23:56
clarkb confirmed for at least that first job23:59

Generated by 2.17.2 by Marius Gedminas - find it at!