| gibi | do we have a zuul outage? I'm getting http 500 from https://zuul.opendev.org/api/tenant/openstack/builds?project=openstack%2Fnova&job_name=tox-cover&result=FAILURE&limit=10 | 09:20 |
|---|---|---|
| gibi | but https://zuul.opendev.org/t/openstack/status opens | 09:22 |
| tore | I have recently had some issues with Zuul, see https://review.opendev.org/c/openstack/designate/+/997376. I have tried multiple rechecks, and every single job has succeeded at least once. This suggests to me that the change itself is fine, but the problem is in Zuul CI. I sparred a bit with Claude, who came to the same conclusion - «looks like resource-contention flakiness | 10:10 |
| tore | on undersized CI worker VMs», saying the next steps would be for an OpenStack infra team member to use "zuul autohold" in order to debug the failures directly on the VM where Zuul runs. Am I in the right place to ask for that? | 10:10 |
| tore | Seems it's not just me, either: https://review.opendev.org/c/openstack/designate/+/992298 | 12:23 |
| fungi | gibi: some database queries are taking too long to complete, it seems like, and getting timed out. it may be additional load from the webui getting crawled. the web and database services are isolated from job running functionality so it shouldn't prevent things from getting tested | 12:50 |
| fungi | tore: yes, unfortunately "sometimes passes every test" doesn't mean the change isn't the cause, and if a change that introduces a nondeterministic/random failure chance gets merged, then the tests or software itself will also start failing at random for every new change (and possibly for users once it gets released) | 12:52 |
| fungi | there are tools available to root out resource contention in tests, it's often times parallel testing interfering between different tests that are poorly isolated from each other in the same job or aren't sufficiently constrained | 12:53 |
| fungi | changes you linked are running jobs on virtual machines with 8gb ram, 4-8 vcpus and 80gb+ of storage (may be local to the hypervisor or over iscsi/ceph) | 12:56 |
| fungi | it looks like the designate-bind9-core job was succeeding reliably until recently: https://zuul.opendev.org/t/openstack/builds?job_name=designate-bind9-core&project=openstack%2Fdesignate&pipeline=gate&skip=0 | 12:57 |
| fungi | the last time it ran for the master branch in the gate pipeline was 2 weeks ago though, so i suspect something happened after that to destabilize those jobs | 12:58 |
| fungi | most of the failures seem to be in tests defined in designate's tempest plugin, though the last changes there were about a month ago | 13:03 |
| fungi | you might check with the devstack/tempest maintainers in the #openstack-qa channel to see if they're aware of any destabilization in devstack or tempest within the past couple of weeks that could account for the change in test behaviors | 13:04 |
| gibi | fungi: ack I don't see effects on patches being tested. But browsing job results are effected unfortunately not just via the API but via the wb | 13:22 |
| gibi | webgui too | 13:22 |
| gibi | https://zuul.opendev.org/t/openstack/builds?job_name=tox-cover&project=openstack%2Fnova&skip=0 | 13:22 |
| fungi | yeah, the webui is essentially just calling the api itself anyway | 13:23 |
| fungi | once i get out of my current meeting i can take a closer look | 13:23 |
| fungi | gibi: aha! corvus spotted the problem with your timing out queries | 13:36 |
| fungi | nova has never run the tox-cover job, so zuul is trying to find the most recent incidence of it and runs out of time scanning the whole table | 13:37 |
| fungi | you probably wanted to query on the openstack-tox-cover job instead? | 13:37 |
| fungi | basically zuul tries to go back to the dawn of time to find results recorded for a job that has never run for that project with that result | 13:38 |
| gibi | ahh thanks | 13:39 |
| fungi | https://zuul.opendev.org/t/openstack/builds?job_name=openstack-tox-cover&project=openstack%2Fnova&skip=0 is returning content for me almost instantly | 13:39 |
| gibi | :facepalm: | 13:39 |
| opendevreview | Jeremy Stanley proposed openstack/project-config master: Update GitHub mirroring credentials for OpenStack https://review.opendev.org/c/openstack/project-config/+/999019 | 20:45 |
| opendevreview | Jeremy Stanley proposed openstack/project-config master: Update GitHub mirroring credentials for OpenStack https://review.opendev.org/c/openstack/project-config/+/999019 | 21:06 |
| opendevreview | Jeremy Stanley proposed openstack/project-config master: Don't print GitHub auth responses as exceptions https://review.opendev.org/c/openstack/project-config/+/999022 | 21:06 |
Generated by irclog2html.py 4.1.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!