| tkajinam | I'd appreciate it if o/ https://review.opendev.org/c/openstack/openstack-zuul-jobs/+/1007297 could get some attention. | 11:47 |
|---|---|---|
| tkajinam | some job updates in neutron repos are pending on it | 11:47 |
| *** haleyb|out is now known as haleyb | 13:30 | |
| opendevreview | Merged openstack/openstack-zuul-jobs master: Add py314-with-*-master jobs https://review.opendev.org/c/openstack/openstack-zuul-jobs/+/1007297 | 14:31 |
| melwitt | I saw something strange just now, a job (nova-whitebox) reported back on a patch as a FAILURE but when I open the job link there is zero information https://zuul.opendev.org/t/openstack/build/ff92381976634653825492d614248fee and shows it still "in progress" and it also isn't in the job build history. anyone seen anything like this before? | 18:49 |
| clarkb | I think that means the zuul database hasn't been updated to reflect the job state (yet?) | 18:52 |
| clarkb | the web ui is driven the by sql db contents, but the internal state zuul runs off of is in zookeeper so the reporting of successs/failure etc is driven by zookeeper state | 18:53 |
| clarkb | melwitt: I think the problem is that the database has filled its disk with all of the information about your ci job runs :) | 18:54 |
| clarkb | I'm moving the conversation to #opendev on matrix since this is a global problem not specific to openstack or any particular job | 18:55 |
| melwitt | ok sure | 18:55 |
| -opendevstatus- NOTICE: Zuul is reporting inconsistent results showing failures in Gerrit comments and in progress status in the Zuul UI. This is due to Zuul database issues that we are working to correct. | 19:03 | |
| clarkb | melwitt: thank you for the heads up. I've paused zuul queues which should prevent this from getting worse and fungi is adding disk to the database server | 19:08 |
| clarkb | melwitt: ok we think we've correct the issue but believe that builds that attempted to report into the db during the disk full period essentially failed and moved on. So there aren't going to be easy links to find for your failure | 19:27 |
| clarkb | melwitt: depending on how important that info asap vs rechecking we can dig that up for you (I think we have to check each of the backends for the path the build uuid would upload to so its doable) | 19:28 |
| clarkb | the rax cdn may make it weird though? | 19:28 |
| clarkb | anyway you can recheck if it isn't urgent. If debugging this specific failure is important we can try our best to dig up the log urls for you | 19:29 |
| melwitt | thank you clarkb. it's nothing urgent, just looked strange to me and I wanted to learn more. no need to debug this specific failure | 19:37 |
Generated by irclog2html.py 4.1.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!