Saturday, 2022-11-12

johnsomFYI, jobs are failing (retry exceeded). I caught a console on one and all it has in it is:00:59
johnsomBuild ID dbeff18b68834c63bedeaf60ee98a2b3 not found00:59
johnsom--- END OF STREAM ---00:59
johnsomThis is the patch I was watching in zuul:
*** dviroel is now known as dviroel|out01:57
fungijohnsom: is it just octavia jobs?02:04
johnsomNo, I saw it on a neutron job too02:05
fungi2022-11-12 00:33:42,543 ERROR zuul.AnsibleJob:   KeyError: 'semaphores'02:09
fungihere's the traceback:
fungii wonder if these started appearing after a zuul service restart02:11
johnsomI know the patch I link didn’t change any job config02:11
fungiit does look like the containers have restarted so that's probably our weekly zuul rolling restart02:13
fungiyeah, the executor where i spotted that in the log is ze02, which restarted at 00:33 utc02:14
fungi shows the restart progress (we do an executor at a time)02:14
fungicorvus: ^ if you're around, any idea why we'd be looking for a semaphores key in the playbook dict?02:15
fungii wonder if this is a transitive incompatibility which will subside once the schedulers also restart02:16
fungilooks like that probably came in with on thursday02:34
fungilooking through that change, i have a feeling it will work for any builds which are enqueued after the schedulers restart in a few hours02:36
fungigiven the current zuul activity level is fairly low, i'm inclined to check it again in the morning02:37
fungiby which time all the services should hopefully be running from a consistent point in the repo02:38
johnsomAck, yeah, I am done for the evening.02:39
fungieverything has updated and looks like jobs are working again12:16

Generated by 2.17.3 by Marius Gedminas - find it at!