| clarkb | gmann: yes its a bug in zuul. Until zuul gets restarted (which happens automatically in a few hours) it will continue to happen | 00:02 |
|---|---|---|
| gmann | clarkb: ok | 00:03 |
| clarkb | normally we might try and restart things more quickly but we're trying to land the bugfix for the other issue that caused us to delay things this morning so that it can be picked up by the restarts too | 00:06 |
| gmann | k | 00:12 |
| opendevreview | Ghanshyam proposed openstack/tempest master: Separate the extra tests to run in a separate job https://review.opendev.org/c/openstack/tempest/+/873055 | 00:12 |
| clarkb | you can track the automated restarts here: https://zuul.opendev.org/components its the scheduler components (which go last after we do all the executors) that we need updated | 00:13 |
| clarkb | you'll see ze01 is paused. That is beacuse we've asked it to stop running any new jobs, complete running jobs, then exit gracefully. | 00:13 |
| clarkb | It should show Running once it has done that and been restarted. In this case very likely with a newer version number as well | 00:17 |
| clarkb | then ze02 will go through the process and so on | 00:18 |
| opendevreview | Ghanshyam proposed openstack/tempest master: Prepare tempest-slow-parallel job and run periodically https://review.opendev.org/c/openstack/tempest/+/873441 | 00:26 |
| opendevreview | Ghanshyam proposed openstack/tempest master: Revert "Mark tempest-multinode-full-py3 as n-v" https://review.opendev.org/c/openstack/tempest/+/873283 | 00:28 |
| opendevreview | Ghanshyam proposed openstack/tempest master: Revert "Mark tempest-multinode-full-py3 as n-v" https://review.opendev.org/c/openstack/tempest/+/873283 | 00:29 |
| opendevreview | Ghanshyam proposed openstack/tempest master: Minimize the tests footprint in multinode job https://review.opendev.org/c/openstack/tempest/+/873442 | 01:39 |
| opendevreview | Ghanshyam proposed openstack/tempest master: Separate the extra tests to run in a separate job https://review.opendev.org/c/openstack/tempest/+/873055 | 01:58 |
| opendevreview | Ghanshyam proposed openstack/tempest master: Revert "Mark tempest-multinode-full-py3 as n-v" https://review.opendev.org/c/openstack/tempest/+/873283 | 01:58 |
| gmann | kopecmartin: dansmith: this is the series to try to decrease the timeout failure, moving some jobs to periodic to make space for the new job and spiting the tester footprints for a few jobs https://review.opendev.org/q/topic:bug%252F2004780 | 05:44 |
| opendevreview | Merged openstack/tempest master: Mark test_minimum_basic_instance_hard_reboot_after_vol_snap_deletion as slow test https://review.opendev.org/c/openstack/tempest/+/872691 | 10:07 |
| dansmith | clarkb: still seeing the config error as of midnight last night.. I see not all the zuul workers are on the same version, so I assume we're still stuck | 14:52 |
| dansmith | glance and nova (and maybe other) gates have been 100% blocked for days because we're trying to land a fix, and that fix has hit the config error several times, among other things | 14:53 |
| opendevreview | Ghanshyam proposed openstack/tempest master: Move a few jobs to periodic https://review.opendev.org/c/openstack/tempest/+/873163 | 19:05 |
| opendevreview | Ghanshyam proposed openstack/tempest master: Prepare tempest-slow-parallel job and run periodically https://review.opendev.org/c/openstack/tempest/+/873441 | 19:05 |
| opendevreview | Ghanshyam proposed openstack/tempest master: Minimize the tests footprint in multinode job https://review.opendev.org/c/openstack/tempest/+/873442 | 19:05 |
| opendevreview | Ghanshyam proposed openstack/tempest master: Separate the extra tests to run in a separate job https://review.opendev.org/c/openstack/tempest/+/873055 | 19:05 |
| opendevreview | Ghanshyam proposed openstack/tempest master: Revert "Mark tempest-multinode-full-py3 as n-v" https://review.opendev.org/c/openstack/tempest/+/873283 | 19:06 |
| opendevreview | Ghanshyam proposed openstack/tempest master: Revert "Mark tempest-multinode-full-py3 as n-v" https://review.opendev.org/c/openstack/tempest/+/873283 | 19:15 |
| opendevreview | Ghanshyam proposed openstack/tempest master: DNM: Test RBAC Nova service role https://review.opendev.org/c/openstack/tempest/+/864595 | 19:50 |
| opendevreview | Ghanshyam proposed openstack/tempest master: Add test for assisted volume snapshot https://review.opendev.org/c/openstack/tempest/+/864839 | 19:56 |
| opendevreview | Ghanshyam proposed openstack/tempest master: DNM: Test RBAC Nova service role https://review.opendev.org/c/openstack/tempest/+/864595 | 19:57 |
| clarkb | dansmith: it was the schedulers which happen last that needed to be upgraded which happend around 9am pacific today | 20:04 |
| dansmith | clarkb: okay saw it several times at 9:10am today, so maybe that was just before | 20:05 |
| dansmith | https://review.opendev.org/c/openstack/tempest/+/873300 for reference | 20:05 |
| clarkb | gmann' change above https://review.opendev.org/c/openstack/tempest/+/873163 seems happy now | 20:05 |
| dansmith | it's in the gate now (again, for the many-eth time) so I'm hoping... | 20:05 |
| gmann | yeah, config error is gone now | 20:06 |
| clarkb | I've just rechecked rosmaita's change with the same problem and it is in check now | 20:06 |
| dansmith | gmann: well, it came and went several times for me, but I hope it's gone for good | 20:08 |
| dansmith | I'd recheck something, config error, immediate recheck, run check, fail in gate with config error, recheck, config error, recheck, run, etc | 20:08 |
| dansmith | presumably because of which worker I was hitting each time or something | 20:08 |
| gmann | ohk. I think I checked ~10ish | 20:09 |
| clarkb | dansmith: I think it was which scheduler. We had separately done some docker upgrade work and I think one of the schedulers may have ended up with the bug fix deployed. But ya everything according to the components list should now be new enough to include the fix | 20:10 |
| dansmith | ack | 20:10 |
| clarkb | and we also deployed the fix for the issue that caused us to have the pause yesterday mornin in processing thing | 20:10 |
| clarkb | we would've deployed the other fix more quickly if we weren't also trying to get that one landed and deployed | 20:10 |
| *** dmellado_ is now known as dmellado | 21:09 | |
| *** dmellado_ is now known as dmellado | 23:47 | |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!