| opendevreview | Ghanshyam proposed openstack/tempest master: DNM: Test RBAC Nova service role https://review.opendev.org/c/openstack/tempest/+/864595 | 01:49 |
|---|---|---|
| opendevreview | Ghanshyam proposed openstack/tempest master: Add test for assisted volume snapshot https://review.opendev.org/c/openstack/tempest/+/864839 | 02:32 |
| opendevreview | Ghanshyam proposed openstack/tempest master: Temporary increase the job timeout https://review.opendev.org/c/openstack/tempest/+/873472 | 02:34 |
| gmann | clarkb: is 3 hr max limit for job timeout ? | 02:44 |
| opendevreview | Ghanshyam proposed openstack/tempest master: Temporary increase the job timeout to 3hrs https://review.opendev.org/c/openstack/tempest/+/873472 | 02:45 |
| dansmith | is that why we're still getting config error? or is the zuul bug not really fixed? | 03:41 |
| dansmith | because it seems like it ran with the 3h timeout -- doesn't seem like the config error is related right? | 03:42 |
| gmann | not sure, >3 hr timeout set end up in config error everytime | 03:43 |
| gmann | but if I remember, tripleo jobs are/were 5 hr long ? | 03:44 |
| clarkb | gmann: its 3 hours for run and then post-run can have a separate time out | 03:48 |
| clarkb | I don't recall what the max is for post-run, but the combo between them lead to the very long tripleo jobs | 03:48 |
| gmann | clarkb: ok | 03:51 |
| clarkb | the idea there is we wanted to give a job that had timed out enough time to copy logs so the situation can be debugged | 03:52 |
| clarkb | so decoupled the timeouts | 03:52 |
| gmann | its run gets timeout. post_run time is not an issue here | 04:00 |
| clarkb | I understand. I'm explaining why the tripleo jobs were able to run longer than 3 hours | 04:01 |
| gmann | k | 04:01 |
| gmann | Actually zuul error was not clear on that and those zuul erorr happening due to the bug in this week created more confusion on root cause of config erorr | 04:02 |
| clarkb | I suspect this particular error has always been an unknown error unfortunately based on storpool reporting the same thing | 04:09 |
| dansmith | clarkb: so this "unknown configuration error" is something in our zuul config or what? | 04:20 |
| dansmith | or is it related to that timeout being set too high? | 04:20 |
| clarkb | dansmith: I'm pretty sure its the error generated when your timeout value is beyond the maximum allowed | 04:21 |
| clarkb | but I haven't loaded ssh keys this evening to confirm. The storpool zuul intall showing the same error when you make the value extreme would corroborate this thoug | 04:22 |
| clarkb | digging around the default max job timeout appears to be 10800. We don't override this in our tenant config: https://opendev.org/openstack/project-config/src/branch/master/zuul/main.yaml That means the issue is https://review.opendev.org/c/openstack/tempest/+/873472/2/zuul.d/integrated-gate.yaml | 04:28 |
| clarkb | There is an exception that is raised that carries the message indicating this i the problem that should get reported so reporting it as an unknown error is probably a bug | 04:28 |
| clarkb | but you've clearly exceeded that value in that change and removing that value allows zuul to proceed so I'm fairly confident that is the issue | 04:30 |
| *** yadnesh|away is now known as yadnesh | 04:31 | |
| dansmith | clarkb: okay I didn't see that gmann increased it between when it ran and started throwing those errors | 04:54 |
| dansmith | since the config error we were getting friday depended on which worker (or scheduler) we ran on, I thought this was more of the same, where it ran sometimes, immediate fail with config error other times | 04:54 |
| dansmith | having never seen that before and then all of it on friday and today, I just figured it was related to the former | 04:55 |
| opendevreview | Merged openstack/tempest master: Move a few jobs to periodic https://review.opendev.org/c/openstack/tempest/+/873163 | 07:48 |
| *** ralonsoh_ is now known as ralonsoh | 08:37 | |
| *** jpena|off is now known as jpena | 08:54 | |
| *** soniya is now known as soniya|afk | 10:36 | |
| *** priteau_ is now known as priteau | 12:25 | |
| *** dmellado_ is now known as dmellado | 12:58 | |
| opendevreview | Ade Lee proposed openstack/devstack master: Modify devstack-base to allow for fips https://review.opendev.org/c/openstack/devstack/+/871606 | 13:02 |
| *** haleyb_ is now known as haleyb | 13:36 | |
| opendevreview | Lajos Katona proposed openstack/devstack master: Rehome functions to enable Neutron's segments integration https://review.opendev.org/c/openstack/devstack/+/818216 | 13:36 |
| *** yadnesh is now known as yadnesh|away | 17:52 | |
| *** jpena is now known as jpena|off | 18:05 | |
| gmann | dansmith: yeah, I was increasing slow job from 3 to 3.5 hr as that also timeout | 18:06 |
| dansmith | gmann: yeah I didn't realize at the time, because I was watching it while doing weekend things | 18:06 |
| gmann | k | 18:06 |
| dansmith | and I thought maybe the spurious/occasional config error thing had returned | 18:06 |
| gmann | kopecmartin: just reminder on election nomination, deadline is 15th. | 18:31 |
| *** jgwentworth is now known as melwitt | 18:33 | |
| gmann | kopecmartin: btw this also ready, placement patch merged https://review.opendev.org/c/openstack/tempest/+/866214 | 19:04 |
| opendevreview | Ghanshyam proposed openstack/tempest master: Add test for assisted volume snapshot https://review.opendev.org/c/openstack/tempest/+/864839 | 19:30 |
| kopecmartin | gmann: yep, thank you, i wanted to wait until tomorrows office hour and nominate after if no one else is interested | 19:57 |
| gmann | kopecmartin: ack | 19:58 |
| opendevreview | Dan Smith proposed openstack/devstack master: Try to reduce mysql memory usage https://review.opendev.org/c/openstack/devstack/+/873646 | 20:21 |
| clarkb | gmann: dansmith: https://review.opendev.org/c/zuul/zuul/+/873645 is the fix for getting an unknown error when pushing a too large timeout value | 20:38 |
| clarkb | it had to do with the yaml anchors and zuul not expecting yaml to use a single object for the acnhor | 20:38 |
| dansmith | cool | 20:38 |
| gmann | thanks | 20:51 |
| *** iurygregory_ is now known as iurygregory | 21:22 | |
| opendevreview | Merged openstack/tempest master: Fix retry_bad_request() context manager https://review.opendev.org/c/openstack/tempest/+/873300 | 21:34 |
| gmann | dansmith: finally ^^ | 21:35 |
| dansmith | ZOMG | 21:35 |
| dansmith | of course the patch to make that job nv was right behind it | 21:36 |
| dansmith | oh it failed | 21:36 |
| gmann | yeah I have put -W on that | 21:36 |
| gmann | we can abandon that now | 21:36 |
| dansmith | jeez | 21:36 |
| dansmith | failed that job for a different reason | 21:36 |
| dansmith | well, at least we have a chance now, cripes | 21:37 |
| opendevreview | Ghanshyam proposed openstack/tempest master: DNM: Test RBAC Nova service role https://review.opendev.org/c/openstack/tempest/+/864595 | 21:54 |
| *** dmellado_ is now known as dmellado | 22:05 | |
| opendevreview | Dan Smith proposed openstack/tempest master: DNM Test mysql memory patch https://review.opendev.org/c/openstack/tempest/+/873653 | 22:15 |
| *** dmellado_ is now known as dmellado | 22:39 | |
| *** dmellado_ is now known as dmellado | 23:02 | |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!