| gmann | kopecmartin: https://review.opendev.org/c/openstack/tempest/+/884804 | 00:03 |
|---|---|---|
| gmann | kopecmartin: and this unblock the nova-lvm job https://review.opendev.org/c/openstack/tempest/+/889895 | 00:04 |
| opendevreview | Ghanshyam Mann proposed openstack/tempest master: Test Nova and Glance RBAC policy old defaults https://review.opendev.org/c/openstack/tempest/+/884764 | 00:18 |
| frickler | gmann: kopecmartin: there are timeouts for tools/generate-tempest-plugins-list.py, I think this was mentioned earlier, but not sure what's the issue there | 07:57 |
| opendevreview | Jakub Skunda proposed openstack/tempest master: [WIP] Catching broken tests in tempest-full-test-account-* jobs https://review.opendev.org/c/openstack/tempest/+/889683 | 09:32 |
| *** gthiemon1e is now known as gthiemonge | 09:52 | |
| opendevreview | Merged openstack/tempest master: Disable dhcpcd in test_port_security_macspoofing_port https://review.opendev.org/c/openstack/tempest/+/889713 | 11:13 |
| *** gibi is now known as gibi_pto | 16:15 | |
| gmann | kopecmartin: this is ready too https://review.opendev.org/c/openstack/tempest/+/884764 | 17:35 |
| gmann | frickler: yeah, locally it takes less than 3 miin but we observed that in particulr provider rax and it might be slow node case? this is not just plugin generate script but we see that timeout in doc job also and on rax provider | 17:37 |
| JayF | TheJulia: ^ gmann frickler: I'll note that we're seeing slower than usual nodes causing failures in Ironic this morning as well | 17:40 |
| gmann | JayF: is it always (msot of the time) rax provider ? | 17:42 |
| * JayF looks at the one glaring example we've been digging | 17:42 | |
| JayF | yes | 17:42 |
| JayF | rax | 17:42 |
| TheJulia | it kind of felt like someone paused the vm on that one, but nothing stood out as a solid point in time' | 17:42 |
| TheJulia | two sepecific spans where things just went weirdly sideways | 17:43 |
| gmann | humm, we might need to check this provider as in tempest we see doc/plugin sanity job which used to take <20 min are timeout a lot now a days | 17:43 |
| TheJulia | we *really* need a generalized failure rate on provider graph | 17:44 |
| TheJulia | it feels like it could be a shame thing, but if we're consistently seeing a higher failure rate on a provider, there is a reason we need to understand | 17:44 |
| TheJulia | wow, a docs job on ovh failed the other day... took like the same task on my desktop 2.5 minutes, there it never finished and had a runtime of ~7 minutes to get half way along | 17:55 |
| TheJulia | 3 minutes to do basic bindep install too | 17:58 |
| dansmith | TheJulia: that's pretty much what we're seeing.. yuuuge delays in the middle of a run | 18:17 |
| dansmith | I'm highly suspicious of major IO problems or throttling going on there | 18:17 |
| dansmith | the docs timeout is a different thing, I think.. it has to survey all the openstack namespace projects and I think it gets stuck doing that sometimes probably for network reasons | 18:18 |
| dansmith | gmann: I thought scenario tests ran single-threaded? | 18:24 |
| dansmith | or did you change that recently? | 18:24 |
| gmann | dansmith: in tempest-slow slow scenario test run in parallel this is what i changed,. but in tempest-full (inherttied jobs) it is all single threaded. | 18:25 |
| dansmith | the nova-next job is running scenario with everything else (and thus multi-threaded | 18:26 |
| dansmith | 2023-07-28 18:11:22.355120 | controller | {0} setUpClass (tempest.scenario.test_volume_migrate_attached.TestVolumeMigrateRetypeAttached) ... SKIPPED: Cinder multi-backend feature disabled | 18:26 |
| dansmith | 2023-07-28 18:12:08.098934 | controller | {1} tempest.scenario.test_volume_boot_pattern.TestVolumeBootPattern.test_volume_boot_pattern [230.835610s] ... ok | 18:26 |
| gmann | yes that is multi threaded https://github.com/openstack/nova/blob/master/.zuul.yaml#L349 | 18:27 |
| gmann | but exclude a few of network one | 18:27 |
| dansmith | okay | 18:27 |
| gmann | we are trying to run all of them in parallel in tempest-full-parallel but seeing some issue | 18:28 |
| dansmith | yeah | 18:28 |
| TheJulia | dansmith: well, yes if all the docs are being built sure, but as a single project example that seems a bit... troublesome when in the past we've seen that job take no more than 10-15 minutes | 18:28 |
| dansmith | I just didn't expect it in this job but sounds like it's intentional | 18:28 |
| dansmith | TheJulia: I'm just saying I've been seeing it fail a lot recently with a timeout, which it really shouldn't | 18:29 |
| TheJulia | dansmith: agree completely | 18:29 |
| melwitt | hah, came here to ask about a openstack-tox-docs job timeout | 18:48 |
| melwitt | seemingly no reason for it that I can see in the job-output.txt | 18:49 |
| dansmith | no earthly reason | 18:51 |
| dansmith | my high school calculus teacher used to prefix every penultimate step in a problem with "and if you've been living your life right...." | 18:52 |
| melwitt | haha, awesome | 18:59 |
| melwitt | I feel like our CI is haunted | 19:10 |
| dansmith | it's definitely not healthy lately | 19:12 |
| dansmith | haunted is probably easier to reconcile | 19:13 |
| melwitt | I guess it is named zuul after all. need to call the ghostbusters | 19:13 |
| gmann | ghost become more active when we try to fix it:) | 19:32 |
| dansmith | maybe I should watch the movie again this weekend and take notes to apply to our activities next week | 19:35 |
| opendevreview | Merged openstack/tempest master: Increase the default concurrency for tempest run https://review.opendev.org/c/openstack/tempest/+/887220 | 19:58 |
| gmann | finally it is merged | 20:12 |
| melwitt | OMG a thing merged | 20:12 |
| gmann | :) | 20:13 |
| gmann | seems ghost is sleeping | 20:13 |
| melwitt | ghostbuster talk must have scared them away momentarily | 20:13 |
| gmann | yeah, we should add that comment during recheck | 20:15 |
| melwitt | haha :) | 20:16 |
| JayF | Somebody called Whoopi :) | 20:42 |
| opendevreview | Merged openstack/tempest master: Test Nova and Glance RBAC policy old defaults https://review.opendev.org/c/openstack/tempest/+/884764 | 22:06 |
| opendevreview | Merged openstack/tempest master: Default to /tmp for scenario (create|get)_timestamp() https://review.opendev.org/c/openstack/tempest/+/884804 | 22:38 |
| melwitt | heyohhh | 22:40 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!