| *** amoralej|off is now known as amoralej | 07:43 | |
| *** Tengu_ is now known as Tengu | 08:37 | |
| opendevreview | daniel.pawlik proposed openstack/ci-log-processing master: Check if directories is not too old even when it got match files https://review.opendev.org/c/openstack/ci-log-processing/+/886587 | 10:01 |
|---|---|---|
| opendevreview | daniel.pawlik proposed openstack/ci-log-processing master: Check if directories is not too old even when it got match files https://review.opendev.org/c/openstack/ci-log-processing/+/886587 | 10:01 |
| opendevreview | daniel.pawlik proposed openstack/ci-log-processing master: Check if directories is not too old even when it got match files https://review.opendev.org/c/openstack/ci-log-processing/+/886587 | 11:32 |
| *** amoralej is now known as amoralej|lunch | 11:45 | |
| *** d34dh0r5- is now known as d34dh0r53 | 12:12 | |
| *** mgoddard- is now known as mgoddard | 12:28 | |
| *** amoralej|lunch is now known as amoralej | 12:49 | |
| *** amoralej is now known as amoralej|off | 17:19 | |
| JayF | Hi infra-root, can you please hold a VM for me next time any Ironic unit test job fails? | 20:12 |
| fungi | JayF: what's the job name? | 20:13 |
| JayF | (all of) https://github.com/jayofdoom.keys or keys from my gerrit config should be sufficient for me to get access | 20:13 |
| JayF | openstack-tox-py3{8,9,10} or ironic-tox-unit-with-driver-libs | 20:13 |
| fungi | i'm not sure if expression matches work for project and job name, but i'll try it | 20:14 |
| JayF | I'm doing that for your benefit | 20:14 |
| JayF | I don't care which job, they all fail fairly randomly | 20:15 |
| JayF | and in similar ways | 20:15 |
| fungi | mmm, also openstack-tox-py3{8,9,10} don't just run on ironic, is there a list of projects you're concerned about? | 20:15 |
| fungi | i mean to say, those aren't ironic unit test jobs | 20:16 |
| fungi | also i need to not burn dinner, so semi-afk right now | 20:16 |
| JayF | openstasck/ironic | 20:17 |
| JayF | basically the listed jobs above are failing semi-randomly on openstack/ironic | 20:17 |
| fungi | for reference here's the subcommand i'm running: https://zuul-ci.org/docs/zuul-client/commands.html#autohold | 20:17 |
| fungi | got it | 20:17 |
| JayF | we're having significant trouble getting it reproduced locally | 20:17 |
| JayF | and I'm hoping that catching a failed node will give me new info or maybe, at worst, a perfect reproducer environment | 20:18 |
| fungi | i did this: sudo zuul-client autohold --tenant=openstack --project=opendev.org/openstack/ironic --job=openstack-tox-py3.* --reason='JayF investigating random ironic unit test failures' | 20:19 |
| fungi | we'll see if that works | 20:19 |
| fungi | you should hopefully see it show up in the list at https://zuul.opendev.org/t/openstack/nodes with state=hold if it ends up catching something | 20:20 |
| JayF | fwiw I'm saving for the same reason julia did | 20:21 |
| JayF | but I think more examples would not be bad anyway | 20:21 |
| fungi | interesting. that one was a hold for an ironic-grenade failure | 20:23 |
| fungi | which you didn't mention this time | 20:23 |
| JayF | we've had a problem that orbits around migrations freezing | 20:23 |
| fungi | got it | 20:23 |
| JayF | it's happening mostly in unit tests | 20:23 |
| JayF | we had something sus in grenade that we were able to get rid of | 20:23 |
| JayF | and fix it | 20:23 |
| JayF | but unit tests remain | 20:24 |
| JayF | that makes me wonder ... does the mysql instrumentation stuff run in unit test jobs? | 20:24 |
| JayF | the ones we disable here for grenade: https://github.com/openstack/ironic/blob/master/zuul.d/ironic-jobs.yaml#L901 | 20:24 |
| * JayF goes digging | 20:24 | |
| JayF | I see no references to it outside of devstack, and aiui our unit tests don't run in devstack so I think that's not the case | 20:28 |
| JayF | Where are the base-level openstack-tox-py3X jobs defined? | 20:28 |
| JayF | aha, openstack-zuul-jobs | 20:30 |
| JayF | hm, parent: tox-fips whcih isn't in that... | 20:32 |
| JayF | fungi: (when you get back from dinner), do you know where the code is that ensures mysql/postgresql are available for unit tests? | 20:37 |
| fungi | okay, done eating, catching up now | 20:51 |
| fungi | used to be we had a job task to run something like tools/setup.sh from the repository of the project being tested, as a counterpart to (and immediately after) installing the packages indicated in its bindep.txt | 20:53 |
| JayF | ack, that would be https://github.com/openstack/ironic/blob/master/tools/test-setup.sh | 20:54 |
| fungi | that sounds right | 20:54 |
| fungi | you should see it getting run in your unit test jobs unless something significant changed while i was distracted | 20:54 |
| JayF | that is unfortunate | 20:54 |
| JayF | no magic hiding anywhere to blame :/ | 20:55 |
| fungi | fwiw, bindep was just one pass at genericizing project boilerplate (after we made pbr and then forklifted the synced "openstack" directories into oslo libs), and trying to converge a lot of the duplication from those test-setup.sh scripts was another planned effort nobody found time to get to | 20:57 |
| JayF | yeah, there's little meaningful (to mysql) difference after comparing a couple | 20:58 |
| JayF | I really hope we catch a held node and I get more data there | 20:58 |
| JayF | I'm going a little crazy | 20:58 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!