| *** mhen_ is now known as mhen | 02:48 | |
| opendevreview | Takashi Kajinami proposed openstack/devstack master: Install OVMF package https://review.opendev.org/c/openstack/devstack/+/966236 | 04:55 |
|---|---|---|
| opendevreview | Takashi Kajinami proposed openstack/tempest master: wip: Add scenario test to verify vTPM support https://review.opendev.org/c/openstack/tempest/+/966203 | 04:56 |
| opendevreview | yatin proposed openstack/devstack stable/2025.1: [Stable/Unmaintained Only] Pin to last compatible tag for py3.9 https://review.opendev.org/c/openstack/devstack/+/966239 | 07:08 |
| opendevreview | yatin proposed openstack/devstack stable/2024.2: [Stable/Unmaintained Only] Pin to last compatible tag for py3.9 https://review.opendev.org/c/openstack/devstack/+/966240 | 07:09 |
| opendevreview | Takashi Kajinami proposed openstack/tempest master: wip: Add scenario test to verify vTPM support https://review.opendev.org/c/openstack/tempest/+/966203 | 12:37 |
| opendevreview | Takashi Kajinami proposed openstack/tempest master: DNM: Testing https://review.opendev.org/c/openstack/tempest/+/966029 | 12:38 |
| opendevreview | yatin proposed openstack/devstack stable/2025.1: [Stable/Unmaintained Only] Pin to last compatible tag for py3.9 https://review.opendev.org/c/openstack/devstack/+/966239 | 13:40 |
| sean-k-mooney | just an fyi the rescent teemept changes appare to have broken all of wathcer stable branch jobs bar 2025.2 https://bugs.launchpad.net/watcher-tempest-plugin/+bug/2130783 | 14:26 |
| sean-k-mooney | joan has been digging into this but it looks like tox is recreating the venv without the tempest plugin | 14:27 |
| sean-k-mooney | that feels like we are missing something in the devstack or tempest side to properly supprot plugins | 14:28 |
| sean-k-mooney | we are using the tox_envlist: all target | 14:29 |
| sean-k-mooney | https://github.com/openstack/tempest/blob/master/tox.ini#L69-L78 | 14:30 |
| sean-k-mooney | i think there is a impcatiablty here with ne version fo tox | 14:32 |
| sean-k-mooney | which nolonve rsupprot share envs between invocation like old version did | 14:32 |
| sean-k-mooney | https://github.com/openstack/tempest/blob/master/tox.ini#L70 | 14:32 |
| stephenfin | (damn good time for me to check this channel) | 14:32 |
| sean-k-mooney | stephenfin: i know you brrought this up before | 14:33 |
| sean-k-mooney | but ya so this is going to start brakign more peopel i think | 14:33 |
| stephenfin | sean-k-mooney: I was looking at that a while back. Tempest/DevStack was using tox as a virtualenv manager of sorts | 14:33 |
| sean-k-mooney | yes it still is | 14:33 |
| sean-k-mooney | i reviewed your devstack patch | 14:33 |
| stephenfin | yup | 14:33 |
| sean-k-mooney | the problem is we use it for 2 things | 14:33 |
| stephenfin | The fix is complicated. I haven't been able to sink my teeth into it properly since | 14:33 |
| sean-k-mooney | 1 to prepare the vnev | 14:33 |
| sean-k-mooney | and 2 to defien the test to run | 14:34 |
| stephenfin | yep | 14:34 |
| sean-k-mooney | so we could hackaround this but i think what we need to do | 14:34 |
| sean-k-mooney | is modify the teox env so we can pass a set of addtional plugins to install | 14:34 |
| stephenfin | I don't believe I ever got the patches to a point that I could push them, but I had explored the concept of a "test suite" in tempest, that would allow us to replace (2) with a new tempest option | 14:35 |
| sean-k-mooney | the problem is that devstack is first creeating the venv then adding the plugins to it and then later executing it | 14:35 |
| stephenfin | and then we could use bog standard virtualenvs (likely created by devstack) for (1) | 14:35 |
| sean-k-mooney | well they are not hardcoded | 14:35 |
| sean-k-mooney | we have several enves that are marpaterized by env vars | 14:36 |
| sean-k-mooney | so the josb defien the include/exclude regex for example | 14:36 |
| sean-k-mooney | tox envs like integrated-network https://github.com/openstack/tempest/blob/master/tox.ini#L175-L187 | 14:37 |
| stephenfin | yeah, I know | 14:37 |
| stephenfin | https://paste.opendev.org/show/budysYJqy3XOTNjDCnX1/ | 14:37 |
| sean-k-mooney | are hardcoded | 14:37 |
| stephenfin | that's the commit message I have. the patch isn't worth pushing (it's not complete) | 14:37 |
| stephenfin | tbh, _someone_ needs to sink a day or two into this to fix it properly | 14:37 |
| sean-k-mooney | so thte thing is i dont think the tests shoudl nessiarly have any annothation on them | 14:38 |
| sean-k-mooney | so anything that require us to merge someing in tempest to change what is run is kind of a deal breaker | 14:38 |
| sean-k-mooney | we need to be able to defien the test suite form nova or watcher or wahtever is executing the tempet job | 14:39 |
| sean-k-mooney | if i read that commit message it wont sovle the standard case that i kare about | 14:39 |
| sean-k-mooney | unless your thinking that i can pass a file or somehting with the regexes ectra that stester woudl consume | 14:40 |
| sean-k-mooney | stephenfin: by the way there is a serial decorator in tempest already | 14:41 |
| sean-k-mooney | any test that must run serially shoudl be using that | 14:41 |
| sean-k-mooney | the senario test that often run seriall dont use that today because they can run in parallel if you have enough capsity | 14:41 |
| sean-k-mooney | if i was sittign down to write this i woudl stop using tox to invoke stestr adn add a role for that | 14:42 |
| stephenfin | sean-k-mooney: serial is about the only one that doesn't need changes. As you say: you can just use the serial option | 14:42 |
| sean-k-mooney | have devstack just create teh venv itslef | 14:42 |
| sean-k-mooney | and supprot defining stest phases in the job | 14:42 |
| sean-k-mooney | stephenfin: no i mean https://github.com/openstack/tempest/tree/master/tempest/serial_tests | 14:43 |
| stephenfin | the issue as I understood it is that various tempest plugin projects are calling some of the tempest tox envs and treating that as an "API" | 14:43 |
| sean-k-mooney | we have an @serial decorate to denote that a test must run seriall even if you have --concurance=4 | 14:43 |
| sean-k-mooney | stephenfin: yes that is the current api provide by the zuul job | 14:44 |
| sean-k-mooney | but we coudl change how the zuul job works and provide a better api | 14:44 |
| sean-k-mooney | im effectivly saying the tempet job shoudl not use tox at all | 14:45 |
| sean-k-mooney | any job that used tox to defien the set of test to run woudl move to beign defien in teh zuul job defintion instead | 14:45 |
| sean-k-mooney | by the way this is only broken on stable | 14:47 |
| sean-k-mooney | on master the jobs are passing | 14:47 |
| sean-k-mooney | so i think is realted to seting uc vai the env | 14:47 |
| stephenfin | Sorry, multitasking (badly) | 14:47 |
| sean-k-mooney | its fine | 14:47 |
| stephenfin | yes, I think we're on the same page | 14:47 |
| stephenfin | the question is just how | 14:48 |
| sean-k-mooney | there are 2 thing i want to do 1 get a short term fix, 2 start a dicsson on how to move away form using tox as the api | 14:48 |
| stephenfin | tbf, I hadn't considered an ansible/zuul-only fix | 14:48 |
| stephenfin | there's no reason we couldn't "translate" the tox_envlist into calls for a shared venv | 14:48 |
| sean-k-mooney | this is obviouly a desion for the tempest team but i woudl prefer to delete most of the tox target in tempest and use that only for local dev | 14:49 |
| stephenfin | *into calls to tempest in the shared venv | 14:49 |
| sean-k-mooney | and decopule the tempet jobs form tox | 14:49 |
| stephenfin | tbh, I'd even get rid of that | 14:49 |
| stephenfin | for local dev | 14:49 |
| sean-k-mooney | tox -e full is nice for local dev but not explcity requried | 14:49 |
| stephenfin | It's always confused me, personally. Tempest is designed to be installed system-wide. We don't e.g. provide a tox target for running nova | 14:50 |
| sean-k-mooney | no | 14:50 |
| sean-k-mooney | tempest master is desied to run on the stable branches | 14:50 |
| stephenfin | sean-k-mooney: right, that's what I was getting at with the idea of a test suite | 14:50 |
| sean-k-mooney | so it has to be in its own venv | 14:50 |
| sean-k-mooney | we cant assuem mater tempest is co installable with stable/* openstack | 14:50 |
| sean-k-mooney | that why its special today | 14:50 |
| stephenfin | so instead of running 'tox -e full', you'd do 'tempest --tbd-flag full' | 14:50 |
| sean-k-mooney | yes but you woudl have to create the venv manully (if not using devstack) | 14:51 |
| stephenfin | which would run | 14:51 |
| stephenfin | tempest run --regex '(?!.*\[.*\bslow\b.*\])(^tempest\.api)' | 14:51 |
| stephenfin | tempest run --combine --serial --regex '(?!.*\[.*\bslow\b.*\])(^tempest\.scenario)|(^tempest\.serial_tests)' | 14:51 |
| stephenfin | in order | 14:51 |
| sean-k-mooney | sure which is a workflow regression | 14:51 |
| stephenfin | what is? | 14:51 |
| stephenfin | the manual creation of a venv? | 14:51 |
| sean-k-mooney | the fact you need to manually manage the venv | 14:51 |
| sean-k-mooney | yes | 14:52 |
| stephenfin | wearing a seatbelt is a comfort regression | 14:52 |
| sean-k-mooney | as i said devstack shoudl probly not use the tox envs but a tempet developer shoudl be able to continue to | 14:52 |
| stephenfin | but there are other things to consider. same here ;) | 14:52 |
| sean-k-mooney | right so are you propsoing we remove py3 form nova | 14:52 |
| sean-k-mooney | because ot me this is exaclty the same | 14:52 |
| sean-k-mooney | asking the tempet team to give up tox entirly does nto sound reasonable | 14:53 |
| sean-k-mooney | having devstack not use tox for tempest is reasonable IMO | 14:53 |
| stephenfin | not the same thing though. nova's test suite isn't pluggable. tempest's is | 14:53 |
| sean-k-mooney | i dont think the plugablity is a good enough arguremnt | 14:54 |
| sean-k-mooney | we coudl supprot the plugin via tox if needed | 14:54 |
| stephenfin | you mean something installing e.g. manila-tempest-plugin when you run 'tox -e full' from tempest? | 14:54 |
| sean-k-mooney | https://github.com/openstack/watcher/blob/master/tox.ini#L19 | 14:54 |
| sean-k-mooney | yes | 14:55 |
| stephenfin | eeew | 14:55 |
| sean-k-mooney | tox also natively suprpot this by the way | 14:55 |
| sean-k-mooney | even without ^ | 14:55 |
| stephenfin | Sorry, but that goes against the whole point of using tox. If it's not reproducible what's the point. | 14:55 |
| sean-k-mooney | it is repoduceable | 14:55 |
| stephenfin | If we need virtualenvs, why can't tempest manage them? | 14:56 |
| sean-k-mooney | it cant manage the installation of itself in a venv | 14:56 |
| sean-k-mooney | if its not installed | 14:56 |
| sean-k-mooney | the same way tox cant manage itself | 14:56 |
| stephenfin | well actually.... | 14:56 |
| sean-k-mooney | i know the .tox venv thing i tuses for building but that is not the same if that is what yoru goign to say | 14:57 |
| stephenfin | I don't see the difference 😇 | 14:57 |
| sean-k-mooney | that is not managing the outer tox | 14:57 |
| sean-k-mooney | that still neeed to be installed somewhere where its requiremetn are ment | 14:58 |
| stephenfin | uv run tempest ... | 14:58 |
| stephenfin | pipx install tempest; tempest ... | 14:58 |
| stephenfin | virtualenv .venv; source .venv/bin/activate; pip install tempest; tempest ... | 14:58 |
| sean-k-mooney | yep but again back to manual depenciy management | 14:58 |
| sean-k-mooney | and if i want to devleop tempst that a regression | 14:58 |
| stephenfin | but don't you already need to have a tempest.conf file somewhere? | 14:59 |
| stephenfin | tox doesn't lay that down, afaik | 15:00 |
| sean-k-mooney | not in general | 15:00 |
| sean-k-mooney | it has sane defaults | 15:00 |
| sean-k-mooney | and if you want to create it you can | 15:00 |
| sean-k-mooney | it has a workspace concept | 15:00 |
| sean-k-mooney | it being tempest | 15:00 |
| stephenfin | so I can just set OS_CLOUD and then run 'tox -e full' on my local machine? | 15:00 |
| stephenfin | and it'll just work? | 15:00 |
| sean-k-mooney | it will not jsut pass | 15:00 |
| stephenfin | so there's bootstrapping needed? | 15:00 |
| sean-k-mooney | yes | 15:01 |
| sean-k-mooney | which devstack does | 15:01 |
| sean-k-mooney | in ci | 15:01 |
| sean-k-mooney | well or locally | 15:01 |
| stephenfin | okay, but tox doesn't | 15:01 |
| sean-k-mooney | correct | 15:01 |
| sean-k-mooney | https://github.com/openstack/tempest/blob/master/.gitignore#L12 | 15:02 |
| sean-k-mooney | so tempest igt ignore chantges ot the .conf files in etc | 15:02 |
| stephenfin | okay, so you already need devstack to configure things for you | 15:02 |
| sean-k-mooney | in the git repo so you can tune them locally | 15:02 |
| stephenfin | otherwise you need to do it yourself | 15:02 |
| sean-k-mooney | and then when you use tox in the tempt repo it uses that | 15:02 |
| sean-k-mooney | ya you need to drop a tempst.conf into the etc directry i belive | 15:03 |
| stephenfin | same as you need to drop a nova.conf in the etc directory | 15:04 |
| stephenfin | and again, we don't provide a tox target for running nova | 15:04 |
| sean-k-mooney | so that is all devstack is doing it creates /opt/stack/tempest/etc/tempest.conf | 15:04 |
| sean-k-mooney | the you can use tox form the root of the repo | 15:05 |
| stephenfin | it also precreated the tox venv, currently | 15:05 |
| stephenfin | iirc | 15:05 |
| stephenfin | *pre-creates | 15:05 |
| sean-k-mooney | it create an addtion tox venv | 15:05 |
| sean-k-mooney | in the /opt/stack/data dir | 15:05 |
| sean-k-mooney | you dont have to use that ro care about it | 15:05 |
| sean-k-mooney | but you can | 15:05 |
| stephenfin | running it with --notest | 15:05 |
| sean-k-mooney | correct | 15:05 |
| stephenfin | see you actually do | 15:05 |
| stephenfin | because tempest plugins get installed into that venv | 15:06 |
| sean-k-mooney | you dont that only in ci | 15:06 |
| sean-k-mooney | locally you can install those yourself | 15:06 |
| stephenfin | or locally, if you running tempest against you devstack | 15:06 |
| sean-k-mooney | into the venv created by tox | 15:06 |
| stephenfin | *your | 15:06 |
| sean-k-mooney | i need to go join a retro... but what tempest does interally is really there chocie | 15:06 |
| sean-k-mooney | i just want to decouple ci form the tox envs | 15:07 |
| sean-k-mooney | so we dont get broken when tox breaks compaitblity liek this | 15:07 |
| stephenfin | on the same page there | 15:07 |
| opendevreview | Stephen Finucane proposed openstack/devstack master: Enable response validation in Keystone https://review.opendev.org/c/openstack/devstack/+/962852 | 15:29 |
| opendevreview | Stephen Finucane proposed openstack/devstack master: Enable response validation in Nova https://review.opendev.org/c/openstack/devstack/+/962853 | 15:29 |
| opendevreview | Stephen Finucane proposed openstack/devstack master: Enable response validation in Keystone https://review.opendev.org/c/openstack/devstack/+/962852 | 15:30 |
| opendevreview | Stephen Finucane proposed openstack/devstack master: Enable response validation in Nova https://review.opendev.org/c/openstack/devstack/+/962853 | 15:30 |
| sean-k-mooney | stephenfin: is there ah particalar reason you split those up | 16:10 |
| sean-k-mooney | doign it in devstack itself makes sense to me as we shoudl also run with this when developeing locally | 16:10 |
| stephenfin | just separation of concerns | 16:10 |
| sean-k-mooney | ack | 16:11 |
| gmaan | sean-k-mooney: stephenfin when there is constraint mismatch happen then tempest venv is recreated (there is no way to tell tox not to do) which uninstall all previously installed plugins | 17:03 |
| stephenfin | yup, we're saying we need to stop using tox and start manually managing a venv | 17:04 |
| stephenfin | like we manually manage the global venv for everything else | 17:04 |
| gmaan | we keep all used constraints as it is but I think in the py3.9 drop we did mistake in https://review.opendev.org/c/openstack/tempest/+/965903 | 17:04 |
| gmaan | yeah, we can do that but we should not able to solve it with tox also., I just logded in. checking it how we can fix it fast | 17:05 |
| sean-k-mooney | gmaan: i was wondering are we jsut missing passing or setting the env var in one of the invocations | 17:07 |
| sean-k-mooney | i.e. are we passign it when creating the env with --no-test adn failing to pass it when runninng the tests or something like htat | 17:09 |
| gmaan | we are not, we have faced this issue in past also. thing is we pined the stable constraints for current stable branch but using tempest master https://review.opendev.org/c/openstack/tempest/+/965903/6/roles/run-tempest/tasks/main.yaml | 17:09 |
| sean-k-mooney | gmaan: can you look at this https://review.opendev.org/c/openstack/devstack/+/966239/comment/00f446b6_cb062314/ and weigh in | 17:09 |
| gmaan | tempest master use do run tox wioth master constraints in devstack/grenade scripts and due to constraints mismatch it end up recreating venv | 17:10 |
| sean-k-mooney | right so i think there are a few thigns hppening. 2025.1 technically supprot python 3.9 | 17:11 |
| gmaan | yes, we should use tempest master but py3.9 drop is the key here which trigger all those changes | 17:11 |
| sean-k-mooney | so yatin want to pin to a tag for 3.9 support | 17:11 |
| gmaan | yeah, I am checking what exact job was broekn for py3.9 | 17:11 |
| sean-k-mooney | but we shoudl be using master in 2025.1 in general | 17:11 |
| sean-k-mooney | not a pin | 17:11 |
| sean-k-mooney | so right now our ubunut 24.04 based jobs that use 3.12 are broken with master | 17:12 |
| gmaan | Ubuntu 22.04 should have py3.100 debian 12 py3.11 | 17:12 |
| opendevreview | Stephen Finucane proposed openstack/devstack master: WIP: lib/tempest: Manually manage virtualenv https://review.opendev.org/c/openstack/devstack/+/959213 | 17:12 |
| opendevreview | Stephen Finucane proposed openstack/devstack master: lib/tempest: Start creating shared tempest venv https://review.opendev.org/c/openstack/devstack/+/966306 | 17:12 |
| sean-k-mooney | or rahter our job on master watcher-tempest-plugin that test with stable/* are broken | 17:12 |
| sean-k-mooney | gmaan: yes | 17:12 |
| sean-k-mooney | gmaan: but https://github.com/openstack/governance/blob/master/reference/runtimes/2025.1.rst#python | 17:13 |
| sean-k-mooney | project generally didnt test 3.9 for 2025.1 out side of tox | 17:13 |
| gmaan | yeah, rocky linux9 jobs can have issue as they have py3.9 but there is way to run tempest on newer tempesty from venv | 17:13 |
| sean-k-mooney | exepct for the small few that had a centos fips job | 17:13 |
| sean-k-mooney | well rocky ectra could use 3.10+ | 17:14 |
| sean-k-mooney | we actully moved c9s to 3.11 i belvie | 17:14 |
| sean-k-mooney | before we added c10s | 17:14 |
| gmaan | yeah, it was CentOS 9-stream job failing , this is whayt ykarel_ reported | 17:14 |
| sean-k-mooney | https://review.opendev.org/c/openstack/watcher-tempest-plugin/+/966256 | 17:15 |
| sean-k-mooney | so the reecne change has broken job that previosly passed | 17:16 |
| sean-k-mooney | because fo https://review.opendev.org/c/openstack/tempest/+/965903/comment/c4c223a8_cb8b8b3d/ as you noted | 17:17 |
| opendevreview | Ghanshyam proposed openstack/tempest master: Revert "Use stable branch upper-constraints till 2025.1" https://review.opendev.org/c/openstack/tempest/+/966308 | 17:17 |
| gmaan | sean-k-mooney: ^^ I have created the revert of it, can you run any DNM change on watcher-tempest-plugin and see if it run all stable jobs there | 17:17 |
| sean-k-mooney | sure | 17:18 |
| gmaan | I am still booting my vm, | 17:18 |
| gmaan | thanks | 17:18 |
| sean-k-mooney | we likely want to jsut add "unmaintained/2024.1", and add a contional in devstack to pin on the sabel branche if its 3.9 or move those jobs ot 3.10 | 17:18 |
| gmaan | my bad on that change not thinking about tox venv creation which I faced many times during EM model | 17:19 |
| sean-k-mooney | well that also new behavior in tox right | 17:19 |
| sean-k-mooney | it only started doing the recreate in this case in the last 2-3 years or so | 17:19 |
| gmaan | pin or those jobs can run the tempest in higher py version in venv. for example. tempest venv can have py3.10 which can test the py3.9 job/cloud | 17:19 |
| sean-k-mooney | oh ya that might be a better idea just move tempest forward | 17:20 |
| sean-k-mooney | have we done that before? | 17:20 |
| gmaan | yeah, that is why tempest being branchless give us benefits. to run it on new things but it can still test old openstack with same set of tempest master tests | 17:21 |
| gmaan | not sure if we did but I have to check. i remember py3.6 drop time has same issue and we recommended to do that but that was long back | 17:21 |
| sean-k-mooney | ack. let me go create that dnm | 17:22 |
| gmaan | and it only need for cs and rocky jobs | 17:22 |
| sean-k-mooney | and only for 2025.1 and older | 17:23 |
| gmaan | yes | 17:23 |
| gmaan | 2204.1 is going to be in unmaintained soon https://review.opendev.org/c/openstack/releases/+/963711 | 17:24 |
| gmaan | so anyways we need to pin tempest there so you can leave that | 17:24 |
| gmaan | 2024.2 and 2025.1 need fix | 17:24 |
| sean-k-mooney | gmaan: also hi how are you today o/ | 17:26 |
| sean-k-mooney | gmaan: we kind of just wen right to the ci failure | 17:26 |
| sean-k-mooney | https://review.opendev.org/c/openstack/watcher-tempest-plugin/+/966311 should give us some result in a bit | 17:27 |
| gmaan | yeah, thought of checking email/irc ping before coffee but I think i do not need it now :) | 17:27 |
| gmaan | perfect | 17:27 |
| sean-k-mooney | oh not having coffee before doing either of those things is alwasy risky | 17:28 |
| gmaan | heh yeah | 17:29 |
| sean-k-mooney | not that my work life blance is an example for anyone but i like to make my coffee in the morning then i bring it upstairs and review my emails and such as i drink it in the morning and think about my day as the caffine kicks in | 17:30 |
| sean-k-mooney | its nice ot thake the first 30-60 mins slow and just read before jumping in to doing work | 17:31 |
| gmaan | yeah, true, I usually open system after breakfast but today I started 30 min late from usual time to thought of checking if anything urgent | 17:33 |
| gmaan | but not a good idea | 17:33 |
| sean-k-mooney | at this point in the cycle gate issues are nto going ot kill anyone so dont stress, enjoy your breakfast if you have not alrady had it | 17:38 |
| gmaan | yeah, going but found another issue of tempest/.devstack stop running ironic jkob because it was rename. | 17:39 |
| gmaan | Zuul ignore to run the job which cannot be find which i think should be error or warning to notice if job is renamed/removed | 17:40 |
| opendevreview | Stephen Finucane proposed openstack/tempest master: Remove run-tempest-26 role https://review.opendev.org/c/openstack/tempest/+/966314 | 17:41 |
| opendevreview | Stephen Finucane proposed openstack/tempest master: Remove deprecated regex opts https://review.opendev.org/c/openstack/tempest/+/966315 | 17:41 |
| opendevreview | Stephen Finucane proposed openstack/tempest master: tools: Remove alias for old exclude-list file https://review.opendev.org/c/openstack/tempest/+/966316 | 17:41 |
| opendevreview | Stephen Finucane proposed openstack/tempest master: Move exclude lists into package https://review.opendev.org/c/openstack/tempest/+/966317 | 17:41 |
| opendevreview | Stephen Finucane proposed openstack/tempest master: Split, rename run-tempest role https://review.opendev.org/c/openstack/tempest/+/966318 | 17:41 |
| stephenfin | gmaan: I'm using the rework-tempest-install topic https://review.opendev.org/q/topic:rework-tempest-install | 17:42 |
| stephenfin | gmaan: sean-k-mooney: my plan is to (a) start creating a parallel self-managed tempest virtualenv in devstack and (b) create a new job that will run tempest from said venv directly with relevant options instead of running tox | 17:44 |
| sean-k-mooney | +1 if you need any help with that let me know | 17:45 |
| stephenfin | I so far have (a) done and some of the prework for (b). I'd likely have (b) finished already were my Ansible skills not so poor 😅 | 17:45 |
| sean-k-mooney | ask claude or cursor :P | 17:45 |
| stephenfin | (there's a high possibility what I've done in the tempest jobs is not correct ansible) | 17:45 |
| stephenfin | nah, this is learning exercise for me ;) | 17:45 |
| sean-k-mooney | that fair too | 17:46 |
| johnsom | I am guessing the above chat is why our 2024.2->2025.1 stable branch jobs started not installing the tempest plugins.... | 17:56 |
| sean-k-mooney | related yes | 17:56 |
| sean-k-mooney | we have a dnm up to test the revert of the suspected cause | 17:57 |
| johnsom | Ack, I will watch this thread | 17:57 |
| sean-k-mooney | https://review.opendev.org/c/openstack/tempest/+/966308 is likely the cause | 17:57 |
| sean-k-mooney | sorry that the revert | 17:57 |
| sean-k-mooney | tldr is the sabel branches shoudl not have been listed in that list | 17:57 |
| sean-k-mooney | we need to fix c9s and rocky jobs diffently | 17:58 |
| johnsom | Yeah, I am not concerned with those at the moment, they are non-voting | 17:59 |
| sean-k-mooney | https://zuul.opendev.org/t/openstack/build/2b9ec2d6586b4aad893a60195e188945 | 17:59 |
| sean-k-mooney | so that passed with the revert via a dnm | 17:59 |
| sean-k-mooney | *depends on | 18:00 |
| johnsom | Nice | 18:00 |
| sean-k-mooney | or watcher-tempest-functional-2024-1 job is failing but i think that branch has been deeleted the patch to remove that job was blocked by the other fialiures | 18:00 |
| clarkb | I haven't really followed this discussion what is the benefit of running tempest without tox (I'm a fan of less tox myself so am curious) | 18:07 |
| sean-k-mooney | clarkb: tox used to allwo sharign venv between tox target but that is not really supprot any more | 18:08 |
| sean-k-mooney | the way we are suing it in devstack where we create the vnve install any plugins and then try to use it is problmeaic | 18:08 |
| clarkb | I think nox would let you do that but I'm not 100% certain :) | 18:08 |
| sean-k-mooney | clarkb: so my proposl which is similar ot stephenfin is lets not use tox for the venv mangenemt or test execution in ci | 18:09 |
| sean-k-mooney | but if tempet cores want ot use tox for local dev they shoudl be able too | 18:09 |
| sean-k-mooney | stephen was suggeting jsut removing it for the local case too | 18:10 |
| clarkb | ya I think that works too | 18:10 |
| clarkb | has tox already dropped the feature so we need to pin it currently? | 18:10 |
| sean-k-mooney | clarkb: effectivly we woudl end up moving the definiton fo what integrate-networkign runs form the tox env into the integrated-netowrkign zuul job | 18:10 |
| sean-k-mooney | clarkb: that not what the pin was for but also kind of yes | 18:11 |
| sean-k-mooney | tox in 4.something will recreat the venv if the venv is annotated as being created for a diffent tox target | 18:12 |
| sean-k-mooney | or if the defintion of the tox target changes | 18:12 |
| sean-k-mooney | clarkb: we think that in this case we are passing diffent values for uc file between when devstack creates it with --no-tests | 18:12 |
| sean-k-mooney | and when we actully run tempest | 18:12 |
| sean-k-mooney | clarkb: when we said pinnign we were dicussing that the patch to pin tempest for unmaintianed did it on stable branches too | 18:13 |
| sean-k-mooney | which was incorect | 18:13 |
| sean-k-mooney | or rahter it pinend the uc version used and there was a stabel only pathc to pin tempest | 18:14 |
| sean-k-mooney | https://review.opendev.org/c/openstack/tempest/+/966308/1/roles/run-tempest/tasks/main.yaml pinned the UC defintion and https://review.opendev.org/c/openstack/devstack/+/966239/2/stackrc was propsoed to pin tempest | 18:15 |
| sean-k-mooney | but that is nto the correct approch to the failing jobs with the removal of py39 | 18:15 |
| clarkb | thanks for the summary | 18:23 |
| gmaan | johnsom: yesh, wait for this to merge, it will unblock stable jobs (except py3.9 which are CS and rocky linux) https://review.opendev.org/c/openstack/tempest/+/966308 | 18:24 |
| gmaan | I will send it on ML as it impacted most of the plugins stable jobs | 18:25 |
| clarkb | https://github.com/wntrblm/nox/issues/167#issuecomment-951728842 nox can do it but in a roundabout way | 18:26 |
| clarkb | but on the positive side of things for that workaroudn it relies on explicit functionality rather than implicit behaviors so should be stable over time hopefully | 18:26 |
| sean-k-mooney | sure or we could jsut have devstack create teh venv, install the deps including the tempest plugins and then just activate the venv and run tempest | 18:29 |
| sean-k-mooney | the only beifit to nox i guess in this case woudl be keepign the deinfiotn of integrated-* runable locally | 18:30 |
| sean-k-mooney | without copy pasting the regex form the zuul.yaml | 18:30 |
| clarkb | I think the main benefit to nox is it hasn't broken functionality and tends to be more explicit because you're writing python to accomplish things. Gives you a lot more control vs being at hte mercy of tox which has made some intersting choices over time | 18:31 |
| clarkb | but I agree there really isn't anything special to making a venv and install deps and running a command in it | 18:31 |
| sean-k-mooney | well that the main benifit over tox right | 18:31 |
| clarkb | can do that with a simple script too (I have an ancient makefile proposal in gerrit somewhere that uses make to drive it too) | 18:32 |
| sean-k-mooney | as in in general not for this specific usecase | 18:32 |
| clarkb | ya | 18:32 |
| sean-k-mooney | bindep moved to using nox right? | 18:32 |
| sean-k-mooney | perhaps nox shoudl be included in the pti dicussion for pytest | 18:32 |
| sean-k-mooney | i think the pti stipulates tox today | 18:33 |
| sean-k-mooney | ya it does https://github.com/openstack/governance/blob/master/reference/pti/python.rst?plain=1#L22 | 18:33 |
| clarkb | ya we moved a few opendev tools because it was at least as much work to review changes to switch to nox as it was to fix tox | 18:33 |
| clarkb | and at that point its an easy decision imo | 18:34 |
| clarkb | sean-k-mooney: I know when I started porting things to nox the thought was it would be impossible to change all of openstack due to the number of projects and branches as so on which is a fair point and changes the cost of switching analysis vs say for bindep and git-review | 18:37 |
| clarkb | but it is interesting to me that tox continues to regularly break things on a 6-12 month schedule | 18:37 |
| opendevreview | Ghanshyam proposed openstack/tempest master: Re-add the ironic job in gate https://review.opendev.org/c/openstack/tempest/+/966339 | 18:38 |
| sean-k-mooney | well this behaivor changed about 2 years ago | 18:38 |
| clarkb | I do think it is problematic that they reinvent the wheel on package building and installation too. There are standard tools that exist for that with different behaviors than tox | 18:39 |
| sean-k-mooney | we just didnt drop a python release until now | 18:39 |
| clarkb | ack | 18:39 |
| sean-k-mooney | the stable branches were incldued to make 3.9 work but as i said that was not the right approch | 18:39 |
| opendevreview | Ghanshyam proposed openstack/devstack master: Re-add the ironic job in gate https://review.opendev.org/c/openstack/devstack/+/966340 | 18:39 |
| opendevreview | Ghanshyam proposed openstack/devstack stable/2025.2: Re-add the ironic job in gate https://review.opendev.org/c/openstack/devstack/+/966341 | 18:40 |
| opendevreview | Ghanshyam proposed openstack/devstack stable/2025.2: Re-add the ironic job in gate https://review.opendev.org/c/openstack/devstack/+/966341 | 18:43 |
| opendevreview | Ghanshyam proposed openstack/devstack stable/2025.1: Re-add the ironic job in gate https://review.opendev.org/c/openstack/devstack/+/966342 | 18:46 |
| gmaan | clarkb: is there any way for zuul to error or warn when jobs does not run because definition is not found (due to rename or removal) ? | 18:48 |
| gmaan | because it ignore the job run and goes unnotice like ^^. | 18:49 |
| clarkb | gmaan: it errors when you try to update the config to remove things | 18:49 |
| clarkb | so these situations generally require us to take action outside of normal zuul operations (which can include branch deletions whcih I think may be the easiest way to sneak this in) | 18:49 |
| gmaan | I manually reviewed the jobs in tempest and found this is not running otherwise it could be unnotice for long time. due to it neutron change in devstack already broke ironic in last cycle | 18:50 |
| clarkb | gmaan: do you know how `ironic-tempest-bios-ipmi-direct-tinyipa` was removed? | 18:50 |
| gmaan | this happened when job was renamed | 18:50 |
| clarkb | yes but you cannot rename or delete builds that are in use. Zuul will error | 18:51 |
| gmaan | https://review.opendev.org/c/openstack/ironic/+/950192/12/zuul.d/ironic-jobs.yaml#1019 | 18:51 |
| gmaan | it did not give any error | 18:51 |
| clarkb | you also cannot add a reference to a job name that doesn't exist | 18:51 |
| gmaan | it was added long back and running fine but last cycle it got renamed and no error from Zuul on renaming change or the one using the old na,me | 18:51 |
| clarkb | gmaan: the job exists https://zuul.opendev.org/t/openstack/job/ironic-tempest-bios-ipmi-direct-tinyipa that is why it wasn't an error | 18:52 |
| gmaan | for master? | 18:52 |
| clarkb | no it is not defined on master (but that doesn't matter to zuul) | 18:52 |
| gmaan | it only exist on stable or non master branch | 18:52 |
| clarkb | you are allowed to define jobs on different branches | 18:53 |
| gmaan | that is why results are false positive. master gate stop running it because it could not find the definition on master and Zuul did not error or warn | 18:53 |
| clarkb | yes I think this is intentional | 18:53 |
| gmaan | I am thinking if job is not run due to any reason but it is added in pipeline then error | 18:54 |
| clarkb | this is what allows us to drop grenade jobs iirc | 18:54 |
| gmaan | but it break things when job just gets ignored without any erorr/warn. | 18:54 |
| clarkb | doing so would break grenande iirc ebcause we stop running it on old branch by deleting the definition on the old branches but not the pipeline entries? I'm pretty sure this specific behavior is tied to openstack grenade sunsetting | 18:54 |
| gmaan | I mean does not test the things which suppposed to be so change break things | 18:55 |
| clarkb | but I need tocheck the job definition to see if therei s some other reason it isn't running | 18:55 |
| clarkb | it could be that zuul did try to run it and decided it shouldn't because of file matchers or something | 18:55 |
| gmaan | I think it is because of job rename | 18:55 |
| gmaan | if not error at least warning in results that xyz job mentioned in pipeline to run is not run due to abc/unknown reason | 18:56 |
| gmaan | that could be helpful to know if jobs are changed (renamed, removed, branch variant change etc) | 18:57 |
| clarkb | ya I could see that being a decent compromise to make maitnenance burdens lower for branch to branch changeovers but call out things that might surprise people | 18:58 |
| clarkb | gmaan: do you want to suggest that to the zuul project? | 18:58 |
| gmaan | sure | 18:58 |
| gmaan | oh its on matrix now not on irc? | 18:59 |
| clarkb | gmaan: yes, you can also send email to the zuul-discuss@lists.zuul-ci.org list | 19:00 |
| clarkb | that will probably get moderated if you aren't a list member but it is low volume if you want to join otherwise I can prod jim to let it through | 19:00 |
| gmaan | I think email is good idea, will do | 19:00 |
| gmaan | I did send earlier but not sure as my email is changed | 19:00 |
| clarkb | I think its helpful when the user having the problem can articulate their use case rather than trying to articulate it for them so I appreciate it | 19:01 |
| clarkb | fwiw there are irrelevant file matchers which shouldn't have been a problem in this case and no explicit branch matchers so I think you are correct that the behavior is explained by the implicit branch matcher behavior | 19:03 |
| clarkb | gmaan: as a side note tehre is a comment in the ironic config that calls out the job is used by others but I guess they didn't follow up on that when removing (renaming) it | 19:06 |
| clarkb | we did have breadcrumbs in place but they were insufficient | 19:07 |
| gmaan | yeah | 19:07 |
| corvus | gmaan: do you have a link to a change that did not run the job you expected? | 19:12 |
| gmaan | yeah | 19:12 |
| corvus | clarkb: i apparently highlight on jim, probably on account of my matrix id. :) | 19:12 |
| gmaan | this one https://review.opendev.org/c/openstack/tempest/+/965903 | 19:12 |
| clarkb | corvus: oh ha that is unexpected | 19:13 |
| corvus | 2025-11-04 19:16:36,408 DEBUG zuul.layout: [e: 4f1a2c3e19ca425d9adfc1b931971e76] No matching parents for job ironic-tempest-bios-ipmi-direct-tinyipa and change <Change 0x74d8feb1c410 openstack/tempest 965903,6> | 19:20 |
| corvus | that's the debug line, and just fyi (i know this isn't what you're asking for), if you did perform a debug enqueue of the change (with "debug: true" in the pipeline config) you would get that output. (i understand you want zuul to tell you that error without you needing to ask for it.) | 19:21 |
| corvus | so if someone did want to make that a warning message, that's the case to look for. | 19:21 |
| gmaan | yeah, warning will be helpful at least | 19:22 |
| corvus | tbh, i think we'd need to think long and hard about that -- because there are a lot of jobs that zuul doesn't run, so any change to start telling users unprompted what zuul is not doing needs to be thought of very carefully. so i don't want to commit to saying that's a good idea and we'd accept it. | 19:22 |
| corvus | but if it isn't too onerous, i think it would not be that hard of a change to make it a buildset warning. | 19:23 |
| gmaan | sure, at least users should be able to audit their pipeline if some notice (if not on every change) can be send | 19:23 |
| corvus | on that date (11-04) opendev logged that 596 times. i don't know how many of those were expected or not, or how many of those would be useful as warnings or just annoying. i think that's the sort of question to answer before making that change. | 19:24 |
| clarkb | as mentioned before I think one of the ideas behind this behavior is it would allow a project like grenadne to drop jobs from its config without needing to reconfigure nova and cinder and so on stable branches to stop running that branch | 19:26 |
| clarkb | I suspect many of the occurrences are due to ^ but that is merely a hunch | 19:26 |
| clarkb | corvus: gmaan: maybe one way to keep the noise down is a simple catch all message in the buildset report? Something like "Not all jobs configured by the project pipeline were executed". This could apply to file matchers too | 19:39 |
| clarkb | jsut a bit of signal that hey something "optimized" the job list and you may have missed that | 19:39 |
| clarkb | though that might end up on like 90% of reports | 19:39 |
| gmaan | ++ yeah that will be enough to ask users to audit/cleanup | 19:40 |
| corvus | and maybe prompt them to do a debug run | 19:45 |
| opendevreview | Ghanshyam proposed openstack/tempest master: End support for stable/2024.1 https://review.opendev.org/c/openstack/tempest/+/966346 | 20:11 |
| opendevreview | Ghanshyam proposed openstack/tempest master: Revert "Use stable branch upper-constraints till 2025.1" https://review.opendev.org/c/openstack/tempest/+/966308 | 20:12 |
| opendevreview | Ghanshyam proposed openstack/tempest master: Pin stable constraints for unmaintained/2024.1 https://review.opendev.org/c/openstack/tempest/+/966348 | 20:19 |
| opendevreview | Ghanshyam proposed openstack/tempest master: Re-add the ironic job in gate https://review.opendev.org/c/openstack/tempest/+/966339 | 20:42 |
| opendevreview | Ghanshyam proposed openstack/tempest master: Pin stable/2025.2 constraint in tox to release 46.0.0 https://review.opendev.org/c/openstack/tempest/+/966350 | 20:46 |
| opendevreview | Ghanshyam proposed openstack/tempest master: Switch back the tox constraint to master https://review.opendev.org/c/openstack/tempest/+/966352 | 20:48 |
| sean-k-mooney | gmaan: im not sure why https://review.opendev.org/c/openstack/tempest/+/966346/1 is before the revert | 21:09 |
| sean-k-mooney | but we will need some approvers to review for the revet to proceed | 21:10 |
| sean-k-mooney | im finsihing up for today so ill check back tommorow | 21:10 |
| gmaan | sean-k-mooney: beacuse 2024.1 job on tempest master started failing | 21:10 |
| gmaan | I need to place them in order | 21:10 |
| sean-k-mooney | oh that makes sense | 21:11 |
| gmaan | sean-k-mooney: I am on it today, and also will propose required change in tempest plugins side also | 21:11 |
| gmaan | you can check tomorrow morning for watcher-tempest-plugin, at the end gate should be green | 21:11 |
| sean-k-mooney | gmaan: awsome thanks, dont burn out on this. | 21:12 |
| gmaan | it usually comes up in tempest and plugins when some old py is dropped and some branch goes to unmaintained | 21:12 |
| sean-k-mooney | as i said eiarlier its not blocking anything critical | 21:12 |
| sean-k-mooney | but it was blocking use droping the 2024.1 job | 21:12 |
| gmaan | i just started so thought of finishing for tempest and all plugins | 21:12 |
| gmaan | yeah | 21:13 |
Generated by irclog2html.py 4.0.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!