| *** mhen_ is now known as mhen | 02:11 | |
| *** ykarel_ is now known as ykarel|off | 03:43 | |
| opendevreview | Merged openstack/tempest master: Pin stable/2026.2 constraint in tox to release new tempest https://review.opendev.org/c/openstack/tempest/+/1014177 | 04:37 |
|---|---|---|
| opendevreview | Merged openstack/devstack-plugin-jaeger master: Restart keystone after enabling oslo.middleware tracing https://review.opendev.org/c/openstack/devstack-plugin-jaeger/+/1012820 | 07:11 |
| gthiemon1e | gmaan: it looks like "Pin stable/2026.2 constraint in tox to release new tempest" breaks the octavia-tempest-plugin jobs | 09:40 |
| gthiemon1e | https://f1404d13ab71c249f917-5e48a2c01e64c14c26ffbc6ffdb010cb.ssl.cf2.rackcdn.com/openstack/8ec6964435104658b746a6a1e8a51a50/job-output.txt | 09:41 |
| gthiemon1e | "Run Tempest" task recreate the venv, and that removes the previously installed plugins | 09:41 |
| gthiemon1e | hmm i assume that https://review.opendev.org/c/openstack/tempest/+/1014178 is going to fix it | 09:44 |
| tkajinam | gthiemon1e, yeah that's intentional but temporal breakage during the release timing | 12:19 |
| frickler | and every cycle that last change seems to have extra trouble passing CI | 13:25 |
| gmaan | gthiemon1e: yes, that is the one will make it green https://review.opendev.org/c/openstack/tempest/+/1014178 | 13:40 |
| gmaan | yeah, i try to approve/merge them together but many times last one stuck | 13:40 |
| gmaan | this time passed the check pipeline and now in gate | 14:08 |
| opendevreview | Eduardo Olivares proposed openstack/tempest master: Add a split-parallel-serial tox environment https://review.opendev.org/c/openstack/tempest/+/1014546 | 15:24 |
| eolivare | gmaan, o/ please let me know if you think this patch ^^^ makes sense - My intention is to use it for this: https://review.opendev.org/c/x/whitebox-neutron-tempest-plugin/+/1014519/2/zuul.d/master_jobs.yaml | 15:29 |
| gmaan | eolivare: ac, adding in my list. most probably will check today | 15:30 |
| eolivare | ty! | 15:30 |
| opendevreview | Merged openstack/tempest master: Switch back the tox constraint to master https://review.opendev.org/c/openstack/tempest/+/1014178 | 16:21 |
| sean-k-mooney | gmaan: it looks liek something has broken the tempest jobs taht run test only form plugins | 17:17 |
| sean-k-mooney | wathcer and cidner jobs are failing with `The specified regex doesn't match with anythingERROR: InvocationError for command /opt/stack/tempest/.tox/tempest/bin/tempest run --regex cinder_tempest_plugin --concurrency=6 ` | 17:17 |
| sean-k-mooney | i woudl assuem other gates are also broken in the same way like cybrog neutron or anything that has a job that only runs test form a plugin rather then core tempest. | 17:21 |
| sean-k-mooney | im not sure what has chaged ot cause this but it may be related to tox and how devstac/plugins are installer ro simialr | 17:22 |
| abhishekk | glance is affected as well, 2026-11-06 15:14:34.016607 | controller | The specified regex doesn't match with anythingERROR: InvocationError for command /opt/stack/tempest/.tox/tempest/bin/tempest run --regex glance_tempest_plugin.tests.rbac --concurrency=2 (exited with code 1) | 17:22 |
| sean-k-mooney | https://zuul.opendev.org/t/openstack/build/d9bada6708eb4111bd331e3603a35486/logs and https://zuul.opendev.org/t/openstack/build/584794c4fb124985a023e26616c1bf7f are two exampels | 17:22 |
| sean-k-mooney | i have not debuged it yet but my guess is tox may be recreting the env or the plugin tests are not discoverable for some other reason | 17:23 |
| sean-k-mooney | devstack and tempest currently are slightly abusing how tox expect virutal envs to be used | 17:24 |
| sean-k-mooney | specificly related to how the plugins are installed into the venv during a ci job | 17:24 |
| sean-k-mooney | stephenfin tried to unwind some of that in the past | 17:25 |
| sean-k-mooney | but over the last several year tox has been gettign stricter about what is in the venv and detecitng if it changes since it was created | 17:25 |
| sean-k-mooney | today we create a vend then we out of band install the plugins then we use a diffent tox env to run the tests | 17:26 |
| sean-k-mooney | im wondering if we need to start explcity passing the flag stephen added to tell tox to jsut skip validatign the venv and use it directly | 17:26 |
| sean-k-mooney | i guess it thinks it is thre https://zuul.opendev.org/t/openstack/build/584794c4fb124985a023e26616c1bf7f/log/controller/logs/devstacklog.txt#29731-29775 | 17:28 |
| sean-k-mooney | but im not seeing any cidner rrelated config option registered | 17:30 |
| sean-k-mooney | https://zuul.opendev.org/t/openstack/build/584794c4fb124985a023e26616c1bf7f/log/controller/logs/tempest_log.txt | 17:30 |
| sean-k-mooney | well outside the standard oen but i guess the plugin migyut not ahve any | 17:31 |
| sean-k-mooney | it should have some https://github.com/openstack/cinder-tempest-plugin/blob/master/cinder_tempest_plugin/config.py | 17:32 |
| sean-k-mooney | i.e. consistency_group | 17:33 |
| sean-k-mooney | so when tempet was invoked it didnt registry them menaing it was not found | 17:33 |
| sean-k-mooney | https://zuul.opendev.org/t/openstack/build/584794c4fb124985a023e26616c1bf7f/log/job-output.txt#31955-31977 | 17:36 |
| sean-k-mooney | 2026-11-06 16:56:51.075830 | TASK [run-tempest : Run Tempest] | 17:36 |
| sean-k-mooney | 2026-11-06 16:56:52.167442 | controller | all recreate: /opt/stack/tempest/.tox/tempest | 17:36 |
| sean-k-mooney | 2026-11-06 16:56:53.478660 | controller | all installdeps: -chttps://releases.openstack.org/constraints/upper/2026.2, -r/opt/stack/tempest/requirements.txt | 17:36 |
| gmaan | sean-k-mooney: it is fixed nw, you can recheck | 17:36 |
| sean-k-mooney | 2026-11-06 16:57:19.347415 | controller | all develop-inst: /opt/stack/tempest | 17:36 |
| sean-k-mooney | 2026-11-06 16:57:28.330796 | controller | all installed: | 17:36 |
| sean-k-mooney | attrs==26.1.0,autopage==0.6.0,bcrypt==5.0.0,certifi==2026.10.23,cffi==2.1.1,charset-normalizer==3.5.0,cliff==4.16.0,cmd2==4.2.0,cryptography==43.0.3,debtcollector==3.1.0,defusedxml==0.7.1,fasteners==0.20,fixtures==4.3.2,idna==3.18,invoke==3.0.3,iso8601==2.1.0,jsonschema==4.26.0,jsonschema-specifications==2025.9.1,markdown-it-py==4.2.0,mdurl==0.1.2,msgpack==1.2.1,netaddr==1.3.0,o | 17:36 |
| sean-k-mooney | slo.concurrency==7.6.1,oslo.config==10.7.0,oslo.context==6.5.0,oslo.i18n==6.9.0,oslo.log==8.3.2,oslo.serialization==5.11.0,oslo.utils==10.2.0,packaging==26.3,paramiko==4.0.0,pbr==7.0.3,prettytable==3.18.0,prompt_toolkit==3.0.53,psutil==7.2.2,pycparser==3.0,Pygments==2.20.0,PyNaCl==1.6.2,pyparsing==3.3.2,pyperclip==1.11.0,python-dateutil==2.9.0.post0,python-subunit==1.4.6,PyYAML= | 17:36 |
| sean-k-mooney | =6.0.3,referencing==0.37.0,requests==2.34.2,rfc3986==2.0.0,rich==15.0.0,rich-argparse==1.8.0,rpds-py==2026.6.3,setuptools==84.2.1,six==1.17.0,stestr==4.2.1,stevedore==5.9.1,-e | 17:36 |
| sean-k-mooney | git+file:///dev/null@2c70bae1c7de14544b99bae0c331f4ac76e14d83#egg=tempest,testscenarios==0.6.2,testtools==2.9.1,tomlkit==0.15.1,typing_extensions==4.16.0,urllib3==2.7.0,voluptuous==0.16.0,wcwidth==0.8.2,wrapt==2.3.0 | 17:36 |
| sean-k-mooney | 2026-11-06 16:57:28.331556 | controller | all run-test-pre: PYTHONHASHSEED='3077721390' | 17:36 |
| sean-k-mooney | 2026-11-06 16:57:28.331728 | controller | all run-test: commands[0] | find . -type f -name '*.pyc' -delete | 17:36 |
| sean-k-mooney | 2026-11-06 16:57:28.457021 | controller | all run-test: commands[1] | tempest run --regex cinder_tempest_plugin --concurrency=6 | 17:36 |
| sean-k-mooney | 2026-11-06 16:57:34.390297 | controller | The specified regex doesn't match with anythingERROR: InvocationError for command /opt/stack/tempest/.tox/tempest/bin/tempest run --regex cinder_tempest_plugin --concurrency=6 (exited with code 1) | 17:37 |
| sean-k-mooney | 2026-11-06 16:57:34.390731 | controller | ___________________________________ summary ____________________________________ | 17:37 |
| sean-k-mooney | 2026-11-06 16:57:34.390791 | controller | ERROR: all: commands failed | 17:37 |
| sean-k-mooney | 2026-11-06 16:57:34.804380 | controller | ERROR | 17:37 |
| sean-k-mooney | 2026-11-06 16:57:34.804748 | controller | { | 17:37 |
| sean-k-mooney | 2026-11-06 16:57:34.804903 | controller | "delta": "0:00:42.717778", | 17:37 |
| sean-k-mooney | 2026-11-06 16:57:34.805013 | controller | "end": "2026-11-06 16:57:34.422609", | 17:37 |
| sean-k-mooney | 2026-11-06 16:57:34.805113 | controller | "msg": "non-zero return code", | 17:37 |
| sean-k-mooney | 2026-11-06 16:57:34.805223 | controller | "rc": 1, | 17:37 |
| sean-k-mooney | 2026-11-06 16:57:34.805331 | controller | "start": "2026-11-06 16:56:51.704831", | 17:37 |
| sean-k-mooney | 2026-11-06 16:57:34.805455 | controller | "warnings": [ | 17:37 |
| sean-k-mooney | 2026-11-06 16:57:34.805572 | controller | "Module remote_tmp /home/tempest/.ansible/tmp did not exist and was created with a mode of 0700, this may cause issues when running as another user. To avoid this, create the remote_tmp dir with the correct permissions manually" | 17:37 |
| sean-k-mooney | 2026-11-06 16:57:34.805665 | controller | ] | 17:37 |
| sean-k-mooney | 2026-11-06 16:57:34.805759 | controller | } | 17:37 |
| sean-k-mooney | ack | 17:37 |
| sean-k-mooney | it looks like the venv was beign recreated when we actully ran the tests ^ | 17:37 |
| sean-k-mooney | i proably should have pastbin'd that | 17:37 |
| sean-k-mooney | was that hte problem or soemthing else | 17:37 |
| gmaan | sean-k-mooney: it was temp outage for releasing tempest and green now. recheck will pass the tests | 17:39 |
| gmaan | i just sent it on ML also in case anyone else wondering about failure | 17:39 |
| sean-k-mooney | odd im not sure why a release woudl cause this type of failure | 17:40 |
| sean-k-mooney | was it form changign the uc file | 17:40 |
| gmaan | i need to pin the constraints in tox for end of support for unmaintained/2025.1 and that end up recreate the venv where plugins were installed. | 17:41 |
| sean-k-mooney | ack | 17:41 |
| sean-k-mooney | ok that makes sense as that is one of the things tox tracks | 17:41 |
| sean-k-mooney | if the constratis change it recretes the env | 17:41 |
| gmaan | i try to do these on friday and merge the pin and revert together but you know gate it merge the pin normally but stuck on revert those constraints :) | 17:42 |
| sean-k-mooney | so if devstack and tempest disagreed on what it shoudl be it woudl do that | 17:42 |
| gmaan | yeah | 17:42 |
| sean-k-mooney | no worries | 17:42 |
| sean-k-mooney | i just started lookign at it because it was mentioned on the wtacher channel tanks for updatdating there as well | 17:43 |
| gmaan | np! | 17:47 |
| frickler | thinking about it, this would likely be a good application for the submitWholeTopic/circular-dependency feature. currently afaict it is only enabled for the zuul repo itself, but maybe we could do the same for tempest (possibly only temporarily) next time this comes up | 18:27 |
| clarkb | I can't remember what sort of granularity you can enable it on. It might be tenant wide? | 18:28 |
| clarkb | Worth looking into if you want to pursue this further | 18:28 |
| frickler | looks like it is per queue, so not the whole tenant, but also not per repo. I don't think we need to deep dive into this right now, but try to keep this in mind for the next release (likely next cycle, unless gmaan has other plans?) | 18:33 |
| gmaan | let me think over it next week if we can fix it at tempest side if not then having such grouping merge can be helpful | 18:35 |
| gmaan | even I am thinking from nova side, many time we hold the base change to approve until last change is good (especially for microversion changes which has base changes of object/RPC etc first and then API microversion) to go and there also it can be helpful | 18:36 |
| opendevreview | Brian Rosmaita proposed openstack/devstack-plugin-ceph master: Add Ceph post-deployment health checks https://review.opendev.org/c/openstack/devstack-plugin-ceph/+/1009412 | 22:37 |
Generated by irclog2html.py 4.1.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!