18:00:57 <gmaan> #startmeeting qa 18:00:57 <opendevmeet> Meeting started Wed Oct 15 18:00:57 2025 UTC and is due to finish in 60 minutes. The chair is gmaan. Information about MeetBot at http://wiki.debian.org/MeetBot. 18:00:57 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 18:00:57 <opendevmeet> The meeting name has been set to 'qa' 18:01:11 <gmaan> today agenda 18:01:13 <gmaan> #link https://wiki.openstack.org/wiki/Meetings/QATeamMeeting#Agenda_for_next_Office_hours 18:02:11 <gmaan> I am back from pto, will run the agenda and periodic CI check quickly 18:02:15 <gmaan> #topic Announcement and Action Item (Optional) 18:02:26 <gmaan> gmaan to plan for PTG topic call and register for PTG accordingly 18:02:56 <gmaan> I sent email about topic to discuss in PTG and it seem none in the etherpad 18:02:57 <gmaan> #link https://etherpad.opendev.org/p/qa-2026.1-ptg 18:03:43 <gmaan> so i did not register for PTG. there will be no QA specific slot but i will be looking on other project or tc etherpad if anything related to QA. 18:04:01 <gmaan> #topic Gate Status / Fix Checks 18:04:14 <gmaan> #link https://review.opendev.org/q/label:Review-Priority%253D%252B2+status:open+(project:openstack/tempest+OR+project:openstack/patrole+OR+project:openstack/devstack+OR+project:openstack/grenade) 18:04:40 <gmaan> it seems two new review, checking 18:07:44 <gmaan> reviewed 18:07:53 <gmaan> #topic Changes with Review-Priority == +1 18:09:52 <gmaan> multiattach test fix is +w 18:10:06 <gmaan> security compliance is hanging there so long, I will re-review it after meeting 18:10:55 <gmaan> image cache operations is new one but not sure if that is priority one. anyways I will try to check this week 18:11:20 <gmaan> #topic Periodic jobs Status Checks 18:11:25 <gmaan> Periodic Master jobs: 18:11:36 <gmaan> #link https://zuul.openstack.org/builds?project=openstack%2Ftempest&project=openstack%2Fdevstack&pipeline=periodic 18:12:00 <gmaan> propose-updates job still failing, I think frickler fix was merged ? 18:12:48 <gmaan> #link https://review.opendev.org/c/openstack/project-config/+/959382 18:14:45 <gmaan> it is running on ubuntu-noble so original error is fixed 18:14:47 <gmaan> #link https://zuul.openstack.org/build/4b4aa9d6c3514283bf2fedcc2ad8ab08/log/job-output.txt#35 18:15:07 <gmaan> but other error from devstack script ModuleNotFoundError: No module named 'requests' 18:15:25 <gmaan> #link https://zuul.openstack.org/build/4b4aa9d6c3514283bf2fedcc2ad8ab08/log/job-output.txt#803 18:15:31 <gmaan> will check after meeting 18:15:44 <gmaan> Periodic Stable jobs: 18:15:59 <gmaan> #link https://zuul.openstack.org/builds?job_name=tempest-full-2023-2&job_name=tempest-full-2024-1&job_name=tempest-full-2024-2&job_name=tempest-full-2025-1&pipeline=periodic-stable&skip=0 18:16:09 <gmaan> #link https://zuul.openstack.org/builds?job_name=tempest-slow-2024-2&job_name=tempest-slow-2024-1&job_name=tempest-slow-2025-1&skip=0 18:16:22 <gmaan> all ok 18:16:25 <gmaan> #link https://zuul.openstack.org/builds?job_name=tempest-multinode-2025-1&job_name=tempest-multinode-2024-1&job_name=tempest-multinode-2024-2&skip=0 18:16:47 <gmaan> tempest-multinode-2024-2 is failing sometime but not 100%, will monitor it with more run 18:16:54 <gmaan> #link https://zuul.openstack.org/builds?job_name=tempest-full-2024-2-extra-tests&job_name=tempest-full-2024-1-extra-tests&job_name=tempest-full-2023-2-extra-tests&job_name=tempest-full-2025-1-extra-tests&skip=0 18:16:56 <gmaan> all ok 18:17:07 <gmaan> #topic Distros check: 18:17:13 <gmaan> Centos 18:17:22 <gmaan> #link https://zuul.openstack.org/builds?job_name=tempest-full-centos-9-stream&job_name=devstack-platform-centos-9-stream&job_name=devstack-platform-centos-10-stream&skip=0 18:17:24 <gmaan> all ok 18:17:33 <gmaan> Rocky Linux 18:17:47 <gmaan> #link https://zuul.openstack.org/builds?job_name=devstack-platform-rocky-blue-onyx&job_name=tempest-full-rocky&skip=0 18:18:13 <gmaan> sometime it is failing but passing in latest run 18:18:21 <gmaan> Ubuntu 18:18:30 <gmaan> #link https://zuul.openstack.org/builds?job_name=devstack-platform-ubuntu-noble-ovn-source&job_name=devstack-platform-ubuntu-noble-ovs&job_name=devstack-platform-ubuntu-jammy&skip=0 18:18:31 <gmaan> all ok 18:18:41 <gmaan> #topic Open Discussion 18:19:13 <gmaan> nothing in open discussion but i will wait for couple of min if anything else to be discussed 18:58:59 <opendevreview> Merged openstack/tempest master: Add multiattach skip check to VolumeRetypeMultiattachTest https://review.opendev.org/c/openstack/tempest/+/960562 20:00:38 <opendevreview> Fernando Ferraz proposed openstack/devstack-plugin-nfs master: zuul: run cinder-tempest-plugin tests too https://review.opendev.org/c/openstack/devstack-plugin-nfs/+/898965 20:01:02 <opendevreview> Fernando Ferraz proposed openstack/devstack-plugin-nfs master: Add job for testing Glance with Cinder as backend https://review.opendev.org/c/openstack/devstack-plugin-nfs/+/940482 20:04:45 <opendevreview> Merged openstack/tempest master: zuul: remove interop-tempest-consistency job https://review.opendev.org/c/openstack/tempest/+/962131 20:07:31 <opendevreview> Fernando Ferraz proposed openstack/devstack-plugin-nfs master: [DNM] Testing remove nconnect on Ubuntu Jammy https://review.opendev.org/c/openstack/devstack-plugin-nfs/+/964104 20:45:36 <opendevreview> Fernando Ferraz proposed openstack/devstack-plugin-nfs master: zuul: run cinder-tempest-plugin tests too https://review.opendev.org/c/openstack/devstack-plugin-nfs/+/898965 20:45:37 <opendevreview> Fernando Ferraz proposed openstack/devstack-plugin-nfs master: Add job for testing Glance with Cinder as backend https://review.opendev.org/c/openstack/devstack-plugin-nfs/+/940482 20:45:38 <opendevreview> Fernando Ferraz proposed openstack/devstack-plugin-nfs master: [DNM] Testing remove nconnect on Ubuntu Jammy https://review.opendev.org/c/openstack/devstack-plugin-nfs/+/964104 21:05:14 <opendevreview> Fernando Ferraz proposed openstack/devstack-plugin-nfs master: Add job for testing Glance with Cinder as backend https://review.opendev.org/c/openstack/devstack-plugin-nfs/+/940482 21:05:15 <opendevreview> Fernando Ferraz proposed openstack/devstack-plugin-nfs master: [DNM] Testing remove nconnect on Ubuntu Jammy https://review.opendev.org/c/openstack/devstack-plugin-nfs/+/964104 21:07:18 <opendevreview> Fernando Ferraz proposed openstack/devstack-plugin-nfs master: [WIP] Add job for testing Glance with Cinder as backend https://review.opendev.org/c/openstack/devstack-plugin-nfs/+/940482 21:07:19 <opendevreview> Fernando Ferraz proposed openstack/devstack-plugin-nfs master: [DNM] Testing remove nconnect on Ubuntu Jammy https://review.opendev.org/c/openstack/devstack-plugin-nfs/+/964104 22:11:15 <sfernand> hey folks! cinder have this patch stale for a while, it setup os_brick to use the same lock path in case glance-cinder-nova are running on the same host 22:11:15 <sfernand> https://review.opendev.org/c/openstack/devstack/+/849329 22:11:15 <sfernand> any core could take a look? 22:12:57 <sfernand> It has some +1s already from some team members 22:15:48 <opendevreview> Merged openstack/tempest master: Add Tempest gate job for stable/2025.2 https://review.opendev.org/c/openstack/tempest/+/961908 23:13:13 <opendevreview> Fernando Ferraz proposed openstack/devstack-plugin-nfs master: zuul: run cinder-tempest-plugin tests too https://review.opendev.org/c/openstack/devstack-plugin-nfs/+/898965 06:35:59 <opendevreview> Yosi Ben Shimon proposed openstack/tempest master: Test coverage for Cinder upload volume back to glance https://review.opendev.org/c/openstack/tempest/+/964163 07:26:40 <opendevreview> Milana Levy proposed openstack/devstack master: WIP - Change default swift test account password https://review.opendev.org/c/openstack/devstack/+/962897 07:41:18 <opendevreview> Milana Levy proposed openstack/tempest master: Adding a password regex test for PCI DSS https://review.opendev.org/c/openstack/tempest/+/954029 08:33:12 <opendevreview> Milana Levy proposed openstack/tempest master: Adding a password regex test for PCI DSS https://review.opendev.org/c/openstack/tempest/+/954029 10:31:26 <opendevreview> Milana Levy proposed openstack/tempest master: Adding a password regex test for PCI DSS https://review.opendev.org/c/openstack/tempest/+/954029 10:35:02 <opendevreview> Milana Levy proposed openstack/tempest master: Adding a password regex test for PCI DSS https://review.opendev.org/c/openstack/tempest/+/954029 10:37:09 <opendevreview> Milana Levy proposed openstack/tempest master: Adding a password regex test for PCI DSS https://review.opendev.org/c/openstack/tempest/+/954029 12:33:57 <gibi> hi folks I see a bungch of wsgi crashes in various devstack jobs. e.g. https://zuul.opendev.org/t/openstack/build/46b6209e205047b8ad212623e0a9f62f/log/controller/logs/screen-keystone.txt Is it a known / tracked issue? 12:38:17 <gibi> did we changed something recently that changed the python version in these jobs from 3.12 to 3.11? 12:42:25 <opendevreview> Milana Levy proposed openstack/tempest master: Adding a password regex test for PCI DSS https://review.opendev.org/c/openstack/tempest/+/954029 12:42:37 <gibi> actually it seems on master it is 3.11 but on stable it is 3.12. Why? 13:23:34 <sfernand> hi folks! do some core have a minute to review https://review.opendev.org/c/openstack/devstack/+/849329 13:23:34 <sfernand> we already have a +2 13:25:26 <sfernand> this is blocking a job we are trying to setup for cinder 13:31:39 <opendevreview> Yosi Ben Shimon proposed openstack/tempest master: Test coverage for Cinder upload volume back to glance https://review.opendev.org/c/openstack/tempest/+/964163 13:38:53 <gibi> sfernand: I reviewed it now but I only have +1 rights in devstack 14:11:27 <opendevreview> Milana Levy proposed openstack/tempest master: Adding a password regex test for PCI DSS https://review.opendev.org/c/openstack/tempest/+/954029 14:17:12 <opendevreview> Milana Levy proposed openstack/tempest master: Adding a password regex test for PCI DSS https://review.opendev.org/c/openstack/tempest/+/954029 15:20:38 <gibi> hi folks could somebody with devstack +2 rights look at https://review.opendev.org/c/openstack/devstack/+/939825 it already has a +2 from gmaan 15:32:04 <gmaan> gibi: just checked the python versions in nova-ceph-multistore job. it changes because we moved this job from noble (default py3.12) to debian Bookworm (py3.11) on master 15:32:42 <gmaan> maybe we can try it on Trixie if that is ready/stable in devstack 15:39:07 <gibi> gmaan: I'm not sure if it is relatet to the recent wsgi crashes 15:39:23 <gibi> just noticed that we run with 3.11 and got surprised 15:39:58 <gibi> did we moved the majority of the master testing to debian? 15:40:17 <gibi> I need to check if we see the failure with the ubuntu versions of python and uwsgi as well in the gate 15:40:18 <gmaan> no. it was just ceph job due to some failure 15:40:37 <gibi> ohh cool then I think I can investigate if it happens on ubuntu or not 15:40:40 <gibi> I will report back 15:41:29 <gmaan> gibi: this one, i think you will remember the ceph issue on ubuntu https://review.opendev.org/q/536ebea559673d694c8f8deeafe1362e2c41c021 15:42:57 <gibi> ahh I never had that time to go back to that issue even though I have an open bug on ceph about it. :/ ETOOMAN 15:43:03 <gibi> ETOOMANY 16:51:32 <gibi> gmaan: so far I only see the wsgi crash with the debian job like in nova-ceph-multistore and cinder-plugin-ceph-tempest 16:51:54 <gibi> this is an example https://zuul.opendev.org/t/openstack/build/c701ae0db630488c8f3abdafb2007ec0/log/controller/logs/screen-keystone.txt 16:55:17 <gmaan> i see, it seems not so bad but worth to check https://zuul.opendev.org/t/openstack/builds?job_name=nova-ceph-multistore&job_name=cinder-plugin-ceph-tempest&skip=0 16:56:48 <gmaan> gouthamr are you aware of any known issue related to wsgi crash on debian jobs ^^ as you might know ceph jobs are running on debian on master (not sure multinode ce[h job also but single node is on debian) 17:38:41 <gouthamr> nope, not aware gmaan 17:57:38 <gouthamr> i don't look at this job usually, but, i just tried finding segfaults like this on opensearch 17:57:40 <gouthamr> https://opensearch.logs.openstack.org/_dashboards/app/data-explorer/discover#?_a=(discover:(columns:!(hosts_region,message,change_url),isDirty:!t,sort:!()),metadata:(indexPattern:'94869730-aea8-11ec-9e6a-83741af3fdcd',view:discover))&_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:now-1M,to:now))&_q=(filters:!(('$state':(store:appState),meta:(alias:!n,disabled:!f,index:'94869730-aea8-11ec-9e6a-83741af3fdcd',key:filename 17:57:40 <gouthamr> .keyword,negate:!f,params:(query:screen-keystone.txt),type:phrase),query:(match_phrase:(filename.keyword:screen-keystone.txt)))),query:(language:kuery,query:'message:%22got%20Segmentation%20Fault%22%20')) 17:58:30 <gouthamr> there are 166 hits from the past month where uwsgi crashed and keystone went unresponsive, not all of those failures are on debian 18:00:22 <gouthamr> actually throw in "build name" to the o/p filters and you'll see a lot of different kind of jobs: grenade, horizon, manila-ui integration, nova-ovs-hybrid-plug etc 18:03:27 <gouthamr> this has happened with "nova-ceph-multistore" 15 times in the past couple of days though (on three changes) 20:35:13 <opendevreview> Merged openstack/devstack master: Add support for shared os_brick file lock path https://review.opendev.org/c/openstack/devstack/+/849329 21:38:32 <sfernand> https://review.opendev.org/c/openstack/devstack/+/849329 has been merged, thanks a lot guys for the reviews! 21:46:11 <opendevreview> Fernando Ferraz proposed openstack/devstack-plugin-nfs master: zuul: run cinder-tempest-plugin tests too https://review.opendev.org/c/openstack/devstack-plugin-nfs/+/898965 21:47:45 <opendevreview> Fernando Ferraz proposed openstack/devstack-plugin-nfs master: [WIP] Add job for testing Glance with Cinder as backend https://review.opendev.org/c/openstack/devstack-plugin-nfs/+/940482 21:48:57 <opendevreview> Fernando Ferraz proposed openstack/devstack-plugin-nfs master: [WIP] Add job for testing Glance with Cinder as backend https://review.opendev.org/c/openstack/devstack-plugin-nfs/+/940482 23:46:17 <opendevreview> Fernando Ferraz proposed openstack/devstack-plugin-nfs master: [DNM] Default to nfs vers=4.1 https://review.opendev.org/c/openstack/devstack-plugin-nfs/+/964265 23:47:11 <opendevreview> Fernando Ferraz proposed openstack/devstack-plugin-nfs master: [DNM] Default to nfs vers=4.2 https://review.opendev.org/c/openstack/devstack-plugin-nfs/+/964266 00:23:06 <opendevreview> Fernando Ferraz proposed openstack/devstack-plugin-nfs master: [DNM] remove nconnect on Ubuntu Jammy https://review.opendev.org/c/openstack/devstack-plugin-nfs/+/964104 01:30:01 <opendevreview> Fernando Ferraz proposed openstack/devstack-plugin-nfs master: [DNM] default to nfs vers=4.0 https://review.opendev.org/c/openstack/devstack-plugin-nfs/+/964269 02:00:48 <opendevreview> Fernando Ferraz proposed openstack/devstack-plugin-nfs master: [DNM] Default to nfs vers=4.2 https://review.opendev.org/c/openstack/devstack-plugin-nfs/+/964266 02:01:53 <opendevreview> Fernando Ferraz proposed openstack/devstack-plugin-nfs master: [WIP] Add job for testing Glance with Cinder as backend https://review.opendev.org/c/openstack/devstack-plugin-nfs/+/940482 02:58:35 <opendevreview> Fernando Ferraz proposed openstack/devstack-plugin-nfs master: zuul: run cinder-tempest-plugin tests too https://review.opendev.org/c/openstack/devstack-plugin-nfs/+/898965 03:41:40 <opendevreview> Fernando Ferraz proposed openstack/devstack-plugin-nfs master: zuul: run cinder-tempest-plugin tests too https://review.opendev.org/c/openstack/devstack-plugin-nfs/+/898965 03:41:41 <opendevreview> Fernando Ferraz proposed openstack/devstack-plugin-nfs master: [WIP] Add job for testing Glance with Cinder as backend https://review.opendev.org/c/openstack/devstack-plugin-nfs/+/940482 03:43:05 <opendevreview> Fernando Ferraz proposed openstack/devstack-plugin-nfs master: [WIP] zuul: add job for testing Glance with Cinder as backend https://review.opendev.org/c/openstack/devstack-plugin-nfs/+/940482 03:43:51 <opendevreview> Fernando Ferraz proposed openstack/devstack-plugin-nfs master: [WIP] zuul: add job for testing Glance with Cinder as backend https://review.opendev.org/c/openstack/devstack-plugin-nfs/+/940482 07:18:24 <opendevreview> Fernando Ferraz proposed openstack/devstack-plugin-nfs master: [WIP] zuul: add job for testing Glance with Cinder as backend https://review.opendev.org/c/openstack/devstack-plugin-nfs/+/940482 07:20:18 <opendevreview> Fernando Ferraz proposed openstack/devstack-plugin-nfs master: [WIP] zuul: add job for testing Glance with Cinder as backend https://review.opendev.org/c/openstack/devstack-plugin-nfs/+/940482 07:21:00 <opendevreview> Fernando Ferraz proposed openstack/devstack-plugin-nfs master: [DNM] remove nconnect on Ubuntu Jammy https://review.opendev.org/c/openstack/devstack-plugin-nfs/+/964104 07:21:12 <opendevreview> Fernando Ferraz proposed openstack/devstack-plugin-nfs master: [WIP] zuul: add job for testing Glance with Cinder as backend https://review.opendev.org/c/openstack/devstack-plugin-nfs/+/940482 07:28:09 <opendevreview> Fernando Ferraz proposed openstack/devstack-plugin-nfs master: [DNM] clear all changes https://review.opendev.org/c/openstack/devstack-plugin-nfs/+/964279 07:31:38 <opendevreview> Fernando Ferraz proposed openstack/devstack-plugin-nfs master: [WIP] zuul: add job for testing Glance with Cinder as backend https://review.opendev.org/c/openstack/devstack-plugin-nfs/+/940482 07:40:13 <opendevreview> Fernando Ferraz proposed openstack/devstack-plugin-nfs master: [DNM] Default to nfs vers=4.2 https://review.opendev.org/c/openstack/devstack-plugin-nfs/+/964266 08:02:41 <opendevreview> Fernando Ferraz proposed openstack/devstack stable/2025.2: Add support for shared os_brick file lock path https://review.opendev.org/c/openstack/devstack/+/964282 08:13:02 <frickler> #endmeeting