Tuesday, 2022-05-24

opendevreviewGhanshyam proposed openstack/tempest master: Use yoga stable constraint in tox to release 31.0.0  https://review.opendev.org/c/openstack/tempest/+/83777700:14
opendevreviewGhanshyam proposed openstack/tempest master: Switch back the tox constraint to master  https://review.opendev.org/c/openstack/tempest/+/83777900:22
opendevreviewGhanshyam proposed openstack/tempest master: Unblock gate to temporary remove the py3.6|7 job  https://review.opendev.org/c/openstack/tempest/+/84282101:22
gmannkopecmartin: ^^ tempest plugin sanity job also got broken before py36|7 things merged. I need to make it non-voting to merge py36|7 patch.01:24
opendevreviewGhanshyam proposed openstack/tempest master: Make tempest plugin sanity job voting again  https://review.opendev.org/c/openstack/tempest/+/84304101:26
gmannkopecmartin: ^^ making it voting with depends-on murano tempest plugin fix01:26
opendevreviewGhanshyam proposed openstack/tempest master: DNM: tetsing tempest master with stable/ussuri constraints  https://review.opendev.org/c/openstack/tempest/+/84304401:39
opendevreviewGhanshyam proposed openstack/tempest master: Use UPPER_CONSTRAINTS_FILE for stable/ussuri testing  https://review.opendev.org/c/openstack/tempest/+/84304501:47
opendevreviewMerged openstack/devstack master: Drop openEuler support  https://review.opendev.org/c/openstack/devstack/+/84253411:34
gryfhello, quick question, what version of ubuntu should be used for gate jobs to avoid some weird issues?13:04
fricklergryf: there are always weird issues. currently Ubuntu 20.04 is still the most tested version, we are working on 22.04 but not 100% there yet13:07
gryffrickler, got it, thanks!13:08
dansmithjohnsom: "luckily" my patch hit the same brick package issue and did the right thing: https://zuul.opendev.org/t/openstack/build/f8f92563509c4245887b9f1ebe53778f/log/job-output.txt#891113:30
opendevreviewyatin proposed openstack/devstack-gate master: Use stable constraints for Tempest venv for stable/train testing  https://review.opendev.org/c/openstack/devstack-gate/+/84314814:12
opendevreviewArtom Lifshitz proposed openstack/tempest master: Make test_server_actions.resource_setup() wait for SSHABLE  https://review.opendev.org/c/openstack/tempest/+/84315514:50
dansmithclarkb: weirdly, I'm seeing "unable to load plugin dbcounter" locally,14:54
dansmithon jammy, even though it works in the job14:54
dansmithI can import it in the python shell, but the entry point seems to be missing or something14:54
clarkbdansmith: with my py_modules changes?14:56
clarkbthat was exactly what I saw without the py_modules update fwiw14:56
clarkbif you look in the python package install area you probably have a dbcounter-stuff.egg-info or whatever the dir is but no dbcounter.py there. There should be a dbcounter.py at the same dir level as the egg info14:57
dansmithyep and it's there14:57
clarkbwithout py_modules dbcounter.py was not installed. With it it was14:57
dansmithand I can import it,14:57
clarkboh so this is a separate issue14:57
dansmithand entry_points.txt is correct,14:58
dansmithbut sqla fails to find it 14:58
dansmith2022-05-24 14:10:05.842 | ERROR keystone sqlalchemy.exc.NoSuchModuleError: Can't load plugin: sqlalchemy.plugins:dbcounter14:58
dansmithbut it's clearly working on jobs14:58
clarkbsounds like similar symptoms but different mechanism for failure. What I observed prior to py_moduels was very similar but was due to dbcounter.py not being installed by pip at all14:58
dansmithso bizarre that it works upstream though.. this is a totes fresh jammy vm14:59
kopecmartin#startmeeting qa15:00
opendevmeetMeeting started Tue May 24 15:00:19 2022 UTC and is due to finish in 60 minutes.  The chair is kopecmartin. Information about MeetBot at http://wiki.debian.org/MeetBot.15:00
opendevmeetUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.15:00
opendevmeetThe meeting name has been set to 'qa'15:00
kopecmartin#link https://wiki.openstack.org/wiki/Meetings/QATeamMeeting#Weekly_QA_Team_meeting15:00
kopecmartinagenda ^^15:00
opendevreviewJames Parker proposed openstack/whitebox-tempest-plugin master: Take into account shared CPUs per NUMA  https://review.opendev.org/c/openstack/whitebox-tempest-plugin/+/84315715:00
kopecmartin#topic Announcement and Action Item (Optional)15:02
kopecmartinwe have just dropped openeuler support from devstack15:03
kopecmartinnothing else from my side here15:04
frickleryes, sadly no continued maintainance once the support was added15:04
frickleropenstacksdk 0.99.0 was released15:04
fricklermainly as a 1.0.0pre1 release. that may trigger some issues15:05
kopecmartinthanks for the heads up15:05
kopecmartinmoving on15:06
kopecmartin#topic Zed Priority Items progress15:06
kopecmartin#link https://etherpad.opendev.org/p/qa-zed-priority15:06
kopecmartinany updates on this front?15:06
gmannnothing special 15:06
kopecmartinnothing special from my side either 15:07
kopecmartin#topic OpenStack Events Updates and Planning15:07
kopecmartinnothing here for now15:07
kopecmartin#topic Gate Status Checks15:07
kopecmartin#link https://review.opendev.org/q/label:Review-Priority%253D%252B2+status:open+(project:openstack/tempest+OR+project:openstack/patrole+OR+project:openstack/devstack+OR+project:openstack/grenade)15:08
kopecmartin2 reviews, both approved, waiting on gates 15:08
gmannyeah, py36|7 is all stuck15:09
gmannand not merged yet15:09
kopecmartinyup, mainly because some jobs are quite unstable lately, mainly the ceph one 15:10
kopecmartinit was a reason of many rechecks 15:10
gmannyeah, let's see15:11
kopecmartin#topic Periodic jobs Status Checks15:11
kopecmartin#link #link https://zuul.openstack.org/builds?job_name=tempest-full-yoga&job_name=tempest-full-xena&job_name=tempest-full-wallaby-py3&job_name=tempest-full-victoria-py3&job_name=tempest-full-ussuri-py3&pipeline=periodic-stable15:12
kopecmartinmaster 15:12
kopecmartin#link https://zuul.openstack.org/builds?project=openstack%2Ftempest&project=openstack%2Fdevstack&pipeline=periodic15:12
fricklerthe "error" thing was me15:12
fricklertriggered the wrong project15:12
fricklerbut then manual triggers are broken in zuul somehow15:13
kopecmartin:) np15:13
fricklereverything else still suspicously green15:13
kopecmartinapart from those 2 errors it's all green 15:13
kopecmartin#topic Distros check15:14
kopecmartin#link https://zuul.openstack.org/builds?job_name=tempest-full-centos-9-stream&job_name=tempest-full-py3-centos-8-stream&job_name=devstack-platform-centos-8-stream&job_name=devstack-platform-centos-9-stream&skip=015:14
kopecmartin#link https://zuul.openstack.org/builds?job_name=devstack-platform-fedora-latest&skip=015:14
kopecmartin#link https://zuul.openstack.org/builds?job_name=devstack-platform-debian-bullseye&skip=015:14
fricklermaybe we should also add a link for the jammy jobs?15:15
kopecmartinnot that bad, could be better though, especially in case of centos as it failed a few gate jobs 15:15
kopecmartingood point15:15
kopecmartinjammy jobs15:17
kopecmartin#link https://zuul.openstack.org/builds?job_name=devstack-platform-ubuntu-jammy&job_name=devstack-platform-ubuntu-jammy-ovn-source&job_name=devstack-platform-ubuntu-jammy-ovs&skip=015:17
fricklerdebian looks a bit flaky, I'll check those later15:18
kopecmartinack, thanks15:19
kopecmartin#topic Sub Teams highlights15:19
fricklerah, those are mainly the ansible v5 checks15:19
kopecmartinChanges with Review-Priority == +1 15:19
kopecmartin#link https://review.opendev.org/q/label:Review-Priority%253D%252B1+status:open+(project:openstack/tempest+OR+project:openstack/patrole+OR+project:openstack/devstack+OR+project:openstack/grenade)15:19
kopecmartinregarding the backports in devstack, do we need to merge them in order? from the latest release back? or can i approve all at once?15:20
gmanneither is fine in devstack bakport15:21
gmannapprove once also work as we do many times15:21
kopecmartinack, thanks15:22
kopecmartin#topic Open Discussion15:23
kopecmartinanything for the open discussion?15:23
fricklerI have one slightly political topic15:23
gmannyeah, I have added two items15:23
gmannfrickler: go ahead first15:23
fricklerI would like to gather some early planning feedback on the next PTG15:23
fricklerthe foundation seems to think that it is a good idea to have people travelling around the world again15:23
fricklerto which I 100% do not agree for multiple reasons15:23
gmannyeah, its physical event 15:24
fricklerso the question is: can we as a team agree to continue to hold a virtual PTG?15:24
fricklermaybe outside of the global PTG if some people are still travelling there?15:24
gmanngood point. even in TC I pointed to ask for community feedback first but that did not happen15:24
gmannfrickler: once challenge in that is we need to make it either virtual only or physical. hybrid can be more complex 15:25
fricklerI for sure won't travel and I also consider boycotting a mixed one, yes15:25
kopecmartini think we can set a date (different from the PTG one) and hold our own virtual meeting to connect with the community 15:26
gmannand no slot in physical one?15:26
kopecmartinconsidering the number of attendees during latest PTGs and number of topics as well15:27
kopecmartini would be ok with no slot in the physical event15:27
gmannbut then it still the issue as frickler pointed 15:27
gmannwe should have in either one. otherwise virtual one will be considered as just a QA call15:28
gmannI might travel as it is in USA but not 100% sure, I am totally not fine on travel and completely agree with frickler concern. 15:29
gmannand I am ok to have virtual one15:29
kopecmartinyes, therefore i'm inclined to hold in virtually only, i said i would be ok with no slot in the physical event15:29
kopecmartin*hold it15:29
fricklerso maybe not appearing at the in-person PTG as a team would be a good sign towards the foundation15:30
fricklerI'll also suggest this to other team I'm participating in15:30
fricklerseems we kind of agree on this15:31
fricklerso gmann you can go ahead with your topics I guess15:31
kopecmartinit seems yes15:31
gmannfirst is Py36|7 support in Tempest 15:31
gmann#link http://lists.openstack.org/pipermail/openstack-discuss/2022-May/028622.html15:32
gmannas you know py36|7 drop in oslo or projects and we though of keeping support in tempest but that is broken15:32
gmannI proposed two option on ML and matthew replied from req perspective option1 and how complex that is15:33
gmannand even if we go with that we are kind of saying Tempest master code will work on py36 but with old oslo so no new feature from olso?15:33
gmannwhich give a clear indication that it is better to drop it from Tempest also as its deps are non-py36 now15:34
gmannone challenge in that Tempest master will not support py36|7 for stable/victoria to stable/yoga as those branch support py36|715:35
gmannwhich again has two solution 1. run tempest in py38 in virtual env and test stable/yoga on py36 2. use old tempest to test them15:35
gmannI am in favor of dropping in tempest, what you say?15:36
gmannmay be i am just typing things you all know :) but just in case..15:36
kopecmartindropping from tempest and testing with py38 in a venv for older releases sounds good15:36
fricklerwhat was the reason again for tempest not to have stable branches?15:36
gmannyeah, that is what we did for py27 drop too15:37
opendevreviewArtom Lifshitz proposed openstack/tempest master: DNM: Test encrypted LM with local attach  https://review.opendev.org/c/openstack/tempest/+/84316315:37
gmannfrickler: at least still Tempest master can be used to test stable branch and master openstack. yes not in all python version but we have way15:38
gmannvia virtual env15:38
gmannand give us interop testing benefits, testing stable+master openstack with same set of tests15:38
opendevreviewArtom Lifshitz proposed openstack/tempest master: DNM: Test encrypted LM with local attach  https://review.opendev.org/c/openstack/tempest/+/84316315:38
kopecmartinalthough if it was branched, it would still work for interop i think15:39
frickleryou could still test yoga openstack against master tempest, even if a yoga tempest branch exists15:39
kopecmartinmaybe it would be even easier 15:39
gmannkopecmartin: well, you will test them with different set of  tests right?15:39
gmannfrickler: but then what is benefit or yoga tempest branch if we continue testing with tempest master15:40
frickleryou do test with a different set of tests now, if <yoga needs tagged tempest15:40
kopecmartinoh, on the other side it would require a lot of changes in the interop tooling 15:40
frickleryou could switch once master stops working15:40
fricklerlike now for py36 jobs15:40
gmannfrickler: no, we will continue testing with master stable/yoga and older supported branch15:40
fricklerbut still backport changes to tests15:40
fricklerI'm not convinced a py38 venv will work on c8s15:41
fricklerdid someone test that?15:41
gmannstable/victoria - stable/yoga tested on ubuntu bionic and it has py38 so tempest virtual env can run on that its just if any downstream cloud want to test py36 stable/yoga then they need to create tempest master virutal env on py3815:42
gmannc8s has py38 which worked for me on fips testing switch 15:42
opendevreviewMerged openstack/devstack stable/yoga: Configure placement section in neutron conf  https://review.opendev.org/c/openstack/devstack/+/84274715:42
fricklerhmm, o.k., I don't want to argue this, I'm not a tempest person really, just tried to understand things15:43
gmannwith tempest branch, concern is it can end up like 'disable this test in stable as we change API in incompatible way' and it will pass stable branch testing as well as master even  API are changed15:43
gmannand tempest branchless catches these things15:44
gmannit is no different than when we dropped py27 but yes it will be same for every py3 minor version drop in OpenStack15:46
kopecmartinyeah, let's workaround the situation with the virtual env and py38 for now15:46
fricklero.k., I think you still had a second topic?15:46
gmannyeah, sorry. I already started thinking on next meeting of release name15:47
gmann(gmann) Stable/ussuri is blocked for Tempest master and master constraints15:47
gmannthis is more of information as I have not finished the work15:47
gmannstable/ussuri is in EM and failing with Tempest master and master constraint15:48
kopecmartinthanks for the info15:48
gmannI started pining tempest for stable/ussuri but then got distracted, I am doing now and should be ready soon https://review.opendev.org/q/topic:ussuri-pin-tempest15:48
gmannand I think once we figure this out then we can make new release of tempest15:48
gmannwhich is this #link https://review.opendev.org/q/topic:ussuri-last15:49
opendevreviewRajat Dhasmana proposed openstack/tempest master: Add test to rebuild volume backed instance  https://review.opendev.org/c/openstack/tempest/+/83101815:49
gmannand we can add poy36|7 drop also in that release or so15:49
gmannbut we agreed on dropping py36|7 from tempest right?15:49
*** whoami-rajat__ is now known as whoami-rajat15:50
kopecmartingmann: we did15:50
gmanncool. I will work on these two gate blocker and should be ready soon15:50
gmannthat is all from me15:50
kopecmartinthank you gmann15:51
fricklero.k., just to complete the ansible thing I mentioned15:51
fricklerzuul currently runs with ansible 2.9 (I think?) which is eol15:51
fricklerzuul 6.0.0 now supports to switch to ansible 5, which I think is ansible-core 2.1215:51
clarkbyes 2.9 is the default15:51
fricklerthere is a test patch up in devstack, which shows some work to do15:52
fricklernot urgent, but also not to be delayed forever15:52
clarkbhttps://review.opendev.org/c/openstack/devstack/+/843033 currently it fails when trying to become the stack user. I don't understand it yet but they have a whole document on the problem at https://docs.ansible.com/ansible-core/2.12/user_guide/become.html#risks-of-becoming-an-unprivileged-user15:52
frickler#link https://review.opendev.org/c/openstack/devstack/+/843033/15:52
fricklermaybe switch to accessing the test nodes as stack user directly instead of zuul sudoing?15:53
frickleror dump the stack user and run as zuul. locally I run as ubuntu all the time15:54
clarkbya I'm not sure yet. Probably good to understand why 2.9 worked but 5 doesn't and go from there15:54
fricklero.k., that's it from me, thx clarkb for filling in the details15:56
kopecmartinthanks for bringing this up frickler15:56
kopecmartini need to run15:56
kopecmartinthank you everyone 15:56
kopecmartinsee you around 15:57
opendevmeetMeeting ended Tue May 24 15:57:01 2022 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)15:57
opendevmeetMinutes:        https://meetings.opendev.org/meetings/qa/2022/qa.2022-05-24-15.00.html15:57
opendevmeetMinutes (text): https://meetings.opendev.org/meetings/qa/2022/qa.2022-05-24-15.00.txt15:57
opendevmeetLog:            https://meetings.opendev.org/meetings/qa/2022/qa.2022-05-24-15.00.log.html15:57
fricklerthanks kopecmartin 15:57
gmannkopecmartin: thanks 15:57
ykarelgmann, can you check https://review.opendev.org/c/openstack/devstack-gate/+/84314816:18
opendevreviewDr. Jens Harbott proposed openstack/devstack master: DNM try devstack and friends with ansible v5  https://review.opendev.org/c/openstack/devstack/+/84303316:24
opendevreviewDr. Jens Harbott proposed openstack/devstack master: DNM: testing ansible testing  https://review.opendev.org/c/openstack/devstack/+/84317016:24
fricklerclarkb: ^^ sorry I had to rebase in order to be able to edit .zuul.yaml. testing now whether the failing task might be special because it is running a custom ansible module16:24
clarkbno problem. I'm more than happy to have others help out on figuring out the ansible v5 stuff :)16:31
opendevreviewJames Parker proposed openstack/whitebox-tempest-plugin master: Take into account shared CPUs per NUMA  https://review.opendev.org/c/openstack/whitebox-tempest-plugin/+/84315716:34
clarkbfrickler: looking at a diff of the source code I wonder if the problem is not having the file be executable on the source side so it isn't preserved on the copy to the remote node16:38
clarkbI'm definitely not seeing how we don't fail already16:44
clarkbbut have a hunch that maybe if we set those files to +x to start that they'd carry that perm over and the chmod would be a noop and not fail?16:44
fricklerclarkb: interesting, I can give that a test in a minute16:44
fricklerclarkb: hmm, the run-devstack task fails the same way and that is just a simple builtin command16:46
fricklerI guess that rules out both your idea and mine16:48
frickleranyway, time for a break, bbl16:48
gmannykarel: yeah, train still has legacy zuulv2 jobs. checking it16:49
*** jpena is now known as jpena|off16:51
clarkbI think the check at this line for being an admin user is why it works in 2.9 https://github.com/ansible/ansible/blob/v2.9.27/lib/ansible/plugins/action/__init__.py#L527 In v5 it falls through unless the command succeeds: https://github.com/ansible/ansible/blob/v2.12.6/lib/ansible/plugins/action/__init__.py#L615-L62716:55
clarkbhrm maybe that isn't it I didn't see that else for the error on line 54216:58
gmannykarel: replied,  legacy-tempest-neutron-full-stable  should run on focal, I missed to migrate that, please do and I will be good on your change.17:02
ykarelgmann, ok will do17:04
gmannykarel: thanks for taking care of it. 17:06
opendevreviewyatin proposed openstack/devstack-gate master: Switch legacy-tempest-neutron-full-stable to focal  https://review.opendev.org/c/openstack/devstack-gate/+/84317617:18
ykarelgmann, pushed ^, let's see how it goes17:18
ykarelif fails will check tomorrow17:18
gmannykarel: no issue, I will update if it does, do not worry and have a nice sleep, already late for you17:19
ykarelThanks gmann, bye17:19
gmannykarel: or may be its normal working hr for you ike whoami-rajat  :P17:19
ykarelit's same as his :)17:20
clarkbfrickler: I'm not much closer because I've discovered that my suse install isn't likely represetnative of the ubuntu test nodes. In particular setfacl seems to work on my btrfs file system. That said ansible-playbook -vvv seems to give sufficient verbosity to see what steps ansible si taking so we can trace it through the old and new versions if run against ubuntu to understand17:21
clarkbwhy one is working and the other doesn't. I need to pause on this now though to get other stuff done17:21
opendevreviewMerged openstack/devstack stable/wallaby: Configure placement section in neutron conf  https://review.opendev.org/c/openstack/devstack/+/84274917:23
opendevreviewDr. Jens Harbott proposed openstack/devstack master: DNM: testing ansible testing  https://review.opendev.org/c/openstack/devstack/+/84317017:59
opendevreviewGhanshyam proposed openstack/devstack stable/ussuri: Test Tempest 26.1.0 on ussuri  https://review.opendev.org/c/openstack/devstack/+/83805118:11
opendevreviewDr. Jens Harbott proposed openstack/devstack master: DNM: testing ansible testing  https://review.opendev.org/c/openstack/devstack/+/84317018:23
opendevreviewGhanshyam proposed openstack/tempest master: Drop py3.6 and py3.7 from Tempest  https://review.opendev.org/c/openstack/tempest/+/84318218:29
opendevreviewDr. Jens Harbott proposed openstack/devstack master: DNM: testing ansible testing  https://review.opendev.org/c/openstack/devstack/+/84317018:33
clarkbfrickler: I'm unable to reproduce this issue with ansible 2.9 or 5 on a jammy held node I have. Both appear to run the setfacl successfully18:37
clarkbthis is using both a local connection and ssh with the controlling user being zuul and the become user being stack18:38
clarkbso now I'm wondering if the reason for the failure is some sort of setfacl issue18:38
clarkbmaybe our base image doesn't install the acl package?18:38
clarkbbut if we do that it will work (and something devstack does pull it in?)18:39
clarkbyes acl is not installed on a jammy node by default18:40
clarkbI wonder if installing thta package would be sufficient to make this work18:40
clarkboh hey you might be 10 minutes ahead of me :)18:42
clarkbthat still doesn't explain why the old version of nasible didn't care, but I suspect progress18:43
fricklerclarkb: yes, I don't grok all the details yet, but I held a node and saw that it was missing the setfacl, so I'm trying now what happens when we install it18:46
fricklertest landed on a rax node, which means 10min of copying /opt before seeing results18:47
opendevreviewMerged openstack/whitebox-tempest-plugin master: Take into account shared CPUs per NUMA  https://review.opendev.org/c/openstack/whitebox-tempest-plugin/+/84315718:54
fricklerdevstack running now https://zuul.opendev.org/t/openstack/stream/113f568178fc4012b7cb862713a48130?logfile=console.log18:56
clarkbit passed with acl installed19:25
opendevreviewDr. Jens Harbott proposed openstack/devstack master: DNM try devstack and friends with ansible v5  https://review.opendev.org/c/openstack/devstack/+/84303320:06
fricklergmann: kopecmartin: I missed to tag this patch in the backport series with RP+1 https://review.opendev.org/c/openstack/devstack/+/84274820:39
gmannfrickler: +A20:40
gmannkopecmartin: can you check this grenade skip job running change  https://review.opendev.org/c/openstack/grenade/+/84279922:11
kopecmartingmann: done22:22
opendevreviewMerged openstack/devstack stable/xena: Configure placement section in neutron conf  https://review.opendev.org/c/openstack/devstack/+/84274822:23
opendevreviewGhanshyam proposed openstack/tempest master: Use UPPER_CONSTRAINTS_FILE for stable/ussuri testing  https://review.opendev.org/c/openstack/tempest/+/84304522:52
opendevreviewGhanshyam proposed openstack/devstack stable/ussuri: Pin Tempest to 26.1.0 tag for stable/ussuri testing  https://review.opendev.org/c/openstack/devstack/+/83805122:55
gmannkopecmartin: vkmc I am not sure you did talk about devstack-plugin-ceph-tempest-py3 job failing for identity error, I am seeing that failure more frequent now a days, open a bug for tracking, we should fix it soon https://bugs.launchpad.net/tempest/+bug/197564823:03
gmanngouthamr: ^^ in case you are aware of this error23:03
opendevreviewGhanshyam proposed openstack/tempest master: DNM: tetsing tempest master with stable/victoria constraints  https://review.opendev.org/c/openstack/tempest/+/84304423:06
opendevreviewGhanshyam proposed openstack/tempest master: End support for stable/victoria  https://review.opendev.org/c/openstack/tempest/+/84322123:13
gmannkopecmartin: ^^ end of support for stable/victoria. I am going to do release for ussuri-last and victoria-last together  (still testing the things)23:14
gmannand may be py36|7 drop also. all three things for 31.0.023:17
opendevreviewMerged openstack/grenade master: Do not run grenade-skip-level job on older stable branches  https://review.opendev.org/c/openstack/grenade/+/84279923:44

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!