15:00:17 <kopecmartin> #startmeeting qa
15:00:17 <opendevmeet> Meeting started Tue Jan  4 15:00:17 2022 UTC and is due to finish in 60 minutes.  The chair is kopecmartin. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:00:17 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:00:17 <opendevmeet> The meeting name has been set to 'qa'
15:00:29 <kopecmartin> #link https://wiki.openstack.org/wiki/Meetings/QATeamMeeting#Weekly_QA_Team_meeting
15:00:32 <kopecmartin> agenda ^^
15:01:21 <gmann> o/
15:01:43 <vhari> o/
15:01:59 <frickler> \o
15:02:21 <jelabarre-rh> o/
15:02:49 <kopecmartin> what an attendance
15:02:53 <kopecmartin> hi there
15:03:00 <kopecmartin> happy new year to all :)
15:03:41 <kopecmartin> let's go through the agenda
15:03:42 <kopecmartin> #topic Announcement and Action Item (Optional)
15:03:56 <kopecmartin> nothing here
15:04:03 <kopecmartin> #topic Yoga Priority Items progress
15:04:08 <kopecmartin> #link https://etherpad.opendev.org/p/qa-yoga-priority
15:04:49 <kopecmartin> any updates here?
15:04:49 <soniya29> hello
15:05:05 <kopecmartin> although i don't expect any due to holidays
15:05:12 <soniya29> kopecmartin, nothing from my side :)
15:05:39 <gmann> I have been doing few microversion schema patches
15:05:50 <gmann> but need to check if they are good to review/passing zuul or not
15:06:00 <gmann> other than that nothing much to share here
15:06:19 <kopecmartin> nothing from my side either, i'm trying to catch up and go through mails and reviews
15:06:24 <jelabarre-rh> still waiting for new patches to get incorporated that I then need to test
15:07:54 <kopecmartin> ack
15:07:56 <kopecmartin> #topic OpenStack Events Updates and Planning
15:07:56 <jelabarre-rh> and then maybe investigate my question on multiple-architecture coverage in Tempest (for what little I understand of the Tempest internals)
15:08:18 <kopecmartin> i was gonna bring that up in the open discussion
15:08:33 <jelabarre-rh> ak
15:09:05 <kopecmartin> good, let's quickly go through the usual and we'll be there in a minute
15:09:13 <kopecmartin> regarding the events, nothing new
15:09:22 <kopecmartin> #topic Gate Status Checks
15:09:31 <kopecmartin> #link https://review.opendev.org/q/label:Review-Priority%253D%252B2+status:open+(project:openstack/tempest+OR+project:openstack/patrole+OR+project:openstack/devstack+OR+project:openstack/grenade+OR+project:openstack/hacking)
15:09:52 <kopecmartin> any gate failures to bring up?
15:09:57 <frickler> gate was broken last week
15:10:20 <frickler> paramiko did a new release which broke rsa key login to cirros
15:11:04 <frickler> and u-c testing didn't catch this, because tempest deployed with committed u-c, not with the patch under test
15:11:27 <kopecmartin> oh
15:11:30 <gmann> yeah
15:11:39 <frickler> ykarel did a fix for the latter and for now paramiko is capped
15:11:51 <kopecmartin> that's great
15:12:02 <frickler> I proposed a fix for tempest https://review.opendev.org/c/openstack/tempest/+/823159
15:12:18 <frickler> but there's also other options like switching to ecdsa possibly
15:12:34 <kopecmartin> the DNM in the title confuses me
15:12:46 <gmann> switching to ecdsa is in progress i think in tempest
15:13:09 <kopecmartin> gmann:  if you mean this
15:13:10 <kopecmartin> #Link https://review.opendev.org/c/openstack/tempest/+/807465
15:13:23 <gmann> yeah
15:13:24 <kopecmartin> it's more like adding a support, not switching
15:13:40 <frickler> kopecmartin: I went for DNM because I wanted to discuss whether to do this unconditionally or add a tempest flag
15:13:40 <gmann> yeah after support we will be able to switch in devstack
15:13:54 <kopecmartin> and i think it's ready , it's the first piece required in order to move closer to fips  goal
15:14:08 <frickler> medium term I'm also hoping to get a new release of cirros done that fixes the issue properly
15:14:23 <frickler> or possibly finally get along with forking cirros
15:14:41 <gmann> frickler: do you have testing patch for 823322 ?
15:15:24 <frickler> gmann: no, since you don't see the issue in CI. I tested locally
15:16:11 <gmann> ok
15:17:46 <kopecmartin> I'd go with https://review.opendev.org/c/openstack/tempest/+/807465
15:17:49 <gmann> frickler: kopecmartin lgtm, +2
15:17:54 <frickler> then we merged the openeuler patch and almost immediately broke it with the fips patch
15:18:07 <kopecmartin> it would help us to bypass the gate failure and test the esdsa support even more
15:18:21 <kopecmartin> as devstack would switch as gmann mentioned
15:20:16 <frickler> and finally mlavalle has a fix up for fedora which is important for neutron testing https://review.opendev.org/c/openstack/devstack/+/823218
15:20:53 <gmann> yeah, slaweq just pinged for review. I will also check after meeting
15:21:06 <frickler> I added the same swap workaround as we have for other platforms already
15:22:16 <gmann> +2
15:22:30 <gmann> on stable/train Tempest pin. Tempest 28.0.0 pin is reverted to unblock gate. I will continue working on finding the compatible Tempest version for stable/train/ currently master is used and gate is green.
15:24:39 <kopecmartin> yeah, on that i thought stestr version was the issue
15:25:43 <gmann> humm. trackback was from tempest run command not stestr so I am not 100% sure
15:26:30 <kopecmartin> the version of stestr which was used didn't have the new args for sure, i checked the stestr's repo manually
15:26:35 <gmann> tempest 26.1.0 has exclude-regex arg in tempest run command but it gave error when job passing it and even help message of tempest run does not show new arg
15:26:50 <kopecmartin> that was strange yes
15:26:58 <gmann> kopecmartin: yeah but in that case tempest run should accept the new arg and stestr should raise error
15:27:27 <gmann> yeah, something strange there. I will check this week.
15:27:47 <kopecmartin> oh, that's right, tempest has a try except block for that situation , hmm
15:28:14 <kopecmartin> ok, thanks
15:28:15 <kopecmartin> moving on
15:28:16 <kopecmartin> #topic Periodic jobs Status Checks
15:28:21 <kopecmartin> #link https://zuul.openstack.org/builds?job_name=tempest-full-xena-py3&job_name=tempest-full-wallaby-py3&job_name=tempest-full-victoria-py3&job_name=tempest-full-ussuri-py3&job_name=tempest-full-train-py3&pipeline=periodic-stable
15:28:25 <kopecmartin> #link https://zuul.openstack.org/builds?project=openstack%2Ftempest&project=openstack%2Fdevstack&pipeline=periodic
15:28:33 <kopecmartin> seems all good here
15:28:57 <kopecmartin> #topic Sub Teams highlights
15:29:00 <kopecmartin> Changes with Review-Priority == +1
15:29:07 <kopecmartin> #link https://review.opendev.org/q/label:Review-Priority%253D%252B1+status:open+(project:openstack/tempest+OR+project:openstack/patrole+OR+project:openstack/devstack+OR+project:openstack/grenade+OR+project:openstack/hacking)
15:29:28 <kopecmartin> there are 4 patches, i've reviewed 2 already, the rest I'll try check later today
15:29:38 <kopecmartin> any changes to bring up?
15:32:03 <kopecmartin> #topic Open Discussion
15:32:15 <kopecmartin> (gmann) Turning off the openstack-health service and so does retirement of it?
15:32:36 <kopecmartin> we've got a volunteer
15:32:39 <kopecmartin> let me find the email
15:33:23 <gmann> +1
15:33:30 <kopecmartin> I summarized what we talked about on the openstack-health meeting before holidays
15:33:33 <kopecmartin> #link http://lists.openstack.org/pipermail/openstack-discuss/2022-January/026487.html
15:33:44 <jelabarre-rh> Arx Cruz?
15:33:48 <gmann> yeah
15:34:46 <kopecmartin> so any recommendations where to start?
15:34:51 <gmann> I think we have repo maintainer now so let's allow them to fix the current issue and once it is up then we will discuss on hosting issue
15:35:18 <gmann> kopecmartin: it will be good to fix it and bring it up http://status.openstack.org/openstack-health/#/
15:35:20 <gmann> #luink http://status.openstack.org/openstack-health/#/
15:35:22 <gmann> #link http://status.openstack.org/openstack-health/#/
15:35:28 <arxcruz> kopecmartin: yo
15:35:31 <gmann> arxcruz: hi
15:35:43 <arxcruz> gmann: congrats on the new heir :)
15:35:53 <gmann> thanks :)
15:35:59 <gmann> arxcruz: thanks for volunteering on maintaining the o-h.
15:36:10 <arxcruz> sure
15:36:21 <gmann> arxcruz: as you might know we have two issues/things here. 1. maintain repo 2. hosting infra
15:36:34 <gmann> currently dashboard is broken #link http://status.openstack.org/openstack-health/#/
15:36:43 <arxcruz> gmann: yes, as I understood, rdo will provide the infra right ?
15:36:50 <arxcruz> we can maintain
15:36:50 <gmann> let's fix that as first step and then we can talk about hosting infra need or so
15:37:30 <arxcruz> gmann: sure, actually, it's because there is no data being populated
15:37:32 <gmann> arxcruz: infra is not yet solved. we will see if upstream infra can be used from new aws credits for e-r or so
15:37:41 <arxcruz> ok
15:37:50 <arxcruz> frenzy_friday: what's the url for our o-h ?
15:37:52 <gmann> arxcruz: yeah, I have not checked it but something wrong in code?
15:38:03 <gmann> frenzy_friday: #link http://status.openstack.org/openstack-health/#/
15:38:21 <arxcruz> gmann: so, i can tell in our case, there's a html site, that is the status.o.o/o-h
15:38:24 <gmann> #link https://opendev.org/openstack/openstack-health
15:38:31 <gmann> ^^this is source code
15:38:39 <arxcruz> and also a backgorund job that do some queries on the logs
15:38:43 <arxcruz> and create a json file
15:38:50 <arxcruz> this background job runs every x hours
15:39:06 <arxcruz> and write down a json file that the website reads and shows the results
15:39:45 <arxcruz> in our case, we create a new branch on openstack-health, because our elasticsearch was newer and we had to make changes in the code for the new api's
15:40:22 <gmann> so fixes can be in existing repo right? #link https://opendev.org/openstack/openstack-health
15:40:38 <arxcruz> gmann: yes, in a branch called rdo
15:41:11 <gmann> did not get?
15:41:38 <gmann> o-h repo is branchless and we do not tag also. like doing /using it from master version itself
15:41:51 <arxcruz> gmann: https://opendev.org/opendev/elastic-recheck/src/branch/rdo
15:42:03 <gmann> ohk you mean e-r one
15:42:15 <gmann> i got confused with o-h repo
15:42:40 <arxcruz> gmann: yeah, sorry, my mistake
15:42:52 <arxcruz> i just come back from vacation, not 100% yet on all the details :)
15:43:16 <gmann> sure, we can discuss those later. but brining it up will be good first step and to check for infra need
15:43:40 <gmann> *bringing dashboard up
15:45:56 <kopecmartin> I'll try to look more into it with arxcruz and let's see what we'll come up with
15:46:02 <kopecmartin> will share next week
15:46:05 <gmann> and we can keep this topic in agenda for infra things or so
15:46:11 <gmann> thanks again arxcruz
15:46:12 <kopecmartin> +1
15:46:26 <kopecmartin> another topic in the agenda is
15:46:38 <kopecmartin> about expanding devstack team with potential candidates
15:46:40 <kopecmartin> any updates here?
15:46:53 <arxcruz> :)
15:46:54 <gmann> ah, forgot about it, I will do this week
15:47:05 <gmann> no further updates on this
15:47:42 <kopecmartin> ok :)
15:47:51 <kopecmartin> then we have the mixed-architecture stack topic
15:47:56 <kopecmartin> #link http://lists.openstack.org/pipermail/openstack-discuss/2022-January/026492.html
15:48:36 <kopecmartin> I was thinking about adding a new opt in tempest which would skip tests which are not meant for different architectures
15:48:41 <gmann> do we need test changes for that? or just run the tests with different image in separate job?
15:48:50 <kopecmartin> the same opt could be then used for tests which are specific to those architecture
15:49:10 <gmann> kopecmartin: skip is not good IMO, if those operation perfoermed by tests cannnot be done in real also then just do not run the test
15:49:30 <gmann> if they can be done then we need to adjust the tests
15:49:52 <kopecmartin> yeah, makes sense, .. then maybe let's start from a job, let's create a job with different arch and let's see what's failing and we can take it from there
15:50:20 <jelabarre-rh> I'm thinking in terms of a system where there might be multiple architectures mixed together.
15:50:24 <gmann> yeah, and that will be good feedback to projects that this operation did not work in this arch so you will fix it or add as a known limit?
15:51:02 <gmann> jelabarre-rh: you mean mixed compute node?
15:51:21 <jelabarre-rh> yes
15:51:35 <gmann> and if tempest test end up creating VM on differrent arch compute node
15:51:47 <jelabarre-rh> there is a customer running x86_64 and ppc64le within the same stack
15:51:56 <frickler> before starting to mix things, how about finishing up the arm devstack job and getting it working?
15:52:13 <gmann> in that case, challenge is how to detect those mixed arch in tempest/
15:52:36 <gmann> Tempest is API driven test suits and not aware much about underlying arch
15:52:46 <gmann> frickler: +1
15:52:55 <jelabarre-rh> I know IBM Power ends up stopping at Train (long story), but I expect the same thing may happen for ARM and maybe RISC-V
15:53:33 <jelabarre-rh> I'm bringing it up now so planning can happen before such systems need to be tested
15:53:51 <jelabarre-rh> a discussion topic rather than an urgen current need
15:54:21 <gmann> yeah, but for tempest it is difficult to know which compute node where test VM is schedule to boot is of what arch
15:54:21 <kopecmartin> a good ptg topic, i'll make a note of it somewhere
15:55:55 <gmann> Tempest being as a user of OpenStack, should not aware of compute node arch. for example, if user boot/migrate  VM on mixed arch then what happen? what are SLA from provide side
15:56:27 <gmann> because user operation may fail on mixed arch right?
15:57:08 <jelabarre-rh> it would be more than just testing one arch then the next, as you'd want to have Affinity and Migration check *how* an attempt to migrate an instance between architectures fails
15:58:12 <gmann> jelabarre-rh: so that is expected behavior right? I mean if user perform those, it fail.
15:58:31 <jelabarre-rh> right
15:59:18 <gmann> so test failing there give the right results. instead skipping test can just hide the things
15:59:52 <jelabarre-rh> I happened upon that more by chance, because I'm mainly testing mixed-architecture deployment, and not running Tempest against x86_64, just ppc64le/
16:00:39 <gmann> k
16:00:43 <gmann> I think we are running out of time. I am +1 to discuss in PTG more.
16:00:46 <gmann> or next meeting
16:01:00 <kopecmartin> yup
16:01:15 <gmann> but as first step, having job and see what all fail will be good input for discussion
16:01:26 <kopecmartin> agree
16:01:38 <jelabarre-rh> sounds good.   Mainly to brainstorm for future revisions
16:01:48 <kopecmartin> #topic Bug Triage
16:01:53 <kopecmartin> #link https://etherpad.opendev.org/p/qa-bug-triage-yoga
16:01:56 <kopecmartin> bug numbers are recorded at ^
16:02:09 <kopecmartin> i have this one fix to share
16:02:10 <kopecmartin> #link https://review.opendev.org/c/openstack/tempest/+/823375
16:02:24 <kopecmartin> that's all from my side
16:02:44 <gmann> kopecmartin: ack, will check schema one
16:02:58 <kopecmartin> thanks
16:03:03 <kopecmartin> thank you all
16:03:08 <kopecmartin> see you around
16:03:09 <gmann> thanks kopecmartin
16:03:18 <kopecmartin> #endmeeting