14:00:07 <kopecmartin> #startmeeting qa
14:00:11 <openstack> Meeting started Tue Apr 13 14:00:07 2021 UTC and is due to finish in 60 minutes.  The chair is kopecmartin. Information about MeetBot at http://wiki.debian.org/MeetBot.
14:00:12 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
14:00:15 <openstack> The meeting name has been set to 'qa'
14:00:18 <kopecmartin> #link https://wiki.openstack.org/wiki/Meetings/QATeamMeeting
14:00:23 <kopecmartin> ^^^^^^ today's agenda
14:00:29 <kopecmartin> hi, who all here today?
14:00:53 <gmann> o/
14:01:22 <kopecmartin> o/
14:01:25 <paras333> o/
14:01:35 <yoctozepto> o/
14:02:04 <rh-jelabarre> o/
14:02:20 <kopecmartin> hi all, perfect, quite an attendance
14:02:23 <kopecmartin> let's start
14:02:28 <kopecmartin> #topic Announcement and Action Item (Optional)
14:02:59 <kopecmartin> only one announcement , PTG is next week!
14:03:11 <kopecmartin> QA topics are recorded here
14:03:13 <kopecmartin> #link https://etherpad.opendev.org/p/qa-xena-ptg
14:03:28 <kopecmartin> if your topic is still not there, don't hesitate to add it
14:04:07 <kopecmartin> looking forward to see you there
14:04:07 <gmann> for secure rbac testing, we need more projects coordination too
14:04:11 <gmann> its cross project topic
14:04:26 <gmann> did we finalize the time or we will do later
14:04:38 <paras333> yes I would say we might need to include lance
14:05:01 <gmann> yeah i pinged him. he is aware of it.
14:05:04 <kopecmartin> i tried to distribute the topics among the days we have booked slots
14:05:06 <gmann> ironic also interested
14:05:09 <paras333> gmann: ok great
14:05:40 <paras333> yeah TheJulia is very interested in this
14:06:35 <gmann> I think 14 UTC is good
14:06:38 <gmann> 14-15
14:07:09 <paras333> gmann: lgtm
14:07:44 <gmann> one more announcement.
14:07:57 <gmann> all the wallaby release and setup patches are merged now
14:07:59 <gmann> #link https://review.opendev.org/q/topic:%2522qa-wallaby-release%2522
14:08:06 <kopecmartin> \o/
14:08:08 <gmann> so we are now done with wallaby things as QA
14:08:17 <kopecmartin> that's great!
14:09:03 <kopecmartin> #topic Wallaby Priority Items progress
14:09:13 <kopecmartin> #link https://etherpad.opendev.org/p/qa-wallaby-priority
14:09:17 <kopecmartin> any updates?
14:09:32 <kopecmartin> Patrole stable release - gmann
14:09:53 <gmann> we did the release for wallaby
14:09:57 <kopecmartin> what about this? we did release patrole, right?
14:10:05 <kopecmartin> yeah, so can be considered that done?
14:10:11 <gmann> for wallaby yes
14:10:18 <kopecmartin> perfect
14:10:27 <gmann> and for next cycle we can discuss in PTG about how to run it on project side gate or so
14:10:36 <gmann> or if anything pending
14:10:59 <kopecmartin> sure, let me write a note somewhere
14:11:20 <gmann> I think I have added it in PTG ethrpad
14:11:33 <kopecmartin> right, i see it now
14:11:40 <kopecmartin> the second day, first topic
14:11:50 <gmann> yeah
14:12:00 <paras333> kopecmartin: can we move run_validation and guest image one to xena cycle, I still have to do some work which I missed in last cycle
14:12:05 <paras333> ?
14:12:10 <kopecmartin> sure
14:12:18 <paras333> kopecmartin: ok, thanks
14:12:25 <gmann> paras333: we will move all those backlog to Xena which are not finished and still needed
14:12:40 <paras333> gmann: ok great, thank you!
14:12:55 <kopecmartin> yeah, once tracked, forever tracked (until finished haha)
14:12:55 <gmann> L70 in PTG etherpad
14:13:06 <paras333> lol
14:13:29 <kopecmartin> l70 but might be moved to the last day, we'll see
14:13:49 <kopecmartin> #topic Gate Status Checks
14:13:59 <kopecmartin> any blockers or urgent patches?
14:14:47 <gmann> we had stable gate blocked, but merged the fixes except stable/train in gate
14:14:56 <gmann> master gate seems green.
14:15:36 <gmann> this thread
14:15:38 <gmann> #link http://lists.openstack.org/pipermail/openstack-discuss/2021-April/021787.html
14:16:28 <kopecmartin> #link https://review.opendev.org/q/If5f14654ab9aee2a140bbfb869b50d63cb289fdf
14:16:40 <kopecmartin> there are the patches, just train pending , as you sai
14:16:41 <kopecmartin> said
14:16:46 <kopecmartin> good then
14:16:48 <gmann> yeah
14:17:02 <kopecmartin> #topic Periodic jobs Status Checks
14:17:07 <kopecmartin> #link https://zuul.openstack.org/builds?job_name=tempest-all&job_name=tempest-full-oslo-master&pipeline=periodic
14:17:36 <kopecmartin> seems ok
14:18:17 <kopecmartin> #topic Sub Teams highlights (Sub Teams means individual projects under QA program)
14:18:24 <kopecmartin> #link https://review.openstack.org/#/q/project:openstack/tempest+status:open
14:18:28 <kopecmartin> #link https://review.openstack.org/#/q/project:openstack/patrole+status:open
14:18:33 <kopecmartin> #link https://review.openstack.org/#/q/project:openstack/devstack+status:open
14:18:37 <kopecmartin> #link https://review.openstack.org/#/q/project:openstack/grenade+status:open
14:18:42 <kopecmartin> #link https://review.opendev.org/#/q/project:openstack/hacking+status:open
14:20:29 <kopecmartin> a lot of patches as always
14:20:43 <kopecmartin> i see several new ones, will try to review as many as possible
14:20:59 <kopecmartin> #topic Gate Blocker Fix / Urgent Change
14:21:06 <kopecmartin> any other blockers than those we have discussed already?
14:21:40 <gmann> nothing afaik
14:21:47 <yoctozepto> ironic has some weird issue though
14:22:02 <gmann> in grenade job?
14:22:06 <yoctozepto> yeah
14:22:09 <yoctozepto> on their branch
14:22:11 <yoctozepto> it works on grenade
14:22:17 <yoctozepto> but not on ironic
14:22:30 <yoctozepto> iurygregory observed this
14:22:43 <yoctozepto> sean-k-mooney found out it's due to not updating non-ironic projects
14:22:44 <paras333> yoctozepto: what's the job and error?
14:22:45 <yoctozepto> which I confirmed
14:22:49 <yoctozepto> yet it works on grenade
14:23:02 <gmann> oh, they upgrade only ironic?
14:23:08 <yoctozepto> 13:54:23 <iurygregory> this may be a bit weird, but ironic-grenade in stable/victoria is broken anyone aware of changes that could cause nova to raise "ImportError: cannot import name 'reraise'" ?
14:23:09 <yoctozepto> 13:55:00 <iurygregory> funny thing is that in grenade we have 4 runs that are green https://zuul.opendev.org/t/openstack/builds?job_name=ironic-grenade&project=openstack%2Fgrenade&branch=stable%2Fvictoria
14:23:09 <yoctozepto> 13:55:16 <iurygregory> but in ironic we are still red https://zuul.opendev.org/t/openstack/builds?job_name=ironic-grenade&project=openstack%2Fironic&branch=stable%2Fvictoria
14:23:09 <yoctozepto> 13:55:34 <iurygregory> always the same problem in nova it seems to me https://zuul.opendev.org/t/openstack/build/432889791d2e4894956b5b3a3adbb6e3/log/controller/logs/screen-n-api.txt?severity=4
14:23:15 <gmann> yeah i see on grenade patches, no issue
14:23:16 <yoctozepto> gmann: not quite on purpose it seems
14:23:31 <yoctozepto> (psst, the times are utc+2)
14:23:31 <gmann> at least nova should be upgraded
14:23:41 <iurygregory> gmann, we have a filter to remove nova when using multinode grenade job only
14:24:07 <gmann> iurygregory: yoctozepto but did not get what is difference between job running grenade vs ironic gate
14:24:15 <gmann> it is same job or different version?
14:24:18 <yoctozepto> I did not either
14:25:09 <iurygregory> gmann, should be the same .-.
14:25:19 <iurygregory> but it's weird that it works in the grenade gate
14:25:22 <iurygregory> but not in ironic
14:25:27 <rpittau> so
14:25:28 <gmann> iurygregory: and that is only for stable/victoria or master also?
14:25:36 <iurygregory> gmann, victoria only
14:25:44 <iurygregory> master and wallaby seems to be fine
14:26:14 <rpittau> gmann, iurygregory, I was checking the logs and grenade config seems a bit different, at least apparently, one is runninng on summary mode only and saving logs in the main log dir, while the ironic one is running on verbose and saving the logs under the "old" dir
14:26:39 <rpittau> is there maybe something overriding the config/
14:26:40 <rpittau> ?
14:27:18 <iurygregory> humm I see some failures in master and wallaby also now D:
14:27:35 <iurygregory> ops master*
14:27:47 <iurygregory> wallaby is green https://zuul.opendev.org/t/openstack/builds?job_name=ironic-grenade&project=openstack%2Fironic&branch=stable%2Fwallaby
14:27:51 <gmann> yeah https://zuul.opendev.org/t/openstack/builds?job_name=ironic-grenade
14:27:54 <rpittau> grenade in ironic is saving the logs under controller/logs/old/devstacklog.txt
14:28:08 <gmann> ok, may be let's check after office hour
14:28:11 <gmann> check/debug
14:28:15 <iurygregory> ack
14:28:24 <rpittau> while grenade in grenade is saving under controller/logs/grenade.sh.log_summary.txt
14:28:29 <rpittau> ok
14:29:20 <kopecmartin> #topic Open Discussion
14:29:43 <kopecmartin> (gmann) is it fine to make devstack-plugin-container branchless
14:29:54 <kopecmartin> this got resolved since last week
14:30:03 <gmann> kopecmartin: that is all sorted out
14:30:14 <kopecmartin> good, i'll remove it from the agenda
14:30:19 <gmann> yeah, it will continue as branched as per plugin team
14:30:51 <kopecmartin> anyone anything else?
14:31:15 <gmann> nothing from me
14:32:13 <kopecmartin> #topic Bug Triage (last 30 min)
14:32:15 <kopecmartin> #link https://etherpad.opendev.org/p/qa-bug-triage-wallaby
14:32:16 <yoctozepto> what about the ceph plugin
14:32:27 <kopecmartin> yoctozepto: what about it?
14:32:36 <yoctozepto> they would prefer to branch :-)
14:33:27 <yoctozepto> copying it from agenda:
14:33:27 <yoctozepto> (yoctozepto) or is it better to branch devstack-plugin-ceph to help the consumers avoid headaches? https://review.opendev.org/c/openstack/devstack-plugin-ceph/+/783844 AND https://review.opendev.org/c/openstack/devstack-plugin-ceph/+/782855
14:33:44 <kopecmartin> oh , i see you added the topic in the agenda, sorry, i missed it, the page didn't rendered correctly for me
14:34:10 <yoctozepto> no problem
14:35:00 * kopecmartin loading link, trying to catch up
14:35:58 <kopecmartin> are there any points against branching?
14:36:12 <yoctozepto> not really
14:36:43 <yoctozepto> cc tbarron, vkmc on devstack-plugin-ceph branching
14:37:24 <kopecmartin> i see that the plugin is currently tracked as project with no release
14:37:24 <gmann> yeah, I think branching will be good but only thing is we need to take care of branch cut for every release.
14:37:32 <gmann> also gouthamr: ^^
14:37:51 <tbarron> I don't know of any reason not to branch.  We talked with gouthamr about it yesterday (still early where he is).
14:37:54 <kopecmartin> not considering tags?
14:38:27 <tbarron> Would the worry be that it's more work than tags?
14:38:55 <kopecmartin> no
14:38:58 <vkmc> o/ +1 branching devstack-plugin-ceph
14:39:07 <vkmc> why tags instead of branches?
14:39:18 <tbarron> The plugin is pretty ugly b/c of all the logic that has to be inserted everywhere to check on the ceph version, or openstack version.
14:39:20 <gmann> yeah branch is much better than tag
14:39:34 <gmann> tbarron: exactly
14:39:35 <openstackgerrit> Merged openstack/grenade stable/train: Fix Grenade DSVM Tempest upper constraints  https://review.opendev.org/c/openstack/grenade/+/785767
14:39:44 <kopecmartin> i agree
14:39:48 <tbarron> I think branches would allow cleaner development branch and eventually the old mess would get retired.
14:39:54 <vkmc> the support matrix has grown and it's hard to keep things sane without branching
14:40:28 <gmann> do we want it from wallaby or Xena ?
14:40:28 <vkmc> we have to keep compatibility between openstack version, ceph version, distro version and also nfs ganesha
14:40:42 <tbarron> It would be nice to be able to use e.g. cephadm for current deployments but fitting that into the existing logic would be really messy.
14:40:47 <gmann> i think we still have time for wallaby branch cut but need to check with release team
14:41:12 <gmann> yoctozepto: can you remove -W from this now, grenade one merged - https://review.opendev.org/c/openstack/devstack/+/785547
14:41:25 <yoctozepto> gmann: as it's a dev thing, I think release will not mind
14:41:27 <yoctozepto> sure thing
14:42:47 <gmann> yeah. but we should do within today or tomorrow max
14:44:17 <yoctozepto> I mean, I doubt they would mind branching after a coordinated release as these are not really parts of a release per se ;-)
14:44:32 <yoctozepto> need to amend gates to use branches here
14:44:42 <yoctozepto> will you ask hberaud then?
14:47:10 <yoctozepto> anyone here? did I get a netsplit again?
14:47:24 <tbarron> yoctozepto: i see you
14:47:33 <yoctozepto> I see you too
14:47:45 <kopecmartin> i'm here
14:47:49 <gmann> yoctozepto: tbarron we need to update governance also first
14:48:01 <yoctozepto> yes, the governance needs updating here
14:48:12 <yoctozepto> though it's mostly cosmetic
14:48:49 <gmann> who is volunteering to push patches?
14:49:07 <yoctozepto> I can but later
14:49:19 <kopecmartin> i can help too
14:50:08 <kopecmartin> ok, let's sync after the office hour
14:50:23 <kopecmartin> so bug triage
14:50:27 <gmann> great, we need to push patch today in order 1. governance change 2. release patch to cut the branch 3.  plugin repo side things
14:50:47 <kopecmartin> #link https://etherpad.opendev.org/p/qa-bug-triage-wallaby
14:51:30 <kopecmartin> gmann: sure, i'll have a look and will ping you probably with questions
14:51:30 <gmann> if no other high priority bug then we can discuss ironic grenade for 10 min?
14:51:38 <gmann> kopecmartin: great, thakns
14:51:44 <kopecmartin> nothing high priority
14:51:46 <kopecmartin> go ahead
14:52:10 <gmann> iurygregory: rpittau yoctozepto ironic greande on master is different issue - https://zuul.opendev.org/t/openstack/build/a0be17b445a24f2f9688f7bfcf62e078/log/controller/logs/grenade.sh_log.txt#1561
14:52:17 <gmann> No Image found for cirros-0.5.1-x86_64-uec
14:52:21 <gmann> seems infra issue?
14:52:26 <yoctozepto> oh my, the fun never ends
14:52:28 <rpittau> most likely
14:52:39 <gmann> checking for victoria
14:52:42 <rpittau> I'm more concerned on victoria
14:52:57 <yoctozepto> you think some images got reverted?
14:53:05 <yoctozepto> the image should be hardcoded in them
14:53:12 <yoctozepto> perhaps some cloud did not refresh images?
14:53:47 <rpittau> AFAIK they're cached, maybe cache has not been updated
14:54:15 <clarkb> the cirros images are cached in the infra test nodes, but devstack/grenade should also download them if not cached
14:54:26 <clarkb> looking at that log the failure is at the upload step though
14:54:30 <rpittau> oh ok
14:54:35 <clarkb> er sorry even after the upload step
14:54:41 <clarkb> it is doing a server create
14:54:57 <yoctozepto> so it's glance that did not get them then
14:55:04 <gmann> yeah
14:55:08 <rpittau> yeah
14:55:11 <yoctozepto> yeah
14:55:30 <clarkb> https://zuul.opendev.org/t/openstack/build/a0be17b445a24f2f9688f7bfcf62e078/log/controller/logs/grenade.sh_log.txt#453-467 the image is on the node
14:55:36 <gmann> does ironic jobs perform glance upgrade or skipped?
14:58:02 <rpittau> it register it for upgrade
14:58:52 <rpittau> so in general it should
14:59:50 <gmann> rpittau: did you try enabling nova in upgrade on stable/victoria and see if that work ?
15:00:16 <rpittau> gmann: err shouldn't that happen already? in grenade itworks fine
15:00:44 <rpittau> the ironic-grenade job in grenade works correctly, in ironic it doesn't
15:00:46 <gmann> rpittau: grenade gate run same job so it should have nova not in upgrade too same as on ironic gate
15:01:01 <gmann> yeah that is not known why
15:01:19 <rpittau> I was saying before that it seems like there is something overriding the config
15:02:07 <yoctozepto> gmann: it's not multinode
15:02:14 <yoctozepto> ironic does not disable nova in single node
15:02:29 <yoctozepto> it's confusing
15:02:40 <gmann> ah
15:02:40 <yoctozepto> all right, we are past time
15:02:53 <yoctozepto> I will start pushing the patches later in the evening
15:03:00 <yoctozepto> so if someone wishes to start earlier
15:03:03 <yoctozepto> be my guest ;-)
15:03:04 <gmann> thanks
15:03:14 <kopecmartin> yoctozepto: i'll try, but it's almost end of my day
15:03:27 <kopecmartin> so maybe you will need to finish them
15:03:30 <kopecmartin> :)
15:03:41 <kopecmartin> let's close the office hour
15:03:45 <kopecmartin> thank you all
15:03:46 <gmann> kopecmartin: can you re-W it to push it on gate pipeline https://review.opendev.org/c/openstack/devstack/+/785547
15:03:53 <kopecmartin> #endmeeting