14:00:07 <kopecmartin> #startmeeting qa 14:00:11 <openstack> Meeting started Tue Apr 13 14:00:07 2021 UTC and is due to finish in 60 minutes. The chair is kopecmartin. Information about MeetBot at http://wiki.debian.org/MeetBot. 14:00:12 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 14:00:15 <openstack> The meeting name has been set to 'qa' 14:00:18 <kopecmartin> #link https://wiki.openstack.org/wiki/Meetings/QATeamMeeting 14:00:23 <kopecmartin> ^^^^^^ today's agenda 14:00:29 <kopecmartin> hi, who all here today? 14:00:53 <gmann> o/ 14:01:22 <kopecmartin> o/ 14:01:25 <paras333> o/ 14:01:35 <yoctozepto> o/ 14:02:04 <rh-jelabarre> o/ 14:02:20 <kopecmartin> hi all, perfect, quite an attendance 14:02:23 <kopecmartin> let's start 14:02:28 <kopecmartin> #topic Announcement and Action Item (Optional) 14:02:59 <kopecmartin> only one announcement , PTG is next week! 14:03:11 <kopecmartin> QA topics are recorded here 14:03:13 <kopecmartin> #link https://etherpad.opendev.org/p/qa-xena-ptg 14:03:28 <kopecmartin> if your topic is still not there, don't hesitate to add it 14:04:07 <kopecmartin> looking forward to see you there 14:04:07 <gmann> for secure rbac testing, we need more projects coordination too 14:04:11 <gmann> its cross project topic 14:04:26 <gmann> did we finalize the time or we will do later 14:04:38 <paras333> yes I would say we might need to include lance 14:05:01 <gmann> yeah i pinged him. he is aware of it. 14:05:04 <kopecmartin> i tried to distribute the topics among the days we have booked slots 14:05:06 <gmann> ironic also interested 14:05:09 <paras333> gmann: ok great 14:05:40 <paras333> yeah TheJulia is very interested in this 14:06:35 <gmann> I think 14 UTC is good 14:06:38 <gmann> 14-15 14:07:09 <paras333> gmann: lgtm 14:07:44 <gmann> one more announcement. 14:07:57 <gmann> all the wallaby release and setup patches are merged now 14:07:59 <gmann> #link https://review.opendev.org/q/topic:%2522qa-wallaby-release%2522 14:08:06 <kopecmartin> \o/ 14:08:08 <gmann> so we are now done with wallaby things as QA 14:08:17 <kopecmartin> that's great! 14:09:03 <kopecmartin> #topic Wallaby Priority Items progress 14:09:13 <kopecmartin> #link https://etherpad.opendev.org/p/qa-wallaby-priority 14:09:17 <kopecmartin> any updates? 14:09:32 <kopecmartin> Patrole stable release - gmann 14:09:53 <gmann> we did the release for wallaby 14:09:57 <kopecmartin> what about this? we did release patrole, right? 14:10:05 <kopecmartin> yeah, so can be considered that done? 14:10:11 <gmann> for wallaby yes 14:10:18 <kopecmartin> perfect 14:10:27 <gmann> and for next cycle we can discuss in PTG about how to run it on project side gate or so 14:10:36 <gmann> or if anything pending 14:10:59 <kopecmartin> sure, let me write a note somewhere 14:11:20 <gmann> I think I have added it in PTG ethrpad 14:11:33 <kopecmartin> right, i see it now 14:11:40 <kopecmartin> the second day, first topic 14:11:50 <gmann> yeah 14:12:00 <paras333> kopecmartin: can we move run_validation and guest image one to xena cycle, I still have to do some work which I missed in last cycle 14:12:05 <paras333> ? 14:12:10 <kopecmartin> sure 14:12:18 <paras333> kopecmartin: ok, thanks 14:12:25 <gmann> paras333: we will move all those backlog to Xena which are not finished and still needed 14:12:40 <paras333> gmann: ok great, thank you! 14:12:55 <kopecmartin> yeah, once tracked, forever tracked (until finished haha) 14:12:55 <gmann> L70 in PTG etherpad 14:13:06 <paras333> lol 14:13:29 <kopecmartin> l70 but might be moved to the last day, we'll see 14:13:49 <kopecmartin> #topic Gate Status Checks 14:13:59 <kopecmartin> any blockers or urgent patches? 14:14:47 <gmann> we had stable gate blocked, but merged the fixes except stable/train in gate 14:14:56 <gmann> master gate seems green. 14:15:36 <gmann> this thread 14:15:38 <gmann> #link http://lists.openstack.org/pipermail/openstack-discuss/2021-April/021787.html 14:16:28 <kopecmartin> #link https://review.opendev.org/q/If5f14654ab9aee2a140bbfb869b50d63cb289fdf 14:16:40 <kopecmartin> there are the patches, just train pending , as you sai 14:16:41 <kopecmartin> said 14:16:46 <kopecmartin> good then 14:16:48 <gmann> yeah 14:17:02 <kopecmartin> #topic Periodic jobs Status Checks 14:17:07 <kopecmartin> #link https://zuul.openstack.org/builds?job_name=tempest-all&job_name=tempest-full-oslo-master&pipeline=periodic 14:17:36 <kopecmartin> seems ok 14:18:17 <kopecmartin> #topic Sub Teams highlights (Sub Teams means individual projects under QA program) 14:18:24 <kopecmartin> #link https://review.openstack.org/#/q/project:openstack/tempest+status:open 14:18:28 <kopecmartin> #link https://review.openstack.org/#/q/project:openstack/patrole+status:open 14:18:33 <kopecmartin> #link https://review.openstack.org/#/q/project:openstack/devstack+status:open 14:18:37 <kopecmartin> #link https://review.openstack.org/#/q/project:openstack/grenade+status:open 14:18:42 <kopecmartin> #link https://review.opendev.org/#/q/project:openstack/hacking+status:open 14:20:29 <kopecmartin> a lot of patches as always 14:20:43 <kopecmartin> i see several new ones, will try to review as many as possible 14:20:59 <kopecmartin> #topic Gate Blocker Fix / Urgent Change 14:21:06 <kopecmartin> any other blockers than those we have discussed already? 14:21:40 <gmann> nothing afaik 14:21:47 <yoctozepto> ironic has some weird issue though 14:22:02 <gmann> in grenade job? 14:22:06 <yoctozepto> yeah 14:22:09 <yoctozepto> on their branch 14:22:11 <yoctozepto> it works on grenade 14:22:17 <yoctozepto> but not on ironic 14:22:30 <yoctozepto> iurygregory observed this 14:22:43 <yoctozepto> sean-k-mooney found out it's due to not updating non-ironic projects 14:22:44 <paras333> yoctozepto: what's the job and error? 14:22:45 <yoctozepto> which I confirmed 14:22:49 <yoctozepto> yet it works on grenade 14:23:02 <gmann> oh, they upgrade only ironic? 14:23:08 <yoctozepto> 13:54:23 <iurygregory> this may be a bit weird, but ironic-grenade in stable/victoria is broken anyone aware of changes that could cause nova to raise "ImportError: cannot import name 'reraise'" ? 14:23:09 <yoctozepto> 13:55:00 <iurygregory> funny thing is that in grenade we have 4 runs that are green https://zuul.opendev.org/t/openstack/builds?job_name=ironic-grenade&project=openstack%2Fgrenade&branch=stable%2Fvictoria 14:23:09 <yoctozepto> 13:55:16 <iurygregory> but in ironic we are still red https://zuul.opendev.org/t/openstack/builds?job_name=ironic-grenade&project=openstack%2Fironic&branch=stable%2Fvictoria 14:23:09 <yoctozepto> 13:55:34 <iurygregory> always the same problem in nova it seems to me https://zuul.opendev.org/t/openstack/build/432889791d2e4894956b5b3a3adbb6e3/log/controller/logs/screen-n-api.txt?severity=4 14:23:15 <gmann> yeah i see on grenade patches, no issue 14:23:16 <yoctozepto> gmann: not quite on purpose it seems 14:23:31 <yoctozepto> (psst, the times are utc+2) 14:23:31 <gmann> at least nova should be upgraded 14:23:41 <iurygregory> gmann, we have a filter to remove nova when using multinode grenade job only 14:24:07 <gmann> iurygregory: yoctozepto but did not get what is difference between job running grenade vs ironic gate 14:24:15 <gmann> it is same job or different version? 14:24:18 <yoctozepto> I did not either 14:25:09 <iurygregory> gmann, should be the same .-. 14:25:19 <iurygregory> but it's weird that it works in the grenade gate 14:25:22 <iurygregory> but not in ironic 14:25:27 <rpittau> so 14:25:28 <gmann> iurygregory: and that is only for stable/victoria or master also? 14:25:36 <iurygregory> gmann, victoria only 14:25:44 <iurygregory> master and wallaby seems to be fine 14:26:14 <rpittau> gmann, iurygregory, I was checking the logs and grenade config seems a bit different, at least apparently, one is runninng on summary mode only and saving logs in the main log dir, while the ironic one is running on verbose and saving the logs under the "old" dir 14:26:39 <rpittau> is there maybe something overriding the config/ 14:26:40 <rpittau> ? 14:27:18 <iurygregory> humm I see some failures in master and wallaby also now D: 14:27:35 <iurygregory> ops master* 14:27:47 <iurygregory> wallaby is green https://zuul.opendev.org/t/openstack/builds?job_name=ironic-grenade&project=openstack%2Fironic&branch=stable%2Fwallaby 14:27:51 <gmann> yeah https://zuul.opendev.org/t/openstack/builds?job_name=ironic-grenade 14:27:54 <rpittau> grenade in ironic is saving the logs under controller/logs/old/devstacklog.txt 14:28:08 <gmann> ok, may be let's check after office hour 14:28:11 <gmann> check/debug 14:28:15 <iurygregory> ack 14:28:24 <rpittau> while grenade in grenade is saving under controller/logs/grenade.sh.log_summary.txt 14:28:29 <rpittau> ok 14:29:20 <kopecmartin> #topic Open Discussion 14:29:43 <kopecmartin> (gmann) is it fine to make devstack-plugin-container branchless 14:29:54 <kopecmartin> this got resolved since last week 14:30:03 <gmann> kopecmartin: that is all sorted out 14:30:14 <kopecmartin> good, i'll remove it from the agenda 14:30:19 <gmann> yeah, it will continue as branched as per plugin team 14:30:51 <kopecmartin> anyone anything else? 14:31:15 <gmann> nothing from me 14:32:13 <kopecmartin> #topic Bug Triage (last 30 min) 14:32:15 <kopecmartin> #link https://etherpad.opendev.org/p/qa-bug-triage-wallaby 14:32:16 <yoctozepto> what about the ceph plugin 14:32:27 <kopecmartin> yoctozepto: what about it? 14:32:36 <yoctozepto> they would prefer to branch :-) 14:33:27 <yoctozepto> copying it from agenda: 14:33:27 <yoctozepto> (yoctozepto) or is it better to branch devstack-plugin-ceph to help the consumers avoid headaches? https://review.opendev.org/c/openstack/devstack-plugin-ceph/+/783844 AND https://review.opendev.org/c/openstack/devstack-plugin-ceph/+/782855 14:33:44 <kopecmartin> oh , i see you added the topic in the agenda, sorry, i missed it, the page didn't rendered correctly for me 14:34:10 <yoctozepto> no problem 14:35:00 * kopecmartin loading link, trying to catch up 14:35:58 <kopecmartin> are there any points against branching? 14:36:12 <yoctozepto> not really 14:36:43 <yoctozepto> cc tbarron, vkmc on devstack-plugin-ceph branching 14:37:24 <kopecmartin> i see that the plugin is currently tracked as project with no release 14:37:24 <gmann> yeah, I think branching will be good but only thing is we need to take care of branch cut for every release. 14:37:32 <gmann> also gouthamr: ^^ 14:37:51 <tbarron> I don't know of any reason not to branch. We talked with gouthamr about it yesterday (still early where he is). 14:37:54 <kopecmartin> not considering tags? 14:38:27 <tbarron> Would the worry be that it's more work than tags? 14:38:55 <kopecmartin> no 14:38:58 <vkmc> o/ +1 branching devstack-plugin-ceph 14:39:07 <vkmc> why tags instead of branches? 14:39:18 <tbarron> The plugin is pretty ugly b/c of all the logic that has to be inserted everywhere to check on the ceph version, or openstack version. 14:39:20 <gmann> yeah branch is much better than tag 14:39:34 <gmann> tbarron: exactly 14:39:35 <openstackgerrit> Merged openstack/grenade stable/train: Fix Grenade DSVM Tempest upper constraints https://review.opendev.org/c/openstack/grenade/+/785767 14:39:44 <kopecmartin> i agree 14:39:48 <tbarron> I think branches would allow cleaner development branch and eventually the old mess would get retired. 14:39:54 <vkmc> the support matrix has grown and it's hard to keep things sane without branching 14:40:28 <gmann> do we want it from wallaby or Xena ? 14:40:28 <vkmc> we have to keep compatibility between openstack version, ceph version, distro version and also nfs ganesha 14:40:42 <tbarron> It would be nice to be able to use e.g. cephadm for current deployments but fitting that into the existing logic would be really messy. 14:40:47 <gmann> i think we still have time for wallaby branch cut but need to check with release team 14:41:12 <gmann> yoctozepto: can you remove -W from this now, grenade one merged - https://review.opendev.org/c/openstack/devstack/+/785547 14:41:25 <yoctozepto> gmann: as it's a dev thing, I think release will not mind 14:41:27 <yoctozepto> sure thing 14:42:47 <gmann> yeah. but we should do within today or tomorrow max 14:44:17 <yoctozepto> I mean, I doubt they would mind branching after a coordinated release as these are not really parts of a release per se ;-) 14:44:32 <yoctozepto> need to amend gates to use branches here 14:44:42 <yoctozepto> will you ask hberaud then? 14:47:10 <yoctozepto> anyone here? did I get a netsplit again? 14:47:24 <tbarron> yoctozepto: i see you 14:47:33 <yoctozepto> I see you too 14:47:45 <kopecmartin> i'm here 14:47:49 <gmann> yoctozepto: tbarron we need to update governance also first 14:48:01 <yoctozepto> yes, the governance needs updating here 14:48:12 <yoctozepto> though it's mostly cosmetic 14:48:49 <gmann> who is volunteering to push patches? 14:49:07 <yoctozepto> I can but later 14:49:19 <kopecmartin> i can help too 14:50:08 <kopecmartin> ok, let's sync after the office hour 14:50:23 <kopecmartin> so bug triage 14:50:27 <gmann> great, we need to push patch today in order 1. governance change 2. release patch to cut the branch 3. plugin repo side things 14:50:47 <kopecmartin> #link https://etherpad.opendev.org/p/qa-bug-triage-wallaby 14:51:30 <kopecmartin> gmann: sure, i'll have a look and will ping you probably with questions 14:51:30 <gmann> if no other high priority bug then we can discuss ironic grenade for 10 min? 14:51:38 <gmann> kopecmartin: great, thakns 14:51:44 <kopecmartin> nothing high priority 14:51:46 <kopecmartin> go ahead 14:52:10 <gmann> iurygregory: rpittau yoctozepto ironic greande on master is different issue - https://zuul.opendev.org/t/openstack/build/a0be17b445a24f2f9688f7bfcf62e078/log/controller/logs/grenade.sh_log.txt#1561 14:52:17 <gmann> No Image found for cirros-0.5.1-x86_64-uec 14:52:21 <gmann> seems infra issue? 14:52:26 <yoctozepto> oh my, the fun never ends 14:52:28 <rpittau> most likely 14:52:39 <gmann> checking for victoria 14:52:42 <rpittau> I'm more concerned on victoria 14:52:57 <yoctozepto> you think some images got reverted? 14:53:05 <yoctozepto> the image should be hardcoded in them 14:53:12 <yoctozepto> perhaps some cloud did not refresh images? 14:53:47 <rpittau> AFAIK they're cached, maybe cache has not been updated 14:54:15 <clarkb> the cirros images are cached in the infra test nodes, but devstack/grenade should also download them if not cached 14:54:26 <clarkb> looking at that log the failure is at the upload step though 14:54:30 <rpittau> oh ok 14:54:35 <clarkb> er sorry even after the upload step 14:54:41 <clarkb> it is doing a server create 14:54:57 <yoctozepto> so it's glance that did not get them then 14:55:04 <gmann> yeah 14:55:08 <rpittau> yeah 14:55:11 <yoctozepto> yeah 14:55:30 <clarkb> https://zuul.opendev.org/t/openstack/build/a0be17b445a24f2f9688f7bfcf62e078/log/controller/logs/grenade.sh_log.txt#453-467 the image is on the node 14:55:36 <gmann> does ironic jobs perform glance upgrade or skipped? 14:58:02 <rpittau> it register it for upgrade 14:58:52 <rpittau> so in general it should 14:59:50 <gmann> rpittau: did you try enabling nova in upgrade on stable/victoria and see if that work ? 15:00:16 <rpittau> gmann: err shouldn't that happen already? in grenade itworks fine 15:00:44 <rpittau> the ironic-grenade job in grenade works correctly, in ironic it doesn't 15:00:46 <gmann> rpittau: grenade gate run same job so it should have nova not in upgrade too same as on ironic gate 15:01:01 <gmann> yeah that is not known why 15:01:19 <rpittau> I was saying before that it seems like there is something overriding the config 15:02:07 <yoctozepto> gmann: it's not multinode 15:02:14 <yoctozepto> ironic does not disable nova in single node 15:02:29 <yoctozepto> it's confusing 15:02:40 <gmann> ah 15:02:40 <yoctozepto> all right, we are past time 15:02:53 <yoctozepto> I will start pushing the patches later in the evening 15:03:00 <yoctozepto> so if someone wishes to start earlier 15:03:03 <yoctozepto> be my guest ;-) 15:03:04 <gmann> thanks 15:03:14 <kopecmartin> yoctozepto: i'll try, but it's almost end of my day 15:03:27 <kopecmartin> so maybe you will need to finish them 15:03:30 <kopecmartin> :) 15:03:41 <kopecmartin> let's close the office hour 15:03:45 <kopecmartin> thank you all 15:03:46 <gmann> kopecmartin: can you re-W it to push it on gate pipeline https://review.opendev.org/c/openstack/devstack/+/785547 15:03:53 <kopecmartin> #endmeeting