Tuesday, 2021-04-13

*** mailingsam has quit IRC00:19
*** enriquetaso has joined #openstack-qa00:28
*** lbragstad_ has joined #openstack-qa00:43
*** lbragstad has quit IRC00:45
*** paras33__ has quit IRC00:51
*** hamalq has quit IRC01:19
*** ajitha has joined #openstack-qa01:57
*** paras333 has joined #openstack-qa02:31
*** paras333 has quit IRC02:35
*** enriquetaso has quit IRC02:40
openstackgerritMerged openstack/grenade master: Fix Grenade DSVM Tempest upper constraints  https://review.opendev.org/c/openstack/grenade/+/78583102:48
*** rcernin has quit IRC02:58
openstackgerritMerged openstack/grenade stable/wallaby: Fix Grenade DSVM Tempest upper constraints  https://review.opendev.org/c/openstack/grenade/+/78560802:59
openstackgerritMerged openstack/grenade stable/victoria: Fix Grenade DSVM Tempest upper constraints  https://review.opendev.org/c/openstack/grenade/+/78560902:59
openstackgerritMerged openstack/grenade stable/ussuri: Fix Grenade DSVM Tempest upper constraints  https://review.opendev.org/c/openstack/grenade/+/78561002:59
*** vishalmanchanda has joined #openstack-qa03:17
*** rcernin has joined #openstack-qa03:19
*** rcernin has quit IRC03:37
*** rcernin has joined #openstack-qa03:37
openstackgerritMerged openstack/grenade master: Make heat and octavia grenade jobs as voting  https://review.opendev.org/c/openstack/grenade/+/78593804:10
*** gcheresh has joined #openstack-qa04:40
*** akahat has quit IRC05:02
*** kopecmartin has quit IRC05:03
*** kopecmartin has joined #openstack-qa05:06
*** akahat has joined #openstack-qa05:14
*** ralonsoh has joined #openstack-qa05:29
*** ralonsoh has quit IRC05:29
*** ralonsoh has joined #openstack-qa05:30
*** Luzi has joined #openstack-qa05:50
*** slaweq has joined #openstack-qa05:52
*** sboyron has joined #openstack-qa06:08
*** lpiwowar has joined #openstack-qa06:21
openstackgerritMerged openstack/devstack-gate master: Update grenade setting for stable/wallaby  https://review.opendev.org/c/openstack/devstack-gate/+/78501006:27
*** rcernin has quit IRC06:38
*** rcernin has joined #openstack-qa06:38
*** eolivare has joined #openstack-qa06:46
*** rcernin has quit IRC06:55
*** jpena|off is now known as jpena07:34
*** sboyron has quit IRC07:37
*** sboyron has joined #openstack-qa07:38
*** sboyron has quit IRC07:38
*** sboyron has joined #openstack-qa07:39
*** rpittau|afk is now known as rpittau07:43
*** tosky has joined #openstack-qa07:49
*** rcernin has joined #openstack-qa07:56
*** rcernin has quit IRC08:00
*** gfidente|afk is now known as gfidente08:06
*** lucasagomes has joined #openstack-qa08:08
*** rcernin has joined #openstack-qa08:08
*** rcernin has quit IRC08:19
*** rcernin has joined #openstack-qa08:24
*** hberaud has left #openstack-qa08:25
*** dtantsur|afk is now known as dtantsur08:31
openstackgerritLajos Katona proposed openstack/tempest master: Add live migration with trunk test  https://review.opendev.org/c/openstack/tempest/+/77468908:35
*** Luzi has quit IRC08:47
*** Luzi has joined #openstack-qa08:48
*** dtantsur is now known as dtantsur|brb08:57
*** rcernin has quit IRC08:58
*** kopecmartin has quit IRC09:28
*** akahat has quit IRC09:28
*** kopecmartin has joined #openstack-qa09:33
masayukigkopecmartin: do we have an office hour today? I'd like to skip it because I'm on leave this week..09:46
kopecmartinmasayukig: we should have .. np, i can chair it09:47
kopecmartinenjoy PTO :)09:47
*** yamamoto has quit IRC09:48
masayukigkopecmartin++ thank you so much! :)09:49
*** rcernin has joined #openstack-qa09:50
*** yamamoto has joined #openstack-qa09:51
*** akahat has joined #openstack-qa09:53
*** rpittau is now known as rpittau|bbl09:54
*** sboyron has quit IRC09:54
*** sboyron has joined #openstack-qa09:55
*** rcernin has quit IRC09:56
*** yamamoto has quit IRC09:58
*** dtantsur|brb is now known as dtantsur10:00
yoctozeptoargh that reliability of train gate10:10
*** rcernin has joined #openstack-qa10:14
*** rcernin has quit IRC10:20
*** yamamoto has joined #openstack-qa10:28
*** yamamoto has quit IRC10:42
openstackgerritLajos Katona proposed openstack/tempest master: Add live migration with trunk test  https://review.opendev.org/c/openstack/tempest/+/77468910:48
*** chaconpiza has joined #openstack-qa10:59
*** tkajinam has quit IRC11:13
*** yamamoto has joined #openstack-qa11:18
*** yamamoto has quit IRC11:29
*** jpena is now known as jpena|lunch11:31
*** enriquetaso has joined #openstack-qa11:52
iurygregorythis may be a bit weird, but ironic-grenade in stable/victoria is broken anyone aware of changes that could cause nova to raise "ImportError: cannot import name 'reraise'" ?11:54
iurygregoryfunny thing is that in grenade we have 4 runs that are green https://zuul.opendev.org/t/openstack/builds?job_name=ironic-grenade&project=openstack%2Fgrenade&branch=stable%2Fvictoria11:55
iurygregorybut in ironic we are still red https://zuul.opendev.org/t/openstack/builds?job_name=ironic-grenade&project=openstack%2Fironic&branch=stable%2Fvictoria11:55
iurygregoryalways the same problem in nova it seems to me https://zuul.opendev.org/t/openstack/build/432889791d2e4894956b5b3a3adbb6e3/log/controller/logs/screen-n-api.txt?severity=411:55
iurygregoryif anyone has some ideas please let me know =)11:56
*** ramishra has quit IRC12:02
*** yamamoto has joined #openstack-qa12:02
*** yamamoto has quit IRC12:06
*** ramishra has joined #openstack-qa12:08
*** artom has joined #openstack-qa12:15
*** nweinber has joined #openstack-qa12:29
*** rpittau|bbl is now known as rpittau12:31
*** jpena|lunch is now known as jpena12:37
*** paras333 has joined #openstack-qa12:52
*** paras333 has quit IRC12:55
*** paras333 has joined #openstack-qa12:56
yoctozeptoiurygregory: is not it the thing that sean-k-mooney commented on #openstack-nova yesterday?13:02
yoctozeptothat ironic stopped upgrading non-ironic projects13:02
iurygregoryI'm not sure, I probably missed that =(13:04
iurygregoryi would say it could be, but is strange that the job has success in grenade but not in ironic..13:05
paras333gmann: kopecmartin: did we cancel today's meeting?13:06
paras333sorry I was on vacation in India and returned last week only.13:07
yoctozeptoparas333: 53 minutes pending till meeting13:07
paras333yoctozepto: Ohh , So it moved to 10 am EST?13:07
yoctozeptoiurygregory: not strange if ironic job upgrades only ironic and its deps13:07
yoctozeptoand these upgraded deps break nova13:07
yoctozeptoquite expected in our release model13:07
yoctozeptoparas333: it's always 14:00 UTC ;-)13:08
yoctozeptoand it's 14 UTC in 51 minutes13:09
yoctozepto:-)13:09
paras333yoctozepto: yeah , it got extended to 10 due to daylight saving here , nvm thanks13:10
yoctozeptohappy to help :-)13:10
iurygregoryyoctozepto, but grenade project uses the ironic job definition so it should also fail no?13:10
yoctozeptoiurygregory: hmm, well, with Sean we analysed the job on the ironic side13:11
yoctozeptothat's interesting, yes13:11
yoctozeptohmm, hmm13:12
*** yamamoto has joined #openstack-qa13:14
iurygregorymaybe we should move required projects to "opendev.org/openstack/******"13:14
iurygregorynot sure if it would have impact13:14
*** mailingsam has joined #openstack-qa13:15
mailingsamgmann regarding the network delete, the networks are being deleted, so you think waiting for network delete is unnecessary?13:15
yoctozeptoiurygregory: well, the same job should work the same unless it specifically has a conditional on the project that it runs from13:16
iurygregoryand if I do remember we only skip nova upgrade in multinode...13:16
iurygregoryagree =)13:17
*** tosky has quit IRC13:19
yoctozeptoiurygregory: well, if you skip nova upgrade and nova fails... that's precisely the reason :D13:22
yoctozeptobut the fact it works on grenade and not ironic is... confusing13:23
iurygregoryonly in the multinode job =D13:23
yoctozeptoah, ah, ok13:23
yoctozeptobut it's not the case here then13:23
iurygregoryand I just noticed we don't have the grenade.sh.log_summary.txt  available O.o weird13:23
yoctozeptopossibly because it failed too abruptly13:24
yoctozeptostill weird13:24
*** tosky has joined #openstack-qa13:25
iurygregoryyeah it makes sense13:25
*** tosky has quit IRC13:29
*** tosky has joined #openstack-qa13:29
gmannmailingsam: i think so, if that is causing failure somewhere then we can debug and change accordingly13:32
openstackgerritGhanshyam proposed openstack/grenade stable/wallaby: Make heat and octavia grenade jobs as voting  https://review.opendev.org/c/openstack/grenade/+/78593013:39
openstackgerritGhanshyam proposed openstack/grenade stable/victoria: Make heat and octavia grenade jobs as voting  https://review.opendev.org/c/openstack/grenade/+/78593113:39
openstackgerritGhanshyam proposed openstack/grenade stable/ussuri: Make heat and octavia grenade jobs as voting  https://review.opendev.org/c/openstack/grenade/+/78593213:40
*** Luzi has quit IRC13:49
openstackgerritGhanshyam proposed openstack/devstack stable/ussuri: DNM: testing neutron-trunk  https://review.opendev.org/c/openstack/devstack/+/78605613:55
*** gfidente has quit IRC13:56
*** gfidente has joined #openstack-qa13:56
kopecmartin#startmeeting qa14:00
openstackMeeting started Tue Apr 13 14:00:07 2021 UTC and is due to finish in 60 minutes.  The chair is kopecmartin. Information about MeetBot at http://wiki.debian.org/MeetBot.14:00
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.14:00
*** openstack changes topic to " (Meeting topic: qa)"14:00
openstackThe meeting name has been set to 'qa'14:00
kopecmartin#link https://wiki.openstack.org/wiki/Meetings/QATeamMeeting14:00
kopecmartin^^^^^^ today's agenda14:00
kopecmartinhi, who all here today?14:00
gmanno/14:00
kopecmartino/14:01
paras333o/14:01
yoctozeptoo/14:01
rh-jelabarreo/14:02
kopecmartinhi all, perfect, quite an attendance14:02
kopecmartinlet's start14:02
kopecmartin#topic Announcement and Action Item (Optional)14:02
*** openstack changes topic to "Announcement and Action Item (Optional) (Meeting topic: qa)"14:02
kopecmartinonly one announcement , PTG is next week!14:02
kopecmartinQA topics are recorded here14:03
kopecmartin#link https://etherpad.opendev.org/p/qa-xena-ptg14:03
kopecmartinif your topic is still not there, don't hesitate to add it14:03
kopecmartinlooking forward to see you there14:04
gmannfor secure rbac testing, we need more projects coordination too14:04
gmannits cross project topic14:04
gmanndid we finalize the time or we will do later14:04
paras333yes I would say we might need to include lance14:04
gmannyeah i pinged him. he is aware of it.14:05
kopecmartini tried to distribute the topics among the days we have booked slots14:05
gmannironic also interested14:05
paras333gmann: ok great14:05
paras333yeah TheJulia is very interested in this14:05
gmannI think 14 UTC is good14:06
gmann14-1514:06
paras333gmann: lgtm14:07
gmannone more announcement.14:07
gmannall the wallaby release and setup patches are merged now14:07
gmann#link https://review.opendev.org/q/topic:%2522qa-wallaby-release%252214:07
kopecmartin\o/14:08
gmannso we are now done with wallaby things as QA14:08
kopecmartinthat's great!14:08
kopecmartin#topic Wallaby Priority Items progress14:09
*** openstack changes topic to "Wallaby Priority Items progress (Meeting topic: qa)"14:09
kopecmartin#link https://etherpad.opendev.org/p/qa-wallaby-priority14:09
kopecmartinany updates?14:09
kopecmartinPatrole stable release - gmann14:09
gmannwe did the release for wallaby14:09
kopecmartinwhat about this? we did release patrole, right?14:09
kopecmartinyeah, so can be considered that done?14:10
gmannfor wallaby yes14:10
kopecmartinperfect14:10
gmannand for next cycle we can discuss in PTG about how to run it on project side gate or so14:10
gmannor if anything pending14:10
*** Luzi has joined #openstack-qa14:10
kopecmartinsure, let me write a note somewhere14:10
gmannI think I have added it in PTG ethrpad14:11
kopecmartinright, i see it now14:11
kopecmartinthe second day, first topic14:11
gmannyeah14:11
paras333kopecmartin: can we move run_validation and guest image one to xena cycle, I still have to do some work which I missed in last cycle14:12
paras333?14:12
kopecmartinsure14:12
paras333kopecmartin: ok, thanks14:12
gmannparas333: we will move all those backlog to Xena which are not finished and still needed14:12
paras333gmann: ok great, thank you!14:12
kopecmartinyeah, once tracked, forever tracked (until finished haha)14:12
gmannL70 in PTG etherpad14:12
paras333lol14:13
kopecmartinl70 but might be moved to the last day, we'll see14:13
kopecmartin#topic Gate Status Checks14:13
*** openstack changes topic to "Gate Status Checks (Meeting topic: qa)"14:13
kopecmartinany blockers or urgent patches?14:13
gmannwe had stable gate blocked, but merged the fixes except stable/train in gate14:14
gmannmaster gate seems green.14:14
gmannthis thread14:15
gmann#link http://lists.openstack.org/pipermail/openstack-discuss/2021-April/021787.html14:15
*** yamamoto has quit IRC14:15
kopecmartin#link https://review.opendev.org/q/If5f14654ab9aee2a140bbfb869b50d63cb289fdf14:16
kopecmartinthere are the patches, just train pending , as you sai14:16
kopecmartinsaid14:16
kopecmartingood then14:16
gmannyeah14:16
kopecmartin#topic Periodic jobs Status Checks14:17
*** openstack changes topic to "Periodic jobs Status Checks (Meeting topic: qa)"14:17
kopecmartin#link https://zuul.openstack.org/builds?job_name=tempest-all&job_name=tempest-full-oslo-master&pipeline=periodic14:17
kopecmartinseems ok14:17
kopecmartin#topic Sub Teams highlights (Sub Teams means individual projects under QA program)14:18
*** openstack changes topic to "Sub Teams highlights (Sub Teams means individual projects under QA program) (Meeting topic: qa)"14:18
kopecmartin#link https://review.openstack.org/#/q/project:openstack/tempest+status:open14:18
kopecmartin#link https://review.openstack.org/#/q/project:openstack/patrole+status:open14:18
kopecmartin#link https://review.openstack.org/#/q/project:openstack/devstack+status:open14:18
kopecmartin#link https://review.openstack.org/#/q/project:openstack/grenade+status:open14:18
kopecmartin#link https://review.opendev.org/#/q/project:openstack/hacking+status:open14:18
kopecmartina lot of patches as always14:20
kopecmartini see several new ones, will try to review as many as possible14:20
kopecmartin#topic Gate Blocker Fix / Urgent Change14:20
*** openstack changes topic to "Gate Blocker Fix / Urgent Change (Meeting topic: qa)"14:21
kopecmartinany other blockers than those we have discussed already?14:21
gmannnothing afaik14:21
yoctozeptoironic has some weird issue though14:21
gmannin grenade job?14:22
yoctozeptoyeah14:22
yoctozeptoon their branch14:22
yoctozeptoit works on grenade14:22
yoctozeptobut not on ironic14:22
yoctozeptoiurygregory observed this14:22
yoctozeptosean-k-mooney found out it's due to not updating non-ironic projects14:22
paras333yoctozepto: what's the job and error?14:22
yoctozeptowhich I confirmed14:22
yoctozeptoyet it works on grenade14:22
gmannoh, they upgrade only ironic?14:23
yoctozepto13:54:23 <iurygregory> this may be a bit weird, but ironic-grenade in stable/victoria is broken anyone aware of changes that could cause nova to raise "ImportError: cannot import name 'reraise'" ?14:23
yoctozepto13:55:00 <iurygregory> funny thing is that in grenade we have 4 runs that are green https://zuul.opendev.org/t/openstack/builds?job_name=ironic-grenade&project=openstack%2Fgrenade&branch=stable%2Fvictoria14:23
yoctozepto13:55:16 <iurygregory> but in ironic we are still red https://zuul.opendev.org/t/openstack/builds?job_name=ironic-grenade&project=openstack%2Fironic&branch=stable%2Fvictoria14:23
yoctozepto13:55:34 <iurygregory> always the same problem in nova it seems to me https://zuul.opendev.org/t/openstack/build/432889791d2e4894956b5b3a3adbb6e3/log/controller/logs/screen-n-api.txt?severity=414:23
gmannyeah i see on grenade patches, no issue14:23
yoctozeptogmann: not quite on purpose it seems14:23
yoctozepto(psst, the times are utc+2)14:23
gmannat least nova should be upgraded14:23
iurygregorygmann, we have a filter to remove nova when using multinode grenade job only14:23
gmanniurygregory: yoctozepto but did not get what is difference between job running grenade vs ironic gate14:24
gmannit is same job or different version?14:24
yoctozeptoI did not either14:24
iurygregorygmann, should be the same .-.14:25
iurygregorybut it's weird that it works in the grenade gate14:25
iurygregorybut not in ironic14:25
rpittauso14:25
gmanniurygregory: and that is only for stable/victoria or master also?14:25
iurygregorygmann, victoria only14:25
iurygregorymaster and wallaby seems to be fine14:25
rpittaugmann, iurygregory, I was checking the logs and grenade config seems a bit different, at least apparently, one is runninng on summary mode only and saving logs in the main log dir, while the ironic one is running on verbose and saving the logs under the "old" dir14:26
rpittauis there maybe something overriding the config/14:26
rpittau?14:26
iurygregoryhumm I see some failures in master and wallaby also now D:14:27
iurygregoryops master*14:27
iurygregorywallaby is green https://zuul.opendev.org/t/openstack/builds?job_name=ironic-grenade&project=openstack%2Fironic&branch=stable%2Fwallaby14:27
gmannyeah https://zuul.opendev.org/t/openstack/builds?job_name=ironic-grenade14:27
rpittaugrenade in ironic is saving the logs under controller/logs/old/devstacklog.txt14:27
gmannok, may be let's check after office hour14:28
gmanncheck/debug14:28
iurygregoryack14:28
rpittauwhile grenade in grenade is saving under controller/logs/grenade.sh.log_summary.txt14:28
rpittauok14:28
*** amodi has quit IRC14:29
kopecmartin#topic Open Discussion14:29
*** openstack changes topic to "Open Discussion (Meeting topic: qa)"14:29
kopecmartin(gmann) is it fine to make devstack-plugin-container branchless14:29
kopecmartinthis got resolved since last week14:29
gmannkopecmartin: that is all sorted out14:30
kopecmartingood, i'll remove it from the agenda14:30
gmannyeah, it will continue as branched as per plugin team14:30
kopecmartinanyone anything else?14:30
gmannnothing from me14:31
*** lbragstad_ is now known as lbragstad14:31
kopecmartin#topic Bug Triage (last 30 min)14:32
*** openstack changes topic to "Bug Triage (last 30 min) (Meeting topic: qa)"14:32
kopecmartin#link https://etherpad.opendev.org/p/qa-bug-triage-wallaby14:32
yoctozeptowhat about the ceph plugin14:32
kopecmartinyoctozepto: what about it?14:32
yoctozeptothey would prefer to branch :-)14:32
yoctozeptocopying it from agenda:14:33
yoctozepto(yoctozepto) or is it better to branch devstack-plugin-ceph to help the consumers avoid headaches? https://review.opendev.org/c/openstack/devstack-plugin-ceph/+/783844 AND https://review.opendev.org/c/openstack/devstack-plugin-ceph/+/78285514:33
kopecmartinoh , i see you added the topic in the agenda, sorry, i missed it, the page didn't rendered correctly for me14:33
yoctozeptono problem14:34
* kopecmartin loading link, trying to catch up14:35
kopecmartinare there any points against branching?14:35
yoctozeptonot really14:36
yoctozeptocc tbarron, vkmc on devstack-plugin-ceph branching14:36
kopecmartini see that the plugin is currently tracked as project with no release14:37
gmannyeah, I think branching will be good but only thing is we need to take care of branch cut for every release.14:37
gmannalso gouthamr: ^^14:37
tbarronI don't know of any reason not to branch.  We talked with gouthamr about it yesterday (still early where he is).14:37
kopecmartinnot considering tags?14:37
tbarronWould the worry be that it's more work than tags?14:38
kopecmartinno14:38
vkmco/ +1 branching devstack-plugin-ceph14:38
vkmcwhy tags instead of branches?14:39
tbarronThe plugin is pretty ugly b/c of all the logic that has to be inserted everywhere to check on the ceph version, or openstack version.14:39
gmannyeah branch is much better than tag14:39
gmanntbarron: exactly14:39
openstackgerritMerged openstack/grenade stable/train: Fix Grenade DSVM Tempest upper constraints  https://review.opendev.org/c/openstack/grenade/+/78576714:39
kopecmartini agree14:39
tbarronI think branches would allow cleaner development branch and eventually the old mess would get retired.14:39
vkmcthe support matrix has grown and it's hard to keep things sane without branching14:39
gmanndo we want it from wallaby or Xena ?14:40
vkmcwe have to keep compatibility between openstack version, ceph version, distro version and also nfs ganesha14:40
tbarronIt would be nice to be able to use e.g. cephadm for current deployments but fitting that into the existing logic would be really messy.14:40
gmanni think we still have time for wallaby branch cut but need to check with release team14:40
gmannyoctozepto: can you remove -W from this now, grenade one merged - https://review.opendev.org/c/openstack/devstack/+/78554714:41
yoctozeptogmann: as it's a dev thing, I think release will not mind14:41
yoctozeptosure thing14:41
gmannyeah. but we should do within today or tomorrow max14:42
yoctozeptoI mean, I doubt they would mind branching after a coordinated release as these are not really parts of a release per se ;-)14:44
yoctozeptoneed to amend gates to use branches here14:44
yoctozeptowill you ask hberaud then?14:44
yoctozeptoanyone here? did I get a netsplit again?14:47
tbarronyoctozepto: i see you14:47
yoctozeptoI see you too14:47
kopecmartini'm here14:47
gmannyoctozepto: tbarron we need to update governance also first14:47
yoctozeptoyes, the governance needs updating here14:48
yoctozeptothough it's mostly cosmetic14:48
gmannwho is volunteering to push patches?14:48
yoctozeptoI can but later14:49
kopecmartini can help too14:49
kopecmartinok, let's sync after the office hour14:50
*** yamamoto has joined #openstack-qa14:50
kopecmartinso bug triage14:50
gmanngreat, we need to push patch today in order 1. governance change 2. release patch to cut the branch 3.  plugin repo side things14:50
kopecmartin#link https://etherpad.opendev.org/p/qa-bug-triage-wallaby14:50
kopecmartingmann: sure, i'll have a look and will ping you probably with questions14:51
gmannif no other high priority bug then we can discuss ironic grenade for 10 min?14:51
gmannkopecmartin: great, thakns14:51
kopecmartinnothing high priority14:51
kopecmartingo ahead14:51
*** Luzi has quit IRC14:51
gmanniurygregory: rpittau yoctozepto ironic greande on master is different issue - https://zuul.opendev.org/t/openstack/build/a0be17b445a24f2f9688f7bfcf62e078/log/controller/logs/grenade.sh_log.txt#156114:52
gmannNo Image found for cirros-0.5.1-x86_64-uec14:52
gmannseems infra issue?14:52
yoctozeptooh my, the fun never ends14:52
rpittaumost likely14:52
gmannchecking for victoria14:52
rpittauI'm more concerned on victoria14:52
yoctozeptoyou think some images got reverted?14:52
yoctozeptothe image should be hardcoded in them14:53
yoctozeptoperhaps some cloud did not refresh images?14:53
rpittauAFAIK they're cached, maybe cache has not been updated14:53
clarkbthe cirros images are cached in the infra test nodes, but devstack/grenade should also download them if not cached14:54
clarkblooking at that log the failure is at the upload step though14:54
rpittauoh ok14:54
clarkber sorry even after the upload step14:54
clarkbit is doing a server create14:54
yoctozeptoso it's glance that did not get them then14:54
gmannyeah14:55
rpittauyeah14:55
yoctozeptoyeah14:55
clarkbhttps://zuul.opendev.org/t/openstack/build/a0be17b445a24f2f9688f7bfcf62e078/log/controller/logs/grenade.sh_log.txt#453-467 the image is on the node14:55
gmanndoes ironic jobs perform glance upgrade or skipped?14:55
rpittauit register it for upgrade14:58
rpittauso in general it should14:58
gmannrpittau: did you try enabling nova in upgrade on stable/victoria and see if that work ?14:59
rpittaugmann: err shouldn't that happen already? in grenade itworks fine15:00
rpittauthe ironic-grenade job in grenade works correctly, in ironic it doesn't15:00
gmannrpittau: grenade gate run same job so it should have nova not in upgrade too same as on ironic gate15:00
gmannyeah that is not known why15:01
rpittauI was saying before that it seems like there is something overriding the config15:01
*** yamamoto has quit IRC15:01
yoctozeptogmann: it's not multinode15:02
yoctozeptoironic does not disable nova in single node15:02
yoctozeptoit's confusing15:02
gmannah15:02
yoctozeptoall right, we are past time15:02
yoctozeptoI will start pushing the patches later in the evening15:02
yoctozeptoso if someone wishes to start earlier15:03
yoctozeptobe my guest ;-)15:03
gmannthanks15:03
kopecmartinyoctozepto: i'll try, but it's almost end of my day15:03
kopecmartinso maybe you will need to finish them15:03
kopecmartin:)15:03
kopecmartinlet's close the office hour15:03
kopecmartinthank you all15:03
gmannkopecmartin: can you re-W it to push it on gate pipeline https://review.opendev.org/c/openstack/devstack/+/78554715:03
kopecmartin#endmeeting15:03
*** openstack changes topic to "Discussion of OpenStack Quality Assurance Projects (tempest, devstack, grenade, and friends) | docs https://wiki.openstack.org/wiki/QA | specs https://review.openstack.org/#/q/project:openstack/qa-specs | channel logs http://eavesdrop.openstack.org/irclogs/%23openstack-qa/"15:03
openstackMeeting ended Tue Apr 13 15:03:53 2021 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)15:03
openstackMinutes:        http://eavesdrop.openstack.org/meetings/qa/2021/qa.2021-04-13-14.00.html15:03
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/qa/2021/qa.2021-04-13-14.00.txt15:03
openstackLog:            http://eavesdrop.openstack.org/meetings/qa/2021/qa.2021-04-13-14.00.log.html15:03
kopecmartingmann: sure, done15:04
gmannthanks15:05
*** jhouser has joined #openstack-qa15:10
openstackgerritJames Parker proposed openstack/whitebox-tempest-plugin master: Test allow disabling CPU flags  https://review.opendev.org/c/openstack/whitebox-tempest-plugin/+/77611215:10
*** lucasagomes has quit IRC16:00
*** gcheresh has quit IRC16:02
*** lpiwowar has quit IRC16:12
*** Yarboa has quit IRC16:14
*** Yarboa has joined #openstack-qa16:17
*** hamalq has joined #openstack-qa16:20
*** rpittau is now known as rpittau|afk16:24
*** rcernin has joined #openstack-qa16:32
*** rcernin has quit IRC16:37
*** jpena is now known as jpena|off16:43
*** ccamposr__ has quit IRC16:59
*** ccamposr__ has joined #openstack-qa16:59
*** dtantsur is now known as dtantsur|afk17:11
*** gfidente is now known as gfidente|afk17:16
*** gcheresh has joined #openstack-qa17:30
kopecmartingmann: i moved the devstack-plugin-ceph from "Project with no release" to "Projects with only Branches" https://wiki.openstack.org/wiki/QA/releases17:39
kopecmartinanything else we should do? (apart from the governance and releases patch)17:39
gmannkopecmartin: perfect, thanks17:40
gmannkopecmartin:I have +A on governance patch17:40
gmannlet's wait for release patch to merge17:40
*** gcheresh has quit IRC17:42
yoctozeptowe need to work on the gate after branching17:57
*** eolivare has quit IRC18:00
toskywill the branches start from wallaby, or will they be retrospectively created for the other branches?18:07
gmannfrom wallaby18:34
gmannyoctozepto: kopecmartin these are backport for grenade job voting  https://review.opendev.org/q/Ic747ac9ddbb21a01e9dc18d8e8ad324d4d7d050d18:35
kopecmartin+218:37
gmannyoctozepto: any specific things we need to care on gate for ceph plugin branch? current stable jobs need to be there as we are starting branch since wallaby only https://github.com/openstack/devstack-plugin-ceph/blob/master/.zuul.yaml18:38
gmannkopecmartin: thanks18:38
gmannyoctozepto: ah, those stable branch we need to remove from wallaby gate18:38
*** yamamoto has joined #openstack-qa18:38
gmannany thing else other than that?18:38
toskygmann: remove them from master, or remove from wallaby? Uhm, shouldn't the wallaby branch be used for <=wallaby?18:40
toskyso master will be used only for xena, and it doesn't need to test older branches18:40
gmanntosky: if we do that we might need a lot of change where ever this plugin is used18:41
toskygmann: but then it doesn't make sense to use master for master and <wallaby18:41
toskyit kind of defeats the purpose of branching18:41
toskybranching is for stabilize the old branches18:41
gmannbut at the same time if using master for <wallaby it will block plugins to do master version specific things18:42
toskymaybe I didn't explain it properly18:42
gmannwell we are branching only from wallaby, using wallaby for older branch will be hack we can do18:42
*** yamamoto has quit IRC18:43
toskythat's why I was asking, maybe it would make sense to artifically create branches for all releases18:43
toskyno special messing around18:43
gmannhumm18:44
gmannand with same hash we can do as they are compatible for current stables.18:44
gmanntosky: I think that will be easy way, i am ok with that approach.18:46
gmanntbarron: gouthamr vkmc yoctozepto what you say? should we cut all stable branches (a least until stable/train) for devstack-plugin-ceph from same current master hash?18:46
tbarrongmann: I think that makes sense, they are presumably working now and if not it's not like we know a better hash point18:48
tbarrongouthamr: vkmc yoctozepto ^^ ?18:48
gmannyeah, same hash is working currently so it would not cause any issue18:49
*** mailingsam has quit IRC18:51
vkmc+1 from me18:57
gouthamr+1 that works, since we put all the effort into being branchless when working on all these releases we care about19:02
*** gcheresh has joined #openstack-qa19:07
*** ramishra has quit IRC19:10
*** ramishra has joined #openstack-qa19:10
*** ralonsoh has quit IRC19:13
*** ajitha has quit IRC19:46
*** gcheresh has quit IRC19:52
*** vishalmanchanda has quit IRC19:55
toskywe may need also the older branches which are still active, or some kind of mangling will be needed too (stein, rocky, queens; sigh; at least both cinder and manila dropped pike)20:17
* kopecmartin edited the patch so that it adds the branches for all currently supported releases: https://review.opendev.org/c/openstack/releases/+/78606920:35
*** nweinber has quit IRC20:40
*** sboyron has quit IRC20:46
* gouthamr hasn't checked the state of the manila gate for queens, rocky and stein - but is sure they're broken21:09
toskygouthamr: but manila is not the only user of the plugin21:11
tbarrontosky: gouthamr yeah which older stable branches are used by cinder/nova/glance?21:12
toskyfor cinder, queens, rocky and stein21:12
gouthamri agree tosky, is a grenade job likely to be impacted as well?21:12
toskygouthamr: the policy so far is that grenade is best effort and can be dropped anytime, but yes, anything which deploys with the plugin21:13
tbarrontosky: and today they are using the main branch, so we can just cut from there for q,r.s, as well, right?21:13
gouthamrso is your recommendation that https://review.opendev.org/c/openstack/releases/+/786069 include branches through queens?21:13
tbarrongouthamr: manila won't work on those either way but so what21:14
*** vhari has quit IRC21:14
toskytbarron: that's the idea21:14
tbarronnobody will be using pike21:14
toskygouthamr: correct, also branch for those21:14
toskycinder has dropped pike so far21:14
toskyiirc also nova, let me see21:14
gouthamrokay, not opposed to it at all - its best effort CI for these q,r,s across projects/jobs  - but if the next change to devstack-plugin-ceph is going to break something on these jobs because we don't have a stable branch, seems like a smaller price of cutting the branches right away21:15
toskynova dropped pike: http://lists.openstack.org/pipermail/openstack-discuss/2020-July/015798.html21:15
toskyand cinder dropped pike too (I can't find the email but I clearly remember it)21:16
tbarrontosky: I seem to remember that too21:17
gmannsupporting all is not feasible i think21:17
tbarrongmann: but cinder and nova have to support them all today, just on the main dev branch, right?  It should be easier for them if there are stable branches.21:19
tbarrongmann: or are you thinking about the administrative overhead of managing releases?21:19
toskyfound it: http://lists.openstack.org/pipermail/openstack-discuss/2020-July/015838.html21:20
gmanntbarron: its both.21:20
toskygmann: but all those old releases won't have new tags21:20
toskyhaving a branch for each version (a simple ref in the repository) allows us to run the jobs against a specific commit without special rules21:21
toskyit's consistency21:21
gmannwe can do for all supported branch and for EM branch we can see if they are broken by what they are using currently then fix21:21
toskybut the fix would be in master21:21
toskyagain, we would have master supporting master openstack and then stein, rocky and queens21:22
gmannthat is how we Tempest policy for stable brnahc support21:22
toskywe pin tempest21:22
gmannonce they start breaking21:22
gmannsame way we can use last cut branch say ussuri21:22
toskybut tempest starts pinning from the last support branch21:22
toskyhere we would have gaps21:22
toskythat's totally different than the tempest case21:23
toskyat some point, later this year, tempest will stop supporting train, and when it breaks we will ping train, but all older openstack releases use an older tag21:23
toskyand newer openstack releases will still use master21:24
toskyhere we have a plugin which support master and some stable branches21:24
toskywhile others have a dedicated branch21:24
toskythis is just about creating a ref in devstack-plugin-ceph; if something breaks, only the affected branch will need a fix (or it will be abandoned)21:25
gmannso who is going to support pike/queens/rocky/stein which are all EM state?21:25
gmanndo we have Extended maintenance team somewhere?21:25
toskyI'm not sure I get it21:25
tbarronwouldn't we follow the devstack model here (which has stable branches) rather than tempest?  Who supports devstack stable/queens?21:26
gmannall Extended maintenance stable branch are supposed to be maintained by Extended maintenance team.21:26
toskywhat if someone want to fix a broken queens job which uses devstack-plugin-ceph from master?21:26
toskyright, but if you don't branch, you prevent that21:26
toskyyou would have to fix things in master (or block changes in master) to fix queens, if you want to do so, and that would be impossible21:27
toskyand that would mean abandoning that branch - which I may agree with, but that's beside the point21:27
gmannwe lives in a very unclear model of Extended maintenance branches and ADDING UP one more stable branch to support every release21:27
gmannand most of the time QA team end up fixing all those21:27
toskywe don't need to support21:27
toskyso please answer the question above: what happens if somehting breaks in a devstack-plugin-ceph job in stain21:28
toskyand you need to fix it in master?21:28
tbarrongmann: I think the teams who think they need EM branches should support rather than putting it on QA team21:28
toskyI'm not explaining myself21:28
gmannyeah ideally that should be case but its not actully21:28
toskynothing I'm saying is going to put more burden than what it is now on the QA team21:29
gmannso I will say if team need then they cut the branch and then we can maintain21:29
*** jhouser has quit IRC21:29
gmannuntil then we are not breaking any EM branch they are keep working21:29
toskyso you are saying that queens job will work with devstack-plugin-ceph from master21:29
*** vhari has joined #openstack-qa21:29
toskythat doesn't make any sense21:29
toskyat all21:29
gmannyes, it work now right?21:29
toskyit's inconsistent21:30
gmannand if anyone want to maintain it separately then we cut the branch21:30
tbarrongmann: but we want to move dev branch ahead without having to worry about breaking stable/queens21:30
toskythen create a tag21:30
toskywhatever21:30
toskyit doesn't make any sense, it doesn't and it wont21:30
toskyand having a branch more won't put more burden21:30
toskyif the QA team doesn't want to support an old branch, they won't fix as it is now21:31
gmanntbarron: it will be overhead to maintain the 8 stable branch for devstack-plugin-ceph team, that is my experience from devstack21:31
toskygmann: again, what will it happen if somethig breaks on stein and you need to fix it but the change will complicate the code again?21:31
gmanntosky: we cut the branch that time21:32
gmannf there is someone there to maintain it21:32
toskygmann: and why not now?21:32
gmann*if there21:32
toskythe difference is that it won't break for sure21:32
toskyif you branch it now when it work, you can stop caring for it21:32
toskyif it breaks for other reasons, and someone want to fix it,that's up to them21:33
toskymy point is that it's not more complicated than it is now21:33
*** ramishra has quit IRC21:33
gmanntosky: because we do not have EM maintainer for devstack-plugin-ceph21:33
gmannif we have name now then I am ok.21:33
toskygmann: it's not different21:33
toskythan what we have now21:33
toskywhy what I described above doesn't apply?21:33
toskyyou will most likely need to branch from the same branch where we are branching now21:34
toskyhaving a branch doesn't make things more complicated21:34
gmannI am fine if you want to cut and maintain. I am sharing my opinion as per my experience for devstack "we keep all branch in EM with hoping if they break EM team will fix BUT QA team end up fixing them most of the time"21:37
toskygmann: well, it was elod who fixed a lot of breakages so fatr21:37
toskyso I hold my point: having the branches now doesn't increase the load on the QA team more than what it is21:38
gmanntosky: no, it is elod or I or yoctozepto or frickler21:38
toskyit just makes the life of whoever who wants to move forward easier21:38
gmannsure, you can try.21:38
toskywell, I mentioned elod because he especially advocated for keeping alive branches I would have buried ages ago21:39
gmannI have spent lot of time on grenade and Temepest constraints fixing for EM branch21:39
toskyso let's say a change to devstack-plugin-ceph breaks stein one week for now: which are the conditions under which you will allow the creation of a branch?21:39
gmannand rocky with py2 is still nightmare to maintain21:40
gmannbut we are maintaining it for many plugins or so21:40
toskyyou are saying that with an open branch there is an expectation that the QA team will fix a breakage in those older branch for devstack-plugin-ceph21:41
gmannpractically yes21:41
gmannthats what happen in devstack21:41
gmanngrenade we stopped running on EM for that reason21:42
toskyand can't you simply state it clearly that those older devstack-plugin-ceph branches are not under QA team responsibility?21:42
gmanntwo clear practical example.21:42
toskya comment in the job description?21:42
toskywouldn't that address the problem?21:42
gmannwe already have that in stable doc. i remember the forum sessions agreement for that when EM model was introduced21:43
gmannanyways, you can cut them now and see if that work or not.21:44
gmannI do not elp much on devstack-plugin-ceph so should not decide it, instead devstack-plugin-ceph team should. just sharing the practicality of what EM branches will be in term of maintenance21:45
gmann*help21:45
openstackgerritMerged openstack/devstack stable/train: Make stackviz tasks not to fail jobs  https://review.opendev.org/c/openstack/devstack/+/78554721:57
*** yamamoto has joined #openstack-qa22:00
gmannfinally merged, one tab closed \o/22:02
gouthamrgmann++ thankless work :)22:06
gouthamrgmann tosky: we're going to do a bit of thinking at the PTG in the manila group as far as rationalizing investment on the EM branches as we do this every cycle; so we may sign up right away to maintain EM branches on devstack-plugin-ceph; if something gives, i don't mind chasing spot fixes to the CI to override the checkout and consider using the stable/train branch on EM branches as appropriate22:10
gouthamrs/we may sign up/may not sign up - big difference22:10
gmannah I thought we got the maintainer before you correct the *not* :P22:11
tbarroni think the people who care about the EM branches should take responsibility for them but cutting them and then declaring them EOL if they don'22:12
tbarront step up could make sense :D22:12
tbarronbut it's not the main issue, really appreciate the work for the current stable branches!22:13
tbarrongmann++22:13
gmannsure. that work. that clear the expectation on "there is no default maintainers". adding it on ML or so will work for notification.22:15
*** tkajinam has joined #openstack-qa22:23
*** yamamoto has quit IRC22:41
*** yamamoto has joined #openstack-qa22:42
*** rcernin has joined #openstack-qa22:52
*** rcernin has quit IRC22:54
*** hyang has joined #openstack-qa22:54
*** rcernin has joined #openstack-qa22:54
*** ramishra has joined #openstack-qa22:55
*** hyang has quit IRC22:57
*** tosky has quit IRC23:34
*** ramishra has quit IRC23:56
*** ajitha has joined #openstack-qa23:56

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!