*** mailingsam has quit IRC | 00:19 | |
*** enriquetaso has joined #openstack-qa | 00:28 | |
*** lbragstad_ has joined #openstack-qa | 00:43 | |
*** lbragstad has quit IRC | 00:45 | |
*** paras33__ has quit IRC | 00:51 | |
*** hamalq has quit IRC | 01:19 | |
*** ajitha has joined #openstack-qa | 01:57 | |
*** paras333 has joined #openstack-qa | 02:31 | |
*** paras333 has quit IRC | 02:35 | |
*** enriquetaso has quit IRC | 02:40 | |
openstackgerrit | Merged openstack/grenade master: Fix Grenade DSVM Tempest upper constraints https://review.opendev.org/c/openstack/grenade/+/785831 | 02:48 |
---|---|---|
*** rcernin has quit IRC | 02:58 | |
openstackgerrit | Merged openstack/grenade stable/wallaby: Fix Grenade DSVM Tempest upper constraints https://review.opendev.org/c/openstack/grenade/+/785608 | 02:59 |
openstackgerrit | Merged openstack/grenade stable/victoria: Fix Grenade DSVM Tempest upper constraints https://review.opendev.org/c/openstack/grenade/+/785609 | 02:59 |
openstackgerrit | Merged openstack/grenade stable/ussuri: Fix Grenade DSVM Tempest upper constraints https://review.opendev.org/c/openstack/grenade/+/785610 | 02:59 |
*** vishalmanchanda has joined #openstack-qa | 03:17 | |
*** rcernin has joined #openstack-qa | 03:19 | |
*** rcernin has quit IRC | 03:37 | |
*** rcernin has joined #openstack-qa | 03:37 | |
openstackgerrit | Merged openstack/grenade master: Make heat and octavia grenade jobs as voting https://review.opendev.org/c/openstack/grenade/+/785938 | 04:10 |
*** gcheresh has joined #openstack-qa | 04:40 | |
*** akahat has quit IRC | 05:02 | |
*** kopecmartin has quit IRC | 05:03 | |
*** kopecmartin has joined #openstack-qa | 05:06 | |
*** akahat has joined #openstack-qa | 05:14 | |
*** ralonsoh has joined #openstack-qa | 05:29 | |
*** ralonsoh has quit IRC | 05:29 | |
*** ralonsoh has joined #openstack-qa | 05:30 | |
*** Luzi has joined #openstack-qa | 05:50 | |
*** slaweq has joined #openstack-qa | 05:52 | |
*** sboyron has joined #openstack-qa | 06:08 | |
*** lpiwowar has joined #openstack-qa | 06:21 | |
openstackgerrit | Merged openstack/devstack-gate master: Update grenade setting for stable/wallaby https://review.opendev.org/c/openstack/devstack-gate/+/785010 | 06:27 |
*** rcernin has quit IRC | 06:38 | |
*** rcernin has joined #openstack-qa | 06:38 | |
*** eolivare has joined #openstack-qa | 06:46 | |
*** rcernin has quit IRC | 06:55 | |
*** jpena|off is now known as jpena | 07:34 | |
*** sboyron has quit IRC | 07:37 | |
*** sboyron has joined #openstack-qa | 07:38 | |
*** sboyron has quit IRC | 07:38 | |
*** sboyron has joined #openstack-qa | 07:39 | |
*** rpittau|afk is now known as rpittau | 07:43 | |
*** tosky has joined #openstack-qa | 07:49 | |
*** rcernin has joined #openstack-qa | 07:56 | |
*** rcernin has quit IRC | 08:00 | |
*** gfidente|afk is now known as gfidente | 08:06 | |
*** lucasagomes has joined #openstack-qa | 08:08 | |
*** rcernin has joined #openstack-qa | 08:08 | |
*** rcernin has quit IRC | 08:19 | |
*** rcernin has joined #openstack-qa | 08:24 | |
*** hberaud has left #openstack-qa | 08:25 | |
*** dtantsur|afk is now known as dtantsur | 08:31 | |
openstackgerrit | Lajos Katona proposed openstack/tempest master: Add live migration with trunk test https://review.opendev.org/c/openstack/tempest/+/774689 | 08:35 |
*** Luzi has quit IRC | 08:47 | |
*** Luzi has joined #openstack-qa | 08:48 | |
*** dtantsur is now known as dtantsur|brb | 08:57 | |
*** rcernin has quit IRC | 08:58 | |
*** kopecmartin has quit IRC | 09:28 | |
*** akahat has quit IRC | 09:28 | |
*** kopecmartin has joined #openstack-qa | 09:33 | |
masayukig | kopecmartin: do we have an office hour today? I'd like to skip it because I'm on leave this week.. | 09:46 |
kopecmartin | masayukig: we should have .. np, i can chair it | 09:47 |
kopecmartin | enjoy PTO :) | 09:47 |
*** yamamoto has quit IRC | 09:48 | |
masayukig | kopecmartin++ thank you so much! :) | 09:49 |
*** rcernin has joined #openstack-qa | 09:50 | |
*** yamamoto has joined #openstack-qa | 09:51 | |
*** akahat has joined #openstack-qa | 09:53 | |
*** rpittau is now known as rpittau|bbl | 09:54 | |
*** sboyron has quit IRC | 09:54 | |
*** sboyron has joined #openstack-qa | 09:55 | |
*** rcernin has quit IRC | 09:56 | |
*** yamamoto has quit IRC | 09:58 | |
*** dtantsur|brb is now known as dtantsur | 10:00 | |
yoctozepto | argh that reliability of train gate | 10:10 |
*** rcernin has joined #openstack-qa | 10:14 | |
*** rcernin has quit IRC | 10:20 | |
*** yamamoto has joined #openstack-qa | 10:28 | |
*** yamamoto has quit IRC | 10:42 | |
openstackgerrit | Lajos Katona proposed openstack/tempest master: Add live migration with trunk test https://review.opendev.org/c/openstack/tempest/+/774689 | 10:48 |
*** chaconpiza has joined #openstack-qa | 10:59 | |
*** tkajinam has quit IRC | 11:13 | |
*** yamamoto has joined #openstack-qa | 11:18 | |
*** yamamoto has quit IRC | 11:29 | |
*** jpena is now known as jpena|lunch | 11:31 | |
*** enriquetaso has joined #openstack-qa | 11:52 | |
iurygregory | this may be a bit weird, but ironic-grenade in stable/victoria is broken anyone aware of changes that could cause nova to raise "ImportError: cannot import name 'reraise'" ? | 11:54 |
iurygregory | funny thing is that in grenade we have 4 runs that are green https://zuul.opendev.org/t/openstack/builds?job_name=ironic-grenade&project=openstack%2Fgrenade&branch=stable%2Fvictoria | 11:55 |
iurygregory | but in ironic we are still red https://zuul.opendev.org/t/openstack/builds?job_name=ironic-grenade&project=openstack%2Fironic&branch=stable%2Fvictoria | 11:55 |
iurygregory | always the same problem in nova it seems to me https://zuul.opendev.org/t/openstack/build/432889791d2e4894956b5b3a3adbb6e3/log/controller/logs/screen-n-api.txt?severity=4 | 11:55 |
iurygregory | if anyone has some ideas please let me know =) | 11:56 |
*** ramishra has quit IRC | 12:02 | |
*** yamamoto has joined #openstack-qa | 12:02 | |
*** yamamoto has quit IRC | 12:06 | |
*** ramishra has joined #openstack-qa | 12:08 | |
*** artom has joined #openstack-qa | 12:15 | |
*** nweinber has joined #openstack-qa | 12:29 | |
*** rpittau|bbl is now known as rpittau | 12:31 | |
*** jpena|lunch is now known as jpena | 12:37 | |
*** paras333 has joined #openstack-qa | 12:52 | |
*** paras333 has quit IRC | 12:55 | |
*** paras333 has joined #openstack-qa | 12:56 | |
yoctozepto | iurygregory: is not it the thing that sean-k-mooney commented on #openstack-nova yesterday? | 13:02 |
yoctozepto | that ironic stopped upgrading non-ironic projects | 13:02 |
iurygregory | I'm not sure, I probably missed that =( | 13:04 |
iurygregory | i would say it could be, but is strange that the job has success in grenade but not in ironic.. | 13:05 |
paras333 | gmann: kopecmartin: did we cancel today's meeting? | 13:06 |
paras333 | sorry I was on vacation in India and returned last week only. | 13:07 |
yoctozepto | paras333: 53 minutes pending till meeting | 13:07 |
paras333 | yoctozepto: Ohh , So it moved to 10 am EST? | 13:07 |
yoctozepto | iurygregory: not strange if ironic job upgrades only ironic and its deps | 13:07 |
yoctozepto | and these upgraded deps break nova | 13:07 |
yoctozepto | quite expected in our release model | 13:07 |
yoctozepto | paras333: it's always 14:00 UTC ;-) | 13:08 |
yoctozepto | and it's 14 UTC in 51 minutes | 13:09 |
yoctozepto | :-) | 13:09 |
paras333 | yoctozepto: yeah , it got extended to 10 due to daylight saving here , nvm thanks | 13:10 |
yoctozepto | happy to help :-) | 13:10 |
iurygregory | yoctozepto, but grenade project uses the ironic job definition so it should also fail no? | 13:10 |
yoctozepto | iurygregory: hmm, well, with Sean we analysed the job on the ironic side | 13:11 |
yoctozepto | that's interesting, yes | 13:11 |
yoctozepto | hmm, hmm | 13:12 |
*** yamamoto has joined #openstack-qa | 13:14 | |
iurygregory | maybe we should move required projects to "opendev.org/openstack/******" | 13:14 |
iurygregory | not sure if it would have impact | 13:14 |
*** mailingsam has joined #openstack-qa | 13:15 | |
mailingsam | gmann regarding the network delete, the networks are being deleted, so you think waiting for network delete is unnecessary? | 13:15 |
yoctozepto | iurygregory: well, the same job should work the same unless it specifically has a conditional on the project that it runs from | 13:16 |
iurygregory | and if I do remember we only skip nova upgrade in multinode... | 13:16 |
iurygregory | agree =) | 13:17 |
*** tosky has quit IRC | 13:19 | |
yoctozepto | iurygregory: well, if you skip nova upgrade and nova fails... that's precisely the reason :D | 13:22 |
yoctozepto | but the fact it works on grenade and not ironic is... confusing | 13:23 |
iurygregory | only in the multinode job =D | 13:23 |
yoctozepto | ah, ah, ok | 13:23 |
yoctozepto | but it's not the case here then | 13:23 |
iurygregory | and I just noticed we don't have the grenade.sh.log_summary.txt available O.o weird | 13:23 |
yoctozepto | possibly because it failed too abruptly | 13:24 |
yoctozepto | still weird | 13:24 |
*** tosky has joined #openstack-qa | 13:25 | |
iurygregory | yeah it makes sense | 13:25 |
*** tosky has quit IRC | 13:29 | |
*** tosky has joined #openstack-qa | 13:29 | |
gmann | mailingsam: i think so, if that is causing failure somewhere then we can debug and change accordingly | 13:32 |
openstackgerrit | Ghanshyam proposed openstack/grenade stable/wallaby: Make heat and octavia grenade jobs as voting https://review.opendev.org/c/openstack/grenade/+/785930 | 13:39 |
openstackgerrit | Ghanshyam proposed openstack/grenade stable/victoria: Make heat and octavia grenade jobs as voting https://review.opendev.org/c/openstack/grenade/+/785931 | 13:39 |
openstackgerrit | Ghanshyam proposed openstack/grenade stable/ussuri: Make heat and octavia grenade jobs as voting https://review.opendev.org/c/openstack/grenade/+/785932 | 13:40 |
*** Luzi has quit IRC | 13:49 | |
openstackgerrit | Ghanshyam proposed openstack/devstack stable/ussuri: DNM: testing neutron-trunk https://review.opendev.org/c/openstack/devstack/+/786056 | 13:55 |
*** gfidente has quit IRC | 13:56 | |
*** gfidente has joined #openstack-qa | 13:56 | |
kopecmartin | #startmeeting qa | 14:00 |
openstack | Meeting started Tue Apr 13 14:00:07 2021 UTC and is due to finish in 60 minutes. The chair is kopecmartin. Information about MeetBot at http://wiki.debian.org/MeetBot. | 14:00 |
openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 14:00 |
*** openstack changes topic to " (Meeting topic: qa)" | 14:00 | |
openstack | The meeting name has been set to 'qa' | 14:00 |
kopecmartin | #link https://wiki.openstack.org/wiki/Meetings/QATeamMeeting | 14:00 |
kopecmartin | ^^^^^^ today's agenda | 14:00 |
kopecmartin | hi, who all here today? | 14:00 |
gmann | o/ | 14:00 |
kopecmartin | o/ | 14:01 |
paras333 | o/ | 14:01 |
yoctozepto | o/ | 14:01 |
rh-jelabarre | o/ | 14:02 |
kopecmartin | hi all, perfect, quite an attendance | 14:02 |
kopecmartin | let's start | 14:02 |
kopecmartin | #topic Announcement and Action Item (Optional) | 14:02 |
*** openstack changes topic to "Announcement and Action Item (Optional) (Meeting topic: qa)" | 14:02 | |
kopecmartin | only one announcement , PTG is next week! | 14:02 |
kopecmartin | QA topics are recorded here | 14:03 |
kopecmartin | #link https://etherpad.opendev.org/p/qa-xena-ptg | 14:03 |
kopecmartin | if your topic is still not there, don't hesitate to add it | 14:03 |
kopecmartin | looking forward to see you there | 14:04 |
gmann | for secure rbac testing, we need more projects coordination too | 14:04 |
gmann | its cross project topic | 14:04 |
gmann | did we finalize the time or we will do later | 14:04 |
paras333 | yes I would say we might need to include lance | 14:04 |
gmann | yeah i pinged him. he is aware of it. | 14:05 |
kopecmartin | i tried to distribute the topics among the days we have booked slots | 14:05 |
gmann | ironic also interested | 14:05 |
paras333 | gmann: ok great | 14:05 |
paras333 | yeah TheJulia is very interested in this | 14:05 |
gmann | I think 14 UTC is good | 14:06 |
gmann | 14-15 | 14:06 |
paras333 | gmann: lgtm | 14:07 |
gmann | one more announcement. | 14:07 |
gmann | all the wallaby release and setup patches are merged now | 14:07 |
gmann | #link https://review.opendev.org/q/topic:%2522qa-wallaby-release%2522 | 14:07 |
kopecmartin | \o/ | 14:08 |
gmann | so we are now done with wallaby things as QA | 14:08 |
kopecmartin | that's great! | 14:08 |
kopecmartin | #topic Wallaby Priority Items progress | 14:09 |
*** openstack changes topic to "Wallaby Priority Items progress (Meeting topic: qa)" | 14:09 | |
kopecmartin | #link https://etherpad.opendev.org/p/qa-wallaby-priority | 14:09 |
kopecmartin | any updates? | 14:09 |
kopecmartin | Patrole stable release - gmann | 14:09 |
gmann | we did the release for wallaby | 14:09 |
kopecmartin | what about this? we did release patrole, right? | 14:09 |
kopecmartin | yeah, so can be considered that done? | 14:10 |
gmann | for wallaby yes | 14:10 |
kopecmartin | perfect | 14:10 |
gmann | and for next cycle we can discuss in PTG about how to run it on project side gate or so | 14:10 |
gmann | or if anything pending | 14:10 |
*** Luzi has joined #openstack-qa | 14:10 | |
kopecmartin | sure, let me write a note somewhere | 14:10 |
gmann | I think I have added it in PTG ethrpad | 14:11 |
kopecmartin | right, i see it now | 14:11 |
kopecmartin | the second day, first topic | 14:11 |
gmann | yeah | 14:11 |
paras333 | kopecmartin: can we move run_validation and guest image one to xena cycle, I still have to do some work which I missed in last cycle | 14:12 |
paras333 | ? | 14:12 |
kopecmartin | sure | 14:12 |
paras333 | kopecmartin: ok, thanks | 14:12 |
gmann | paras333: we will move all those backlog to Xena which are not finished and still needed | 14:12 |
paras333 | gmann: ok great, thank you! | 14:12 |
kopecmartin | yeah, once tracked, forever tracked (until finished haha) | 14:12 |
gmann | L70 in PTG etherpad | 14:12 |
paras333 | lol | 14:13 |
kopecmartin | l70 but might be moved to the last day, we'll see | 14:13 |
kopecmartin | #topic Gate Status Checks | 14:13 |
*** openstack changes topic to "Gate Status Checks (Meeting topic: qa)" | 14:13 | |
kopecmartin | any blockers or urgent patches? | 14:13 |
gmann | we had stable gate blocked, but merged the fixes except stable/train in gate | 14:14 |
gmann | master gate seems green. | 14:14 |
gmann | this thread | 14:15 |
gmann | #link http://lists.openstack.org/pipermail/openstack-discuss/2021-April/021787.html | 14:15 |
*** yamamoto has quit IRC | 14:15 | |
kopecmartin | #link https://review.opendev.org/q/If5f14654ab9aee2a140bbfb869b50d63cb289fdf | 14:16 |
kopecmartin | there are the patches, just train pending , as you sai | 14:16 |
kopecmartin | said | 14:16 |
kopecmartin | good then | 14:16 |
gmann | yeah | 14:16 |
kopecmartin | #topic Periodic jobs Status Checks | 14:17 |
*** openstack changes topic to "Periodic jobs Status Checks (Meeting topic: qa)" | 14:17 | |
kopecmartin | #link https://zuul.openstack.org/builds?job_name=tempest-all&job_name=tempest-full-oslo-master&pipeline=periodic | 14:17 |
kopecmartin | seems ok | 14:17 |
kopecmartin | #topic Sub Teams highlights (Sub Teams means individual projects under QA program) | 14:18 |
*** openstack changes topic to "Sub Teams highlights (Sub Teams means individual projects under QA program) (Meeting topic: qa)" | 14:18 | |
kopecmartin | #link https://review.openstack.org/#/q/project:openstack/tempest+status:open | 14:18 |
kopecmartin | #link https://review.openstack.org/#/q/project:openstack/patrole+status:open | 14:18 |
kopecmartin | #link https://review.openstack.org/#/q/project:openstack/devstack+status:open | 14:18 |
kopecmartin | #link https://review.openstack.org/#/q/project:openstack/grenade+status:open | 14:18 |
kopecmartin | #link https://review.opendev.org/#/q/project:openstack/hacking+status:open | 14:18 |
kopecmartin | a lot of patches as always | 14:20 |
kopecmartin | i see several new ones, will try to review as many as possible | 14:20 |
kopecmartin | #topic Gate Blocker Fix / Urgent Change | 14:20 |
*** openstack changes topic to "Gate Blocker Fix / Urgent Change (Meeting topic: qa)" | 14:21 | |
kopecmartin | any other blockers than those we have discussed already? | 14:21 |
gmann | nothing afaik | 14:21 |
yoctozepto | ironic has some weird issue though | 14:21 |
gmann | in grenade job? | 14:22 |
yoctozepto | yeah | 14:22 |
yoctozepto | on their branch | 14:22 |
yoctozepto | it works on grenade | 14:22 |
yoctozepto | but not on ironic | 14:22 |
yoctozepto | iurygregory observed this | 14:22 |
yoctozepto | sean-k-mooney found out it's due to not updating non-ironic projects | 14:22 |
paras333 | yoctozepto: what's the job and error? | 14:22 |
yoctozepto | which I confirmed | 14:22 |
yoctozepto | yet it works on grenade | 14:22 |
gmann | oh, they upgrade only ironic? | 14:23 |
yoctozepto | 13:54:23 <iurygregory> this may be a bit weird, but ironic-grenade in stable/victoria is broken anyone aware of changes that could cause nova to raise "ImportError: cannot import name 'reraise'" ? | 14:23 |
yoctozepto | 13:55:00 <iurygregory> funny thing is that in grenade we have 4 runs that are green https://zuul.opendev.org/t/openstack/builds?job_name=ironic-grenade&project=openstack%2Fgrenade&branch=stable%2Fvictoria | 14:23 |
yoctozepto | 13:55:16 <iurygregory> but in ironic we are still red https://zuul.opendev.org/t/openstack/builds?job_name=ironic-grenade&project=openstack%2Fironic&branch=stable%2Fvictoria | 14:23 |
yoctozepto | 13:55:34 <iurygregory> always the same problem in nova it seems to me https://zuul.opendev.org/t/openstack/build/432889791d2e4894956b5b3a3adbb6e3/log/controller/logs/screen-n-api.txt?severity=4 | 14:23 |
gmann | yeah i see on grenade patches, no issue | 14:23 |
yoctozepto | gmann: not quite on purpose it seems | 14:23 |
yoctozepto | (psst, the times are utc+2) | 14:23 |
gmann | at least nova should be upgraded | 14:23 |
iurygregory | gmann, we have a filter to remove nova when using multinode grenade job only | 14:23 |
gmann | iurygregory: yoctozepto but did not get what is difference between job running grenade vs ironic gate | 14:24 |
gmann | it is same job or different version? | 14:24 |
yoctozepto | I did not either | 14:24 |
iurygregory | gmann, should be the same .-. | 14:25 |
iurygregory | but it's weird that it works in the grenade gate | 14:25 |
iurygregory | but not in ironic | 14:25 |
rpittau | so | 14:25 |
gmann | iurygregory: and that is only for stable/victoria or master also? | 14:25 |
iurygregory | gmann, victoria only | 14:25 |
iurygregory | master and wallaby seems to be fine | 14:25 |
rpittau | gmann, iurygregory, I was checking the logs and grenade config seems a bit different, at least apparently, one is runninng on summary mode only and saving logs in the main log dir, while the ironic one is running on verbose and saving the logs under the "old" dir | 14:26 |
rpittau | is there maybe something overriding the config/ | 14:26 |
rpittau | ? | 14:26 |
iurygregory | humm I see some failures in master and wallaby also now D: | 14:27 |
iurygregory | ops master* | 14:27 |
iurygregory | wallaby is green https://zuul.opendev.org/t/openstack/builds?job_name=ironic-grenade&project=openstack%2Fironic&branch=stable%2Fwallaby | 14:27 |
gmann | yeah https://zuul.opendev.org/t/openstack/builds?job_name=ironic-grenade | 14:27 |
rpittau | grenade in ironic is saving the logs under controller/logs/old/devstacklog.txt | 14:27 |
gmann | ok, may be let's check after office hour | 14:28 |
gmann | check/debug | 14:28 |
iurygregory | ack | 14:28 |
rpittau | while grenade in grenade is saving under controller/logs/grenade.sh.log_summary.txt | 14:28 |
rpittau | ok | 14:28 |
*** amodi has quit IRC | 14:29 | |
kopecmartin | #topic Open Discussion | 14:29 |
*** openstack changes topic to "Open Discussion (Meeting topic: qa)" | 14:29 | |
kopecmartin | (gmann) is it fine to make devstack-plugin-container branchless | 14:29 |
kopecmartin | this got resolved since last week | 14:29 |
gmann | kopecmartin: that is all sorted out | 14:30 |
kopecmartin | good, i'll remove it from the agenda | 14:30 |
gmann | yeah, it will continue as branched as per plugin team | 14:30 |
kopecmartin | anyone anything else? | 14:30 |
gmann | nothing from me | 14:31 |
*** lbragstad_ is now known as lbragstad | 14:31 | |
kopecmartin | #topic Bug Triage (last 30 min) | 14:32 |
*** openstack changes topic to "Bug Triage (last 30 min) (Meeting topic: qa)" | 14:32 | |
kopecmartin | #link https://etherpad.opendev.org/p/qa-bug-triage-wallaby | 14:32 |
yoctozepto | what about the ceph plugin | 14:32 |
kopecmartin | yoctozepto: what about it? | 14:32 |
yoctozepto | they would prefer to branch :-) | 14:32 |
yoctozepto | copying it from agenda: | 14:33 |
yoctozepto | (yoctozepto) or is it better to branch devstack-plugin-ceph to help the consumers avoid headaches? https://review.opendev.org/c/openstack/devstack-plugin-ceph/+/783844 AND https://review.opendev.org/c/openstack/devstack-plugin-ceph/+/782855 | 14:33 |
kopecmartin | oh , i see you added the topic in the agenda, sorry, i missed it, the page didn't rendered correctly for me | 14:33 |
yoctozepto | no problem | 14:34 |
* kopecmartin loading link, trying to catch up | 14:35 | |
kopecmartin | are there any points against branching? | 14:35 |
yoctozepto | not really | 14:36 |
yoctozepto | cc tbarron, vkmc on devstack-plugin-ceph branching | 14:36 |
kopecmartin | i see that the plugin is currently tracked as project with no release | 14:37 |
gmann | yeah, I think branching will be good but only thing is we need to take care of branch cut for every release. | 14:37 |
gmann | also gouthamr: ^^ | 14:37 |
tbarron | I don't know of any reason not to branch. We talked with gouthamr about it yesterday (still early where he is). | 14:37 |
kopecmartin | not considering tags? | 14:37 |
tbarron | Would the worry be that it's more work than tags? | 14:38 |
kopecmartin | no | 14:38 |
vkmc | o/ +1 branching devstack-plugin-ceph | 14:38 |
vkmc | why tags instead of branches? | 14:39 |
tbarron | The plugin is pretty ugly b/c of all the logic that has to be inserted everywhere to check on the ceph version, or openstack version. | 14:39 |
gmann | yeah branch is much better than tag | 14:39 |
gmann | tbarron: exactly | 14:39 |
openstackgerrit | Merged openstack/grenade stable/train: Fix Grenade DSVM Tempest upper constraints https://review.opendev.org/c/openstack/grenade/+/785767 | 14:39 |
kopecmartin | i agree | 14:39 |
tbarron | I think branches would allow cleaner development branch and eventually the old mess would get retired. | 14:39 |
vkmc | the support matrix has grown and it's hard to keep things sane without branching | 14:39 |
gmann | do we want it from wallaby or Xena ? | 14:40 |
vkmc | we have to keep compatibility between openstack version, ceph version, distro version and also nfs ganesha | 14:40 |
tbarron | It would be nice to be able to use e.g. cephadm for current deployments but fitting that into the existing logic would be really messy. | 14:40 |
gmann | i think we still have time for wallaby branch cut but need to check with release team | 14:40 |
gmann | yoctozepto: can you remove -W from this now, grenade one merged - https://review.opendev.org/c/openstack/devstack/+/785547 | 14:41 |
yoctozepto | gmann: as it's a dev thing, I think release will not mind | 14:41 |
yoctozepto | sure thing | 14:41 |
gmann | yeah. but we should do within today or tomorrow max | 14:42 |
yoctozepto | I mean, I doubt they would mind branching after a coordinated release as these are not really parts of a release per se ;-) | 14:44 |
yoctozepto | need to amend gates to use branches here | 14:44 |
yoctozepto | will you ask hberaud then? | 14:44 |
yoctozepto | anyone here? did I get a netsplit again? | 14:47 |
tbarron | yoctozepto: i see you | 14:47 |
yoctozepto | I see you too | 14:47 |
kopecmartin | i'm here | 14:47 |
gmann | yoctozepto: tbarron we need to update governance also first | 14:47 |
yoctozepto | yes, the governance needs updating here | 14:48 |
yoctozepto | though it's mostly cosmetic | 14:48 |
gmann | who is volunteering to push patches? | 14:48 |
yoctozepto | I can but later | 14:49 |
kopecmartin | i can help too | 14:49 |
kopecmartin | ok, let's sync after the office hour | 14:50 |
*** yamamoto has joined #openstack-qa | 14:50 | |
kopecmartin | so bug triage | 14:50 |
gmann | great, we need to push patch today in order 1. governance change 2. release patch to cut the branch 3. plugin repo side things | 14:50 |
kopecmartin | #link https://etherpad.opendev.org/p/qa-bug-triage-wallaby | 14:50 |
kopecmartin | gmann: sure, i'll have a look and will ping you probably with questions | 14:51 |
gmann | if no other high priority bug then we can discuss ironic grenade for 10 min? | 14:51 |
gmann | kopecmartin: great, thakns | 14:51 |
kopecmartin | nothing high priority | 14:51 |
kopecmartin | go ahead | 14:51 |
*** Luzi has quit IRC | 14:51 | |
gmann | iurygregory: rpittau yoctozepto ironic greande on master is different issue - https://zuul.opendev.org/t/openstack/build/a0be17b445a24f2f9688f7bfcf62e078/log/controller/logs/grenade.sh_log.txt#1561 | 14:52 |
gmann | No Image found for cirros-0.5.1-x86_64-uec | 14:52 |
gmann | seems infra issue? | 14:52 |
yoctozepto | oh my, the fun never ends | 14:52 |
rpittau | most likely | 14:52 |
gmann | checking for victoria | 14:52 |
rpittau | I'm more concerned on victoria | 14:52 |
yoctozepto | you think some images got reverted? | 14:52 |
yoctozepto | the image should be hardcoded in them | 14:53 |
yoctozepto | perhaps some cloud did not refresh images? | 14:53 |
rpittau | AFAIK they're cached, maybe cache has not been updated | 14:53 |
clarkb | the cirros images are cached in the infra test nodes, but devstack/grenade should also download them if not cached | 14:54 |
clarkb | looking at that log the failure is at the upload step though | 14:54 |
rpittau | oh ok | 14:54 |
clarkb | er sorry even after the upload step | 14:54 |
clarkb | it is doing a server create | 14:54 |
yoctozepto | so it's glance that did not get them then | 14:54 |
gmann | yeah | 14:55 |
rpittau | yeah | 14:55 |
yoctozepto | yeah | 14:55 |
clarkb | https://zuul.opendev.org/t/openstack/build/a0be17b445a24f2f9688f7bfcf62e078/log/controller/logs/grenade.sh_log.txt#453-467 the image is on the node | 14:55 |
gmann | does ironic jobs perform glance upgrade or skipped? | 14:55 |
rpittau | it register it for upgrade | 14:58 |
rpittau | so in general it should | 14:58 |
gmann | rpittau: did you try enabling nova in upgrade on stable/victoria and see if that work ? | 14:59 |
rpittau | gmann: err shouldn't that happen already? in grenade itworks fine | 15:00 |
rpittau | the ironic-grenade job in grenade works correctly, in ironic it doesn't | 15:00 |
gmann | rpittau: grenade gate run same job so it should have nova not in upgrade too same as on ironic gate | 15:00 |
gmann | yeah that is not known why | 15:01 |
rpittau | I was saying before that it seems like there is something overriding the config | 15:01 |
*** yamamoto has quit IRC | 15:01 | |
yoctozepto | gmann: it's not multinode | 15:02 |
yoctozepto | ironic does not disable nova in single node | 15:02 |
yoctozepto | it's confusing | 15:02 |
gmann | ah | 15:02 |
yoctozepto | all right, we are past time | 15:02 |
yoctozepto | I will start pushing the patches later in the evening | 15:02 |
yoctozepto | so if someone wishes to start earlier | 15:03 |
yoctozepto | be my guest ;-) | 15:03 |
gmann | thanks | 15:03 |
kopecmartin | yoctozepto: i'll try, but it's almost end of my day | 15:03 |
kopecmartin | so maybe you will need to finish them | 15:03 |
kopecmartin | :) | 15:03 |
kopecmartin | let's close the office hour | 15:03 |
kopecmartin | thank you all | 15:03 |
gmann | kopecmartin: can you re-W it to push it on gate pipeline https://review.opendev.org/c/openstack/devstack/+/785547 | 15:03 |
kopecmartin | #endmeeting | 15:03 |
*** openstack changes topic to "Discussion of OpenStack Quality Assurance Projects (tempest, devstack, grenade, and friends) | docs https://wiki.openstack.org/wiki/QA | specs https://review.openstack.org/#/q/project:openstack/qa-specs | channel logs http://eavesdrop.openstack.org/irclogs/%23openstack-qa/" | 15:03 | |
openstack | Meeting ended Tue Apr 13 15:03:53 2021 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 15:03 |
openstack | Minutes: http://eavesdrop.openstack.org/meetings/qa/2021/qa.2021-04-13-14.00.html | 15:03 |
openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/qa/2021/qa.2021-04-13-14.00.txt | 15:03 |
openstack | Log: http://eavesdrop.openstack.org/meetings/qa/2021/qa.2021-04-13-14.00.log.html | 15:03 |
kopecmartin | gmann: sure, done | 15:04 |
gmann | thanks | 15:05 |
*** jhouser has joined #openstack-qa | 15:10 | |
openstackgerrit | James Parker proposed openstack/whitebox-tempest-plugin master: Test allow disabling CPU flags https://review.opendev.org/c/openstack/whitebox-tempest-plugin/+/776112 | 15:10 |
*** lucasagomes has quit IRC | 16:00 | |
*** gcheresh has quit IRC | 16:02 | |
*** lpiwowar has quit IRC | 16:12 | |
*** Yarboa has quit IRC | 16:14 | |
*** Yarboa has joined #openstack-qa | 16:17 | |
*** hamalq has joined #openstack-qa | 16:20 | |
*** rpittau is now known as rpittau|afk | 16:24 | |
*** rcernin has joined #openstack-qa | 16:32 | |
*** rcernin has quit IRC | 16:37 | |
*** jpena is now known as jpena|off | 16:43 | |
*** ccamposr__ has quit IRC | 16:59 | |
*** ccamposr__ has joined #openstack-qa | 16:59 | |
*** dtantsur is now known as dtantsur|afk | 17:11 | |
*** gfidente is now known as gfidente|afk | 17:16 | |
*** gcheresh has joined #openstack-qa | 17:30 | |
kopecmartin | gmann: i moved the devstack-plugin-ceph from "Project with no release" to "Projects with only Branches" https://wiki.openstack.org/wiki/QA/releases | 17:39 |
kopecmartin | anything else we should do? (apart from the governance and releases patch) | 17:39 |
gmann | kopecmartin: perfect, thanks | 17:40 |
gmann | kopecmartin:I have +A on governance patch | 17:40 |
gmann | let's wait for release patch to merge | 17:40 |
*** gcheresh has quit IRC | 17:42 | |
yoctozepto | we need to work on the gate after branching | 17:57 |
*** eolivare has quit IRC | 18:00 | |
tosky | will the branches start from wallaby, or will they be retrospectively created for the other branches? | 18:07 |
gmann | from wallaby | 18:34 |
gmann | yoctozepto: kopecmartin these are backport for grenade job voting https://review.opendev.org/q/Ic747ac9ddbb21a01e9dc18d8e8ad324d4d7d050d | 18:35 |
kopecmartin | +2 | 18:37 |
gmann | yoctozepto: any specific things we need to care on gate for ceph plugin branch? current stable jobs need to be there as we are starting branch since wallaby only https://github.com/openstack/devstack-plugin-ceph/blob/master/.zuul.yaml | 18:38 |
gmann | kopecmartin: thanks | 18:38 |
gmann | yoctozepto: ah, those stable branch we need to remove from wallaby gate | 18:38 |
*** yamamoto has joined #openstack-qa | 18:38 | |
gmann | any thing else other than that? | 18:38 |
tosky | gmann: remove them from master, or remove from wallaby? Uhm, shouldn't the wallaby branch be used for <=wallaby? | 18:40 |
tosky | so master will be used only for xena, and it doesn't need to test older branches | 18:40 |
gmann | tosky: if we do that we might need a lot of change where ever this plugin is used | 18:41 |
tosky | gmann: but then it doesn't make sense to use master for master and <wallaby | 18:41 |
tosky | it kind of defeats the purpose of branching | 18:41 |
tosky | branching is for stabilize the old branches | 18:41 |
gmann | but at the same time if using master for <wallaby it will block plugins to do master version specific things | 18:42 |
tosky | maybe I didn't explain it properly | 18:42 |
gmann | well we are branching only from wallaby, using wallaby for older branch will be hack we can do | 18:42 |
*** yamamoto has quit IRC | 18:43 | |
tosky | that's why I was asking, maybe it would make sense to artifically create branches for all releases | 18:43 |
tosky | no special messing around | 18:43 |
gmann | humm | 18:44 |
gmann | and with same hash we can do as they are compatible for current stables. | 18:44 |
gmann | tosky: I think that will be easy way, i am ok with that approach. | 18:46 |
gmann | tbarron: gouthamr vkmc yoctozepto what you say? should we cut all stable branches (a least until stable/train) for devstack-plugin-ceph from same current master hash? | 18:46 |
tbarron | gmann: I think that makes sense, they are presumably working now and if not it's not like we know a better hash point | 18:48 |
tbarron | gouthamr: vkmc yoctozepto ^^ ? | 18:48 |
gmann | yeah, same hash is working currently so it would not cause any issue | 18:49 |
*** mailingsam has quit IRC | 18:51 | |
vkmc | +1 from me | 18:57 |
gouthamr | +1 that works, since we put all the effort into being branchless when working on all these releases we care about | 19:02 |
*** gcheresh has joined #openstack-qa | 19:07 | |
*** ramishra has quit IRC | 19:10 | |
*** ramishra has joined #openstack-qa | 19:10 | |
*** ralonsoh has quit IRC | 19:13 | |
*** ajitha has quit IRC | 19:46 | |
*** gcheresh has quit IRC | 19:52 | |
*** vishalmanchanda has quit IRC | 19:55 | |
tosky | we may need also the older branches which are still active, or some kind of mangling will be needed too (stein, rocky, queens; sigh; at least both cinder and manila dropped pike) | 20:17 |
* kopecmartin edited the patch so that it adds the branches for all currently supported releases: https://review.opendev.org/c/openstack/releases/+/786069 | 20:35 | |
*** nweinber has quit IRC | 20:40 | |
*** sboyron has quit IRC | 20:46 | |
* gouthamr hasn't checked the state of the manila gate for queens, rocky and stein - but is sure they're broken | 21:09 | |
tosky | gouthamr: but manila is not the only user of the plugin | 21:11 |
tbarron | tosky: gouthamr yeah which older stable branches are used by cinder/nova/glance? | 21:12 |
tosky | for cinder, queens, rocky and stein | 21:12 |
gouthamr | i agree tosky, is a grenade job likely to be impacted as well? | 21:12 |
tosky | gouthamr: the policy so far is that grenade is best effort and can be dropped anytime, but yes, anything which deploys with the plugin | 21:13 |
tbarron | tosky: and today they are using the main branch, so we can just cut from there for q,r.s, as well, right? | 21:13 |
gouthamr | so is your recommendation that https://review.opendev.org/c/openstack/releases/+/786069 include branches through queens? | 21:13 |
tbarron | gouthamr: manila won't work on those either way but so what | 21:14 |
*** vhari has quit IRC | 21:14 | |
tosky | tbarron: that's the idea | 21:14 |
tbarron | nobody will be using pike | 21:14 |
tosky | gouthamr: correct, also branch for those | 21:14 |
tosky | cinder has dropped pike so far | 21:14 |
tosky | iirc also nova, let me see | 21:14 |
gouthamr | okay, not opposed to it at all - its best effort CI for these q,r,s across projects/jobs - but if the next change to devstack-plugin-ceph is going to break something on these jobs because we don't have a stable branch, seems like a smaller price of cutting the branches right away | 21:15 |
tosky | nova dropped pike: http://lists.openstack.org/pipermail/openstack-discuss/2020-July/015798.html | 21:15 |
tosky | and cinder dropped pike too (I can't find the email but I clearly remember it) | 21:16 |
tbarron | tosky: I seem to remember that too | 21:17 |
gmann | supporting all is not feasible i think | 21:17 |
tbarron | gmann: but cinder and nova have to support them all today, just on the main dev branch, right? It should be easier for them if there are stable branches. | 21:19 |
tbarron | gmann: or are you thinking about the administrative overhead of managing releases? | 21:19 |
tosky | found it: http://lists.openstack.org/pipermail/openstack-discuss/2020-July/015838.html | 21:20 |
gmann | tbarron: its both. | 21:20 |
tosky | gmann: but all those old releases won't have new tags | 21:20 |
tosky | having a branch for each version (a simple ref in the repository) allows us to run the jobs against a specific commit without special rules | 21:21 |
tosky | it's consistency | 21:21 |
gmann | we can do for all supported branch and for EM branch we can see if they are broken by what they are using currently then fix | 21:21 |
tosky | but the fix would be in master | 21:21 |
tosky | again, we would have master supporting master openstack and then stein, rocky and queens | 21:22 |
gmann | that is how we Tempest policy for stable brnahc support | 21:22 |
tosky | we pin tempest | 21:22 |
gmann | once they start breaking | 21:22 |
gmann | same way we can use last cut branch say ussuri | 21:22 |
tosky | but tempest starts pinning from the last support branch | 21:22 |
tosky | here we would have gaps | 21:22 |
tosky | that's totally different than the tempest case | 21:23 |
tosky | at some point, later this year, tempest will stop supporting train, and when it breaks we will ping train, but all older openstack releases use an older tag | 21:23 |
tosky | and newer openstack releases will still use master | 21:24 |
tosky | here we have a plugin which support master and some stable branches | 21:24 |
tosky | while others have a dedicated branch | 21:24 |
tosky | this is just about creating a ref in devstack-plugin-ceph; if something breaks, only the affected branch will need a fix (or it will be abandoned) | 21:25 |
gmann | so who is going to support pike/queens/rocky/stein which are all EM state? | 21:25 |
gmann | do we have Extended maintenance team somewhere? | 21:25 |
tosky | I'm not sure I get it | 21:25 |
tbarron | wouldn't we follow the devstack model here (which has stable branches) rather than tempest? Who supports devstack stable/queens? | 21:26 |
gmann | all Extended maintenance stable branch are supposed to be maintained by Extended maintenance team. | 21:26 |
tosky | what if someone want to fix a broken queens job which uses devstack-plugin-ceph from master? | 21:26 |
tosky | right, but if you don't branch, you prevent that | 21:26 |
tosky | you would have to fix things in master (or block changes in master) to fix queens, if you want to do so, and that would be impossible | 21:27 |
tosky | and that would mean abandoning that branch - which I may agree with, but that's beside the point | 21:27 |
gmann | we lives in a very unclear model of Extended maintenance branches and ADDING UP one more stable branch to support every release | 21:27 |
gmann | and most of the time QA team end up fixing all those | 21:27 |
tosky | we don't need to support | 21:27 |
tosky | so please answer the question above: what happens if somehting breaks in a devstack-plugin-ceph job in stain | 21:28 |
tosky | and you need to fix it in master? | 21:28 |
tbarron | gmann: I think the teams who think they need EM branches should support rather than putting it on QA team | 21:28 |
tosky | I'm not explaining myself | 21:28 |
gmann | yeah ideally that should be case but its not actully | 21:28 |
tosky | nothing I'm saying is going to put more burden than what it is now on the QA team | 21:29 |
gmann | so I will say if team need then they cut the branch and then we can maintain | 21:29 |
*** jhouser has quit IRC | 21:29 | |
gmann | until then we are not breaking any EM branch they are keep working | 21:29 |
tosky | so you are saying that queens job will work with devstack-plugin-ceph from master | 21:29 |
*** vhari has joined #openstack-qa | 21:29 | |
tosky | that doesn't make any sense | 21:29 |
tosky | at all | 21:29 |
gmann | yes, it work now right? | 21:29 |
tosky | it's inconsistent | 21:30 |
gmann | and if anyone want to maintain it separately then we cut the branch | 21:30 |
tbarron | gmann: but we want to move dev branch ahead without having to worry about breaking stable/queens | 21:30 |
tosky | then create a tag | 21:30 |
tosky | whatever | 21:30 |
tosky | it doesn't make any sense, it doesn't and it wont | 21:30 |
tosky | and having a branch more won't put more burden | 21:30 |
tosky | if the QA team doesn't want to support an old branch, they won't fix as it is now | 21:31 |
gmann | tbarron: it will be overhead to maintain the 8 stable branch for devstack-plugin-ceph team, that is my experience from devstack | 21:31 |
tosky | gmann: again, what will it happen if somethig breaks on stein and you need to fix it but the change will complicate the code again? | 21:31 |
gmann | tosky: we cut the branch that time | 21:32 |
gmann | f there is someone there to maintain it | 21:32 |
tosky | gmann: and why not now? | 21:32 |
gmann | *if there | 21:32 |
tosky | the difference is that it won't break for sure | 21:32 |
tosky | if you branch it now when it work, you can stop caring for it | 21:32 |
tosky | if it breaks for other reasons, and someone want to fix it,that's up to them | 21:33 |
tosky | my point is that it's not more complicated than it is now | 21:33 |
*** ramishra has quit IRC | 21:33 | |
gmann | tosky: because we do not have EM maintainer for devstack-plugin-ceph | 21:33 |
gmann | if we have name now then I am ok. | 21:33 |
tosky | gmann: it's not different | 21:33 |
tosky | than what we have now | 21:33 |
tosky | why what I described above doesn't apply? | 21:33 |
tosky | you will most likely need to branch from the same branch where we are branching now | 21:34 |
tosky | having a branch doesn't make things more complicated | 21:34 |
gmann | I am fine if you want to cut and maintain. I am sharing my opinion as per my experience for devstack "we keep all branch in EM with hoping if they break EM team will fix BUT QA team end up fixing them most of the time" | 21:37 |
tosky | gmann: well, it was elod who fixed a lot of breakages so fatr | 21:37 |
tosky | so I hold my point: having the branches now doesn't increase the load on the QA team more than what it is | 21:38 |
gmann | tosky: no, it is elod or I or yoctozepto or frickler | 21:38 |
tosky | it just makes the life of whoever who wants to move forward easier | 21:38 |
gmann | sure, you can try. | 21:38 |
tosky | well, I mentioned elod because he especially advocated for keeping alive branches I would have buried ages ago | 21:39 |
gmann | I have spent lot of time on grenade and Temepest constraints fixing for EM branch | 21:39 |
tosky | so let's say a change to devstack-plugin-ceph breaks stein one week for now: which are the conditions under which you will allow the creation of a branch? | 21:39 |
gmann | and rocky with py2 is still nightmare to maintain | 21:40 |
gmann | but we are maintaining it for many plugins or so | 21:40 |
tosky | you are saying that with an open branch there is an expectation that the QA team will fix a breakage in those older branch for devstack-plugin-ceph | 21:41 |
gmann | practically yes | 21:41 |
gmann | thats what happen in devstack | 21:41 |
gmann | grenade we stopped running on EM for that reason | 21:42 |
tosky | and can't you simply state it clearly that those older devstack-plugin-ceph branches are not under QA team responsibility? | 21:42 |
gmann | two clear practical example. | 21:42 |
tosky | a comment in the job description? | 21:42 |
tosky | wouldn't that address the problem? | 21:42 |
gmann | we already have that in stable doc. i remember the forum sessions agreement for that when EM model was introduced | 21:43 |
gmann | anyways, you can cut them now and see if that work or not. | 21:44 |
gmann | I do not elp much on devstack-plugin-ceph so should not decide it, instead devstack-plugin-ceph team should. just sharing the practicality of what EM branches will be in term of maintenance | 21:45 |
gmann | *help | 21:45 |
openstackgerrit | Merged openstack/devstack stable/train: Make stackviz tasks not to fail jobs https://review.opendev.org/c/openstack/devstack/+/785547 | 21:57 |
*** yamamoto has joined #openstack-qa | 22:00 | |
gmann | finally merged, one tab closed \o/ | 22:02 |
gouthamr | gmann++ thankless work :) | 22:06 |
gouthamr | gmann tosky: we're going to do a bit of thinking at the PTG in the manila group as far as rationalizing investment on the EM branches as we do this every cycle; so we may sign up right away to maintain EM branches on devstack-plugin-ceph; if something gives, i don't mind chasing spot fixes to the CI to override the checkout and consider using the stable/train branch on EM branches as appropriate | 22:10 |
gouthamr | s/we may sign up/may not sign up - big difference | 22:10 |
gmann | ah I thought we got the maintainer before you correct the *not* :P | 22:11 |
tbarron | i think the people who care about the EM branches should take responsibility for them but cutting them and then declaring them EOL if they don' | 22:12 |
tbarron | t step up could make sense :D | 22:12 |
tbarron | but it's not the main issue, really appreciate the work for the current stable branches! | 22:13 |
tbarron | gmann++ | 22:13 |
gmann | sure. that work. that clear the expectation on "there is no default maintainers". adding it on ML or so will work for notification. | 22:15 |
*** tkajinam has joined #openstack-qa | 22:23 | |
*** yamamoto has quit IRC | 22:41 | |
*** yamamoto has joined #openstack-qa | 22:42 | |
*** rcernin has joined #openstack-qa | 22:52 | |
*** rcernin has quit IRC | 22:54 | |
*** hyang has joined #openstack-qa | 22:54 | |
*** rcernin has joined #openstack-qa | 22:54 | |
*** ramishra has joined #openstack-qa | 22:55 | |
*** hyang has quit IRC | 22:57 | |
*** tosky has quit IRC | 23:34 | |
*** ramishra has quit IRC | 23:56 | |
*** ajitha has joined #openstack-qa | 23:56 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!