| opendevreview | yatin proposed openstack/devstack stable/zed: [OVN] Ensure socket files are absent in init_ovn https://review.opendev.org/c/openstack/devstack/+/870505 | 05:47 |
|---|---|---|
| opendevreview | yatin proposed openstack/devstack stable/yoga: [OVN] Ensure socket files are absent in init_ovn https://review.opendev.org/c/openstack/devstack/+/870506 | 05:48 |
| *** soniya29|rover is now known as soniya29|rover|brb | 07:12 | |
| *** soniya29|rover|brb is now known as soniya29|rover | 08:15 | |
| *** jpena|off is now known as jpena | 08:21 | |
| *** soniya29|rover is now known as soniya29|rover|lunch | 09:16 | |
| *** soniya29|rover|lunch is now known as soniya29|rover | 10:44 | |
| *** gthiemon1e is now known as gthiemonge | 12:40 | |
| *** soniya is now known as soniya|rover | 14:34 | |
| kopecmartin | #startmeeting qa | 15:00 |
| opendevmeet | Meeting started Tue Jan 17 15:00:31 2023 UTC and is due to finish in 60 minutes. The chair is kopecmartin. Information about MeetBot at http://wiki.debian.org/MeetBot. | 15:00 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 15:00 |
| opendevmeet | The meeting name has been set to 'qa' | 15:00 |
| kopecmartin | #link https://wiki.openstack.org/wiki/Meetings/QATeamMeeting#Agenda_for_next_Office_hours | 15:00 |
| kopecmartin | agenda ^^^ | 15:00 |
| lpiwowar | hi o/ | 15:00 |
| frickler | \o | 15:00 |
| gmann | o/ | 15:01 |
| kopecmartin | o/ | 15:01 |
| kopecmartin | let's get started | 15:03 |
| kopecmartin | #topic Announcement and Action Item (Optional) | 15:03 |
| kopecmartin | hmm, none from my side | 15:03 |
| kopecmartin | #topic Antelope Priority Items progress | 15:04 |
| kopecmartin | #link https://etherpad.opendev.org/p/qa-antelope-priority | 15:04 |
| kopecmartin | any updates on the priority items? | 15:04 |
| * kopecmartin checking the doc | 15:04 | |
| frickler | I added the failure mode for ceph plugin on 22.04 | 15:04 |
| gmann | on RBAC, nova has changed the default which is to enable the scope checks and new defaults by default | 15:04 |
| frickler | needs some tempest people to look at it | 15:05 |
| kopecmartin | frickler: thank, i'll have a look | 15:06 |
| kopecmartin | gmann: that's cool | 15:06 |
| gmann | #link https://review.opendev.org/c/openstack/nova/+/866218 | 15:06 |
| frickler | seems nova has broken kolla with that, but we will pin to disabled scope for now | 15:07 |
| gmann | and we have new job running new defaults on tempest as well as on nova gate. in Devstack we are keeping old defaults by default until enabling new defaults by defaults is released | 15:07 |
| gmann | frickler: all scope to nova policy are projects which means no change even we enable or disable. enable will only fail if anyone using system token for nova APIs | 15:08 |
| gmann | frickler: can you give me the failure link, I will look into that | 15:08 |
| frickler | gmann: https://zuul.opendev.org/t/openstack/build/33d2dc93c40a457d83adad4c7720e9be for example | 15:09 |
| gmann | and also keeping legacy admin behavior same as before | 15:10 |
| gmann | frickler: thanks, I will check those todayu | 15:10 |
| frickler | https://72b1f029e56da2cad7f4-0417878d098db6313bb6384c3905736a.ssl.cf2.rackcdn.com/870535/1/gate/kolla-ansible-ubuntu-source/33d2dc9/primary/logs/kolla/nova/nova-api.txt | 15:11 |
| frickler | that has the failure | 15:11 |
| gmann | k | 15:12 |
| kopecmartin | moving on | 15:13 |
| kopecmartin | #topic OpenStack Events Updates and Planning | 15:13 |
| kopecmartin | the date of the next virtual PTG is March 27-31, 2023 | 15:13 |
| kopecmartin | #link https://etherpad.opendev.org/p/qa-bobcat-ptg | 15:13 |
| kopecmartin | i've just created the etherpad | 15:13 |
| kopecmartin | but if anyone has any ideas for a topic to discuss, feel free to draft it there | 15:14 |
| kopecmartin | (I'll add more formatting and notes soon) | 15:14 |
| kopecmartin | #topic Gate Status Checks | 15:15 |
| kopecmartin | #link https://review.opendev.org/q/label:Review-Priority%253D%252B2+status:open+(project:openstack/tempest+OR+project:openstack/patrole+OR+project:openstack/devstack+OR+project:openstack/grenade) | 15:15 |
| kopecmartin | nothing there, anything urgent to review? | 15:15 |
| kopecmartin | #topic Bare rechecks | 15:16 |
| kopecmartin | #link https://etherpad.opendev.org/p/recheck-weekly-summary | 15:16 |
| kopecmartin | this looks good | 15:16 |
| kopecmartin | #topic Periodic jobs Status Checks | 15:16 |
| kopecmartin | stable | 15:16 |
| kopecmartin | #link https://zuul.openstack.org/builds?job_name=tempest-full-yoga&job_name=tempest-full-xena&job_name=tempest-full-wallaby-py3&job_name=tempest-full-victoria-py3&job_name=tempest-full-ussuri-py3&job_name=tempest-full-zed&pipeline=periodic-stable | 15:16 |
| kopecmartin | master | 15:16 |
| kopecmartin | #link https://zuul.openstack.org/builds?project=openstack%2Ftempest&project=openstack%2Fdevstack&pipeline=periodic | 15:16 |
| kopecmartin | all green \o/ | 15:17 |
| kopecmartin | #topic Distros check | 15:17 |
| kopecmartin | cs-9 | 15:17 |
| kopecmartin | #link https://zuul.openstack.org/builds?job_name=tempest-full-centos-9-stream&job_name=devstack-platform-centos-9-stream&skip=0 | 15:17 |
| kopecmartin | fedora | 15:17 |
| kopecmartin | #link https://zuul.openstack.org/builds?job_name=devstack-platform-fedora-latest&skip=0 | 15:17 |
| kopecmartin | debian | 15:17 |
| kopecmartin | #link https://zuul.openstack.org/builds?job_name=devstack-platform-debian-bullseye&skip=0 | 15:17 |
| kopecmartin | focal | 15:17 |
| kopecmartin | #link https://zuul.opendev.org/t/openstack/builds?job_name=devstack-platform-ubuntu-focal&skip=0 | 15:17 |
| kopecmartin | rocky | 15:17 |
| kopecmartin | #link https://zuul.openstack.org/builds?job_name=devstack-platform-rocky-blue-onyx | 15:17 |
| kopecmartin | isn't missing something? | 15:18 |
| frickler | euler? | 15:18 |
| kopecmartin | yeah | 15:18 |
| kopecmartin | openEuler | 15:20 |
| kopecmartin | #link https://zuul.openstack.org/builds?job_name=devstack-platform-openEuler-22.03-ovn-source&job_name=devstack-platform-openEuler-22.03-ovs&skip=0 | 15:20 |
| kopecmartin | all good, all green, nice | 15:20 |
| kopecmartin | maybe we want to consider marking the openEuler as voting? | 15:21 |
| kopecmartin | #topic Sub Teams highlights | 15:22 |
| kopecmartin | Changes with Review-Priority == +1 | 15:22 |
| kopecmartin | #link https://review.opendev.org/q/label:Review-Priority%253D%252B1+status:open+(project:openstack/tempest+OR+project:openstack/patrole+OR+project:openstack/devstack+OR+project:openstack/grenade) | 15:22 |
| gmann | it is difficult to see the failure frequency in zuul builds, can we prepare the query in new dashbaord and know what rate it is passing? | 15:22 |
| kopecmartin | that should be possible, what do you think lpiwowar? | 15:23 |
| lpiwowar | gmann: I think it might be possible. I can check it out. | 15:23 |
| gmann | we should start using that dashbaord which can help on voting/non-voting jobs | 15:23 |
| gmann | thanks | 15:23 |
| kopecmartin | +1 | 15:23 |
| lpiwowar | gmann: I just need to check it out. Sometimes it is bit tricky to create the query. | 15:24 |
| gmann | lpiwowar: ohk, sure. thanks | 15:24 |
| kopecmartin | we have 2 reviews with priority | 15:25 |
| kopecmartin | one for devstack | 15:25 |
| kopecmartin | #link https://review.opendev.org/c/openstack/devstack/+/837207 | 15:25 |
| kopecmartin | one for tempest | 15:25 |
| kopecmartin | #link https://review.opendev.org/c/openstack/tempest/+/869440 | 15:25 |
| kopecmartin | the tempest's one got already tested by lpiwowar in the stable branches | 15:26 |
| lpiwowar | kopecmartin: yes. It should be ok. Thanks for mentioning it. | 15:26 |
| kopecmartin | seems good | 15:26 |
| gmann | nice, I will check it today then. | 15:26 |
| gmann | cool | 15:26 |
| frickler | I wonden when ovn will work without 100s of workarounds | 15:26 |
| frickler | wonder even | 15:26 |
| gmann | :) | 15:26 |
| kopecmartin | let's not make it easy :D | 15:27 |
| kopecmartin | #topic Open Discussion | 15:27 |
| kopecmartin | anything for the open discussion? | 15:27 |
| frickler | yes | 15:27 |
| lpiwowar | frickler: What fixed the tempest-full-multinode-py3 job? | 15:27 |
| lpiwowar | #link https://review.opendev.org/c/openstack/tempest/+/866692 | 15:28 |
| lpiwowar | I was just curious. | 15:28 |
| frickler | lpiwowar: zuul dropped setting private IPs in some situations, that has been reverted for now | 15:28 |
| lpiwowar | frickler: Ok, thanks | 15:28 |
| frickler | there was a bug opened about opensuse support. and I wonder whether we should rather drop that distro completely | 15:29 |
| frickler | https://bugs.launchpad.net/devstack/+bug/2002900 | 15:29 |
| frickler | would clean up quite a bit of special casing | 15:29 |
| frickler | and we don't run any CI on it anymore | 15:29 |
| gmann | lpiwowar: just curious, are you testing /getting failure with devstack change of 'cirros version to 0.6.1' ? or in normal run also? seeing depends-on to devstack patch ? | 15:30 |
| gmann | or testing if cirrors vesion fix the failure? | 15:30 |
| gmann | +1 on dropping opensuse completely as we do not test it | 15:31 |
| kopecmartin | makes sense, if it's not tested, let's drop the support | 15:33 |
| kopecmartin | gmann: the multinode was failing even with newer cirros image (0.6.1? don't remember) if that's the question | 15:33 |
| lpiwowar | gmann: The tempest-multinode-full-py3 job was failing also outside of the patch. So I think that the bump up of the cirros version did not help. | 15:33 |
| lpiwowar | kopecmartin: +1 | 15:34 |
| gmann | ok, yeah i saw failure sometime I thikn the basic scenario test right? but that was not 100% failing | 15:34 |
| gmann | is there bug open to know traceback and exact traceback? | 15:34 |
| lpiwowar | The scenario tests failed only sometimes. But recently the failure rate went up. | 15:35 |
| gmann | and we can check the frequency of failure too. as it is not failure 100% I am sure it is race or timing issue | 15:35 |
| gmann | which test? | 15:35 |
| lpiwowar | gmann: Give me a moment I will tell you exactly. | 15:36 |
| gmann | no hurry, let's add it in bug and we can track there | 15:36 |
| lpiwowar | gmann: ack | 15:37 |
| gmann | lpiwowar: thanks for debugging gate such failure which will help our CI testing a lot. | 15:38 |
| lpiwowar | gmann: thanks! | 15:38 |
| lpiwowar | I remember seeing failures of: tempest.scenario.test_minimum_basic.TestMinimumBasicScenario.test_minimum_basic_scenario and tempest.scenario.test_minimum_basic.TestMinimumBasicScenario.test_minimum_basic_instance_hard_reboot_after_vol_snap_deletion | 15:39 |
| gmann | k | 15:39 |
| lpiwowar | But from time to time there were other tests failing as well. | 15:39 |
| lpiwowar | Just adding for documentation. Here is a link for the bug regarding the tempest-multinode-full-py3 job | 15:42 |
| lpiwowar | #link https://bugs.launchpad.net/tempest/+bug/1998916 | 15:42 |
| kopecmartin | thanks | 15:42 |
| kopecmartin | cool, anything else? | 15:42 |
| gmann | thanks | 15:43 |
| lpiwowar | nothing from my side | 15:43 |
| kopecmartin | #topic Bug Triage | 15:43 |
| kopecmartin | #link https://etherpad.openstack.org/p/qa-bug-triage-antelope | 15:43 |
| kopecmartin | etherpad updated | 15:44 |
| kopecmartin | i wanted to bring up the suse bug, but we already talked about that | 15:44 |
| kopecmartin | that's all from my side | 15:44 |
| frickler | gmann: can https://bugs.launchpad.net/devstack/+bug/1999183 be closed? | 15:45 |
| frickler | or at least degraded from critical? | 15:45 |
| gmann | frickler: not closed as we still have tox4 pin on devstack jobs on master. but I will change priority for this | 15:45 |
| frickler | ok, thx | 15:46 |
| gmann | done, I have this in my list but did not get time. | 15:47 |
| frickler | tox4 has turned out to be a fast moving target, yes | 15:49 |
| kopecmartin | yeah, thanks all, if there isn't anything else, let's close today's office hour | 15:50 |
| kopecmartin | #endmeeting | 15:51 |
| opendevmeet | Meeting ended Tue Jan 17 15:51:19 2023 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 15:51 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/qa/2023/qa.2023-01-17-15.00.html | 15:51 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/qa/2023/qa.2023-01-17-15.00.txt | 15:51 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/qa/2023/qa.2023-01-17-15.00.log.html | 15:51 |
| lpiwowar | kopecmartin: thanks o/ | 15:51 |
| gmann | thanks kopecmartin | 15:51 |
| *** soniya|rover is now known as soniya|rover|out | 15:52 | |
| frickler | gmann: fyi https://review.opendev.org/c/openstack/kolla-ansible/+/870804 is the workaround for kolla for now | 16:02 |
| gmann | frickler: yeah, i saw that. failure is due to kolla using system scope token which has changed in zed cycle on new RBAC direction (except ironic and keystone service). I am checking it and will propose the change to run nova/other services with correct scope | 16:04 |
| opendevreview | Ghanshyam proposed openstack/tempest master: Do not run failing interop-tempest-consistency job https://review.opendev.org/c/openstack/tempest/+/870838 | 16:22 |
| gmann | kopecmartin: tempest gate is blocked by interop-tempest-consistency . commenting it for now until issue is fixed to unblock gate ^^ | 16:23 |
| *** jpena is now known as jpena|off | 17:26 | |
| opendevreview | Ghanshyam proposed openstack/tempest master: Introduce @serial test execution decorator https://review.opendev.org/c/openstack/tempest/+/821732 | 19:05 |
| dansmith | gmann: I'm now getting this when devstack tries to install keystone: | 21:27 |
| dansmith | pkg_resources.extern.packaging.version.InvalidVersion: Invalid version: '0.23ubuntu1' | 21:27 |
| gmann | dansmith: not seen this either. on focal right? | 21:29 |
| dansmith | gmann: yeah still focal | 21:29 |
| dansmith | I'm not sure if that's complaining about keystone itself or a pip package, but seems like it must be something installed with distro packages that pip is trying to parse right? | 21:30 |
| dansmith | $ dpkg -l | grep 0.23ubuntu1 | 21:30 |
| dansmith | shows me distro-info and python3-distro-info | 21:30 |
| dansmith | which is part of ubuntu-minimal | 21:31 |
| dansmith | maybe I need to apt-upgrade | 21:31 |
| dansmith | mm, nope | 21:31 |
| dansmith | well, I can upgrade to jammy but seems like this should work | 21:32 |
| gmann | yeah, not sure why it is failing. but my last week installation was on jammy. have not tried it on focal recently | 21:32 |
| dansmith | if this is some distro package problem I probably need to do a clean reinstall I'd guess | 21:42 |
| dansmith | lol same difference after an in-place upgrade to jammy, but now it complains about a different version: 1.1build1 | 23:21 |
| dansmith | ...which is now the version of distro-info | 23:22 |
| dansmith | I give up | 23:22 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!