14:03:13 <ykarel> #startmeeting neutron_ci 14:03:13 <opendevmeet> Meeting started Mon Oct 28 14:03:13 2024 UTC and is due to finish in 60 minutes. The chair is ykarel. Information about MeetBot at http://wiki.debian.org/MeetBot. 14:03:13 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 14:03:13 <opendevmeet> The meeting name has been set to 'neutron_ci' 14:03:25 <ykarel> Ping list: bcafarel, lajoskatona, slawek, mlavalle, mtomaska, ralonsoh, ykarel, jlibosva, elvira 14:03:29 <lajoskatona> o/ 14:03:32 <ralonsoh> hello 14:03:32 <ykarel> Have another video meeting now, so going on IRC today 14:03:33 <mtomaska> o/ 14:03:42 <ralonsoh> (I saw the meeting in qa hahaha) 14:04:13 <lajoskatona> It's Monday, and in Hungary at least just after daytime saving change :-) 14:04:31 <lajoskatona> so anything can happen.... 14:05:09 <bcafarel> :) 14:05:23 <slaweq> o/ 14:05:56 <elvira> o/ 14:06:40 <ykarel> k let's start with topics 14:06:42 <ykarel> #topic Actions from previous meetings 14:06:51 <ykarel> slawek to check https://bugs.launchpad.net/neutron/+bug/2083609 14:07:34 <slaweq> I didn't had time to check it again 14:07:36 <slaweq> sorry 14:08:11 <ykarel> np, will readd for now. atleast not happening too frequently 14:08:21 <ykarel> #action slawek to check https://bugs.launchpad.net/neutron/+bug/2083609 14:08:25 <slaweq> thx 14:08:31 <ykarel> ralonsoh to check for issues which may be triggered by wsgi switch 14:08:44 <ralonsoh> yes, I think I found the biggest problem 14:08:49 <ralonsoh> (one sec) 14:08:55 <ralonsoh> https://bugs.launchpad.net/neutron/+bug/2085543 14:09:07 <ralonsoh> I'm currently talking to otherwiseguy to solve that 14:09:34 <ykarel> ok thx ralonsoh for following it, thats the main one hitting us badly in CI currently 14:09:44 <ralonsoh> I know, I'm actively working on this 14:09:48 <lajoskatona> frightening what kind of issues are jumping forward with wsgi changes.... 14:10:07 <ralonsoh> (wait for the WSGI to ASGI migration...) 14:10:36 <lajoskatona> :'( 14:10:57 <ykarel> haleyb to update grafana dashboard 14:11:06 <ykarel> https://review.opendev.org/c/openstack/project-config/+/931682 was proposed and merged 14:11:12 <ykarel> #topic Stable branches 14:11:20 <ykarel> bcafarel, any updates here? 14:11:44 <ykarel> atleast i am aware stable/2023.1 and before are broken since py3.8 constraint dropped last week 14:12:04 <bcafarel> you typed faster than me :) 14:12:20 <bcafarel> that is the main thing I saw but did not have time to dig more in 14:12:32 <ykarel> following it on #openstack-qa if it's already being tracked 14:12:44 <ykarel> k thx bcafarel 14:13:09 <ykarel> #topic Stadium projects 14:13:15 <ykarel> all green in periodic 14:13:21 <ykarel> lajoskatona, anything else to add here? 14:13:22 <lajoskatona> yes, all green 14:13:45 <lajoskatona> Takashi pushed some patches for stadiums as well, really small ones for py38 to py39 14:14:04 <lajoskatona> if you have some time please check those (and any open reviews on stadiums please) :-) 14:14:36 <ykarel> thx lajoskatona for the updates 14:14:47 <ykarel> #topic Rechecks 14:15:18 <ykarel> due to those wsgi issues we have many rechecks 14:15:39 <ykarel> just 2/17 bare rechecks, let's keep doing it i.e no bare rechecks 14:15:51 <ykarel> and check failures and report bugs if not already logged 14:16:11 <ykarel> #topic fullstack/functional 14:16:16 <ykarel> https://bugs.launchpad.net/neutron/+bug/2083609 14:16:24 <ykarel> already discussed above 14:16:30 <ykarel> #topic Tempest/Scenario 14:16:36 <ykarel> - https://5c06aefe35fb69c8caeb-e8808aa78747cb96cfa3bd19c511369e.ssl.cf2.rackcdn.com/933132/1/gate/neutron-tempest-plugin-ovn/7a8d94f/testr_results.html 14:16:36 <ykarel> - https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_7b4/periodic/opendev.org/openstack/neutron/master/neutron-ovn-tempest-plugin-ovs-master/7b45261/testr_results.html 14:16:42 <ykarel> - https://bugs.launchpad.net/neutron/+bug/2085462 14:16:42 <ykarel> - https://bugs.launchpad.net/neutron/+bug/2085543 14:16:55 <ykarel> This also discussed above which ralonsoh looking into 14:17:03 <ykarel> https://bugs.launchpad.net/neutron/+bug/2083287 14:17:12 <ykarel> i reopened it as still seeing those failures 14:17:47 <ralonsoh> 2083287 is ok, at least the reported issue 14:17:56 <ralonsoh> the problem is in LP#2085543 14:18:03 <ralonsoh> this week I'll have a patch 14:18:55 <ykarel> ralonsoh, you mean no additional fix needed for 2083287 in tempest side? and other fixes for 2085543 will take care of those failures as well? 14:19:08 <ykarel> atleast i saw same traces as before in the linked failures so reopened 14:19:12 <ralonsoh> no, tempest is good now 14:19:21 <ralonsoh> the problem is the DB race condition in Neutron 14:19:27 <ralonsoh> (between Neutron DB and OVN DB) 14:20:01 <ykarel> ok thx, if you can add the relevant links for this race there and re close 14:20:13 <ralonsoh> for sure 14:21:17 <ykarel> thx much 14:21:40 <ykarel> #topic Periodic 14:21:52 <ykarel> 2023.2 openstacksdk job test fail openstack.tests.functional.dns.v2.test_service_status.TestServiceStatus 14:21:58 <ykarel> Reported https://bugs.launchpad.net/openstacksdk/+bug/2084427 and pushed https://review.opendev.org/c/openstack/openstacksdk/+/933278 14:22:11 <ykarel> the fix merged today, so next runs should be good 14:22:18 <ykarel> Broken translation job https://zuul.openstack.org/builds?job_name=propose-translation-update&project=openstack/neutron 14:22:26 <ykarel> - triggered by py3.8 drop https://review.opendev.org/c/openstack/requirements/+/925201 14:22:35 <ykarel> following this on #opendev 14:22:49 <ykarel> Broken tempest jobs in stable/2023.1 and before 14:22:55 <ykarel> - https://zuul.openstack.org/builds?job_name=ironic-tempest-ipa-wholedisk-bios-agent_ipmitool-tinyipa&project=openstack%2Fneutron&branch=stable%2F2023.1&skip=0 14:22:55 <lajoskatona> thanks for taking care 14:22:55 <ykarel> - https://zuul.openstack.org/builds?project=openstack%2Fneutron&pipeline=periodic-weekly&skip=0 14:23:08 <ykarel> This also triggered by py3.8 constraints drop https://review.opendev.org/c/openstack/requirements/+/925201 14:23:32 <ykarel> as tempest in stable branches post wallaby runs with master constraints 14:23:39 <ykarel> following this on #openstack-qa 14:24:00 <ykarel> #topic Grafana 14:24:10 <ykarel> https://grafana.opendev.org/d/f913631585/neutron-failure-rate 14:24:24 <ykarel> let's have a look here too to see if we see any other issue 14:26:57 <ralonsoh> we have a fullstack spike 14:28:23 <ykarel> for fullstack we had one issue which mentioned above 14:28:30 <ralonsoh> right 14:28:46 <ykarel> and also noticed some other failure, but were not recurring, so not reported yet 14:28:49 <ykarel> will check again 14:29:16 <ykarel> also seems something wrong with dashboard, as it's showing spike in gate for openvswitch jobs, but that looks wrong 14:29:34 <ykarel> so will check what's wrong 14:29:57 <ykarel> #action ykarel to check grafana dashboard issue 14:31:14 <ykarel> there are spikes in check too, but those looks due to the known issues we have, i checked last few days failure and most of those were trunk subport failure 14:31:30 <ykarel> that's it, anything else to add? 14:32:37 <lajoskatona> nothing from me 14:32:46 <ralonsoh> no, thanks 14:32:56 <ykarel> k moving 14:32:57 <ykarel> #topic On Demand 14:33:12 <ykarel> i have one topic 14:33:19 <lajoskatona> neither me 14:33:22 <ykarel> New Meeting time to be considered, from next week will have this meeting conflict with our Internal meeting 14:33:40 <ykarel> any suggestion for the new timing? 14:33:52 <ralonsoh> 1400UTC (one hour later) 14:33:58 <ralonsoh> same day, Monday 14:34:29 <ykarel> it's already 1400 UTC, u mean 1500 UTC, right? 14:34:35 <lajoskatona> should work 14:34:38 <ralonsoh> sorry, my bad 14:34:44 <ralonsoh> I'm still adding +2... 14:34:47 <bcafarel> DST fun never ends 14:34:49 <ralonsoh> I mean 1500UTC 14:35:00 <ralonsoh> yeah... I'm still in summer time 14:35:41 <ykarel> any objection for 1500 UTC? or good for everyone? 14:36:11 <ralonsoh> good for me 14:36:15 <slaweq> first half of it will overlap with my internal meeting, but I can just be lurking here 14:36:32 <lajoskatona> +1 for 1500UTC 14:36:35 <slaweq> so you don't need to look for other time just because of me 14:38:39 <ralonsoh> thanks for the time slot update! 14:38:42 <ykarel> if no other objection i think we can try this and see how it goes and reconsider if needed in few weeks 14:39:06 <ykarel> will send a email and patch for the time change 14:39:15 <ykarel> thx everyone for joining 14:39:21 <ykarel> #endmeeting