Tuesday, 2022-07-26

« Monday, 2022-07-25 Index Wednesday, 2022-07-27 »
*** amoralej|off is now known as amoralej06:18
opendevreviewSharon Koech proposed openstack/neutron master: Network Cascade Deletion Extension  https://review.opendev.org/c/openstack/neutron/+/84977606:47
slaweqmlavalle: thx, I will look into it today06:59
opendevreviewLajos Katona proposed openstack/neutron stable/yoga: Test: mock out _check_netfilter_for_bridges in unit tests  https://review.opendev.org/c/openstack/neutron/+/85092007:35
opendevreviewFernando Royo proposed openstack/ovn-octavia-provider stable/victoria: Ensure members without subnet belong to VIP subnet or fail  https://review.opendev.org/c/openstack/ovn-octavia-provider/+/85099207:36
bauzasmlavalle: thanks for the bug report update, looking07:49
bauzasmlavalle: about your question about live migrations, I'll look at Gerrit07:49
slaweqbauzas: hi, I just found out that this issue happens a lot in various neutron jobs too08:00
opendevreviewFernando Royo proposed openstack/ovn-octavia-provider stable/ussuri: Ensure members without subnet belong to VIP subnet or fail  https://review.opendev.org/c/openstack/ovn-octavia-provider/+/85099608:01
bauzasslaweq: yep, saw your comment08:05
bauzasyesterday, some nova folks were debating the tempest logic08:06
bauzasI'm about to possibly mark the bug incomplete on the nova side08:06
bauzasbut before that, I'm checking whether some change happened on July 21st08:06
bauzasgibi: around ?08:07
bauzasslaweq: https://meetings.opendev.org/irclogs/%23openstack-nova/%23openstack-nova.2022-07-25.log.html#t2022-07-25T16:20:1908:07
gibilooking back...08:08
gibibauzas: the last thing nova merged was on 19th08:08
gibiand between 19-21 there was normal success rate08:08
bauzashttps://review.opendev.org/q/project:openstack/nova+is:merged08:09
bauzasand we haven't released os-vif, right?08:09
slaweqI will check logs today08:10
gibias Lajos mentioned in the bug we have os-vif 3.0.008:10
gibiright around the time window08:10
slaweqmaybe I will find something there08:10
bauzasgibi: I can shoot a DNM patch08:10
slaweqbauzas: gibi would be good to check IMHO08:11
gibibauzas: sure, go ahead08:11
gibithis was the os-vif 3.0.0 release https://review.opendev.org/c/openstack/releases/+/84954408:12
bauzasyup, and  75b290f Delete trunk bridges to avoid race with Neutron seems a bit related08:13
bauzasan usual suspect, per se.08:13
bauzashttps://review.opendev.org/c/openstack/nova/+/850998 is created, will see08:17
lajoskatonagibi, bauzas, slaweq: If I am right https://review.opendev.org/c/openstack/neutron/+/837780 can be the solution, that pathc waited for the os-vif release (more testing is needed for sure)08:32
lajoskatonahttps://bugs.launchpad.net/nova/+bug/1940425/comments/908:32
bauzasguess so08:33
bauzassome logic changed and now we hit like 100% failure rate08:33
bauzasso, either we revert the os-vif change, or we go full steam ahead and let os-vif fully handle the trunks08:34
ralonsohlajoskatona, bauzas the point is that we need to ensure the os-vif change is being used by Nova08:35
bauzasthat's an upgrade problem08:36
ralonsohif not, this neutron patch will break any deployment with Nova using an older os-vif version too08:36
ralonsohbauzas, but how to handle it?08:36
bauzaskeeping in mind nova supports yoga computes too08:36
bauzasprobably why grenade and nova-next are failing 08:36
ralonsohbauzas, exactly08:36
bauzasralonsoh: I personnally feel the best is to wait for 1 release to happen08:37
bauzasand only delete the trunk creation in Neutron once you're sure that all computes are fully upgraded08:37
bauzasbut, there is still something wrong in the os-vif release08:37
ralonsohbauzas, so wait for this Neutron patch until A release?08:37
bauzasos-vif shouldn't handle this08:37
bauzasralonsoh: I honestly feel this is a design discussion but yeah that could be one option08:38
ralonsohok then08:38
bauzaseither Neutron is able to know Nova is fully upgraded08:38
bauzasor Neutron has to wait for a graceful period08:38
bauzasIMHO, the immediate action would be to revert the broken logic in os-vif08:39
bauzasand then, discuss between nova and neutron how we can handle this upgrade problem08:40
opendevreviewMerged openstack/neutron-specs master: OVS: multiple routed provider segments per host  https://review.opendev.org/c/openstack/neutron-specs/+/82382308:50
slaweqbauzas: ralonsoh gibi lajoskatona so it seems that for now to unblock the gate we should blacklist os-vif 3.0.0, right?08:55
slaweqand then revert that patch and make new releaser08:56
slaweq*release08:56
ralonsohyeah, we should do this08:56
ralonsohslaweq, maybe we can solve this without reverting08:56
slaweqralonsoh: ok, will You propose a patch?08:56
ralonsohok08:56
slaweqralonsoh: yeah, but for now we should block that os-vif version IMO08:56
ralonsohyes08:56
slaweqmaybe revert will not be needed but some fix and new release for sure08:57
lajoskatonaralonsoh, slaweq, bauzas: the login in os-vif 3.0.0 is ok with https://review.opendev.org/c/openstack/neutron/+/837780, or do we miss something?08:59
ralonsohlajoskatona, yes but we can't ensure nova and neutron are sync09:00
ralonsohor at least the Neutron has this patch09:00
ralonsohwe can run Nova Zed and Neutron Yoga09:00
lajoskatonaralonsoh, slaweq, bauzas: isn't it possible to make it optional in os-vif, like we planned in Neutron with a cfg option, and when the full fix is ready we set it to use by default?09:00
ralonsohlajoskatona, this is what I think09:00
ralonsohif we push this from Neutron09:00
lajoskatonaralonsoh: ok09:00
ralonsohso we can "mark" the trunk bridge09:00
ralonsohwith an external-ids tag (whatever name)09:01
ralonsohwhen os-vif reads this tag from the trunk bridge, it will delete the trunk bridge09:01
ralonsohthat means Neutron is updated09:01
ralonsohnow all the logic responsibility falls under Neutron09:01
ralonsohwe need to check if the bridge is there (or not)09:01
bauzasslaweq: there is a DNM patch https://review.opendev.org/c/openstack/nova/+/85099809:02
ralonsohand apply the new logic (just remove the port) or remove the bridge and unwite the ports09:02
gibislaweq: I agree to blacklist 3.0.0 for now to see if that unblocks our gate09:02
bauzasif we see this works and we're sure we'll provide a new revision, I can remove the DNM info and we'll merge it09:02
bauzasabout how to fix the problem, this is a larger discussion 09:03
slaweqbauzas: yeah09:03
ralonsohrequirements patch09:05
ralonsohhttps://review.opendev.org/c/openstack/requirements/+/85100209:05
bauzasthanks09:05
sahidMorning guys09:05
bauzaswill rebase my DNM09:05
sahidlajoskatona: with regards to the multi-segment spec, any else needed, or any questionning that I may have missed to respond? :-)09:06
opendevreviewSlawek Kaplonski proposed openstack/neutron master: DNM Just test os-vif != 3.0.0  https://review.opendev.org/c/openstack/neutron/+/85100309:07
slaweqralonsoh: gibi bauzas lajoskatona ^^ and neutron test on top of ralonsoh's patch09:07
bauzasslaweq: I just updated mine https://review.opendev.org/c/openstack/nova/+/85099809:07
bauzasZuul failed because I forgot to propose the reqs patch, shit09:08
slaweqbauzas: ok, lets check both. If all jobs will be green we will be good to go IMO09:08
lajoskatonasahid: Hi, we can continue with the reviews09:09
sean-k-mooneyo/09:09
gibislaweq: ack09:09
sean-k-mooneyso im not in favor of blocking os-vif 3.0.009:09
sean-k-mooneyat least not in the zed release09:10
sean-k-mooneyif we want to temporary do so while this is fixed on the neutron side im ok with that09:10
ralonsohwe can't guarantee what version of nova or neutron is installed09:10
ralonsohwe need a way to sync that09:10
sean-k-mooneyno you dont09:11
sean-k-mooneythe nova version does not matter09:11
ralonsohNova/os-vif09:11
sean-k-mooneythe os-vif change did not depend on nova changes09:11
ralonsohok: what version of os-vif / Neutron you have in your system09:11
sean-k-mooneythe neutron agent just need to try and delete the bridge if its not already deleted09:11
bauzassean-k-mooney: we need to deliver a 3.0.1 at least, or ask Neutron to revert some change related to the os-vif change, if they can09:12
sean-k-mooneythe change in neutron to drop the bridge cleanup code will just need to wait till AA09:12
sean-k-mooneyyes i am asking for the neutron change to be reverted09:13
ralonsohwe have nothing in Neutron yet09:13
ralonsohthere is nothing in Neutron to be reverted09:13
bauzasbut it looks like something is broken due to the new os-vif trunk handling09:13
ralonsohbecause we don't expect the trunk port to be deleted09:13
bauzasand my guess is that if you have two computes, one old and one master09:13
sean-k-mooneyall that was added was deleting the trunk bridge09:14
ralonsohthat's the issue09:14
sean-k-mooneywhich only affecct ml2/ovs09:14
slaweqexactly as ralonsoh said, we didn't merged neutron part yet https://review.opendev.org/c/openstack/neutron/+/83778009:14
sean-k-mooneyi tough this was breakign with ml2/ovn09:14
ralonsohnot the trunk port, the trunk bridge09:14
ralonsohso, as commented 20 mins ago09:14
ralonsohwe can sync all of this from Neutron09:14
ralonsohwe can mark the trunk bridge, in the external-ids dict09:14
ralonsohand if os-vif detects this, it will delete the trunk bridge, as in 3.0.0 (but with this check)09:15
ralonsohif not, that means Neutron is an old version and won't understand why the trunk bridge is deleted09:15
sean-k-mooneyis it recreating the bridge09:16
ralonsohit is first deleting the bridge and then creating it again09:16
sean-k-mooneyok that sound like a bug09:17
sean-k-mooneysince the nutorn agent is never ment to creat the bridge09:17
ralonsohno we don't create it09:17
sean-k-mooneythe trunk creation was always done by os-vif09:17
opendevreviewFernando Royo proposed openstack/ovn-octavia-provider stable/ussuri: Ensure members without subnet belong to VIP subnet or fail  https://review.opendev.org/c/openstack/ovn-octavia-provider/+/85099609:17
ralonsohbut we delete it09:17
ralonsohat least now09:17
sean-k-mooneyyes so neutron used to delete the bridge09:17
sean-k-mooneynow os-vif can too09:18
sean-k-mooneybut why is it recreated?09:18
ralonsohwhen it is re-created?09:18
sahidlajoskatona: ack thank you, let me know if anything is needed09:18
bauzascan we all assume that something is broken by the new release of os-vif and until we figure out a backup plan, we need to revert this version ?09:18
bauzasit's holding all our merges as of now09:18
ralonsohyes, we should block 3.0.009:19
sean-k-mooneybauzas: honestly im not a fan of that but have the ci results come in with the pin yet09:19
bauzaswe could later revert the skip if we find a solution that doesn't require an os-vif patch09:19
bauzassean-k-mooney: they're running as I speak09:19
sean-k-mooneycan you clarify one thing09:19
sean-k-mooneyis this also affecting ovn09:19
bauzasideally, we should verify the exact change in cause09:19
bauzasand maybe provide a 3.0.1 without just the patch 09:20
sean-k-mooneyi dont want to merge anything in os-vif until we know why this happens09:20
bauzasas 3.0.0 ships more than just the trunk logic modification09:20
bauzassean-k-mooney: don't disagree with you09:20
bauzasbut we haven't bisected yet09:20
bauzaswe're just testing that rolling back to 2.8.0 will work09:21
sean-k-mooneyso im still waitign for a an answer to "have we seen this on any ovn job"09:21
slaweqsean-k-mooney: You mean this live-migration with trunks failure on ovn job? If yes, I didn't saw any issue like that there09:22
slaweqonly on ovs jobs09:22
sean-k-mooneyack09:22
sean-k-mooneyif we had seen this fail on ovn it would rule out that patch09:22
sean-k-mooneysince that code only runs on ml2/ovs09:22
sean-k-mooneywell not quite but ml2/ovn does not have trunk bridge so its a noop09:23
sean-k-mooneythere are only 2 functionl change in 3.0.009:23
bauzascould we verify that  "75b290f Delete trunk bridges to avoid race with Neutron" is the faulty patch ?09:23
sean-k-mooneyhttps://github.com/openstack/os-vif/commit/75b290fb2a8f706583e0c12c5c5a4c0fc80e648109:23
sean-k-mooneyand https://github.com/openstack/os-vif/commit/9ace551db2262d1134ad984ae8a10436a02cdac609:24
bauzas75b290f Delete trunk bridges to avoid race with Neutron could also be a suspect, nope ?09:24
bauzassnap, bad paste09:24
bauzasyeah, the hybrid plugin paste09:24
bauzas$ git log --oneline --no-merges 2.8.0..3.0.009:24
bauzas771dfff update ci since linuxbridge is now experimental09:24
bauzas1651a73 Drop lower-constraints.txt and its testing09:24
bauzas75b290f Delete trunk bridges to avoid race with Neutron09:24
bauzasa12edbf update job template to zed09:24
bauzas9ace551 Check for hybrid plugging in OVS09:24
bauzas95fbe6a Change minversion of tox to 3.18.009:24
sean-k-mooneythe second patch is one we started backporting downstream09:24
sean-k-mooneyso if we have an issue with that we would need to revert it there too09:25
bauzaswe have 75b290f and 9ace551 that are suspects09:25
sean-k-mooneyfor 17.009:25
bauzasexcellent...09:25
bauzashence the bisect I feel09:25
sean-k-mooneyi mean if we really want to i can have it test the indiviual commits09:26
sean-k-mooneyi just need to mofiy the job to use os-vif form git09:26
bauzasthe other option is two gerrit branches09:26
sean-k-mooneyor that09:26
bauzasone testing the revert of 9ace551 and the second one testing revert of 75b290f, both using os-vif master09:26
sean-k-mooneyslaweq: ralonsoh  are we seeign any excption or error on the l2 agent09:27
ralonsohsean-k-mooney, let me check09:27
sean-k-mooneybauzas: 9ace551 is unlikely to be inovlved here since that code only takes effect if you go form hybrid-plug true to false09:27
sean-k-mooneysuch as ml2/ovs to ml2/ovn09:28
sean-k-mooneythe only way im aware of that 75b290f, could break things is if neutron was not able to proceed if the bridge was deleted09:29
bauzassnap, does the depends-on now require an URL instead of a Change-Id ?09:31
sean-k-mooneyno09:31
sean-k-mooneyboth work09:31
sean-k-mooneybut url is required for non gerrit sources09:31
sean-k-mooneythat said url has been prefered for a long long time09:31
bauzasthat was my guess but https://zuul-ci.org/docs/zuul/latest/gating.html wasn't explicit09:31
sean-k-mooneyurl was added for github gitlab and other sourcs that doen use change ides09:32
bauzasbut yeah, then, I should switch into using URLs09:32
bauzasanyway, the status page shows the dependency, which is good09:33
bauzasoh f***09:33
bauzashttps://zuul.openstack.org/status#85099809:33
bauzaswe only trigger functional and unittests09:34
bauzasthat's probably why we were able to merge a broken os-vif release09:34
bauzasI need another DNM patch for testing it09:35
sean-k-mooneyfor os-vif yes in the requirement repo09:35
opendevreviewFernando Royo proposed openstack/ovn-octavia-provider stable/ussuri: Ensure members without subnet belong to VIP subnet or fail  https://review.opendev.org/c/openstack/ovn-octavia-provider/+/85099609:36
sean-k-mooneywe dont have multinode os-vif test if i rememebr corrctly it was on my todolist to add at somepoint09:36
bauzassean-k-mooney: no, also in nova09:36
bauzasif you only update requirements.txt09:36
sean-k-mooneynova is not the gating thing for os-vif09:36
bauzasI'm unclear09:36
sean-k-mooneynovas gate jobs are not the important thing for os-vif release09:37
sean-k-mooneyos-vifs jobs are09:37
bauzasI'm saying that bumping or dropping an os-vif release in nova's reqs.txt doesn't run the full gate09:37
sean-k-mooneybecause you did not touch on of the code files09:38
bauzasin other words, while we verify os-vif itself, we don't verify how we use it in nova, in particular within an upgrade scenario09:38
bauzassean-k-mooney: correct09:39
bauzashence the DNM I need to file09:39
sean-k-mooneyright but this is not where that verification should be happenign normally09:39
sean-k-mooneyif we want to test upgrade we shoudl add grenade to the os-vif repo09:39
sean-k-mooneywe have an eperimental job in nova for testing unrelased os-vif by the way09:41
sean-k-mooneyhttps://github.com/openstack/nova/blob/master/.zuul.yaml#L753-L754=09:41
sean-k-mooneybut in general any upgrade/multi node testing for os-vif should happen in the os-vif repo if we want to test that09:41
sean-k-mooneywhen i brought this up in the past people did not seam to care about testing it so i have kept the bare minium set of jobs working09:42
sean-k-mooneyand never had time to expand them09:42
bauzasyup I don't disagree with you09:42
bauzasbut this explains why we were able to merge a change that broke our gate09:43
sean-k-mooneynope09:44
sean-k-mooneybecause your asserting it could be prevented by nova jobs09:44
sean-k-mooneyunless your refering to the upper constrait patch09:44
sean-k-mooneybut again it is not nova's job ot test os-vif09:45
bauzasI'm not asserting anything about what we should do09:46
sean-k-mooneyif the bug is cause by the os-vif trunk delete patch the only place it should have been caught was a multinode job on the os-vif check pipeline09:47
bauzasmy point was only to say "oh, ok, I understand now how we were able to merge the os-vif release bump into nova reqs.txt without having the check and gate jobs yelling"09:47
bauzasand agreed on the fact this is an os-vif covering issue09:48
sean-k-mooneywe didnt update novas requirements.txt09:48
sean-k-mooneywe raised the max version in upper-constraits.txt09:48
sean-k-mooneywe are currently relying on tempst full09:50
sean-k-mooneyhttps://github.com/openstack/requirements/blob/master/.zuul.d/project.yaml#L60=09:50
sean-k-mooneywe dont have any ml2/ovs coverage09:50
sean-k-mooneyand likely no trunk coverages09:50
sean-k-mooneyso if we wanted to catch the upper-constratis bump we would have needed a multinode ml2/ovs job09:51
sean-k-mooneythis still feels like a neutron bug. nueton should not break if the bridge is deleted so we likely need to find out why that happens and fix it and backport that anyway09:52
sean-k-mooneythat was the whole premis behind the approch we agreed at the ptg. and how the upgrade was ment to work09:53
bauzassean-k-mooney: my bad, yeah I was referring to the nova's change in reqs, which is u-c09:53
ralonsohsean-k-mooney, why not?09:53
ralonsohwhy Neutron should not fail? Neutron is expecting to find this bridge09:54
sean-k-mooneybecause it should not be at that point09:54
sean-k-mooneywe depete all the patch ports09:54
sean-k-mooneyunpluuing the subports and parent port09:54
sean-k-mooneybefore we remove the bridge09:54
lajoskatonabauzas, sean-k-mooney: do we need new jobs to avoid such cases in future? I mean neutron-tempest-plugin with os-vif master(that could be a periodic which need to be checked regularly) or os-vif job with neutron/nova master?09:54
sean-k-mooneyso neutorn shoudl no longer be expect the bridge to exist09:54
sean-k-mooneybecause the port is nolonger plugged on that host09:54
sean-k-mooneylajoskatona: we should just make the os-vif gate jobs multinode. its been a todo item for years. and maybe add greneade09:55
sean-k-mooneylajoskatona: in the requirements repo we might want to have an ml2/ovs multi node job too in general but that should not be needed09:56
sean-k-mooneyos-vif already uses nova/neutron master in its master branch09:56
sean-k-mooneybut because its only singel node it did not run this test09:56
lajoskatonasean-k-mooney: ok, thanks09:57
opendevreviewRodolfo Alonso proposed openstack/os-ken master: Remove "nose" library  https://review.opendev.org/c/openstack/os-ken/+/85063109:58
opendevreviewElvira García Ruiz proposed openstack/neutron master: [OVN] Log drop events per security group  https://review.opendev.org/c/openstack/neutron/+/83501410:20
opendevreviewsean mooney proposed openstack/os-vif master: [WIP] make os-vif jobs multinode and enable trunk testing  https://review.opendev.org/c/openstack/os-vif/+/85101110:26
sean-k-mooney^ is probably incomplete but its more or less what should be needed to enable multinode testing and trunk port testing in os-vif's ci10:27
opendevreviewFernando Royo proposed openstack/networking-ovn stable/train: Ensure members without subnet belong to VIP subnet or fail  https://review.opendev.org/c/openstack/networking-ovn/+/85101310:48
lajoskatonasean-k-mooney: thanks10:52
opendevreviewFernando Royo proposed openstack/networking-ovn stable/train: Ensure members without subnet belong to VIP subnet or fail  https://review.opendev.org/c/openstack/networking-ovn/+/85101311:09
opendevreviewMerged openstack/networking-bagpipe master: py310: Add required-projects list to py310 job  https://review.opendev.org/c/openstack/networking-bagpipe/+/84191711:10
*** froyo is now known as froyo|lunch12:07
*** froyo|lunch is now known as froyo12:07
bauzasso, the os-vif block workarounds the problem and unblocks the gate12:39
bauzasI'm in favor of merging the skip by now 12:39
bauzasand not wait for finding the root cause in order to make a 3.0.112:39
bauzaswe could deliver 3.0.1 later once we agree12:40
opendevreviewRodolfo Alonso proposed openstack/neutron master: Add a default table in "ip rule" command  https://review.opendev.org/c/openstack/neutron/+/85022312:50
opendevreviewayenachew molla proposed openstack/neutron-tempest-plugin master: Delete tcp rule from a security group  https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/84826812:57
lajoskatonabauzas: sounds ok, I added this topic to the Neutron meeting agenda, to have it discussed13:01
bauzasmaybe the root cause couldn't be in os-vif13:01
bauzasif so, we could revert the release skip13:01
bauzastbc, the skip is still temporary until we figure out the proper fix13:01
*** amoralej is now known as amoralej|lunch13:09
*** dasm|off is now known as dasm13:19
*** amoralej|lunch is now known as amoralej13:41
lajoskatona#startmeeting networking14:00
opendevmeetMeeting started Tue Jul 26 14:00:12 2022 UTC and is due to finish in 60 minutes.  The chair is lajoskatona. Information about MeetBot at http://wiki.debian.org/MeetBot.14:00
opendevmeetUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.14:00
opendevmeetThe meeting name has been set to 'networking'14:00
lajoskatonao/14:00
mlavalleo/14:00
ralonsohhi14:00
averdaguero/14:00
obondarevhi14:00
skoech[m]\o14:00
isabekHi14:01
elvirao/14:01
lajoskatonaLet's start the meeting14:02
lajoskatonaZed schedule: https://releases.openstack.org/zed/schedule.html14:02
lajoskatonaRepeating the release dates which are coming from : [release] Release countdown for week R-11, Jul 18 - 22: (#link https://lists.openstack.org/pipermail/openstack-discuss/2022-July/029583.html)14:02
lajoskatonaNon-client library freeze: August 26th, 2022 (R-6 week)14:03
lajoskatonaClient library freeze: September 1st, 2022 (R-5 week)14:03
lajoskatonaZed-3 milestone: September 1st, 2022 (R-5 week)14:03
slaweqo/14:03
lajoskatonaThe A release will be Antelope14:04
lajoskatonaAntelope PTG etherpad: https://etherpad.opendev.org/p/neutron-antelope-ptg14:04
mlavalleNice!14:04
lajoskatonaif you have topics which you would like to discuss please add the topics to it14:04
lajoskatonaI added a few lines to collect Operator's engagement topics also14:05
lajoskatonaplease check and write down if you have anything you think can be interesting on a discussion with operators14:05
lajoskatonaSome not so official thing:14:06
lajoskatonaI will be on PTO and back on 15 August14:07
lajoskatonaSo I plan to cancel next 2 team meeting and drivers meeting14:07
lajoskatonawhat do you think?14:07
slaweq++14:07
mlavalle++14:07
ralonsoh+114:08
slaweqI will be also off 8-16 Aug14:08
mlavallelajoskatona: when do you start your time off?14:08
bcafarellate o/14:08
lajoskatonanext Monday14:08
mlavalleack14:08
mlavallehave fun!14:08
lajoskatonathanks :-)14:08
lajoskatonaI will send out mail about it14:09
mlavallelet's hope we don't wreck the place while you are away14:09
lajoskatonayou can wait for me with that14:09
lajoskatonaanyway it usually happens around release time so we have few weeks14:10
skoech[m]:D14:10
lajoskatonaIf you have no more questions comments for the announcements we can move on14:10
slaweqactually it already started with os-vif :P14:10
lajoskatonayeah we can count that as 114:11
slaweqLOL14:11
lajoskatona:-)14:11
lajoskatonaOk, let's go to next topic14:12
lajoskatona#topic Bugs14:12
lajoskatonaReport from obondarev: https://lists.openstack.org/pipermail/openstack-discuss/2022-July/029707.html14:12
lajoskatonaI saw one bug without owner:14:12
lajoskatona[OVN] metadata does not work when using neutron-dhcp-agent for baremetal ports (#link https://bugs.launchpad.net/neutron/+bug/1982569 )14:12
lajoskatonaobondarev do you have anything more to add for the last week?14:12
obondarevjust an RFE to be discussed on the next drivers: https://bugs.launchpad.net/neutron/+bug/198254114:13
obondarev[RFE] OVN E/W routing14:13
obondarevfor external (baremetal) VLAN ports14:13
lajoskatonathanks14:14
lajoskatonaand perhaps we can have a few words of the hot topic of today: https://bugs.launchpad.net/neutron/+bug/194042514:15
lajoskatonathis is the os-vif 3.0.0 release hell slaweq mentioned14:16
lajoskatonaor more the usual x-project hell14:16
slaweqit seems it's good that we merged and released it in this direction as otherwise we wouldn't even find this issue14:17
slaweqand it would be later reported by some user14:17
lajoskatonaif I understand well now we block os-vif 3.0.0 with https://review.opendev.org/c/openstack/requirements/+/85100214:17
lajoskatonaslaweq: thanks, this is really positive view of this, and true14:18
slaweqyw :)14:18
lajoskatonaIf I understand well we don't know really which commit was the one that causes trouble, but there was not so many:14:19
lajoskatonahttps://bugs.launchpad.net/neutron/+bug/1940425/comments/714:19
slaweqIIUC it was "75b290f Delete trunk bridges to avoid race with Neutron"14:20
ralonsohsean-k-mooney, is integrating grenade on os-vif CI14:20
slaweqas neutron now still expects trunk bridge to be there but os-vif is deleting it14:20
ralonsohwe'll be able to check that14:20
mlavalleyeah, that's the obvious culprit14:20
lajoskatonaslaweq: I have the same feeling14:20
ralonsohbut most probably yes, this is "Delete trunk bridges to avoid race with Neutron14:20
ralonsoha12edbf update job template to zed"14:20
sean-k-mooneyralonsoh: well multinode jobs first we can add grenede too if we want14:21
sean-k-mooneythe test that was failing was not running on the os-vif ci previously14:21
ralonsohexactly14:21
sean-k-mooneyhttps://review.opendev.org/c/openstack/os-vif/+/85101114:22
sean-k-mooneyit failed in that run by the way https://067d6e188ecb98d71167-c128921194a2b01210c7e01142fc6470.ssl.cf2.rackcdn.com/851011/1/check/os-vif-ovs-iptables/46df59a/testr_results.html14:22
sean-k-mooneytest_live_migration_with_trunk[i14:22
sean-k-mooneybut that said other live migration test filed so the job config might not be right yet14:22
ralonsohabout the possible solution, I would comment on the LP bug14:23
ralonsohto avoid mail chains14:23
sean-k-mooneywe can but form my perspective there is no bug in nova or os-vif14:24
mlavalleI also want to mention that the neutron counterpart to the os-vif commit is https://review.opendev.org/c/openstack/neutron/+/837780. In it, I'm adding a config knob that will allow us to handle when the os-vif side deletes the trunk's bridge14:24
sean-k-mooneywe have unplugged the trunk port and subports before we remvoe the bridge14:24
sean-k-mooneyso neutron should not be expecting it to exsit anymore14:24
ralonsohsean-k-mooney, we can comment on the LP bug14:24
sean-k-mooneysure.14:25
lajoskatonamlavalle: would be good to know if that solves the issue14:25
lajoskatonaI mean if yes we can think about how to make it work with Neutron-Nova14:25
sean-k-mooneyalthough i have my hands full with downstream escalation so i wont be able to focus on this much for a few days14:25
lajoskatonaI suspect this can be a good topic for the PTG :-)14:25
sean-k-mooneywe decied to go the current directly at the last  ptg14:26
sean-k-mooneybut sure14:26
sean-k-mooneyi have not had time to look at why this is causeing issues for neutorn.14:26
sean-k-mooneyif that can be sumemrised that would be good14:27
sean-k-mooneyin the bug14:27
ralonsohI'll do14:27
sean-k-mooneythanks14:27
lajoskatonaralonsoh: thanks14:27
mlavalleI will also follow up in the LP bug14:28
lajoskatonamlavalle: thanks14:28
lajoskatonaThis week isabek is the deputy and next week elvira will be (and I have to shuffle the list...)14:29
elvirayes o/14:29
isabeko/14:29
ralonsohyeah, we need an assignee for the 2 week of august14:29
lajoskatonaelvira, isabek: thanks14:29
lajoskatonaI will check the table today or tomorrow14:30
lajoskatonaok, that's it for bugs14:31
lajoskatona#topic On Demand Agenda14:31
lajoskatonado you have anything to discuss?14:32
mlavallenot me14:32
slaweqnothing from me14:32
bcafarelnothing except enjoy your time off lajoskatona :)14:33
ralonsoh+114:33
lajoskatonathanks14:33
amotokienjoy your vacation lajoskatona14:33
obondarev+1!14:33
elvira+114:33
lajoskatonawe go to live in a tent, so perhaps this will be the note for the weather to have some rain around Hungary :-)14:33
slaweq:)14:34
mlavalleLOL14:34
amotoki:)14:34
lajoskatonaOk, I think we can close the meeting14:34
mlavallewe are also praying for rain in Texas14:34
lajoskatona#endmeeting14:34
opendevmeetMeeting ended Tue Jul 26 14:34:26 2022 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)14:34
opendevmeetMinutes:        https://meetings.opendev.org/meetings/networking/2022/networking.2022-07-26-14.00.html14:34
opendevmeetMinutes (text): https://meetings.opendev.org/meetings/networking/2022/networking.2022-07-26-14.00.txt14:34
opendevmeetLog:            https://meetings.opendev.org/meetings/networking/2022/networking.2022-07-26-14.00.log.html14:34
lajoskatonao/14:34
slaweqo/14:34
mlavalleo/14:34
amotokio/14:34
obondarevo/14:34
mlavalleslaweq: ci meeting ove irc today?14:34
slaweqmlavalle: yes14:35
slaweqin 25 minutes14:35
mlavallethanks!14:35
lajoskatonaslaweq: Hi, could you please check the Network Cascade Delete patches which are up for review when you have some free time: https://review.opendev.org/q/topic:cascade_delete_extension+status:open14:42
lajoskatonaslaweq: I don't know why gerrit mark them as "merge conflict", because they are on top of master (at least when I last checked them)14:43
slaweqlajoskatona: sure14:46
lajoskatonaslaweq: thanks14:49
opendevreviewRodolfo Alonso proposed openstack/neutron master: [OVN][Placement] Drive binding by placement allocation  https://review.opendev.org/c/openstack/neutron/+/78647814:50
opendevreviewRodolfo Alonso proposed openstack/os-ken master: Remove "six" library  https://review.opendev.org/c/openstack/os-ken/+/85076714:53
opendevreviewSlawek Kaplonski proposed openstack/neutron master: Switch Fedora based job to Centos Stream  https://review.opendev.org/c/openstack/neutron/+/84433514:55
opendevreviewMerged openstack/ovn-octavia-provider stable/yoga: Ensure members without subnet belong to VIP subnet or fail  https://review.opendev.org/c/openstack/ovn-octavia-provider/+/85055614:55
opendevreviewMerged openstack/ovn-octavia-provider stable/xena: Ensure members without subnet belong to VIP subnet or fail  https://review.opendev.org/c/openstack/ovn-octavia-provider/+/85055714:55
slaweq#startmeeting neutron_ci15:00
opendevmeetMeeting started Tue Jul 26 15:00:12 2022 UTC and is due to finish in 60 minutes.  The chair is slaweq. Information about MeetBot at http://wiki.debian.org/MeetBot.15:00
opendevmeetUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.15:00
opendevmeetThe meeting name has been set to 'neutron_ci'15:00
mlavalleo/15:00
slaweqo/15:00
lajoskatonao/15:00
slaweqlets wait for other folks too15:01
slaweqralonsoh: ykarel bcafarel ping, neutron CI meeting15:01
ralonsohsi sorry15:01
ralonsohhi*15:01
slaweqok, lets start15:02
slaweqGrafana dashboard: https://grafana.opendev.org/d/f913631585/neutron-failure-rate?orgId=115:02
slaweqPlease open now :)15:02
slaweq#topic Actions from previous meetings15:02
ykarelo/15:02
slaweqslaweq to fix functiona/fullstack failures on centos 9 stream: https://bugs.launchpad.net/neutron/+bug/197632315:02
slaweqstill no progress with that one, but I will get to it :)15:03
slaweq#action     slaweq to fix functiona/fullstack failures on centos 9 stream: https://bugs.launchpad.net/neutron/+bug/197632315:03
slaweqnext one15:03
slaweqslaweq to move fedora periodic job to centos9 stream15:03
slaweqno progress yet but I just started working on it few minutes ago15:03
slaweqso I hope I will have something soon15:03
slaweq#action slaweq to move fedora periodic job to centos9 stream15:03
slaweqnext one15:04
slaweq    lajoskatona to open LP bugs related to the stable branches timeouts15:04
lajoskatonaI think I have, just a sec15:04
ralonsohhttps://review.opendev.org/q/119b82f1b10df50a696e936e80672c7ddb00436f15:04
lajoskatonahttps://bugs.launchpad.net/neutron/+bug/198220615:05
lajoskatonathanks ralonsoh15:05
slaweq++15:05
slaweqthx15:05
slaweqnext one then15:05
slaweqmlavalle to report issue with propose-translation-update job and investigate that issue15:05
mlavalleI did investigate15:05
mlavallethis job is failing massively for all the projects15:05
mlavallesince this merged: https://review.opendev.org/c/zuul/zuul-jobs/+/84624815:06
mlavalleso I proposed a revert. Not because I thought necessarily that was the best solution, but to start a discussion: https://review.opendev.org/c/zuul/zuul-jobs/+/85091715:06
lajoskatonaRevert of revert :-)15:07
ralonsohthere is an alternative proposal: https://review.opendev.org/c/openstack/project-config/+/85096215:07
mlavalleand indeed I think I got the conversation going: https://review.opendev.org/c/openstack/project-config/+/85096215:07
mlavalleso we are getting there15:07
slaweqthx mlavalle for taking care of it15:08
slaweqwith that we checked all action items from last week15:08
mlavalleso most likely I will end up abandoning the revert of the revert and hopefully will find a solution with the latest patch from Ian15:08
slaweq++15:08
ralonsoh+115:08
ykarel+115:08
slaweq#topic Stable branches15:08
slaweqbcafarel: any updates here?15:08
mlavalleas I said, the revert of the revert was more a provocation than anything else15:08
lajoskatona:-)15:09
bcafarelas long as the "revert of the revert of the revert..." line still fits in the screen :)15:09
lajoskatonafor stable branches please check this for train: https://review.opendev.org/c/openstack/requirements/+/85082815:10
bcafarelfor stable branches, we have these timeout issues - some in UT fixed by lajoskatona to backport and others, still looking as they timeout during installation of packages15:10
bcafarelhttps://bugs.launchpad.net/neutron/+bug/1982206 and https://bugs.launchpad.net/neutron/+bug/1982720 (that later one breaking train grenade)15:10
lajoskatonaahh ok, so we have bug for the train issue, I forgot that15:11
slaweqso grenade timeout happens only on train?15:11
bcafarelat least from what I have seen yes15:13
bcafarel~2 hours to get paste "Processing triggers for libc-bin" step apparently :/15:13
slaweqhttps://zuul.opendev.org/t/openstack/build/1a4f23e400a3491b88b161e50878753a/log/job-output.txt#249815:13
slaweqregarding this grenade job, is this really neutron issue?15:14
lajoskatonagood question15:14
slaweqfor me this doesn't seems like an neutron issue really15:15
slaweqI can try to investigate it a bit more15:15
ralonsohif in Train we are still using 18.04, that means the default binary is python3.615:16
ralonsohif I'm not wrong15:16
ralonsohso that error will be persistent15:16
slaweqralonsoh: yes, it can be15:16
slaweqI will investigate it15:16
slaweq#action slaweq to check neutron-grenade failurs on stable/train: https://bugs.launchpad.net/neutron/+bug/198272015:16
ralonsohthanks15:16
lajoskatonathanks15:16
ykarelseems https://ff10fb94d3bc910c8640-4a6b1d9d45ac1c9671941502c3743ab2.ssl.cf1.rackcdn.com/850828/1/check/neutron-grenade/1a4f23e/logs/devstack-gate-setup-workspace-old.txt the actual error?15:16
slaweqaha, so is it maybe because stable/stein is EM already?15:17
ralonsohah right, could be15:17
opendevreviewMerged openstack/ovn-octavia-provider stable/wallaby: Ensure members without subnet belong to VIP subnet or fail  https://review.opendev.org/c/openstack/ovn-octavia-provider/+/85055815:18
slaweqno, it's something different15:18
slaweqanyway, lets not spent too much time on this today, I will check it and we will see15:18
ykarelli+115:18
ralonsohIMO, we should not spend too much time on grenade on Train15:18
lajoskatona+115:19
slaweqtrue15:19
ykarel+115:19
slaweqok, lets move on now15:19
slaweq#topic Stadium projects15:19
slaweqlajoskatona: anything new?15:19
lajoskatonaonly bagpipe is failing as I remember from this morning I have to check it in detail15:20
lajoskatonait's possible that only the periodic job failed15:20
lajoskatonathat's it for the stadiums15:21
slaweqok, thx15:21
slaweq#topic Grafana15:21
slaweq#link https://grafana.opendev.org/d/f913631585/neutron-failure-rate15:22
slaweqthere is not much there as in check queue we have pretty high numbers on some jobs15:22
slaweqlike e.g. multinode ovs jobs - issue related to os-vif15:22
slaweqlast week we also had issue with pyroute2 which blocked our gate15:23
ralonsohyeah, both libraries15:23
slaweqwith both of those issues, I don't have much to say about rechecks neighter15:24
slaweqas we didn't merged too many patches recently15:24
slaweqRegarding bare rechecks:15:24
slaweq+---------+---------------+--------------+-------------------+... (full message at https://matrix.org/_matrix/media/r0/download/matrix.org/TKTpTbrxBQEuJgvvIDGanBWX)15:24
slaweqhere are the stats15:24
lajoskatonais it lower than last week?15:25
slaweqnot much15:25
slaweqbut I need to update this script finally15:25
lajoskatonaok, thanks for taking care of it15:25
slaweqas it is counting all rechecks from patches updated in last X days15:25
slaweqeven rechecks which are older15:25
slaweqand that's all from me for the rechecks and grafana15:26
slaweqany questions/comments?15:26
ralonsohno thanks15:26
slaweqif no, I think we can move on15:27
slaweqto the last topic for today15:27
slaweq#topic Periodic15:27
slaweqhere there is one new issue15:27
slaweqopenstack-tox-py39-with-oslo-master is failing since 21.07.2022 every day15:27
slaweqI reported bug https://bugs.launchpad.net/neutron/+bug/198281815:27
slaweqand failure example is https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_9d6/periodic/opendev.org/openstack/neutron/master/openstack-tox-py39-with-oslo-master/9d672a3/testr_results.html15:27
slaweqanyone wants to check it maybe?15:27
slaweqif not I can take a look15:27
ralonsohyeah15:28
ralonsohmaybe related to new DB 15:28
ralonsohI can check it15:28
slaweqthx ralonsoh 15:28
slaweq#action ralonsoh to check failing openstack-tox-py39-with-oslo-master periodic job15:28
lajoskatonaI added comment to the bug,15:28
lajoskatonaseems oslo.db lkast release15:28
ralonsohahhh yes15:29
slaweqthx lajoskatona 15:29
lajoskatonaThe patch "a530cbf Remove the 'Session.autocommit' parameter" was guilty in my env15:30
lajoskatonafrom oslo.db, but I had no more time to see what it is really15:30
slaweqanyone have any other topic related to neutron CI to discuss today?15:32
slaweqif not, I will give You some time back today :)15:32
lajoskatonanothing from me15:33
ralonsohfine for me15:33
mlavallenothing from me15:33
slaweqok, thx for attending the meeting then15:33
slaweqand see You all online15:33
slaweqo/15:33
ralonsohbye15:33
slaweq#endmeeting15:33
opendevmeetMeeting ended Tue Jul 26 15:33:32 2022 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)15:33
opendevmeetMinutes:        https://meetings.opendev.org/meetings/neutron_ci/2022/neutron_ci.2022-07-26-15.00.html15:33
opendevmeetMinutes (text): https://meetings.opendev.org/meetings/neutron_ci/2022/neutron_ci.2022-07-26-15.00.txt15:33
opendevmeetLog:            https://meetings.opendev.org/meetings/neutron_ci/2022/neutron_ci.2022-07-26-15.00.log.html15:33
lajoskatonao/15:33
ykarelo/15:33
ralonsohfolks, missing topic from Team meeting15:33
ralonsohos-ken15:33
ralonsoh--> https://review.opendev.org/q/project:openstack/os-ken+status:open15:33
ralonsohif you have time, thanks!!15:33
mlavalleo/15:33
slaweqralonsoh: sure15:33
* bcafarel looks15:34
lajoskatonaralonsoh: checking15:35
*** amoralej is now known as amoralej|off16:16
*** dasm is now known as dasm|off22:17
opendevreviewMiguel Lavalle proposed openstack/neutron master: Avoid race condition when deleting trunk bridges  https://review.opendev.org/c/openstack/neutron/+/83778023:27
« Monday, 2022-07-25 Index Wednesday, 2022-07-27 »

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!