Wednesday, 2022-05-04

opendevreviewMerged openstack/nova stable/wallaby: Invalidate provider tree when compute node disappears  https://review.opendev.org/c/openstack/nova/+/81180700:45
*** EugenMayer7 is now known as EugenMayer02:30
*** dasm|ruck is now known as dasm|ruck|off05:13
*** prometheanfire is now known as Guest006:23
opendevreviewanguoming proposed openstack/nova master: Add catching InstanceNotFound exception when call live_migration_abort  https://review.opendev.org/c/openstack/nova/+/84042907:59
opendevreviewanguoming proposed openstack/nova master: Add catching InstanceNotFound exception when call live_migration_abort  https://review.opendev.org/c/openstack/nova/+/84042908:09
opendevreviewanguoming proposed openstack/nova master: Add catching InstanceNotFound exception when call live_migration_abort  https://review.opendev.org/c/openstack/nova/+/84042908:10
opendevreviewanguoming proposed openstack/nova master: Add catching InstanceNotFound exception when call live_migration_abort  https://review.opendev.org/c/openstack/nova/+/84042908:13
opendevreviewanguoming proposed openstack/nova master: Add catching InstanceNotFound exception when call live_migration_abort  https://review.opendev.org/c/openstack/nova/+/84042908:23
opendevreviewBalazs Gibizer proposed openstack/nova master: Allow claiming PCI PF if child VF is unavailable  https://review.opendev.org/c/openstack/nova/+/83855508:57
gibimelwitt, sean-k-mooney: ^^ thanks for noting the missing test coverage, I extended the patch now08:57
EugenMayerHello. Today an instance just shutdown. I checked the audit log and see that it has been stopped by 'nobody' means, the user UUID is '-' - what could that mean?09:55
bauzasgmann: as you prefer, I just want to make sure that both the LP name, the spec name and the link to the LP BP in the spec are the same :)10:24
bauzasgmann: if you modify the LP name tho, the link in the yoga spec won't work, but that's fine10:24
opendevreviewMerged openstack/nova-specs master: Amend unified limits spec to explain "API limit" enforcement  https://review.opendev.org/c/openstack/nova-specs/+/82941310:56
opendevreviewanguoming proposed openstack/nova master: Add catching InstanceNotFound exception when call live_migration_abort  https://review.opendev.org/c/openstack/nova/+/84042911:11
gibifyi reported a gate failure https://bugs.launchpad.net/neutron/+bug/1971563 it is not super frequent (two hits in 14 days) but I needed a bug number to recheck ;)11:17
sean-k-mooneyok so that in ml2/ovs with iptables11:24
sean-k-mooneyam both the dhcp agent and the l2 agent have to set porvisioning completed for the neutron server to send the event so maybe only one of those completed in time11:26
gibiyeah I tagged neutron in the bug as I only see that the port state goes from ACTIVE -> ACTIVE on neutron side after nova started waiting for the vif-plug but I don't see the notification sending in the neutron server logs11:27
sean-k-mooneyrovisioning for port b6dc2b79-ed38-4907-86e2-bdff1c5a9b9f completed by entity L2. 11:27
sean-k-mooneyso ya l2 agent completed wiring it up but not dhcp agent11:27
sean-k-mooneyMay 03 16:47:04.331996 ubuntu-focal-ovh-bhs1-0029531414 neutron-dhcp-agent[90791]: INFO neutron.agent.dhcp.agent [None req-89d4fdba-e0f4-4778-b944-ed87f5102ff1 None None] DHCP configuration for ports {'b6dc2b79-ed38-4907-86e2-bdff1c5a9b9f'} is completed11:29
sean-k-mooneyhttps://zuul.opendev.org/t/openstack/build/518f8641b9a7419391b0f99f795f26bd/log/controller/logs/screen-q-dhcp.txt#414511:29
sean-k-mooneyso ya the dhcp agent things it completed but we did not see the event on the neutron server marking provisioning complete by the dhcp agent so this looks like a bug in the dhcp agent11:31
sean-k-mooneynormally it compeltes first so perhaps there is a bug where it wont send the event if the port is already active or something like that11:32
gibisean-k-mooney: thanks for looking into this, could you please add these details to the bug11:39
sean-k-mooneysure just responeing to a review comment and ill add them11:39
gibithank you11:40
sean-k-mooneygibi: actully the dhcp agent has marked it as complete11:55
sean-k-mooneyin the neutron server log11:55
sean-k-mooneygibi: neutron did send the event12:02
sean-k-mooneyINFO neutron.notifiers.nova [-] Nova event matching ['req-17e4fee0-ad06-4350-af09-1db0d331d6b5'] response: {'server_uuid': '3a81145d-d263-4e1d-8ec3-faf38fed34f2', 'name': 'network-vif-plugged', 'status': 'completed', 'tag': 'b6dc2b79-ed38-4907-86e2-bdff1c5a9b9f', 'code': 200}12:02
gibisean-k-mooney: timing doesn't add up12:05
gibinova plugged the vif at May 03 16:48:41.23753812:05
sean-k-mooneywe recvie it at 16:47:2212:05
sean-k-mooneyya so this is proably because of the dhcp agent race12:06
gibineutron should not send the plugged event _before_ nova plugs the vif12:06
sean-k-mooneyi bet we dont have the config option set in neutron12:06
sean-k-mooneygibi: there was a race in neutron where it would not wait for both the dhcp and l2 agent to finish 12:06
sean-k-mooneyit was fixed by https://review.opendev.org/c/openstack/neutron/+/76627712:07
sean-k-mooneyalthough hum12:07
sean-k-mooneythat ws for live migration12:08
gibithis is not a live migration12:08
gibithis is evacuate12:08
sean-k-mooneythe same could happen there12:08
gibiand I still not get it. Can neutron send a vif-plugged event _before_ nova even plugs the vif via os-vif?12:08
sean-k-mooneylet me check if that is enabled or not12:08
sean-k-mooneygibi: its simple12:09
sean-k-mooneythe port was active on the host  we are evacuating form12:09
sean-k-mooneyso it thinks the l2 agent is finished doign its work12:09
sean-k-mooneyso when the dhcp agent responds it sends the event12:09
sean-k-mooneythat is proably what is happening here12:09
sean-k-mooneythat is what happend for live migration12:10
sean-k-mooneyi expect the same behavior for evacuate12:10
sean-k-mooneygibi: https://zuul.opendev.org/t/openstack/build/518f8641b9a7419391b0f99f795f26bd/log/controller/logs/etc/neutron/neutron_conf.txt#1322-133412:11
sean-k-mooneyits disabled12:11
sean-k-mooneywe should try enabling that and see if it fixes the problem12:11
sean-k-mooneygibi: the fix in neutron is based on the presence of migratin_to in the port profile12:12
gibiI thought the expected sequence would be: 1) nova bounds the port to the target host 2) nova plugs the vif on the target host 3) neutron agents plugs the other end of the vif on the target host 4) neutron sends the vif-plugged event to nova12:13
gibibut based on what you said 3) and 4) happens before 2)12:14
sean-k-mooneyright but what actuly happens is the l2 agent on the souce host say the port is alreay bound the dhcp agent say the dhcp configurtion is correct and then neuton sends the event12:14
gibiI see, 12:14
sean-k-mooneyhttps://review.opendev.org/c/openstack/neutron/+/766277/10/neutron/agent/rpc.py12:14
sean-k-mooneyadd filterign so that we only consider updates form  the host that migrating_to points too12:15
gibiOK12:15
gibibased on the comment in the config, the live_migration_events flag should be removed in neutron in Zed already12:16
sean-k-mooneywell it will be removed in zed and always enabled12:16
sean-k-mooneybut i dont know if they have done that yet12:16
gibiOK12:16
sean-k-mooneyhttps://github.com/openstack/neutron/blob/master/neutron/conf/common.py#L182-L199=12:17
sean-k-mooneystill there12:17
gibiOK, I will push a patch to enable that flag in the hybrid plug job12:17
sean-k-mooneyack ralonsoh  do you have patches to remove https://github.com/openstack/neutron/blob/master/neutron/conf/common.py#L182-L199=12:17
sean-k-mooneyand alway smake that the correct behvior12:18
opendevreviewBalazs Gibizer proposed openstack/nova master: Enable live_migration_events in nova-ovs-hybrid-plug  https://review.opendev.org/c/openstack/nova/+/84044612:23
gibisean-k-mooney: ^^12:23
sean-k-mooneyack just realised im meent to be on a call12:24
sean-k-mooneyill take a look after12:24
gibithanks12:24
gibibtw this error was pretty infrequent so we might not know if this fixes it12:24
sean-k-mooneyok ya if its what i think it is its an race that we ocationly loose12:26
sean-k-mooneynormaly the event arrives after we start waiting but not always12:26
ralonsohsean-k-mooney, I'll do it now12:34
sean-k-mooneyralonsoh: thanks no rush but better to land that earlier then late in the cycle 12:35
ralonsohsean-k-mooney, btw, just to confirm: the value will be True now12:35
sean-k-mooneyyes12:35
ralonsohperfect12:35
sean-k-mooneywell you are removing the option yes12:35
ralonsohsean-k-mooney, https://review.opendev.org/c/openstack/neutron/+/84044812:45
ralonsohI'll wait until your reviews12:46
sean-k-mooneyralonsoh: it looks good to me but one nit12:47
ralonsohsure12:47
sean-k-mooneyyou do not have a release note for this12:47
ralonsohright, it deserves one12:47
ralonsohI'll add it12:47
sean-k-mooney+0 while you adress that but otherwise +112:48
sean-k-mooneyi do want to also see the ci run on this too but it should be fine12:48
ralonsohperfect12:49
gmannbauzas: ah, good point on yoga spec link. in that case, let me keep it same name then and in detail i can mention what all things this BP is targeting. I will update zed proposed spec file13:09
bauzasgmann: ack ok13:09
*** dasm|ruck|off is now known as dasm|ruck13:43
opendevreviewRico Lin proposed openstack/nova-specs master: Add vIOMMU device support for libvirt driver  https://review.opendev.org/c/openstack/nova-specs/+/84031013:47
sean-k-mooneygibi: so rodolfo has a patch to make this the default but do we ant to proceed with https://review.opendev.org/c/openstack/nova/+/840446 anyway and perhaps backport that to the relevent branches ?14:11
gibiI haven't see hits of this bug on stable. do we have the hybrid job on stable?14:11
sean-k-mooneyi think artom is adding it we had before yoga too since that was the devstack default14:12
sean-k-mooneygibi: https://github.com/openstack/nova/blob/stable/yoga/.zuul.yaml#L650=14:14
gibithen I think it make sense to land this now and backport it14:15
gibithen we can drop the flag from master when ralonsoh's patch lands14:15
sean-k-mooneythis https://review.opendev.org/c/openstack/nova/+/828413/2 and https://review.opendev.org/c/openstack/nova/+/828418 will be adding it to xena and wallaby14:16
sean-k-mooneycool14:16
opendevreviewRico Lin proposed openstack/nova-specs master: Add vIOMMU device support for libvirt driver  https://review.opendev.org/c/openstack/nova-specs/+/84031016:04
ricolinsean-k-mooney: Thanks for your very detailed review, just update the spec accordingly :) 16:05
opendevreviewMerged openstack/nova-specs master: Re-propose remove tenant_id  https://review.opendev.org/c/openstack/nova-specs/+/83778916:34
opendevreviewGhanshyam proposed openstack/nova-specs master: Re-propose allow Project admin to list allowed hypervisors  https://review.opendev.org/c/openstack/nova-specs/+/83316516:44
gmanngibi: dansmith ^^ as you reviewed it in Yoga cycle. re-proposing the spec.16:45
*** whoami-rajat__ is now known as whoami-rajat17:25
*** Guest0 is now known as prometheanfire19:47

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!