Friday, 2023-05-19

gmanndansmith: this is enabling the nova rbac new defaults in devstack by default and nova change to add a new job to test old default please check https://review.opendev.org/c/openstack/devstack/+/883556 https://review.opendev.org/c/openstack/nova/+/88355700:30
dansmithgmann: got it00:36
gmannthanks00:41
*** dmellado90 is now known as dmellado905:09
*** EugenMayer45 is now known as EugenMayer406:17
opendevreviewDanylo Vodopianov proposed openstack/nova master: Packed virtqueue support was added.  https://review.opendev.org/c/openstack/nova/+/87607512:45
opendevreviewDanylo Vodopianov proposed openstack/nova-specs master: Add support for Napatech LinkVirt SmartNICs  https://review.opendev.org/c/openstack/nova-specs/+/85929013:16
opendevreviewDanylo Vodopianov proposed openstack/nova-specs master: Add support for Napatech LinkVirt SmartNICs  https://review.opendev.org/c/openstack/nova-specs/+/85929013:41
opendevreviewDanylo Vodopianov proposed openstack/nova-specs master: Add support for Napatech LinkVirt SmartNICs  https://review.opendev.org/c/openstack/nova-specs/+/85929013:43
opendevreviewAmit Uniyal proposed openstack/nova master: [WIP] add initial healthcheck support  https://review.opendev.org/c/openstack/nova/+/82501513:44
opendevreviewAmit Uniyal proposed openstack/nova master: [WIP] add healthcheck manager to manager base  https://review.opendev.org/c/openstack/nova/+/82784413:44
opendevreviewAmit Uniyal proposed openstack/nova master: [WIP] add healthcheck tracker to nova context  https://review.opendev.org/c/openstack/nova/+/82946813:44
opendevreviewAmit Uniyal proposed openstack/nova master: [WIP] add healthcheck utils and constants  https://review.opendev.org/c/openstack/nova/+/82946913:44
opendevreviewAmit Uniyal proposed openstack/nova master: add healthcheck endpoint to proxy commands  https://review.opendev.org/c/openstack/nova/+/83070313:44
dansmithsean-k-mooney: this is the binding error I've been seeing quite a bit latey: https://a6fc37e91c861c55cf2e-59e8bddca242bc843b9f9be8c2ce73c4.ssl.cf2.rackcdn.com/879905/5/check/nova-live-migration/e3a0c46/testr_results.html15:01
dansmithit says to check neutron logs, so maybe it's just a neutron thing, but does that look familiar at all?15:01
dansmithactually I though it used to say something about os-vif in the message now that I think about it so maybe it's something different15:01
opendevreviewyatin proposed openstack/nova master: Add config option to configure TB cache size  https://review.opendev.org/c/openstack/nova/+/86841915:09
opendevreviewArtom Lifshitz proposed openstack/nova master: POC: Re-order and parallelize calls to Neutron and Cinder in post_live_migration  https://review.opendev.org/c/openstack/nova/+/88367815:10
* artom waits for fireworks15:11
dansmithartom: the GIL has nothing to do with your threading there15:17
dansmithwhat you mean, I think, is "as concurrently as the activities are green(ed)"15:18
sean-k-mooneydansmith:  only because you linked it to me like two weeks ago15:20
sean-k-mooneyill tak a look at it again i didnt get around to it then15:20
dansmithsean-k-mooney: thanks15:20
sean-k-mooney Refusing to bind port e3308a61-39ff-4064-abb2-76de0d2139dc to dead agent:  <neutron.plugins.ml2.drivers.ovn.agent.neutron_agent.ControllerAgent object at 0x7f6a7a6d2950>15:32
dansmithdoes that mean the agent on the compute died or something?15:32
sean-k-mooneyi think that meanst the ovn metadtaa agent is dead since ovn its slef is agent less15:33
sean-k-mooneybut yes neutorn think the agent is dead on the destination host15:34
sean-k-mooneyit bound fine orgianly on the other host15:34
artomdansmith, right, I meant it in the sense of GIL only allowing one execution thread at a time, so we're counting to eventlet to do its yield thing15:34
artomI'll just remove it from the commit message :P15:35
sean-k-mooneyill see if i can figure out why15:35
sean-k-mooneydansmith: so it soudn like its hitting the code for https://github.com/openstack/neutron/commit/8a55f091925fd5e6742fb92783c524450843f5a015:48
sean-k-mooneyhum so at the time of the port bidning15:50
sean-k-mooneythere are no errro in the metadta aganet log but there are gaps for 3-6 seconds at a tiem and its interacting with both ovs and privsep15:51
opendevreviewArtom Lifshitz proposed openstack/nova master: POC: Re-order and parallelize calls to Neutron and Cinder in post_live_migration  https://review.opendev.org/c/openstack/nova/+/88367816:12
opendevreviewArtom Lifshitz proposed openstack/nova master: POC: Call Neutron immediately upon _post_live_migration() start  https://review.opendev.org/c/openstack/nova/+/88368216:12
sean-k-mooneydansmith: so my best guess is its related to thsi change https://github.com/openstack/neutron/commit/628442aed7400251f12809a45605bd717f494c4e16:15
sean-k-mooney7 mounts ago they started trying to spread the agent heatbeats16:16
sean-k-mooneyim seeing logs to the effect fo delaying update to the cachs table for 10 seconds16:16
sean-k-mooneyaround when the agent prior to the agent being detected as dead16:17
sean-k-mooneymy guess is if the agent is doign somthign like writing to the ovs db 16:17
sean-k-mooneyit can miss the heatbeat16:17
sean-k-mooney Delaying updating chassis table for 23 seconds {{(pid=38857) run /opt/stack/neutron/neutron/agent/ovn/metadata/agent.py:243}}16:18
sean-k-mooneyim seeign quite a spread16:18
dansmithartom: ack I figured, probably better to make it accurate though yeah :)16:20
dansmithsean-k-mooney: ah, interesting16:20
dansmithsean-k-mooney: so like under heavy load they're missing some heartbeats maybe16:20
sean-k-mooneyya perhaps16:20
sean-k-mooneyim goign to put up a tiny patch to change that form cfg.CONF.agent_down_time // 2  to cfg.CONF.agent_down_time // 316:21
sean-k-mooneythat will make it heat beat a little more often16:21
dansmithack cool16:22
sean-k-mooneythat was recently done for rabbit 2 -> 3 for similar reasons16:22
sean-k-mooneyoh its not merged yet https://review.opendev.org/c/openstack/oslo.messaging/+/87561516:26
sean-k-mooneydansmith: i assume there isnt a bug currently16:32
dansmithsean-k-mooney: not that I've opened16:32
sean-k-mooneyok ill file one quickly with some of the errors i was seeing16:32
sean-k-mooneythe logs are not super helpful 16:33
dansmithsweet thanks16:40
sean-k-mooneyhttps://bugs.launchpad.net/neutron/+bug/202021516:44
sean-k-mooneyi will push a patch once i run the unit/functional tests and see what breaks16:45
*** efried1 is now known as efried16:48
opendevreviewArtom Lifshitz proposed openstack/nova master: POC: Parallelize calls to Neutron and Cinder in post_live_migration  https://review.opendev.org/c/openstack/nova/+/88367816:59
*** melwitt_ is now known as melwitt17:11
sean-k-mooneydansmith: i think https://review.opendev.org/c/openstack/neutron/+/883687 will help but its hard to tell if not then https://bugs.launchpad.net/neutron/+bug/2020215 might give the neutron folks another idea17:11
dansmithack thanks for chasing that17:11
sean-k-mooneyim going to finish there for today o/17:11
dansmiththanks, enjoy the weekend17:11
opendevreviewArtom Lifshitz proposed openstack/nova master: POC: Parallelize calls to Neutron and Cinder in post_live_migration  https://review.opendev.org/c/openstack/nova/+/88367817:25
opendevreviewArtom Lifshitz proposed openstack/nova master: POC: Parallelize calls to Neutron and Cinder in post_live_migration  https://review.opendev.org/c/openstack/nova/+/88367820:06
opendevreviewArtom Lifshitz proposed openstack/nova master: POC: Parallelize calls to Neutron and Cinder in post_live_migration  https://review.opendev.org/c/openstack/nova/+/88367823:01

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!