| *** ministry is now known as __ministry | 08:41 | |
| slaweq | ralonsoh ykarel please review https://review.opendev.org/c/openstack/neutron/+/868364 when You will have few minutes | 08:53 |
|---|---|---|
| slaweq | thx in advance | 08:53 |
| slaweq | and also https://review.opendev.org/q/topic:stateless-security-groups-testing :) | 08:54 |
| slaweq | thx a lot | 08:54 |
| ralonsoh | sure, let me check | 08:58 |
| ralonsoh | slaweq, so basically we need to wait for the mac to be written | 09:00 |
| slaweq | ralonsoh TBH I'm not 100% sure but this is only logical explanation of why it fails sometimes as in those cases MAC isn't set to the correct one | 09:01 |
| slaweq | thus DHCP request fails | 09:01 |
| opendevreview | Slawek Kaplonski proposed openstack/neutron master: Ignore python warnings in the neutron-status functional tests https://review.opendev.org/c/openstack/neutron/+/868379 | 09:16 |
| slaweq | ralonsoh with ^^ I think that functional tests with tox4 on Your patch should pass | 09:16 |
| opendevreview | Slawek Kaplonski proposed openstack/neutron master: [WIP] Enable tox4 testing https://review.opendev.org/c/openstack/neutron/+/867554 | 09:16 |
| slaweq | I will take a look at fullstack tests too | 09:17 |
| opendevreview | Slawek Kaplonski proposed openstack/neutron master: Testing https://review.opendev.org/c/openstack/neutron/+/867554 https://review.opendev.org/c/openstack/neutron/+/867616 | 09:18 |
| ralonsoh | slaweq, yes, the deprecation warnings in the FT test | 09:18 |
| ralonsoh | this one is easy hehehe | 09:18 |
| ralonsoh | but I don't know what is hapening with the fullstack ones | 09:19 |
| ralonsoh | slaweq, ^^ can you update the bug reference?? | 09:19 |
| ralonsoh | this is not correct | 09:19 |
| opendevreview | Slawek Kaplonski proposed openstack/neutron master: Ignore python warnings in the neutron-status functional tests https://review.opendev.org/c/openstack/neutron/+/868379 | 09:20 |
| opendevreview | Slawek Kaplonski proposed openstack/neutron master: [WIP] Enable tox4 testing https://review.opendev.org/c/openstack/neutron/+/867554 | 09:20 |
| opendevreview | Slawek Kaplonski proposed openstack/neutron master: Testing https://review.opendev.org/c/openstack/neutron/+/867554 https://review.opendev.org/c/openstack/neutron/+/867616 | 09:20 |
| slaweq | sure | 09:20 |
| slaweq | I'm checking fullstack now | 09:20 |
| ykarel | slaweq, ack will check in some time | 09:32 |
| opendevreview | Slawek Kaplonski proposed openstack/neutron master: DNM Trying to get fullstack logs with tox4 https://review.opendev.org/c/openstack/neutron/+/868380 | 10:13 |
| ralonsoh | slaweq, https://review.opendev.org/c/openstack/neutron/+/868380/1/tox.ini | 10:13 |
| ralonsoh | ohhhh now way | 10:13 |
| ralonsoh | no way! | 10:13 |
| ralonsoh | that was the issue??? | 10:13 |
| slaweq | ralonsoh I'm not sure yet but I suspect that it can be | 10:14 |
| ralonsoh | (I'm dumber every day...) | 10:14 |
| ralonsoh | thanks!! | 10:14 |
| slaweq | as in https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_92c/867554/14/check/neutron-fullstack-with-uwsgi/92cd1e5/tox/dsvm-fullstack-gate/4-commands%5B2%5D.log I don't see that variable | 10:14 |
| ralonsoh | right so at least we'll know what is failing | 10:15 |
| ralonsoh | (I hope is rootwrap only) | 10:15 |
| slaweq | it works fine with tox3 but I think that Omer was talking about something similar in tobiko yesterday | 10:15 |
| slaweq | I hope that with this we will know what's failing - lets wait for the results now :) | 10:15 |
| opendevreview | Merged openstack/ovn-octavia-provider master: Fix listener provisioning_status after HM created/deleted https://review.opendev.org/c/openstack/ovn-octavia-provider/+/867974 | 11:35 |
| opendevreview | Fernando Royo proposed openstack/ovn-octavia-provider stable/zed: Fix listener provisioning_status after HM created/deleted https://review.opendev.org/c/openstack/ovn-octavia-provider/+/868383 | 11:45 |
| opendevreview | Fernando Royo proposed openstack/ovn-octavia-provider stable/yoga: Fix listener provisioning_status after HM created/deleted https://review.opendev.org/c/openstack/ovn-octavia-provider/+/868384 | 11:46 |
| opendevreview | Fernando Royo proposed openstack/ovn-octavia-provider stable/xena: Fix listener provisioning_status after HM created/deleted https://review.opendev.org/c/openstack/ovn-octavia-provider/+/868385 | 11:46 |
| opendevreview | Fernando Royo proposed openstack/ovn-octavia-provider stable/wallaby: Fix listener provisioning_status after HM created/deleted https://review.opendev.org/c/openstack/ovn-octavia-provider/+/868386 | 11:47 |
| opendevreview | Fernando Royo proposed openstack/ovn-octavia-provider stable/zed: Uncouple HM status of member statuses https://review.opendev.org/c/openstack/ovn-octavia-provider/+/868388 | 11:54 |
| opendevreview | Fernando Royo proposed openstack/ovn-octavia-provider stable/yoga: Uncouple HM status of member statuses https://review.opendev.org/c/openstack/ovn-octavia-provider/+/868389 | 11:55 |
| opendevreview | Fernando Royo proposed openstack/ovn-octavia-provider stable/xena: Uncouple HM status of member statuses https://review.opendev.org/c/openstack/ovn-octavia-provider/+/868410 | 11:55 |
| opendevreview | Fernando Royo proposed openstack/ovn-octavia-provider stable/wallaby: Uncouple HM status of member statuses https://review.opendev.org/c/openstack/ovn-octavia-provider/+/868411 | 11:55 |
| admin1 | hi all .. in case of using sriov agent , when does privsep daemon starts ? .. is it after a vm has initiated, or always running | 12:19 |
| admin1 | i have 1 node where its running and works fine, in another node, vms fail to deploy and the diff is in working one, privsep process is also running.. in non working one, only sr-iov agent is running | 12:20 |
| opendevreview | Merged openstack/neutron master: [Fullstack] Wait 10 seconds to ensure that MAC address is configured https://review.opendev.org/c/openstack/neutron/+/868364 | 12:48 |
| ralonsoh | slaweq, thanks for the patch checking fullstack | 13:04 |
| ralonsoh | check this: https://paste.opendev.org/show/bbdbgTwm6X6Y34azxsHP/ | 13:04 |
| ralonsoh | admin1, daemon will start running the first time a privsep call is done | 13:04 |
| ralonsoh | before that, you won't see it | 13:04 |
| noonedeadpunk | Hey there! I'm back with the same neutron regression that happened on Zed between commits add538d7ff26d843fd43ca85f7dca9385dc3ecc1 and 6927dfbb3e52ba298928362da5cce32d49b1e1f8 - I believe it was backported from master as does hit it as well | 13:21 |
| noonedeadpunk | We see it in Zuul and it's reproducable | 13:22 |
| noonedeadpunk | Basically, neutron-server jsut timeouts when it's attempted to create a network with all-in-one bare metal setup | 13:22 |
| noonedeadpunk | So neutron-server is not isolated from OVN services in a container | 13:22 |
| noonedeadpunk | When it is - issue does not exist | 13:23 |
| noonedeadpunk | Here's the paste https://paste.openstack.org/show/bOnNJ7ySJomi4Jk3RwYL/ | 13:23 |
| noonedeadpunk | And that's what we see in logs https://zuul.opendev.org/t/openstack/build/aaca275a240544c89b046dc31a126e04/log/logs/host/neutron-server.service.journal-11-54-45.log.txt#2429 | 13:23 |
| noonedeadpunk | Now I have a sandbox to play in where issue is reproduced | 13:24 |
| noonedeadpunk | I will try to narrow down to specific commit that brought that regression | 13:28 |
| admin1 | hi ralonsoh, my logs are here => https://paste.openstack.org/show/bmcOiXMrl1uG9Wx3iMOt/ | 13:33 |
| admin1 | tldr, 2 identical servers, works in 1 , not in another | 13:33 |
| admin1 | configs and logs in the paste | 13:33 |
| noonedeadpunk | Ok, I quite sure it's https://opendev.org/openstack/neutron/commit/753e7ceb2d04f6e723613407f0703b9221c5da1e that brought regression | 13:38 |
| noonedeadpunk | oh damn, it was backported back to wallaby | 13:41 |
| ralonsoh | noonedeadpunk, what is the issue? | 14:00 |
| noonedeadpunk | openstack network create/delete hangs until timeout | 14:03 |
| ralonsoh | noonedeadpunk, yes but why this commit is related to this? | 14:03 |
| ralonsoh | https://opendev.org/openstack/neutron/commit/753e7ceb2d04f6e723613407f0703b9221c5da1e | 14:03 |
| noonedeadpunk | Well, because when I'm installing previous one it's working as a charm | 14:03 |
| noonedeadpunk | ralonsoh: https://paste.openstack.org/show/bR2lZs2i11WHFWtwlqP3/ | 14:03 |
| noonedeadpunk | Neutron SHA is the only thing that's being changed | 14:04 |
| ralonsoh | noonedeadpunk, and did you try using metadata_workers=1? | 14:04 |
| ralonsoh | or other value than 0? | 14:04 |
| ralonsoh | in any case, I see this patch not related to the error | 14:04 |
| noonedeadpunk | hm. I think jamesdenton has found the issue in setup. And it's indeed related to the lockpath for the neutron-ovn-metadata that must differ from neutron-server | 14:07 |
| ykarel | i see Error during notification for neutron.plugins.ml2.drivers.ovn.mech_driver.mech_driver.OVNMechanismDriver.post_fork_initialize-1516394 process, after_init: PermissionError: [Errno 13] Permission denied: b'/run/lock/neutron/hash_ring_probe_lock' which looks related | 14:07 |
| noonedeadpunk | It wasn't a requirement before though. | 14:08 |
| noonedeadpunk | ANd it's quite un-obvious I would say | 14:08 |
| noonedeadpunk | ralonsoh: metadata_workers=1 is in the config jsut in case | 14:09 |
| *** dasm|off is now known as dasm | 14:10 | |
| slaweq | ralonsoh I have logs from fullstack job :) | 14:18 |
| slaweq | it seems that agents aren't starting due to error like: | 14:19 |
| slaweq | 2022-12-22 10:40:22.781 20156 WARNING oslo.privsep.daemon [-] privsep log: sudo: privsep-helper: command not found | 14:19 |
| slaweq | I will try to check why it's like that and I will update Your patch if I will find something | 14:19 |
| ralonsoh | slaweq, yeah, I pinged you before with that | 14:21 |
| slaweq | ahh, sorry | 14:21 |
| slaweq | I missed Your ping | 14:21 |
| slaweq | ralonsoh why is this needed https://review.opendev.org/c/openstack/neutron/+/867554/16/tox.ini#105 ? | 14:33 |
| ralonsoh | slaweq, I modified that because I though the problem was in rootwrap | 14:34 |
| ralonsoh | slaweq, I'll push a new patch removing that and I'll rebase your testing path on top | 14:34 |
| ralonsoh | ok? | 14:34 |
| opendevreview | Rodolfo Alonso proposed openstack/neutron master: [WIP] Enable tox4 testing https://review.opendev.org/c/openstack/neutron/+/867554 | 14:35 |
| opendevreview | Rodolfo Alonso proposed openstack/neutron master: DNM Trying to get fullstack logs with tox4 https://review.opendev.org/c/openstack/neutron/+/868380 | 14:35 |
| slaweq | ok | 14:38 |
| ralonsoh | admin1, where are the neutron server logs? | 14:42 |
| opendevreview | Slawek Kaplonski proposed openstack/neutron master: Testing https://review.opendev.org/c/openstack/neutron/+/867554 https://review.opendev.org/c/openstack/neutron/+/867616 | 14:44 |
| opendevreview | Rodolfo Alonso proposed openstack/neutron master: Add ``devlink.get_port`` method for devlink ports https://review.opendev.org/c/openstack/neutron/+/867526 | 14:55 |
| opendevreview | yatin proposed openstack/neutron-tempest-plugin master: [DNM] Test tb-cache workaround for jammy https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/867934 | 15:02 |
| admin1 | ralonsoh, i do not have anything for those req-id in the server logs | 15:03 |
| ralonsoh | admin1, the port is not binding and the reason is in the neutron api logs | 15:04 |
| ralonsoh | so please check them | 15:04 |
| opendevreview | Balazs Gibizer proposed openstack/os-vif master: Make tox.ini tox 4.0.0 compatible https://review.opendev.org/c/openstack/os-vif/+/868420 | 15:09 |
| opendevreview | Slawek Kaplonski proposed openstack/neutron-tempest-plugin master: Run security group scenario tests against stateless security groups too https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/867802 | 15:44 |
| opendevreview | Slawek Kaplonski proposed openstack/neutron stable/zed: [Fullstack] Wait 10 seconds to ensure that MAC address is configured https://review.opendev.org/c/openstack/neutron/+/868393 | 15:45 |
| opendevreview | Slawek Kaplonski proposed openstack/neutron stable/yoga: [Fullstack] Wait 10 seconds to ensure that MAC address is configured https://review.opendev.org/c/openstack/neutron/+/868394 | 15:45 |
| opendevreview | Slawek Kaplonski proposed openstack/neutron stable/xena: [Fullstack] Wait 10 seconds to ensure that MAC address is configured https://review.opendev.org/c/openstack/neutron/+/868395 | 15:45 |
| opendevreview | Slawek Kaplonski proposed openstack/neutron stable/wallaby: [Fullstack] Wait 10 seconds to ensure that MAC address is configured https://review.opendev.org/c/openstack/neutron/+/868396 | 15:45 |
| opendevreview | Slawek Kaplonski proposed openstack/neutron stable/victoria: [Fullstack] Wait 10 seconds to ensure that MAC address is configured https://review.opendev.org/c/openstack/neutron/+/868397 | 15:45 |
| opendevreview | Slawek Kaplonski proposed openstack/neutron stable/ussuri: [Fullstack] Wait 10 seconds to ensure that MAC address is configured https://review.opendev.org/c/openstack/neutron/+/868398 | 15:46 |
| opendevreview | Slawek Kaplonski proposed openstack/neutron stable/train: [Fullstack] Wait 10 seconds to ensure that MAC address is configured https://review.opendev.org/c/openstack/neutron/+/868399 | 15:46 |
| opendevreview | Balazs Gibizer proposed openstack/os-vif master: Make tox.ini tox 4.0.0 compatible https://review.opendev.org/c/openstack/os-vif/+/868420 | 15:51 |
| *** labedz is now known as labedz_ | 16:45 | |
| opendevreview | Merged openstack/neutron master: Add extra logs when checking if FIP GW port has been created https://review.opendev.org/c/openstack/neutron/+/868328 | 16:54 |
| opendevreview | Merged openstack/ovn-octavia-provider stable/zed: Uncouple HM status of member statuses https://review.opendev.org/c/openstack/ovn-octavia-provider/+/868388 | 17:21 |
| opendevreview | Merged openstack/ovn-octavia-provider stable/yoga: Uncouple HM status of member statuses https://review.opendev.org/c/openstack/ovn-octavia-provider/+/868389 | 17:22 |
| opendevreview | Merged openstack/neutron master: Rename setup-mtu-t1 argument because it is misleading https://review.opendev.org/c/openstack/neutron/+/864158 | 17:28 |
| opendevreview | Rodolfo Alonso proposed openstack/neutron master: [OVN] Add vnic_type and binding profile capabilities to LSP info https://review.opendev.org/c/openstack/neutron/+/867359 | 17:40 |
| opendevreview | Rodolfo Alonso proposed openstack/neutron master: DNM -- [OVN] OVN monitor agent https://review.opendev.org/c/openstack/neutron/+/866480 | 17:40 |
| opendevreview | Merged openstack/neutron master: [OVN] Implement abstract "post_connect" in "MetadataAgentOvnSbIdl" https://review.opendev.org/c/openstack/neutron/+/866601 | 17:54 |
| opendevreview | Merged openstack/ovn-octavia-provider stable/wallaby: Uncouple HM status of member statuses https://review.opendev.org/c/openstack/ovn-octavia-provider/+/868411 | 18:14 |
| opendevreview | Merged openstack/neutron stable/wallaby: [stable-only] Load config options importing ``common_config`` https://review.opendev.org/c/openstack/neutron/+/868197 | 18:23 |
| opendevreview | Merged openstack/neutron stable/xena: [stable-only] Load config options importing ``common_config`` https://review.opendev.org/c/openstack/neutron/+/868196 | 18:23 |
| opendevreview | Merged openstack/neutron stable/yoga: [stable-only] Load config options importing ``common_config`` https://review.opendev.org/c/openstack/neutron/+/868195 | 18:23 |
| opendevreview | Merged openstack/neutron master: [OVN] Initialize the mech driver variable in ``OVNL3RouterPlugin`` https://review.opendev.org/c/openstack/neutron/+/867206 | 18:23 |
| opendevreview | Merged openstack/ovn-octavia-provider stable/yoga: Fix listener provisioning_status after HM created/deleted https://review.opendev.org/c/openstack/ovn-octavia-provider/+/868384 | 18:47 |
| opendevreview | Merged openstack/neutron stable/yoga: [Fullstack] Wait 10 seconds to ensure that MAC address is configured https://review.opendev.org/c/openstack/neutron/+/868394 | 19:07 |
| opendevreview | Merged openstack/neutron stable/xena: [Fullstack] Wait 10 seconds to ensure that MAC address is configured https://review.opendev.org/c/openstack/neutron/+/868395 | 19:07 |
| opendevreview | Merged openstack/neutron stable/wallaby: [Fullstack] Wait 10 seconds to ensure that MAC address is configured https://review.opendev.org/c/openstack/neutron/+/868396 | 19:07 |
| opendevreview | Merged openstack/neutron stable/zed: [Fullstack] Wait 10 seconds to ensure that MAC address is configured https://review.opendev.org/c/openstack/neutron/+/868393 | 20:27 |
| opendevreview | Merged openstack/neutron stable/train: [Fullstack] Wait 10 seconds to ensure that MAC address is configured https://review.opendev.org/c/openstack/neutron/+/868399 | 20:27 |
| opendevreview | Merged openstack/ovn-octavia-provider stable/wallaby: Fix listener provisioning_status after HM created/deleted https://review.opendev.org/c/openstack/ovn-octavia-provider/+/868386 | 21:55 |
| *** ianw is now known as ianw_pto | 23:43 | |
| *** dasm is now known as dasm|off | 23:57 | |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!