| opendevreview | Takashi Kajinami proposed openstack/neutron master: Remove comment about removed logic https://review.opendev.org/c/openstack/neutron/+/963346 | 01:24 |
|---|---|---|
| opendevreview | Dai Dang Van proposed openstack/neutron master: Add dns forwarder l2 extension https://review.opendev.org/c/openstack/neutron/+/951390 | 04:08 |
| opendevreview | Lajos Katona proposed openstack/neutron-tempest-plugin master: dns_forwarder extension: add extension to distributed dhcp job https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/958775 | 04:11 |
| opendevreview | Lajos Katona proposed openstack/neutron master: dns_forwarder extension: add devstack plugin https://review.opendev.org/c/openstack/neutron/+/958774 | 04:13 |
| opendevreview | Vladimir Kozhukalov proposed openstack/neutron master: Fix periodic_resync thread in DHCP agent https://review.opendev.org/c/openstack/neutron/+/962871 | 08:18 |
| ralonsoh | TheJulia, can you share some CI logs? | 08:23 |
| ralonsoh | please open a LP with some information, we'll triage it ASAP | 08:24 |
| opendevreview | Vladimir Kozhukalov proposed openstack/neutron master: Fix periodic_resync thread in DHCP agent https://review.opendev.org/c/openstack/neutron/+/962871 | 08:51 |
| *** iurygregory_ is now known as iurygregory | 12:08 | |
| *** ykarel_ is now known as ykarel | 12:56 | |
| opendevreview | Lajos Katona proposed openstack/neutron master: WIP: [eventlet]: Handle stop DHCP agent https://review.opendev.org/c/openstack/neutron/+/963386 | 13:06 |
| lajoskatona | ralonsoh: Hi, for DHCP agent hanging issue------^ | 13:07 |
| opendevreview | Ashish Gupta proposed openstack/neutron-tempest-plugin master: Testing https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/963287 | 13:07 |
| ralonsoh | lajoskatona, let me check locally | 13:07 |
| lajoskatona | ralonsoh: The problem is that the ProcessMonitor._check_child_processes are still hanging :-( | 13:08 |
| lajoskatona | ralonsoh: perhaps if you have some idea where is a point where the ProcessMonitor can be stopped when we stop the dhcp_agent | 13:09 |
| opendevreview | Elod Illes proposed openstack/neutron unmaintained/xena: [CI][um-only] Force use of ansible 9 for py36 job https://review.opendev.org/c/openstack/neutron/+/963387 | 13:13 |
| ralonsoh | lajoskatona, the DHCP agent process is using service.launch() to spawn the service class. This service class is defined in the upper line | 13:19 |
| ralonsoh | server = neutron_service.Service.create( | 13:19 |
| ralonsoh | binary=constants.AGENT_PROCESS_DHCP, | 13:19 |
| ralonsoh | topic=topics.DHCP_AGENT, | 13:19 |
| ralonsoh | report_interval=cfg.CONF.AGENT.report_interval, | 13:19 |
| ralonsoh | manager='neutron.agent.dhcp.agent.DhcpAgentWithStateReport') | 13:19 |
| TheJulia | ralonsoh: https://bugs.launchpad.net/neutron/+bug/2127038 has links | 13:20 |
| ralonsoh | `DhcpAgentWithStateReport`, that inherits from `DhcpAgent`, also inherits from `Manager` | 13:20 |
| ralonsoh | we should implement the "stop()" method in `DhcpAgent`, stopping the _process_monitor | 13:21 |
| ralonsoh | lajoskatona, agree? | 13:21 |
| ralonsoh | TheJulia, thanks! | 13:22 |
| ralonsoh | TheJulia, so they are failing 100% of the times these jobs | 13:23 |
| ralonsoh | ok, that means something is broken in Neutron | 13:23 |
| TheJulia | ralonsoh: the ipv4 version, basically 100% of the time, the ipv6 version, maybe 80% | 13:24 |
| ralonsoh | TheJulia, can you check depending on a Neutron patch? | 13:25 |
| TheJulia | Generally for v4, we've almost never seen a successful dhcp in the logs until we ended up changing cleaning which likely pushed a race condition around the proverbial board by making the job have a different run time and performance profile. v6 sometimes works, sometimes fails, but sporatically enough that the jobs generally fail | 13:25 |
| TheJulia | ralonsoh: which depending patch? (still waking up) | 13:25 |
| ralonsoh | not proposed yet, just asking if that is possible | 13:26 |
| TheJulia | ralonsoh: I've pondered pulling out the patches individually on neutron just to identify it | 13:26 |
| TheJulia | ralonsoh: oh, absolutely | 13:26 |
| ralonsoh | I think the first one you sent https://review.opendev.org/c/openstack/neutron/+/962191 is the culprit | 13:26 |
| TheJulia | Yeah, that is the one I'm suspecting as well | 13:26 |
| lajoskatona | ralonsoh: I added the stop() to the DhcpAgent class, and based on the logs it is executed | 13:26 |
| TheJulia | I shifted gears to try and reproduce locally but haven't quite gotten there entirely yet | 13:26 |
| TheJulia | (unfortunately, gaming laptop with lots of ram has an entirely different performance profile to CI) | 13:27 |
| opendevreview | Rodolfo Alonso proposed openstack/neutron master: WIP == Revert "[OVN] Always exclude the local chassis creating ``HA_Chassis_Group``" https://review.opendev.org/c/openstack/neutron/+/963390 | 13:27 |
| ralonsoh | TheJulia, ^ | 13:27 |
| TheJulia | k, going to create a wip | 13:28 |
| TheJulia | ralonsoh: I'll keep an eye on https://zuul.opendev.org/t/openstack/status?change=963342 and let you know what the results are in ~1hr | 13:29 |
| TheJulia | ralonsoh: thanks! | 13:30 |
| opendevreview | Lajos Katona proposed openstack/neutron-vpnaas-dashboard master: Migrate from neutronclient to SDK https://review.opendev.org/c/openstack/neutron-vpnaas-dashboard/+/960388 | 13:30 |
| opendevreview | Lajos Katona proposed openstack/neutron-vpnaas-dashboard master: Migrate from neutronclient to SDK https://review.opendev.org/c/openstack/neutron-vpnaas-dashboard/+/960388 | 13:33 |
| opendevreview | Vladimir Kozhukalov proposed openstack/neutron master: Fix periodic_resync thread in DHCP agent https://review.opendev.org/c/openstack/neutron/+/962871 | 13:54 |
| opendevreview | Slawek Kaplonski proposed openstack/neutron-fwaas master: Fix bindep for rpm platforms https://review.opendev.org/c/openstack/neutron-fwaas/+/963405 | 14:44 |
| TheJulia | ralonsoh: Both of ironic's ovn jobs passed with that revert, I'll recheck it for good measure to ensure its not just a fluke. That being said both console logs are problem free as it relates to dhcping | 14:47 |
| ralonsoh | TheJulia, tomorrow I'll investigate this patch and propose a solution. By default, I'll propose this revert. Give me until tomorrow morning | 14:48 |
| TheJulia | No worries, thanks for the assistance! | 14:48 |
| *** sfinucan is now known as stephenfin | 15:44 | |
| opendevreview | Jakub Libosvar proposed openstack/neutron master: WIP: new job tempest-multinode-with-bgp https://review.opendev.org/c/openstack/neutron/+/962188 | 16:31 |
| opendevreview | Lajos Katona proposed openstack/neutron master: WIP: [eventlet]: Handle stop DHCP agent https://review.opendev.org/c/openstack/neutron/+/963386 | 17:34 |
| opendevreview | Vladimir Kozhukalov proposed openstack/neutron master: Fix periodic_resync thread in DHCP agent https://review.opendev.org/c/openstack/neutron/+/962871 | 17:47 |
| opendevreview | Brian Haley proposed openstack/neutron master: Make sure OVN PG is present before adding a SG rule https://review.opendev.org/c/openstack/neutron/+/946216 | 20:43 |
| opendevreview | Jakub Libosvar proposed openstack/neutron master: WIP: new job tempest-multinode-with-bgp https://review.opendev.org/c/openstack/neutron/+/962188 | 20:58 |
Generated by irclog2html.py 4.0.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!