| haleyb | ozzzo: well, it does say you have 4 dead agents, usually that output is a table. either way you should start looking at log files, there | 00:05 |
|---|---|---|
| haleyb | there's not much to go on in that pasted, and it's eod for me... | 00:05 |
| ozzzo | haleyb: right. That 2nd paste is what I get when I have dead agents. I posted that to contrast it with what I have now (first paste), in which I have no dead agents, and no 2-minute build delay, but still get the "Unable to schedule network" warning | 02:23 |
| opendevreview | Fernando Royo proposed openstack/ovn-octavia-provider master: Replace python-neutronclient with openstacksdk https://review.opendev.org/c/openstack/ovn-octavia-provider/+/870514 | 07:01 |
| opendevreview | Lajos Katona proposed openstack/neutron-specs master: ERSPAN for tap-as-a-service https://review.opendev.org/c/openstack/neutron-specs/+/882151 | 07:44 |
| opendevreview | Elvira García Ruiz proposed openstack/neutron stable/zed: [OVN] Update ovn meter when neutron server reloads https://review.opendev.org/c/openstack/neutron/+/882689 | 08:02 |
| opendevreview | Elvira García Ruiz proposed openstack/neutron stable/yoga: [OVN] Update ovn meter when neutron server reloads https://review.opendev.org/c/openstack/neutron/+/882527 | 08:04 |
| opendevreview | Elvira García Ruiz proposed openstack/neutron stable/wallaby: [OVN] Update ovn meter when neutron server reloads https://review.opendev.org/c/openstack/neutron/+/882711 | 08:09 |
| opendevreview | Elvira García Ruiz proposed openstack/neutron stable/xena: [OVN] Update ovn meter when neutron server reloads https://review.opendev.org/c/openstack/neutron/+/882809 | 08:10 |
| opendevreview | Arnau Verdaguer proposed openstack/neutron master: [OVN trunk][WIP] Add bound info on subport when parent is bound https://review.opendev.org/c/openstack/neutron/+/882581 | 08:42 |
| opendevreview | Rodolfo Alonso proposed openstack/neutron-lib master: Add port hardware offload extension https://review.opendev.org/c/openstack/neutron-lib/+/882726 | 08:45 |
| obondarev | ralonsoh: Hi Rodolfo, I apologise for being dumb that I forgot to send bug report for the last week, I got lost in vacation & holidays. Please assign the next vacant shift to me, and one more as a penalty | 09:21 |
| ralonsoh | obondarev, no problem, don't worry, for the next cycle | 09:21 |
| ralonsoh | elvira, is doing the triage this week | 09:21 |
| obondarev | and thanks for covering me | 09:22 |
| ralonsoh | yw | 09:24 |
| opendevreview | Rodolfo Alonso proposed openstack/neutron master: Replace "tenant_id" with "project_id" in IPAM engine https://review.opendev.org/c/openstack/neutron/+/877533 | 09:28 |
| opendevreview | Merged openstack/ovn-octavia-provider master: Replace python-neutronclient with openstacksdk https://review.opendev.org/c/openstack/ovn-octavia-provider/+/870514 | 09:36 |
| opendevreview | Fernando Royo proposed openstack/ovn-octavia-provider master: Fix update member action https://review.opendev.org/c/openstack/ovn-octavia-provider/+/880945 | 09:37 |
| elvira | ralonsoh: Hmm Wasn't isabek this week? I mean I can do this week with no problem but I think I have next week | 09:40 |
| ralonsoh | ahh sorry, yes | 09:40 |
| ralonsoh | my bad | 09:40 |
| elvira | np :) | 09:40 |
| opendevreview | Merged openstack/ovn-octavia-provider master: Update doc about ovn-octavia HM type limitation https://review.opendev.org/c/openstack/ovn-octavia-provider/+/879978 | 10:02 |
| opendevreview | Felix Huettner proposed openstack/neutron-dynamic-routing master: fix unittest issues caused by neutron rbac changes https://review.opendev.org/c/openstack/neutron-dynamic-routing/+/882815 | 10:17 |
| ralonsoh | slaweq, ^^ because of https://review.opendev.org/c/openstack/neutron/+/879827 | 10:32 |
| ralonsoh | we should push a new neutron release today | 10:32 |
| ralonsoh | this change is important enough to be released asap | 10:32 |
| slaweq | ralonsoh we are in the M-1 week of Bobcat cycle so we can do M1 release | 10:34 |
| ralonsoh | exactly | 10:34 |
| ralonsoh | I'll push the change now | 10:34 |
| ralonsoh | and thank for the confirmation | 10:34 |
| slaweq | thank You | 10:38 |
| lajoskatona | slaweq, ralonsoh: thanks for the coming neutron release as I see other stadiums also affected | 10:52 |
| opendevreview | Slawek Kaplonski proposed openstack/neutron-tempest-plugin master: [S-RBAC] Update DSCP marking rule create API test https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/882818 | 10:52 |
| ralonsoh | I'm asking in openstack-release | 10:52 |
| ralonsoh | just to be sure | 10:52 |
| opendevreview | Slawek Kaplonski proposed openstack/neutron master: [S-RBAC] Get QoS rule types API available for READER role https://review.opendev.org/c/openstack/neutron/+/882414 | 10:53 |
| slaweq | ralonsoh ^^ now I think it should be finally green with all QoS related tests :) | 10:53 |
| ralonsoh | slaweq, checking now | 10:53 |
| opendevreview | Luis Tomas Bolivar proposed openstack/ovsdbapp master: Add support for ecmp routes https://review.opendev.org/c/openstack/ovsdbapp/+/882268 | 10:54 |
| ralonsoh | folks: https://review.opendev.org/c/openstack/releases/+/882819 | 11:21 |
| ralonsoh | Neutron beta 1 release ^^ please use that for the stadium projects, in the requirements | 11:22 |
| opendevreview | Luis Tomas Bolivar proposed openstack/ovsdbapp master: Add support for ecmp routes https://review.opendev.org/c/openstack/ovsdbapp/+/882268 | 12:06 |
| opendevreview | Fernando Royo proposed openstack/ovn-octavia-provider master: Apply admin_state_up on a new member creation https://review.opendev.org/c/openstack/ovn-octavia-provider/+/880737 | 12:49 |
| sahid | o/ we have a query that is happening quite often, i'm trying to identify which coponent is generating it | 12:53 |
| sahid | https://pastebin.ubuntu.com/p/NCNNfrgJRv/ | 12:54 |
| sahid | perhaps slaweq you may have an idea as it is related to rbacs | 12:54 |
| sahid | this query returns more than 400 000 rows | 12:54 |
| sahid | number of networks * number of subnets * numbers of rbacs | 12:55 |
| slaweq | sahid give me few minutes, I will check it | 12:55 |
| sahid | slaweq: sure nothing hurry, thanks a lot | 12:55 |
| slaweq | sahid I'm not sure TBH but it's probably from the RBAC code: https://github.com/openstack/neutron/blob/master/neutron/db/rbac_db_models.py#L109 | 13:03 |
| slaweq | it's "different" RBAC than our API policies which we are calling "Secure RBAC" | 13:04 |
| slaweq | it's probably checking somewhere if network is shared or not | 13:04 |
| sahid | thank you :-) i will investigate further | 13:07 |
| opendevreview | Luis Tomas Bolivar proposed openstack/ovsdbapp master: Ensure LrpAddCommand may_exits works without peers https://review.opendev.org/c/openstack/ovsdbapp/+/882269 | 13:11 |
| ozzzo_work | When I create a VM on my kolla-ansible train cluster I see this in neutron-server.log: https://paste.openstack.org/show/busV3eny89jYcT7HBd8O/ | 13:13 |
| ozzzo_work | Does the "Unable to schedule network" warning indicate a problem? | 13:14 |
| ozzzo_work | It's not a dead agent; all of my agents are :-) | 13:15 |
| ozzzo_work | and when I do have dead agents, I see a different set of symptoms | 13:15 |
| ozzzo_work | The VM is created fine, without the 2-minute delay that I see when I have dead agents, but it throws this error for every VM that I create on every cluster (kolla-ansible Train) | 13:16 |
| ralonsoh | this is because you don't have agents assigned to this network | 13:17 |
| ozzzo_work | ralonsoh: Is there a document on managing agent->network assignments? | 13:17 |
| ralonsoh | I don't recall this | 13:18 |
| ralonsoh | do you have routed networks? | 13:18 |
| ralonsoh | do you have different segments? | 13:18 |
| ralonsoh | ozzzo_work, what is the output of | 13:19 |
| ralonsoh | openstack network agent list --network <networkid> | 13:19 |
| ozzzo_work | I don't remember the answer to your routed networks question. I haven't looked at our networking for a while. | 13:21 |
| ozzzo_work | I get a blank list when I list agents by network | 13:22 |
| ralonsoh | so you don't have any dhcp agent attending to this network | 13:22 |
| ralonsoh | check the dhcp agents status | 13:22 |
| ralonsoh | most probably they are in fail state | 13:22 |
| ralonsoh | do you have HA? | 13:22 |
| ozzzo_work | When you say HA, do you mean 3 controllers? | 13:23 |
| ralonsoh | yes but specifically what is the value for dhcp_agents_per_network? | 13:24 |
| ralonsoh | this is what schedules the networks to the dhcp agents | 13:25 |
| ozzzo_work | This is what my network agents look like: | 0ef02d8f-5c3c-42e2-a95a-3cfd38a27bac | Open vSwitch agent | qde3-compute502.cloud.ourdomain.com | None | :-) | UP | neutron-openvswitch-agent | | 13:25 |
| ozzzo_work | group_vars/de6/vars.yml:dhcp_agents_per_network: 2 | 13:25 |
| ralonsoh | so per network you should have 2 dhcp agents | 13:26 |
| ralonsoh | is that is not the case, then there is something wrong in the dhcp agents | 13:26 |
| ralonsoh | check the logs | 13:26 |
| ralonsoh | or try restarting them, 1 by 1 | 13:26 |
| ozzzo_work | Are dhcp agents different from network agents? | 13:26 |
| ralonsoh | dhcp agents are specific network agents | 13:26 |
| ralonsoh | you have have ovs agent, linux bridge agent, metadata agent, etc | 13:27 |
| ozzzo_work | is there a document I should read to understand neutron agents? When I list agents I only see neutron-openvswitch-agent | 13:28 |
| ozzzo_work | 1 per compute | 13:28 |
| ralonsoh | https://docs.openstack.org/neutron/pike/admin/archives/config-agents.html | 13:29 |
| ozzzo_work | my clusters are working, it's just that I see this error. Could it be that we're not using DHCP and we're hitting this bug: https://bugs.launchpad.net/networking-ovn/+bug/1647421 | 13:29 |
| ozzzo_work | ok reading, ty! | 13:29 |
| ralonsoh | hold on | 13:29 |
| ralonsoh | hold on | 13:29 |
| ralonsoh | are you using OVN? | 13:29 |
| ralonsoh | ozzzo_work, ^ | 13:30 |
| ozzzo_work | ovn is openvswitch? | 13:31 |
| ralonsoh | no, what ML2 mech driver are you using? | 13:31 |
| ralonsoh | [ml2] | 13:31 |
| ralonsoh | mechanism_drivers = ... | 13:31 |
| ozzzo_work | my co-workers tell me that we're not using ovn | 13:32 |
| ralonsoh | so the ovn bug is not related | 13:33 |
| ralonsoh | you should check the status of your dhcp agents | 13:33 |
| ralonsoh | and health them | 13:33 |
| ozzzo_work | should the dhcp agents show up in "network agent list" output? | 13:34 |
| ralonsoh | this is a standalone (one single computer) deployment | 13:34 |
| ralonsoh | https://paste.opendev.org/show/bPOjGJ0Md3X0vWPtn1vw | 13:34 |
| ozzzo_work | neutron_plugin_agent: "openvswitch" | 13:34 |
| ozzzo_work | my list looks different from that; I only have 1 agent per compute; it's the neutron-openvswitch-agent | 13:35 |
| ozzzo_work | but my VMs get an IP somehow | 13:35 |
| ralonsoh | because you are using config drive | 13:36 |
| ozzzo_work | ralonsoh: I don't see anything about config drive in the config-agents document. Where can I read about that? | 13:40 |
| ralonsoh | this is in nova documentation | 13:40 |
| ozzzo_work | ok I'm reading https://docs.openstack.org/nova/train/admin/config-drive.html | 13:44 |
| opendevreview | Rodolfo Alonso proposed openstack/neutron master: Add a "port" child table "porthardwareoffload" https://review.opendev.org/c/openstack/neutron/+/882832 | 14:23 |
| *** iurygregory_ is now known as iurygregory | 14:25 | |
| opendevreview | Rodolfo Alonso proposed openstack/neutron master: [DNM] [WIP] [OVN] Use the API context in the ``OVNClient._add_router_ext_gw`` method https://review.opendev.org/c/openstack/neutron/+/882857 | 14:57 |
| opendevreview | Rodolfo Alonso proposed openstack/neutron master: [sqlalchemy-20] Add the transaction context to the upgrade checks methods https://review.opendev.org/c/openstack/neutron/+/882865 | 15:01 |
| opendevreview | Rodolfo Alonso proposed openstack/ovn-octavia-provider master: [DNM] Test issue with "test_port_forwarding" https://review.opendev.org/c/openstack/ovn-octavia-provider/+/882872 | 15:16 |
| ralonsoh | froyo, ^ | 15:16 |
| froyo | ralonsoh, thx | 15:17 |
| opendevreview | Rodolfo Alonso proposed openstack/neutron master: [OVN] Use the API context in ``OVNClient._add_router_ext_gw`` method https://review.opendev.org/c/openstack/neutron/+/882857 | 15:54 |
| *** gthiemon1e is now known as gthiemonge | 15:55 | |
| opendevreview | Rodolfo Alonso proposed openstack/ovn-octavia-provider master: [DNM] Test issue with "test_port_forwarding" https://review.opendev.org/c/openstack/ovn-octavia-provider/+/882872 | 15:55 |
| opendevreview | Merged openstack/neutron master: Mark "ipv6_pd_enabled" as deprecated and experimental. https://review.opendev.org/c/openstack/neutron/+/879030 | 16:10 |
| ozzzo_work | ralonsoh: It looks like we are using provider networking. Does that mean that we have an external DHCP server? | 16:35 |
| ralonsoh | what does it mean provider networking? | 16:36 |
| opendevreview | Stephen Finucane proposed openstack/neutron master: docs: Deindent code blocks https://review.opendev.org/c/openstack/neutron/+/882885 | 16:38 |
| ozzzo_work | https://docs.openstack.org/install-guide/launch-instance-networks-provider.html | 16:38 |
| ralonsoh | so you are using an external network | 16:39 |
| ralonsoh | you can have DHCP agents on these networks but you can also use external ones | 16:39 |
| ralonsoh | up to you | 16:39 |
| ozzzo_work | I think we must have an external DHCP server, because there are no DHCP agents in my "os network agent list" output | 16:41 |
| ozzzo_work | We are using config drive. Would the config drive be getting the IP from an external DHCP server? | 16:50 |
| opendevreview | Rodolfo Alonso proposed openstack/ovn-octavia-provider master: [DNM] Test issue with "test_port_forwarding" https://review.opendev.org/c/openstack/ovn-octavia-provider/+/882872 | 16:54 |
| ralonsoh | ozzzo_work, please, collect the information of your deployment and open a LP bug. Before raising a question, please check that you have all the information | 16:55 |
| ozzzo_work | I don't think it's an Openstack bug; I think we just have a weird config | 17:26 |
| opendevreview | Miguel Lavalle proposed openstack/neutron master: [WIP][DNM][OVN] Change the default firewall policy https://review.opendev.org/c/openstack/neutron/+/839066 | 22:49 |
| opendevreview | Miguel Lavalle proposed openstack/neutron master: [WIP][DNM][OVN] Change the default firewall policy https://review.opendev.org/c/openstack/neutron/+/839066 | 22:52 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!