| opendevreview | Merged openstack/openstack-ansible master: Use nodepool centos and epel mirrors for CI jobs on centos-9 https://review.opendev.org/c/openstack/openstack-ansible/+/845112 | 01:59 |
|---|---|---|
| opendevreview | Merged openstack/openstack-ansible master: Freeze Ansible Roles for Beta release https://review.opendev.org/c/openstack/openstack-ansible/+/844874 | 02:10 |
| *** ysandeep|out is now known as ysandeep | 06:37 | |
| jrosser_ | so we must be ready to make an RC | 07:34 |
| *** chkumar|ruck is now known as raukadah | 07:54 | |
| noonedeadpunk | jrosser_: yes! thanks! | 08:19 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Unfreeze roles after milestone https://review.opendev.org/c/openstack/openstack-ansible/+/845092 | 08:20 |
| noonedeadpunk | lets's also land this^ | 08:20 |
| *** ysandeep is now known as ysandeep|lunch | 08:22 | |
| noonedeadpunk | I pushed https://review.opendev.org/c/openstack/releases/+/845239 for release | 09:07 |
| *** ysandeep|lunch is now known as ysandeep | 10:09 | |
| *** dviroel|afk is now known as dviroel | 11:25 | |
| jrosser_ | ooh brace for branching i think....... | 11:25 |
| noonedeadpunk | well yeah, we need to branch indeed. But I was thinking to improve some things across roles next week. And we must release until June 23rd indeed | 11:30 |
| noonedeadpunk | interesting thing - why we don't have any content inside /etc/yum in CI logs? | 11:35 |
| noonedeadpunk | as $stream in repos should go from /etc/yum/vars/stream | 11:36 |
| mgariepy | is there any issue that needs to be taken care of ? or need some eyes ? | 11:52 |
| *** ysandeep is now known as ysandeep|out | 12:11 | |
| *** ysandeep|out is now known as ysandeep|afk | 12:12 | |
| *** ysandeep|afk is now known as ysandeep | 12:53 | |
| opendevreview | Merged openstack/openstack-ansible master: Unfreeze roles after milestone https://review.opendev.org/c/openstack/openstack-ansible/+/845092 | 13:41 |
| jamesdenton | Hi damiandabrowski[m] - re: https://bugs.launchpad.net/neutron/+bug/1859832 - I assume you experience some L3 HA failover issues which prompted your patch? Do you recall what you were seeing? | 14:38 |
| damiandabrowski[m] | jamesdenton: hi! I've described my case in another bug report, the fix was merged ~2 weeks ago https://bugs.launchpad.net/neutron/+bug/1952907 | 14:40 |
| jamesdenton | ahh there it is, thank you | 14:40 |
| jamesdenton | do you recall ever seeing a situation where qg didn't come up at all? | 14:41 |
| damiandabrowski[m] | not really :/ in my case I've seen issues only when specific floating IP was assigned to several routers in a short period of time(4h in my case == cache time on a switch) | 14:44 |
| damiandabrowski[m] | so let's imagine You spawn a router an add floating IP 1.2.3.4, then You delete this router, create another one with the same floating ip but it lands on different net node. | 14:44 |
| jamesdenton | oh arp cache issues huh | 14:44 |
| damiandabrowski[m] | Then the switch will have outdated entry in update mac-address-table because neutron fails to send GARPs. | 14:44 |
| jamesdenton | yep ok | 14:44 |
| jamesdenton | we had a U->V upgrade two weeks ago where the router "failed over" via keepalived but qg didn't come up and there was no route insertion, and without debug it was hard to tell what exactly happened or didn't happen. The fix was the pop into the namespace and 'ip list set qg up; ip route add default', but i sorta suspect the underlying condition is/was related to what your patch addresses. So i'm going with that :D | 14:46 |
| damiandabrowski[m] | good luck :D i've seen some other issues with l3 agents, noonedeadpunk may remember more details, but I can recall some stale keepalived processes(we had to kill them manually to get routers back working) | 14:49 |
| damiandabrowski[m] | but during V upgrade we've evacuated net nodes prior to the upgrade | 14:49 |
| jamesdenton | ^^^ what did this entail? | 14:49 |
| damiandabrowski[m] | killing keepalived processes or evacuating net nodes prior to the upgrade? | 14:50 |
| jamesdenton | yeah, i guess i'm just curious what you mean by evacuating the net nodes | 14:51 |
| damiandabrowski[m] | ahh, moving all l3 and dhcp agents somewhere else | 14:52 |
| damiandabrowski[m] | so during openstack upgrade, net node was empty | 14:53 |
| damiandabrowski[m] | (we were upgrading these nodes one by one) | 14:53 |
| jamesdenton | gotcha - wasn't sure how many net nodes you had and # l3 or dhcp agents per network/route | 14:53 |
| jamesdenton | oh i see, is that your typical process then? | 14:53 |
| jamesdenton | for V or any other? | 14:53 |
| damiandabrowski[m] | IIRC we try to have less than 200-300 l3 agents per net node, otherwise You may hit weird problems | 14:55 |
| damiandabrowski[m] | we have a custom script to "evacuate" all agents from net node | 14:55 |
| damiandabrowski[m] | then we perform an upgrade and move agents back there | 14:55 |
| jamesdenton | s/agents/routers i assume | 14:56 |
| jamesdenton | gotcha | 14:56 |
| damiandabrowski[m] | yup | 14:56 |
| jamesdenton | how many net nodes do you have, then? | 14:56 |
| damiandabrowski[m] | but AFAIK during X upgrade we stopped evacuating net nodes, however i heard about some issues during an upgrade | 14:56 |
| jamesdenton | X upgrade... living on the edge :D | 14:56 |
| damiandabrowski[m] | depends on the region, 3-10 | 14:56 |
| damiandabrowski[m] | :D | 14:57 |
| jamesdenton | are you OVS now? | 14:57 |
| damiandabrowski[m] | yeah, we're thinking about moving to OVN but we have other prios atm :/ | 14:57 |
| jamesdenton | understood | 14:58 |
| jamesdenton | appreciate the help here! | 14:58 |
| damiandabrowski[m] | You're welcome! | 15:00 |
| *** dviroel is now known as dviroel|lunch | 15:01 | |
| *** ysandeep is now known as ysandeep|out | 15:29 | |
| mgariepy | damiandabrowski[m], is your script to evacuate the agent somewhere i can see it ? | 15:53 |
| admin1 | what i have seen working is if for any reasons in upgrade a l3/dhcp node does not work, simply migrate it to another node . and it will recreate | 16:00 |
| admin1 | with script, can do this in an instant | 16:00 |
| *** dviroel|lunch is now known as dviroel | 16:19 | |
| *** ysandeep|out is now known as ysandeep | 16:34 | |
| *** ysandeep is now known as ysandeep|out | 17:29 | |
| circ-user-JxoNj | Sorry - first time on IRC. Anybody there? | 22:13 |
| *** circ-user-JxoNj is now known as kdhallbgm_ | 22:15 | |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!