| *** mmalchuk_ is now known as mmalchuk | 07:20 | |
| jakeyip | I'm aaround if anyone wants to talk magnum | 08:57 |
|---|---|---|
| andrewbonney | Hi. We wanted to bring up https://bugs.launchpad.net/magnum/+bug/2067345 at the meeting if possible | 08:58 |
| jakeyip | ok | 09:02 |
| jakeyip | #startmeeting magnum | 09:03 |
| opendevmeet | Meeting started Wed Aug 28 09:03:03 2024 UTC and is due to finish in 60 minutes. The chair is jakeyip. Information about MeetBot at http://wiki.debian.org/MeetBot. | 09:03 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 09:03 |
| opendevmeet | The meeting name has been set to 'magnum' | 09:03 |
| jakeyip | #link https://etherpad.opendev.org/p/magnum-weekly-meeting | 09:03 |
| jakeyip | Please put your topics into to Agenda | 09:03 |
| jakeyip | #topic Roll Call | 09:03 |
| jakeyip | o/ | 09:03 |
| jakeyip | mnasiadka / dalees if you are around | 09:04 |
| mnasiadka | o/ | 09:04 |
| mnasiadka | I'm here | 09:04 |
| dalees | o/ | 09:04 |
| dalees | I'm around, sort of. | 09:04 |
| jakeyip | cool let's get on with it :) | 09:06 |
| jakeyip | #topic QueuePool limit bug | 09:07 |
| jakeyip | #link https://bugs.launchpad.net/magnum/+bug/2067345 | 09:07 |
| jakeyip | andrewbonney: I believe this is from you | 09:07 |
| andrewbonney | Yeah, I just wanted to raise it again as we've seen the same as others since upgrading to C | 09:07 |
| andrewbonney | It's pretty major as we have to restart Magnum services frequently to keep things working | 09:07 |
| jakeyip | did the patches fix things? | 09:08 |
| andrewbonney | I haven't applied them personally yet as patching oslo.db is a little involved, but given other services are using oslo.db without issue I was a little surprised that might be required | 09:08 |
| jakeyip | yeah I am not sure where the bug is, as I haven't encountered it in prod (we are still at B). | 09:11 |
| jakeyip | I was planning to get to C then I can debug, but unfortunately I had to chase down a few bugs in other places affecting our deployment of Magnum, so C upgrade got delayed | 09:12 |
| jakeyip | how about mnasiadka or dalees ? | 09:12 |
| dalees | Likewise, not running C yet; CAPI driver has got most of my attention for now and Magnum version isn't the limitation anymore. | 09:14 |
| jakeyip | if I was to guess, it may have been something introduced by us trying to bring sqlalchemy up to date | 09:16 |
| andrewbonney | I did have a look at the code around those changes but nothing jumped out unfortunately | 09:17 |
| jakeyip | andrewbonney: are you able to help us test by rolling back those commits? | 09:17 |
| jakeyip | #link https://review.opendev.org/c/openstack/magnum/+/910722 | 09:17 |
| jakeyip | #link https://review.opendev.org/c/openstack/magnum/+/910512 | 09:17 |
| mnasiadka | we are going to work on upgrades to C - so sooner or later this year we'll probably stumble on the same issue | 09:18 |
| jakeyip | andrewbonney: which driver are you using? | 09:18 |
| andrewbonney | We're running the vexxhost CAPI integration | 09:19 |
| andrewbonney | I'm happy to try rolling stuff back, but that will also involve pinning oslo.db back to ensure compatibility with the autocommit changes | 09:21 |
| jakeyip | will reverting just the autocommit change https://review.opendev.org/c/openstack/magnum/+/910722 fail ? | 09:25 |
| andrewbonney | If we stick with oslo.db 15 from upper-constraints I believe so yes | 09:25 |
| jakeyip | what are the magnum / oslo.db versions you are running now? | 09:26 |
| andrewbonney | Magnum 18.0.1, oslo.db 15.0.0 | 09:27 |
| jakeyip | sqlalchemy? | 09:28 |
| andrewbonney | 1.4.51 | 09:28 |
| dalees | andrewbonney: what is the pattern you see with db connections, how quickly do they rise with approx how many clusters? similar to https://bugs.launchpad.net/magnum/+bug/2067345/comments/12 ? | 09:29 |
| andrewbonney | I can go away and collect some data. Looking at our logs it takes maybe 3 days from service restart to start seeing errors, but this is with 1-3 clusters present at any one time | 09:31 |
| andrewbonney | We're running all this in a staging environment at present | 09:31 |
| dalees | I'll try an upgrade in development env soon, and see if I can reproduce the issues. | 09:34 |
| jrosser | what andrewbonney is describing is an environment where we do man create/delete of a small number of clusters | 09:35 |
| jrosser | rather than having a large number of clusters that is long lived | 09:35 |
| jrosser | *many create/delete | 09:35 |
| jakeyip | andrewbonney: another thing you can try is try this patch https://review.opendev.org/c/openstack/magnum/+/926626 | 09:35 |
| andrewbonney | Will do, ta | 09:36 |
| jakeyip | this switches over the code from the legacy facade to the new one introduced in 2024.1, possibly fixing the issue | 09:36 |
| jakeyip | no sorry, not introduced in 2024.1, introduced many years ago | 09:37 |
| jakeyip | I think rolling forward to https://review.opendev.org/c/openstack/magnum/+/926626 is prob the best choice | 09:40 |
| andrewbonney | I'll give that a go and feed back in the issue after I've got some data on connections | 09:44 |
| jakeyip | thanks! | 09:45 |
| jakeyip | anything else? | 09:49 |
| jakeyip | ok thanks everyone for coming | 09:54 |
| jakeyip | #endmeeting | 09:54 |
| opendevmeet | Meeting ended Wed Aug 28 09:54:07 2024 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 09:54 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/magnum/2024/magnum.2024-08-28-09.03.html | 09:54 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/magnum/2024/magnum.2024-08-28-09.03.txt | 09:54 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/magnum/2024/magnum.2024-08-28-09.03.log.html | 09:54 |
| jakeyip | BTW anyone going to OpenInfra Asia next week? | 09:54 |
| mnasiadka | Yes, I'll be there | 10:25 |
| opendevreview | Andrew Bonney proposed openstack/magnum-ui master: Fix master_lb_enabled not following template during cluster create https://review.opendev.org/c/openstack/magnum-ui/+/927387 | 12:55 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!