| oschwart | hey folks | 13:05 |
|---|---|---|
| oschwart | I am using the threading backend here https://review.opendev.org/c/openstack/designate/+/942773 | 13:06 |
| oschwart | but designate doesn't really run - it looks like (designate component) central either doesn't get messages from the api, or cannot respond | 13:06 |
| oschwart | I see the following logs | 13:07 |
| oschwart | DEBUG oslo_messaging._drivers.amqpdriver [-] AMQPListener connection timeout | 13:07 |
| oschwart | DEBUG oslo_messaging._drivers.amqpdriver [-] Listener is running | 13:07 |
| oschwart | DEBUG oslo_messaging._drivers.amqpdriver [-] AMQPListener connection consume | 13:07 |
| oschwart | I am not really sure how to debug those, any ideas? | 13:07 |
| hberaud[m] | What is your oslo.messaging executor? | 13:08 |
| oschwart | threading | 13:11 |
| oschwart | Jun 17 13:01:16 ubuntu-debug designate-central[205469]: DEBUG oslo_messaging._utils [-] Using a threading executor {{(pid=205469) get_executor_with_context /opt/stack/data/venv/lib/python3.10/site-packages/oslo_messaging/_utils.py:90}} | 13:11 |
| oschwart | (I was just grepping the logs, no other meaning for that specific log print) | 13:12 |
| hberaud[m] | ack | 13:33 |
| hberaud[m] | Can you check if heartbeat_in_pthread is enabled, disabled? | 13:35 |
| oschwart | from the logs I see that it is disabled | 13:36 |
| hberaud[m] | Maybe there is an issue with the way your amqp heartbeat is running, making your connection to be resets | 13:37 |
| hberaud[m] | Just to isolate some potential side bugs, Is oslo.messaging's threading executor works properly when the oslo.service' threading backend is disabled? | 13:41 |
| hberaud[m] | I can see you call twice this kind of code: import oslo_service.backend as service | 13:42 |
| hberaud[m] | service.init_backend(service.BackendType.THREADING) | 13:42 |
| hberaud[m] | which is, if I correctly remember, something that should be only did once, or at least ignored if made several times damani : correct? | 13:44 |
| hberaud[m] | damani: For context ^ | 13:44 |
| damani[m] | hberaud, correcte about the call, but normally i have made a patch to fix that | 13:46 |
| hberaud[m] | damani: thanks. Please, do you mind having a look at oschwart's patch to see if you observe something that could explain the problem. | 13:48 |
| damani[m] | yes i'm on it | 13:52 |
| damani[m] | but at the moment doesn't really see something weird | 13:52 |
| damani[m] | but i continue to work on it | 13:52 |
| oschwart | thanks folks hberaud++ damani++ | 13:55 |
| hberaud[m] | oschwart: when you speak about "central" did you mean that you observe those failure in the unit tests about "central"? Just trying to understand your context https://review.opendev.org/c/openstack/designate/+/942773/17/designate/tests/functional/central/test_service.py | 14:00 |
| oschwart | oh I would ignore the unit tests (at least for now) - I see in the task summary (here) https://zuul.opendev.org/t/openstack/build/771471dddc6d4d4993161b735ba4ebc9 | 14:03 |
| oschwart | CRITICAL designate.manage.pool [None designate-manage None None] No response received from designate-central. Check it is running, and retry: oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to message ID 07ebc8eb6fd54e93b79c7403c8682f37 | 14:04 |
| hberaud[m] | ack | 14:04 |
| oschwart | when we deploy designate we run a `designate-manage pool` task to define a default pool. it should use oslo.messaging there and I think that's where it fails | 14:05 |
| hberaud[m] | ack | 14:07 |
| hberaud[m] | #startmeeting openstack-eventlet-removal | 15:02 |
| hberaud[m] | Ping list: amorin, JayF, gouthamr, ralonsoh,gibi | 15:02 |
| hberaud[m] | https://etherpad.opendev.org/p/flamingo-eventlet-tracking | 15:02 |
| JayF | \o | 15:06 |
| hberaud[m] | o/ | 15:06 |
| hberaud[m] | I think we can start | 15:06 |
| hberaud[m] | #topic Sharing updates | 15:07 |
| hberaud[m] | So | 15:07 |
| hberaud[m] | I've a couple of good news from the eventlet side | 15:07 |
| * gibi lurks but on a paralle meeting as well | 15:07 | |
| hberaud[m] | the python 3.13 problems seems under control, Itamar found the root causes and proposed fix that is now merged | 15:08 |
| hberaud[m] | the problem was in the usage of fork, you can find more details here https://github.com/eventlet/eventlet/pull/1047 | 15:08 |
| hberaud[m] | and also into the previous discussion of this channel (days ago | 15:09 |
| hberaud[m] | ) | 15:09 |
| hberaud[m] | Guillaume confirmed that it solved the Nova problem originally reported. | 15:10 |
| JayF | \o/ | 15:10 |
| hberaud[m] | But we still observe the garbage collector problem, even with that patch https://github.com/eventlet/eventlet/issues/1032 | 15:10 |
| hberaud[m] | The root cause of this one is not still identified | 15:11 |
| hberaud[m] | This morning I started to prepare a new release of eventlet: 0.40.1 | 15:12 |
| hberaud[m] | This new version should land in the coming days | 15:12 |
| hberaud[m] | In paralllel of that I started a side discussion (in DM) with Daniel about the impact of fork into oslo.service and in openstack in general. I made a tiny analyze that I'll forward to Daniel, this way we should be able to tackle the problem identified by Itamar on this side of the fence. | 15:14 |
| JayF | itamarst ^ did you end up starting any work on the oslo.service side for this? | 15:15 |
| JayF | I know you were talking about how much it was needed :D | 15:15 |
| hberaud[m] | You're right to ask him the question :) | 15:16 |
| hberaud[m] | lets continue the conversation | 15:17 |
| hberaud[m] | do you have things that you want to add now? | 15:18 |
| JayF | Ironic is likely to merge our eventlet wsgi removal this week https://review.opendev.org/c/openstack/ironic/+/951054 | 15:18 |
| hberaud[m] | awesome | 15:18 |
| JayF | We're fixing an unrelated CI issue just to see the green on the -nv job before landing it :) | 15:18 |
| hberaud[m] | good job | 15:18 |
| JayF | good job cid :D | 15:18 |
| hberaud[m] | indeed kudos cid | 15:19 |
| JayF | I would highly recommend looking at cheroot for any other projects using the eventlet.wsgi server directly | 15:19 |
| JayF | the patterns appear to match up very well | 15:19 |
| hberaud[m] | yeah, I read the patch, that seems interesting, I was not aware of cheroot | 15:19 |
| JayF | and we saw no meaningful performance difference in benchmarking (although evnetlet is still monkey patched, so we're on fake threads) | 15:19 |
| JayF | if you've heard of cherrypy -- it's the backend library for that | 15:19 |
| hberaud[m] | okok | 15:20 |
| hberaud[m] | good to know that | 15:20 |
| JayF | As a note; I will not be at this meeting for the next 6 weeks as I'll be on an extended sabbatical leave. | 15:20 |
| hberaud[m] | ok move on | 15:20 |
| hberaud[m] | no problem, enjoy your abbatical leave | 15:20 |
| hberaud[m] | thanks for your heads up | 15:21 |
| hberaud[m] | s/sabbatical/ | 15:21 |
| hberaud[m] | #topic https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack.org/thread/BIC7BTAN72X6AA4BE6VVNSP7FYFOC362/ | 15:21 |
| hberaud[m] | I currently writing my answer to that thread | 15:21 |
| hberaud[m] | I wanted to let the other sub discussion to ends up before replying | 15:22 |
| JayF | I have been, based on oslo deprecation guidelines, telling my leadership that 2026.2 is the deadline for this eventlet migration from an Ironic/my standpoint. | 15:22 |
| JayF | I can't promise that means we *won't* help if something goes upside down if folks go beyond that, we're very nice people, but we have zero plans to at this point. | 15:23 |
| hberaud[m] | ack | 15:23 |
| JayF | Honestly, I'm more than a little frustrated my post regarding that existing advertised deadline was completely ignored. | 15:23 |
| hberaud[m] | My initial proposal regarding 2027.2 was made on the basis that at that point in the discussion we did not have to remove this type of dual concurrency mode - which is considered a feature removal and therefore falls within the SLURP policy. | 15:26 |
| JayF | This is a situation where a policy can say whatever it wants; python will march forward without us -- and without eventlet. IMO, it's a disservice to our users to even suggest we'll be able to keep it going that long. I'm worried folks will end up making promises which, in reality, will not be able to be fulfilled. | 15:27 |
| JayF | Ironic is not, and has zero plans to enable a dual-running mode, or to treat this as an operator-facing API deserving of deprecation, beyond any operator-facing configs that may be moving around. | 15:28 |
| hberaud[m] | I see several problem about B: it will require additional maintenance and resources (who are already more than scarse), and it will surely enter us into the support of python 3.14 and even potentially 3.15 | 15:28 |
| hberaud[m] | agree with you | 15:29 |
| hberaud[m] | I'd rather suggest to consider that question as a simple policy question and involving the TC and the release team to see if compromise could not be found | 15:30 |
| JayF | That's not really a can of worms I have an ability to open given my imminent leave. I think you'll do a good job of representing the "please don't try to keep eventlet on life support for another 2 years" argument | 15:32 |
| hberaud[m] | Python 3.14 comes with new post-GIL changes, meaning that it will surely impact eventlet in one way or another, and switching back our execution model to something broken won't help us in this situation | 15:32 |
| hberaud[m] | and the same is true with Python 3.15, where, this time, this is RLocks that are at the agenda... | 15:33 |
| hberaud[m] | RLocks is simply a nightmare with eventlet, I cannot count the number of back and forth we made due to rlocks | 15:34 |
| hberaud[m] | So yeah, here is my position in some sort (A.2) and I'd love to try to find a political compromise for this situation | 15:36 |
| hberaud[m] | That won't be the first that an mainstream open source project made some compromise with its internal policy | 15:37 |
| hberaud[m] | Anyway, I'll send my reply by tomorrow and I'll invite the TC and the release at the party | 15:38 |
| JayF | I think there's a small element of, even if operators asked for a switch, they may not know what's best for themselves. | 15:38 |
| hberaud[m] | Yes | 15:39 |
| hberaud[m] | #topic open discussion | 15:39 |
| hberaud[m] | anything else that you want to discuss today? | 15:39 |
| JayF | I'm good, thanks for running the meeting hberaud[m] o/ | 15:40 |
| hberaud[m] | You are welcome | 15:40 |
| hberaud[m] | thanks for joining | 15:40 |
| hberaud[m] | #endmeeting | 15:41 |
| JayF | looks like the startmeeting had a space before it | 15:41 |
| JayF | so this will be logged but not as an actual meeting | 15:41 |
| hberaud[m] | ah indeed, good catch | 15:41 |
| hberaud[m] | bad copy pasting | 15:41 |
| gibi | sorry I was not able to engage in this meeting due to the parallel one. I will read the mailing list replies but I want to note that it pretty possible that even if oslo want's to drop eventlet in 2026.2 nova will not be ready to drop it. | 15:56 |
| gibi | I'm doing my best to raise that the progress in nova is too slow both upstream and in my employer downstream | 15:57 |
| gibi | but the progress is what it is | 15:57 |
| gibi | unfortunately RHEL 10 is on py3.12 so I have limited leverage downstream | 15:58 |
| itamarst | JayF: I have some unrelated work that should probably be done first since it's already in progress | 16:07 |
| hberaud[m] | gibi: No problem. You make an excellent job, we do not want to put more pressure on you. In all the oslo won't drop anything until services are not ready. We just think that there is rooms for compromize between these oslo deprecation for 2026.2 and an extend of eventlet to 2028.1 | 16:24 |
| hberaud[m] | @gibi In all the case you did very well to raise this point on the mailing list, that's the kind of action that help to remain constructive and that help to remove fences between us and our goal. | 16:27 |
| hberaud[m] | this topic is a fact that we cannot ignore, and so the sooner we will address it the better it will be solved and the confident our customers will be. | 16:30 |
| hberaud[m] | s/In all the oslo/In all the case, the oslo world won't/ | 16:31 |
| -opendevstatus- NOTICE: Zuul jobs reporting POST_FAILURE were due to an incident with one of our cloud providers; this provider has been temporarily disabled and changes can be rechecked | 22:36 | |
Generated by irclog2html.py 4.0.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!