15:00:03 <hberaud[m]> #startmeeting openstack-eventlet-removal
15:00:03 <opendevmeet> Meeting started Tue Nov 19 15:00:03 2024 UTC and is due to finish in 60 minutes.  The chair is hberaud[m]. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:00:03 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:00:03 <opendevmeet> The meeting name has been set to 'openstack_eventlet_removal'
15:00:19 <hberaud[m]> #link https://etherpad.opendev.org/p/epoxy-eventlet-tracking
15:00:34 <gouthamr> o/ (here but on mobile)
15:00:41 <hberaud[m]> Ping: amorin, JayF, gouthamr
15:00:43 <amorin> o/
15:01:17 <hberaud[m]> we are around line 39 of the etherpad
15:01:35 <ralonsoh> hi
15:01:49 <hberaud[m]> lets wait 1 or 2 minutes for people
15:02:04 <tkajinam> o/
15:03:04 <hberaud[m]> here we go!
15:03:08 <hberaud[m]> #topic Review task completion
15:04:08 <hberaud[m]> the only things in our tasks was the meetbot setup, kindly configured by Goutham, thanks gouthamr
15:04:31 <hberaud[m]> it seems to work so I think we can close this topic
15:04:55 <hberaud[m]> I just have one question
15:05:25 <hberaud[m]> is the "title" passed to the startmeeting command have an importance?
15:06:17 <hberaud[m]> I mean, today I used "openstack-eventlet-removal" but what if we use "eventlet-removal" during the next meeting?
15:06:27 <tkajinam> I guess it should match the meeting_id
15:06:33 <hberaud[m]> is our logs will be moved elsewhere?
15:07:27 <hberaud[m]> ok
15:07:45 <hberaud[m]> then I propose to stick to "openstack-eventlet-removal"
15:07:54 <tkajinam> I guess the log goes to https://meetings.opendev.org/meetings/<title>/
15:08:01 <hberaud[m]> ok, next topc
15:08:03 <hberaud[m]> topic
15:08:13 <hberaud[m]> #topic general updates
15:08:33 <tkajinam> (I see the directory is created in https://meetings.opendev.org/meetings/openstack-eventlet-removal/ )
15:09:09 <gouthamr> ++
15:09:32 <hberaud[m]> this dir will surely be fulfilled once I'll trigger the endmeeting command
15:09:42 <tkajinam> yup
15:09:47 <hberaud[m]> great
15:10:02 <hberaud[m]> thanks for your confirmations
15:10:38 <hberaud[m]> so today we can announce that the support of Python 3.13 in Eventlet is now merged and released
15:10:52 <tkajinam> \o/
15:10:55 <hberaud[m]> the patch was merged yesterday
15:11:06 <hberaud[m]> and I released it this morning
15:11:13 <hberaud[m]> https://github.com/eventlet/eventlet/issues/991
15:11:36 <hberaud[m]> Eventlet 0.38.0 is the version that introduce the support of Python 3.13
15:11:44 <hberaud[m]> but...
15:13:17 <hberaud[m]> unfortunatelly, for now, we are still blocked in our openstack upgrade path by some neutron failures https://review.opendev.org/c/openstack/requirements/+/933257
15:13:53 <hberaud[m]> Roldolfo kindly put comments and created a launchpad to track them
15:14:03 <ralonsoh> yes, working on the Neutron's patch
15:15:04 <hberaud[m]> so IIUC you need the os-ken upgrade to start working on it, exact?
15:15:12 <hberaud[m]> (no-rush)
15:15:51 <ralonsoh> that is done, this is just a matter of solving an issue in the CI (other not related to os-ken)
15:16:05 <hberaud[m]> we have a almost complete series to solve these neutron failures
15:16:16 <hberaud[m]> good, thanks
15:16:45 <hberaud[m]> the other aspects of that topic are the mistral and manilla failures
15:16:53 <hberaud[m]> but their jobs are non voting
15:17:31 <hberaud[m]> gouthamr: what should we do if these both deliverables are not properly aligned by the end of Epoxy?
15:18:00 <amorin> on mistral side, I tried to fix everything
15:18:09 <amorin> is there still something wrong?
15:18:15 <hberaud[m]> s/manilla/masakari/
15:18:45 <tkajinam> 73 RLock(s) were not greened, to fix this error make sure you run eventlet.monkey_patch() before importing any other modules.
15:18:46 <hberaud[m]> amorin: good question...
15:18:55 <hberaud[m]> https://zuul.opendev.org/t/openstack/build/faa17300315042ecbe588c034aad66ea
15:19:12 <tkajinam> I wonder what this log implies. we do call monkey_patch right after importing eventlet and idk how we can fix it really
15:19:12 <hberaud[m]> do you released the fixes?
15:20:16 <hberaud[m]> indeed
15:20:18 <amorin> I will take a look at that, I have no idea
15:20:26 <tkajinam> I see heat unit tests timed out and I suspect the log is slowing down the test execution
15:20:59 <hberaud[m]> amorin: do you mind if I put a task related on it on our tracker, just to be sure to keep an eye on that?
15:21:07 <amorin> yes please
15:21:14 <hberaud[m]> thanks
15:22:14 <hberaud[m]> amorin: I put your name on this task
15:22:44 <hberaud[m]> tkajinam: do you want to inspect further the heat side?
15:23:38 <gouthamr> hberaud[m]: regarding masakari ( and any other component cross jobs that failed ), we can post on the ML.. and get this wider attention
15:23:41 <tkajinam> I can try but I likely need some help as I don't see any apparent errors and have no idea what's going on at this moment
15:24:09 <hberaud[m]> gouthamr: is masakari still actively maintained?
15:24:17 <gouthamr> hberaud[m]: yes
15:24:24 <hberaud[m]> ok
15:25:27 <hberaud[m]> I assigned the ML discussion to myself
15:26:48 <hberaud[m]> do you want to add anything else concerning this topic?
15:27:16 <hberaud[m]> I think we can put the heat investigation aside for now and see if they repeat again in the coming days
15:28:08 <hberaud[m]> ok, move on
15:28:17 <hberaud[m]> #topic update from HTTP/SGI working group
15:28:47 <gouthamr> +1 thank you; since the jobs failing are non voting, we can proceed with merging the requirements bump, and track fixes as bugs… if the projects are unable to fix this soon in this release, I’ll discuss this within the TC
15:29:00 <gouthamr> (hit send late)
15:29:16 <hberaud[m]> so we have no updates from working group for now
15:29:48 <ralonsoh> sorry for the question, what is this group? who belongs to it?
15:30:34 <hberaud[m]> for now I think we mostly try to sparkle it
15:30:57 <hberaud[m]> there is not yet official representents
15:31:24 <hberaud[m]> I'd encourage at least one team member by group if possible
15:31:58 <hberaud[m]> preferably people interested by the various aspects of these groups
15:32:57 <hberaud[m]> I think our priority is to take a decision about uWSGI as an official alternative or not
15:34:11 <hberaud[m]> some deliverables are already choosen to follows the uWSGI path as migration path
15:34:17 <hberaud[m]> by example glance
15:34:39 <hberaud[m]> we had a tiny discussion this morning on this channel related to glance and uWSGI
15:34:53 <sean-k-mooney> so uwsigin is the default wsgi servr we use for all testing
15:35:11 <sean-k-mooney> is glance looking at using it as more the a server
15:35:45 <hberaud[m]> IMO this is also the most easy path for migrating away from eventlet.wsgi
15:36:02 <sean-k-mooney> well we are mixing too things
15:36:16 <sean-k-mooney> eventlet.wsgi is used for 2 diffent usecase
15:36:24 <sean-k-mooney> providing a  webserver to host the wsgi ap
15:36:27 <sean-k-mooney> *app
15:36:41 <sean-k-mooney> and i belive it can be used as a wsgi framework to some degree
15:36:58 <hberaud[m]> but the goal of the SGI working group was to ensure that we all agree on the path we want to follow
15:37:00 <sean-k-mooney> so for the former moving to uswgi make sense
15:37:29 <sean-k-mooney> right but we cant mandante a single wsgi server implemention
15:37:41 <hberaud[m]> AFAIK glance use routes + eventlet.wsgi or uWSGI
15:38:09 <hberaud[m]> not a single but more a by default
15:38:28 <sean-k-mooney> most service now supprot using the api under a wsgi server and in devstack we use uWSGI as the defautl and have doen for many years
15:38:44 <sean-k-mooney> keyston was the first to fully drop the eventlet console script
15:39:21 <sean-k-mooney> nova's plan currently is just to remove that console-script (nova-api) without a replacment and jsut tell pepopel to use the wsgi applciaiton using any wsgi server
15:39:36 <sean-k-mooney> we will continue to test with uWSGI as the defualt
15:39:37 <hberaud[m]> ok
15:40:15 <hberaud[m]> so yeah the goal of these working groups is to host this kind of discussion
15:40:36 <hberaud[m]> how do you want to proceed?
15:40:45 <amorin> maybe dumb question, why not using only apache, without uwsgi?
15:41:12 <amorin> apache + mod_wsgi
15:41:14 <sean-k-mooney> amorin: partly because that was buggy in the past with the python 3 transtion
15:41:39 <sean-k-mooney> we could not proplery test on diffent pyton version as siply as we coud with uwsgi
15:41:44 <hberaud[m]> I heard many people complaining about apache, but I personally have no opinion about it
15:42:20 <amorin> hum, ok, we are using it on production here, no known issue (for nova, neutron, cinder, mistral)
15:42:22 <sean-k-mooney> the intent is that any wsgi server could be used so if you wanted to test with mod_wsgi you could
15:42:47 <amorin> ok, note that I have nothing against uwsgi anyway :)
15:42:55 * gouthamr notes no issue with Manila either… we have a job testing it always
15:42:58 <sean-k-mooney> amorin: we ahve it in our downstream ready product currently too but it does nhave a lot of memory overhead in comparison but it works
15:43:17 <tkajinam> yeah
15:43:31 <sean-k-mooney> gouthamr: as noted the problem was testing with the non default pythyon version
15:43:45 <tkajinam> uwsgi is much lightweight. If you run all services in container then using apache may add more and more overheads
15:43:58 <sean-k-mooney> if your installing apache form the package manager usign python form the venv can be tricky
15:44:11 <tkajinam> and for baremetal deployment hosting all api services on httpd is a mess, because it messes up the deployment resource dependencies
15:44:14 <sean-k-mooney> and if that python is not the defaut python then that is harder still
15:44:25 <amorin> that's true, we have a custom mod_wsgi.so inside the venv for this
15:44:39 <sean-k-mooney> anyway i dont want to derail on that histroy too much
15:45:08 <hberaud[m]> sean-k-mooney: we should also notice that the SGI also aim to provide a way to guide folks through the application layer, so I think it join your initial remarks about mixed aspect of eventlet.wsgi  https://wiki.openstack.org/wiki/Eventlet-removal#The_HTTP_SGI_Working_Group
15:45:19 <hberaud[m]> no problem
15:46:16 <hberaud[m]> time flies and we have other topic, I'll send a ML thread concerning this specific working group to encourage having this discussion in parallel
15:47:01 <sean-k-mooney> so i joined partrly to say that since i have moved team internaly i likely wont be able to move the eventlet removal forward much in nova this cycle
15:47:26 <hberaud[m]> thanks for the heads up
15:47:27 <sean-k-mooney> but since i will be workign on watcher a bit this cycle maybe ill find time to start it there....
15:48:15 <hberaud[m]> I saw the nova ML PTG summary which spoke about gibi and the main interlocutor concerning the eventlet aspect on Nova
15:48:20 <sean-k-mooney> one of the things ill add to my todo list is to see if watcher currently supprot runnign without eventlet.wsgi and make sure that is tested properly in ci
15:48:50 <hberaud[m]> sean-k-mooney: in all case thank you for all your efforts, much appreciated
15:48:51 <sean-k-mooney> yes gibi will try and move that forward when they have tiem and will help to some degree
15:49:20 <sean-k-mooney> i and gibi just will have less time to work on this hten we had orginally planend
15:49:31 <hberaud[m]> ack
15:49:46 <hberaud[m]> move on to the next topic
15:49:50 <hberaud[m]> #topic update from task working group
15:50:03 <hberaud[m]> so the oslo.service specs are now merged
15:50:42 <hberaud[m]> Daniel (damani) told us that he will start the implementation with Mike in the coming days
15:51:03 <hberaud[m]> so we should soon observe regulare update concerning this aspect
15:51:15 <tkajinam> ok
15:51:27 <hberaud[m]> #topic update from interoperability working group
15:52:36 <hberaud[m]> for now we have real update concerning this group, we are meeting each teams on by one, so once our tour will be done I think we will have a better understanding/oveview of this specific aspect
15:52:48 <hberaud[m]> s/we have no real/
15:53:13 <hberaud[m]> #topic oslo.log and the pipemutex
15:53:35 <hberaud[m]> well the pipemutex topic come back to us
15:54:33 <hberaud[m]> my eventlet colleague (Itamar) proposed a way to workaround it in the case we want to use the asyncio hub https://github.com/eventlet/eventlet/issues/432#issuecomment-2483439460
15:54:34 <amorin> is there anything wrong with that?
15:54:59 <hberaud[m]> we made back and forth with the removal of that mutex
15:55:13 <sean-k-mooney> i saw i think that is the wrong approch
15:55:30 <sean-k-mooney> i would prefer to fully remove the pipemutex
15:55:42 <hberaud[m]> me too
15:56:03 <hberaud[m]> Itamar proposed an parallel approach too
15:56:08 <hberaud[m]> let me retrieve it
15:56:19 <sean-k-mooney> i think that shoudl eb relitivly easy to do if we port the synconise decorator and lockign form oslo.utils?
15:56:36 <hberaud[m]> https://github.com/eventlet/eventlet/issues/874#issuecomment-2483425977
15:56:58 <hberaud[m]> good question
15:57:20 <hberaud[m]> you mean, easy to remove the pipemutex?
15:57:23 <sean-k-mooney> we cant just import it because of the deps between them
15:57:36 <sean-k-mooney> yes so the pipe mutex also require the debug flag
15:57:47 <hberaud[m]> maybe
15:57:57 <sean-k-mooney> so to me just disabling it when using asyncio is not a good approch
15:58:06 <sean-k-mooney> i dont really want that in nova even when not using asyncio
15:58:20 <hberaud[m]> yeah same thing for me
15:58:41 <hberaud[m]> I'd not prefer to have this kind of tricks anywhere in oslo
15:59:12 <sean-k-mooney> so the lockign we use for the syncronize decorate uses a filesystem level look that is also a fare lock
15:59:30 <hberaud[m]> let me create a task to revive this discussion
15:59:30 <sean-k-mooney> that has all of the charteristics we need that the pipemutex was trying to provide
16:00:09 <sean-k-mooney> i cant promise to fix this but im kind of wonderign if i can poc replacing it quickly
16:00:15 <tkajinam> you are talking about the lock implementation in oslo.concurrency, not oslo.utils, right ?
16:00:16 <sean-k-mooney> so i might try and spend an hour on it today
16:00:25 <sean-k-mooney> tkajinam: yes probaly
16:00:37 <tkajinam> one thing we have to be aware of the file based lock in oslo.concucurrency is that it requires lock_path
16:00:50 <sean-k-mooney> its using a fateners reader writer lock combined with a file on disk
16:00:52 <tkajinam> I mean the lock_path option should be set, otherwise the lock fails with an exception
16:01:14 <tkajinam> so switching it causes deployment impact, as was seen in rabbitmq queue manager implementation in the past
16:01:32 <sean-k-mooney> yep but we coudl have that use a reasonable default and most uses oslo.log already use it i think
16:02:00 <sean-k-mooney> that is  a complication worth nothing
16:02:01 <hberaud[m]> sean-k-mooney: I also put your name in parallel of mine on this pipemutex task topic
16:02:18 <tkajinam> maybe. now RDO overrides lock_path for only limited number of services(nova, glance and cinder for os-brick and neutron for its purpose)
16:02:26 <gouthamr> ~~ time check on this meeting ~~
16:02:41 <hberaud[m]> :)
16:02:42 <tkajinam> oops
16:02:57 <hberaud[m]> lets finish with the agenda
16:02:57 <gouthamr> we can continue discussing :)
16:03:04 <hberaud[m]> #topic cleaning up workaround logic for old eventlet
16:03:12 <tkajinam> that's just fyi
16:03:24 <tkajinam> we just need to get these merged
16:03:46 <tkajinam> so we can move on
16:03:47 <hberaud[m]> thanks for the heads up
16:04:33 <hberaud[m]> will ping the team for the patches not yet merged
16:04:43 <hberaud[m]> #topic Open Discussion
16:05:12 <hberaud[m]> Anything else that you want to discuss before I close this meeting?
16:06:05 <hberaud[m]> calling 1
16:06:07 <hberaud[m]> 2
16:06:10 <hberaud[m]> 3
16:06:24 <hberaud[m]> thanks everyone!
16:06:30 <hberaud[m]> #endmeeting