| hberaud[m] | done | 06:38 |
|---|---|---|
| hberaud[m] | thanks | 06:38 |
| zigo | hberaud[m]: Re: our discussion in #openstack-oslo | 08:03 |
| zigo | FYI, we tried using uwsgi for swift a few releases ago, and ... managed to undertand it was completely broken. So swift would need some work to make this happen. Hopefully, sooner than later, they should work on that. The Swift project had the tendency to lag behind this type of stuff, for example when moving to Python 3, hopefully, they wont do the same mistake again. | 08:03 |
| zigo | As for Glance, I found out that if using uwsgi only (ie: without Apache proxy in front), doing backup restor is currently broken. | 08:04 |
| zigo | There's no reason to use Apache in front, it's a very annoying setup. | 08:04 |
| zigo | So I'd advise the Glance team to try fixing the devstack setup to stop using Apache, so they can resolve this annoying bug. | 08:04 |
| hberaud[m] | zigo: thanks for the heads up. I suppose it is related to your previous warning on the mailing list (over the devstack/uwsgi topic https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack.org/message/FZFN6TBAZLPKHCZQKTTSDIKMXLJYTWMV/), isn't? | 08:34 |
| tkajinam | zigo, have you shared the details of that issue with glance ? I was asked in the previous thread but AFAIR no detail has been shared | 08:50 |
| tkajinam | I don't know how we can fix "the issue" without details. so far no similar problems are caught in CI which uses uwsgi | 08:51 |
| tkajinam | if you have already reported a bug then please ignore it | 08:51 |
| tkajinam | s/I was asked/It was asked/g | 08:51 |
| zigo | tkajinam: As I wrote, the CI uses uwsgi + apache, the way to fix would be running tempest without apache. | 09:21 |
| zigo | Then you'll see the backup restore failing. | 09:21 |
| tkajinam | zigo, there is a job without tls-proxy (apache) but it has been passing https://zuul.opendev.org/t/openstack/builds?job_name=devstack-no-tls-proxy | 09:31 |
| tkajinam | of cause glance is run by uwsgi | 09:32 |
| tkajinam | and it runs full set of tempest tests it seems https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_0e4/periodic/opendev.org/openstack/devstack/master/devstack-no-tls-proxy/0e4f202/testr_results.html | 09:33 |
| tkajinam | so I believe we really need a bug report with actual details to dig into your problems. There may be some differences in deployment pattern but we can't know these without any details | 09:34 |
| zigo | Ok, I'll try to reproduce it then. | 09:34 |
| hberaud[m] | thanks | 09:59 |
| hberaud[m] | friendly reminder our bi-weekly-odd meeting starts in ~30 min | 14:30 |
| hberaud[m] | #startmeeting openstack-eventlet-removal | 15:00 |
| opendevmeet | Meeting started Tue Nov 19 15:00:03 2024 UTC and is due to finish in 60 minutes. The chair is hberaud[m]. Information about MeetBot at http://wiki.debian.org/MeetBot. | 15:00 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 15:00 |
| opendevmeet | The meeting name has been set to 'openstack_eventlet_removal' | 15:00 |
| hberaud[m] | #link https://etherpad.opendev.org/p/epoxy-eventlet-tracking | 15:00 |
| gouthamr | o/ (here but on mobile) | 15:00 |
| hberaud[m] | Ping: amorin, JayF, gouthamr | 15:00 |
| amorin | o/ | 15:00 |
| hberaud[m] | we are around line 39 of the etherpad | 15:01 |
| ralonsoh | hi | 15:01 |
| hberaud[m] | lets wait 1 or 2 minutes for people | 15:01 |
| tkajinam | o/ | 15:02 |
| hberaud[m] | here we go! | 15:03 |
| hberaud[m] | #topic Review task completion | 15:03 |
| hberaud[m] | the only things in our tasks was the meetbot setup, kindly configured by Goutham, thanks gouthamr | 15:04 |
| hberaud[m] | it seems to work so I think we can close this topic | 15:04 |
| hberaud[m] | I just have one question | 15:04 |
| hberaud[m] | is the "title" passed to the startmeeting command have an importance? | 15:05 |
| hberaud[m] | I mean, today I used "openstack-eventlet-removal" but what if we use "eventlet-removal" during the next meeting? | 15:06 |
| tkajinam | I guess it should match the meeting_id | 15:06 |
| hberaud[m] | is our logs will be moved elsewhere? | 15:06 |
| hberaud[m] | ok | 15:07 |
| hberaud[m] | then I propose to stick to "openstack-eventlet-removal" | 15:07 |
| tkajinam | I guess the log goes to https://meetings.opendev.org/meetings/<title>/ | 15:07 |
| hberaud[m] | ok, next topc | 15:08 |
| hberaud[m] | topic | 15:08 |
| hberaud[m] | #topic general updates | 15:08 |
| tkajinam | (I see the directory is created in https://meetings.opendev.org/meetings/openstack-eventlet-removal/ ) | 15:08 |
| gouthamr | ++ | 15:09 |
| hberaud[m] | this dir will surely be fulfilled once I'll trigger the endmeeting command | 15:09 |
| tkajinam | yup | 15:09 |
| hberaud[m] | great | 15:09 |
| hberaud[m] | thanks for your confirmations | 15:10 |
| hberaud[m] | so today we can announce that the support of Python 3.13 in Eventlet is now merged and released | 15:10 |
| tkajinam | \o/ | 15:10 |
| hberaud[m] | the patch was merged yesterday | 15:10 |
| hberaud[m] | and I released it this morning | 15:11 |
| hberaud[m] | https://github.com/eventlet/eventlet/issues/991 | 15:11 |
| hberaud[m] | Eventlet 0.38.0 is the version that introduce the support of Python 3.13 | 15:11 |
| hberaud[m] | but... | 15:11 |
| hberaud[m] | unfortunatelly, for now, we are still blocked in our openstack upgrade path by some neutron failures https://review.opendev.org/c/openstack/requirements/+/933257 | 15:13 |
| hberaud[m] | Roldolfo kindly put comments and created a launchpad to track them | 15:13 |
| ralonsoh | yes, working on the Neutron's patch | 15:14 |
| hberaud[m] | so IIUC you need the os-ken upgrade to start working on it, exact? | 15:15 |
| hberaud[m] | (no-rush) | 15:15 |
| ralonsoh | that is done, this is just a matter of solving an issue in the CI (other not related to os-ken) | 15:15 |
| hberaud[m] | we have a almost complete series to solve these neutron failures | 15:16 |
| hberaud[m] | good, thanks | 15:16 |
| hberaud[m] | the other aspects of that topic are the mistral and manilla failures | 15:16 |
| hberaud[m] | but their jobs are non voting | 15:16 |
| hberaud[m] | gouthamr: what should we do if these both deliverables are not properly aligned by the end of Epoxy? | 15:17 |
| amorin | on mistral side, I tried to fix everything | 15:18 |
| amorin | is there still something wrong? | 15:18 |
| hberaud[m] | s/manilla/masakari/ | 15:18 |
| tkajinam | 73 RLock(s) were not greened, to fix this error make sure you run eventlet.monkey_patch() before importing any other modules. | 15:18 |
| hberaud[m] | amorin: good question... | 15:18 |
| hberaud[m] | https://zuul.opendev.org/t/openstack/build/faa17300315042ecbe588c034aad66ea | 15:18 |
| tkajinam | I wonder what this log implies. we do call monkey_patch right after importing eventlet and idk how we can fix it really | 15:19 |
| hberaud[m] | do you released the fixes? | 15:19 |
| hberaud[m] | indeed | 15:20 |
| amorin | I will take a look at that, I have no idea | 15:20 |
| tkajinam | I see heat unit tests timed out and I suspect the log is slowing down the test execution | 15:20 |
| hberaud[m] | amorin: do you mind if I put a task related on it on our tracker, just to be sure to keep an eye on that? | 15:20 |
| amorin | yes please | 15:21 |
| hberaud[m] | thanks | 15:21 |
| hberaud[m] | amorin: I put your name on this task | 15:22 |
| hberaud[m] | tkajinam: do you want to inspect further the heat side? | 15:22 |
| gouthamr | hberaud[m]: regarding masakari ( and any other component cross jobs that failed ), we can post on the ML.. and get this wider attention | 15:23 |
| tkajinam | I can try but I likely need some help as I don't see any apparent errors and have no idea what's going on at this moment | 15:23 |
| hberaud[m] | gouthamr: is masakari still actively maintained? | 15:24 |
| gouthamr | hberaud[m]: yes | 15:24 |
| hberaud[m] | ok | 15:24 |
| hberaud[m] | I assigned the ML discussion to myself | 15:25 |
| hberaud[m] | do you want to add anything else concerning this topic? | 15:26 |
| hberaud[m] | I think we can put the heat investigation aside for now and see if they repeat again in the coming days | 15:27 |
| hberaud[m] | ok, move on | 15:28 |
| hberaud[m] | #topic update from HTTP/SGI working group | 15:28 |
| gouthamr | +1 thank you; since the jobs failing are non voting, we can proceed with merging the requirements bump, and track fixes as bugs… if the projects are unable to fix this soon in this release, I’ll discuss this within the TC | 15:28 |
| gouthamr | (hit send late) | 15:29 |
| hberaud[m] | so we have no updates from working group for now | 15:29 |
| ralonsoh | sorry for the question, what is this group? who belongs to it? | 15:29 |
| hberaud[m] | for now I think we mostly try to sparkle it | 15:30 |
| hberaud[m] | there is not yet official representents | 15:30 |
| hberaud[m] | I'd encourage at least one team member by group if possible | 15:31 |
| hberaud[m] | preferably people interested by the various aspects of these groups | 15:31 |
| hberaud[m] | I think our priority is to take a decision about uWSGI as an official alternative or not | 15:32 |
| hberaud[m] | some deliverables are already choosen to follows the uWSGI path as migration path | 15:34 |
| hberaud[m] | by example glance | 15:34 |
| hberaud[m] | we had a tiny discussion this morning on this channel related to glance and uWSGI | 15:34 |
| sean-k-mooney | so uwsigin is the default wsgi servr we use for all testing | 15:34 |
| sean-k-mooney | is glance looking at using it as more the a server | 15:35 |
| hberaud[m] | IMO this is also the most easy path for migrating away from eventlet.wsgi | 15:35 |
| sean-k-mooney | well we are mixing too things | 15:36 |
| sean-k-mooney | eventlet.wsgi is used for 2 diffent usecase | 15:36 |
| sean-k-mooney | providing a webserver to host the wsgi ap | 15:36 |
| sean-k-mooney | *app | 15:36 |
| sean-k-mooney | and i belive it can be used as a wsgi framework to some degree | 15:36 |
| hberaud[m] | but the goal of the SGI working group was to ensure that we all agree on the path we want to follow | 15:36 |
| sean-k-mooney | so for the former moving to uswgi make sense | 15:37 |
| sean-k-mooney | right but we cant mandante a single wsgi server implemention | 15:37 |
| hberaud[m] | AFAIK glance use routes + eventlet.wsgi or uWSGI | 15:37 |
| hberaud[m] | not a single but more a by default | 15:38 |
| sean-k-mooney | most service now supprot using the api under a wsgi server and in devstack we use uWSGI as the defautl and have doen for many years | 15:38 |
| sean-k-mooney | keyston was the first to fully drop the eventlet console script | 15:38 |
| sean-k-mooney | nova's plan currently is just to remove that console-script (nova-api) without a replacment and jsut tell pepopel to use the wsgi applciaiton using any wsgi server | 15:39 |
| sean-k-mooney | we will continue to test with uWSGI as the defualt | 15:39 |
| hberaud[m] | ok | 15:39 |
| hberaud[m] | so yeah the goal of these working groups is to host this kind of discussion | 15:40 |
| hberaud[m] | how do you want to proceed? | 15:40 |
| amorin | maybe dumb question, why not using only apache, without uwsgi? | 15:40 |
| amorin | apache + mod_wsgi | 15:41 |
| sean-k-mooney | amorin: partly because that was buggy in the past with the python 3 transtion | 15:41 |
| sean-k-mooney | we could not proplery test on diffent pyton version as siply as we coud with uwsgi | 15:41 |
| hberaud[m] | I heard many people complaining about apache, but I personally have no opinion about it | 15:41 |
| amorin | hum, ok, we are using it on production here, no known issue (for nova, neutron, cinder, mistral) | 15:42 |
| sean-k-mooney | the intent is that any wsgi server could be used so if you wanted to test with mod_wsgi you could | 15:42 |
| amorin | ok, note that I have nothing against uwsgi anyway :) | 15:42 |
| * gouthamr notes no issue with Manila either… we have a job testing it always | 15:42 | |
| sean-k-mooney | amorin: we ahve it in our downstream ready product currently too but it does nhave a lot of memory overhead in comparison but it works | 15:42 |
| tkajinam | yeah | 15:43 |
| sean-k-mooney | gouthamr: as noted the problem was testing with the non default pythyon version | 15:43 |
| tkajinam | uwsgi is much lightweight. If you run all services in container then using apache may add more and more overheads | 15:43 |
| sean-k-mooney | if your installing apache form the package manager usign python form the venv can be tricky | 15:43 |
| tkajinam | and for baremetal deployment hosting all api services on httpd is a mess, because it messes up the deployment resource dependencies | 15:44 |
| sean-k-mooney | and if that python is not the defaut python then that is harder still | 15:44 |
| amorin | that's true, we have a custom mod_wsgi.so inside the venv for this | 15:44 |
| sean-k-mooney | anyway i dont want to derail on that histroy too much | 15:44 |
| hberaud[m] | sean-k-mooney: we should also notice that the SGI also aim to provide a way to guide folks through the application layer, so I think it join your initial remarks about mixed aspect of eventlet.wsgi https://wiki.openstack.org/wiki/Eventlet-removal#The_HTTP_SGI_Working_Group | 15:45 |
| hberaud[m] | no problem | 15:45 |
| hberaud[m] | time flies and we have other topic, I'll send a ML thread concerning this specific working group to encourage having this discussion in parallel | 15:46 |
| sean-k-mooney | so i joined partrly to say that since i have moved team internaly i likely wont be able to move the eventlet removal forward much in nova this cycle | 15:47 |
| hberaud[m] | thanks for the heads up | 15:47 |
| sean-k-mooney | but since i will be workign on watcher a bit this cycle maybe ill find time to start it there.... | 15:47 |
| hberaud[m] | I saw the nova ML PTG summary which spoke about gibi and the main interlocutor concerning the eventlet aspect on Nova | 15:48 |
| sean-k-mooney | one of the things ill add to my todo list is to see if watcher currently supprot runnign without eventlet.wsgi and make sure that is tested properly in ci | 15:48 |
| hberaud[m] | sean-k-mooney: in all case thank you for all your efforts, much appreciated | 15:48 |
| sean-k-mooney | yes gibi will try and move that forward when they have tiem and will help to some degree | 15:48 |
| sean-k-mooney | i and gibi just will have less time to work on this hten we had orginally planend | 15:49 |
| hberaud[m] | ack | 15:49 |
| hberaud[m] | move on to the next topic | 15:49 |
| hberaud[m] | #topic update from task working group | 15:49 |
| hberaud[m] | so the oslo.service specs are now merged | 15:50 |
| hberaud[m] | Daniel (damani) told us that he will start the implementation with Mike in the coming days | 15:50 |
| hberaud[m] | so we should soon observe regulare update concerning this aspect | 15:51 |
| tkajinam | ok | 15:51 |
| hberaud[m] | #topic update from interoperability working group | 15:51 |
| hberaud[m] | for now we have real update concerning this group, we are meeting each teams on by one, so once our tour will be done I think we will have a better understanding/oveview of this specific aspect | 15:52 |
| hberaud[m] | s/we have no real/ | 15:52 |
| hberaud[m] | #topic oslo.log and the pipemutex | 15:53 |
| hberaud[m] | well the pipemutex topic come back to us | 15:53 |
| hberaud[m] | my eventlet colleague (Itamar) proposed a way to workaround it in the case we want to use the asyncio hub https://github.com/eventlet/eventlet/issues/432#issuecomment-2483439460 | 15:54 |
| amorin | is there anything wrong with that? | 15:54 |
| hberaud[m] | we made back and forth with the removal of that mutex | 15:54 |
| sean-k-mooney | i saw i think that is the wrong approch | 15:55 |
| sean-k-mooney | i would prefer to fully remove the pipemutex | 15:55 |
| hberaud[m] | me too | 15:55 |
| hberaud[m] | Itamar proposed an parallel approach too | 15:56 |
| hberaud[m] | let me retrieve it | 15:56 |
| sean-k-mooney | i think that shoudl eb relitivly easy to do if we port the synconise decorator and lockign form oslo.utils? | 15:56 |
| hberaud[m] | https://github.com/eventlet/eventlet/issues/874#issuecomment-2483425977 | 15:56 |
| hberaud[m] | good question | 15:56 |
| hberaud[m] | you mean, easy to remove the pipemutex? | 15:57 |
| sean-k-mooney | we cant just import it because of the deps between them | 15:57 |
| sean-k-mooney | yes so the pipe mutex also require the debug flag | 15:57 |
| hberaud[m] | maybe | 15:57 |
| sean-k-mooney | so to me just disabling it when using asyncio is not a good approch | 15:57 |
| sean-k-mooney | i dont really want that in nova even when not using asyncio | 15:58 |
| hberaud[m] | yeah same thing for me | 15:58 |
| hberaud[m] | I'd not prefer to have this kind of tricks anywhere in oslo | 15:58 |
| sean-k-mooney | so the lockign we use for the syncronize decorate uses a filesystem level look that is also a fare lock | 15:59 |
| hberaud[m] | let me create a task to revive this discussion | 15:59 |
| sean-k-mooney | that has all of the charteristics we need that the pipemutex was trying to provide | 15:59 |
| sean-k-mooney | i cant promise to fix this but im kind of wonderign if i can poc replacing it quickly | 16:00 |
| tkajinam | you are talking about the lock implementation in oslo.concurrency, not oslo.utils, right ? | 16:00 |
| sean-k-mooney | so i might try and spend an hour on it today | 16:00 |
| sean-k-mooney | tkajinam: yes probaly | 16:00 |
| tkajinam | one thing we have to be aware of the file based lock in oslo.concucurrency is that it requires lock_path | 16:00 |
| sean-k-mooney | its using a fateners reader writer lock combined with a file on disk | 16:00 |
| tkajinam | I mean the lock_path option should be set, otherwise the lock fails with an exception | 16:00 |
| tkajinam | so switching it causes deployment impact, as was seen in rabbitmq queue manager implementation in the past | 16:01 |
| sean-k-mooney | yep but we coudl have that use a reasonable default and most uses oslo.log already use it i think | 16:01 |
| sean-k-mooney | that is a complication worth nothing | 16:02 |
| hberaud[m] | sean-k-mooney: I also put your name in parallel of mine on this pipemutex task topic | 16:02 |
| tkajinam | maybe. now RDO overrides lock_path for only limited number of services(nova, glance and cinder for os-brick and neutron for its purpose) | 16:02 |
| gouthamr | ~~ time check on this meeting ~~ | 16:02 |
| hberaud[m] | :) | 16:02 |
| tkajinam | oops | 16:02 |
| hberaud[m] | lets finish with the agenda | 16:02 |
| gouthamr | we can continue discussing :) | 16:02 |
| hberaud[m] | #topic cleaning up workaround logic for old eventlet | 16:03 |
| tkajinam | that's just fyi | 16:03 |
| tkajinam | we just need to get these merged | 16:03 |
| tkajinam | so we can move on | 16:03 |
| hberaud[m] | thanks for the heads up | 16:03 |
| hberaud[m] | will ping the team for the patches not yet merged | 16:04 |
| hberaud[m] | #topic Open Discussion | 16:04 |
| hberaud[m] | Anything else that you want to discuss before I close this meeting? | 16:05 |
| hberaud[m] | calling 1 | 16:06 |
| hberaud[m] | 2 | 16:06 |
| hberaud[m] | 3 | 16:06 |
| hberaud[m] | thanks everyone! | 16:06 |
| hberaud[m] | #endmeeting | 16:06 |
| opendevmeet | Meeting ended Tue Nov 19 16:06:30 2024 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 16:06 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/openstack_eventlet_removal/2024/openstack_eventlet_removal.2024-11-19-15.00.html | 16:06 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/openstack_eventlet_removal/2024/openstack_eventlet_removal.2024-11-19-15.00.txt | 16:06 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/openstack_eventlet_removal/2024/openstack_eventlet_removal.2024-11-19-15.00.log.html | 16:06 |
| ralonsoh | bye | 16:06 |
| tkajinam | thank you | 16:06 |
| amorin | thanks | 16:07 |
| gouthamr | thank you! | 16:08 |
| JayF | Thanks for the ping; couldn't make it today but plan to make it generally | 17:35 |
| hberaud[m] | JayF: no problem, thank you | 17:42 |
| JayF | hberaud[m]: https://github.com/eventlet/eventlet/issues/994 | 17:44 |
| JayF | Itamar is helping with an Ironic SNMP driver thing; the library migrated to asyncio, and he is running into various things which I suspect are just frontrunning issues we'll see migrating other openstack stuff | 17:44 |
| JayF | looks like he already has a fix enroute \o/ | 17:45 |
| hberaud[m] | awesome | 18:24 |
| damani[m] | hi | 19:31 |
| JayF | o/ | 19:35 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!