Friday, 2022-09-02

tobias-urdinyoctozepto: yeah, if you check my previous patchsets there i initially tried using the official async lib without success, and we all know how fun it also is to troubleshoot threading and eventlet issues :p06:41
tobias-urdinunfortunately there is prob some stuff that needs to be fixed in lib before we can consider using it as well, the async lib has a lot of retry logic etc but we could just potentially port the same logic over to that library06:43
tobias-urdinso i guess first plan of action is continue to investigate that library what needs to be done, propose a spec and work on that until we're satisfied06:43
tobias-urdini would also like to take some time to investigate the old spec and old proposed implementation as well so that we can learn from it06:44
hberaudyoctozepto: I'd like to add that eventlet/greenlet are not really active projects, see https://twitter.com/ossmkitty/status/156275063889082777707:56
hberaudyoctozepto: Python core maintainers and distros maintainers raised warning about the current situation of these projects (eventlet/greenlet) https://github.com/python-greenlet/greenlet/issues/30507:58
yoctozeptotobias-urdin: agreed, doing the same07:58
yoctozeptohberaud: long-term I would love to see OpenStack moving to asyncio (I prefer explicit async to implicit async)07:59
yoctozeptobut I have no idea how feasible that is07:59
yoctozeptoit would involve all libs07:59
hberaudWouldn't it be a good time to consider starting moving away from eventlet?08:00
yoctozeptowe need a plan to support two models at once08:00
yoctozeptohberaud: probably, yeah08:00
hberaudMaybe libs should be transitioned first08:00
hberaudAnd then services08:00
hberaudI don't have the big picture of eventlet on Openstack08:01
hberaudBut you are right, moving away from eventlet/greenletseems to be a real journey08:03
yoctozeptoindeed08:04
hberaudHowever I really think it would be beneficial for us, as eventlet is often the root cause of many of our pains08:04
yoctozeptowell, it goes against the grain of python preferring explicit to implicit08:04
yoctozeptoand goes low-level with that08:04
hberaudyeah08:05
yoctozeptoI wonder if anyone thinks the opposite, i.e. that staying with greenlet is beneficial08:08
yoctozepto(not including the migrations costs of course, i.e. it is *directly* beneficial)08:08
hberaudyoctozepto: I left some comments related to our previous discussion here: https://review.opendev.org/c/openstack/oslo.messaging/+/84833808:18
tobias-urdingood question I'm interested in that as well, I don't have enough knowledge on the internals to comment on it, but I don't like blackboxes08:18
yoctozeptohberaud: ack08:23
hberaudDepending on the libs we decide to use (nats.py, nats-python, etc) I think the specs should reflect how to make eventlet and async codes cohabit.08:23
yoctozeptoanother rmq question on the openstack-discuss08:23
yoctozeptoaye, as you saw, it seems it is generally impossible to have the two used in the same process without stepping onto bugs08:24
yoctozeptoI would be more inclined to discuss the effort to have the libs support either model (i.e. not two at once, but be usable with similar interfaces with any - choose one and go go go)08:25
hberaudgood idea08:33
*** sean-k-mooney1 is now known as sean-k-mooney09:31
sean-k-mooneyhberaud:heh i forgot i reviewed https://review.opendev.org/c/openstack/oslo-specs/+/692784 also ussiri is 2 years ago how...09:41
yoctozeptohappy to remind you all, seems we have gathered quite a strong teams along this idea; let's hope for the best ;p09:44
sean-k-mooneyhberaud: so to continue the conversation i was starting regarding https://review.opendev.org/c/openstack/oslo.messaging/+/848338/17#message-0d8a5e811a0f54e24081f244e8287122bf9dba7611:27
sean-k-mooneynats.py you see as a roadblock because of asycio11:27
sean-k-mooneyand the fact it really is not happy with eventlets11:27
sean-k-mooneybut you mention oslo.db11:27
sean-k-mooneyhowever we user eventlest in may ohter context were we would use oslo.messaging too11:28
sean-k-mooneyone suboptimal approch would be to take a privespe styple approch11:28
sean-k-mooneyrun the oslo-messaging driver in a sepreate processs using asyncio11:28
sean-k-mooneyso that is isolated form the rest fo the client application11:29
sean-k-mooneywe have hadd isseus usign pthread with the heatbeat work so i dont think that an option here11:29
sean-k-mooneythe downside of that is all messge woudl have to flow though a unix socket or named pip between the processes11:30
hberaudyes11:30
hberaudif we are able to release the two new drivers in the same time that would be ok 11:31
hberaud(your ones based on unix socket and the one based on nats)11:32
sean-k-mooneyoh i wasnt even thinking about that11:32
sean-k-mooneythat might be a way to do it11:32
sean-k-mooneyi mentioned privsep more because it already does this with its channels11:33
hberaudok11:33
hberaudcould be an option indeed11:33
sean-k-mooneybut i guess ya you coudl resues the unix socket dirver as a bridge maybe11:33
sean-k-mooneyanyway do you see https://github.com/Gr1N/nats-python as a blocker to move forward11:34
sean-k-mooneyah the pr has not been adressed in some time11:35
sean-k-mooneyso the challange is in finding a lib that is maintained and eventlet compatible11:36
hberaudsean-k-mooney: yes I see it as a blocker, I left related comments onto the nats review11:36
hberaudand the lib (nats-python) seems to lack of a lot of basic features11:38
sean-k-mooneyya11:39
sean-k-mooneyjust re read your comments11:39
hberaudCould be worth to create a comparative table to define what we need, what we want, and what we have11:39
hberaud(in those libs)11:39
sean-k-mooneyam i think movign the serveices away form evently woudl be a multi cycle effort by the way11:39
sean-k-mooney*away form eventlet11:39
sean-k-mooneyits threading model is deeply baked into nova11:40
sean-k-mooneyto the point that nova cant really run un monkey patched11:40
sean-k-mooneyit will endup in an infinit loop waiting for RPC messages 11:41
sean-k-mooneyin the compute-agent at least11:41
hberaudI see11:42
sean-k-mooneythe topic has come up before but we expect it would take a substaital rewrite to adress11:42
sean-k-mooneywe conepmplated starting a new agent based on asyncio (dont have the time) as a poc and then seeing if we could prot service so tehy could run under either mode11:43
sean-k-mooneybut i dont see that happing in the next year or two unless new contibutors signing up to work on it11:44
sean-k-mooneypart of the concern is if we woudl be abel to share code adn move things bit by bit or not11:44
yoctozeptohberaud, sean-k-mooney : the legacy lib cannot handle more than one server, so it's not feasible unless we do NATS with a TCP loadbalancer12:44
hberaudgood to know12:45
yoctozeptoyeah, plus it lacks at least those features I mentioned in gerrit12:47
yoctozeptoall in all, it does not make sense to base off of it unless we plan to build upon it quite a bit12:47
sean-k-mooneyack12:54
sean-k-mooneyso short term i think runing the nats part in a spereate process spawned by the oslo driver and bridiging it  within the nats driver woudl be "simple" way to move to ths offial asyncio implmationtion12:55
sean-k-mooneybut its not the nicest approch12:55
sean-k-mooneyit shoudl jsut work however12:55
sean-k-mooneyas it will provide a clean seam between the eventlet part and asyncio part12:56
yoctozeptoRPC to get RPC...12:57
yoctozepto😂12:57
sean-k-mooneyIPC to get RPC12:58
sean-k-mooneybut kind of12:58
sean-k-mooneyit could be a mmaped queue or any other IPC mechanium12:58
sean-k-mooneyyou jsut need to kep the two eventloops in differnt processes12:59
yoctozeptoyeah13:03
yoctozeptothough I am curious about final eventlet removal13:04
hberaudI prefer this solution, however, it will surely have performances impacts somewhere13:04
hberaud(IPC to get RPC)13:05
yoctozeptoyeah13:05
hberaudhowever, that's not worst than being stuck by eventlet somewhere13:05
yoctozeptoI must say I am curious about removing the dep on eventlet13:06
hberaudthis is a kind of start...13:07
yoctozeptoindeed, although I am still trying to wrap my head around it more holistically13:13
yoctozeptoI mean, staying with that IPC will be ugly13:14
yoctozeptowe need to have a path to eventlet-free openstack13:14
tobias-urdinseems like there was some (way way back) planning for that https://wiki.openstack.org/wiki/Oslo/blueprints/asyncio https://blueprints.launchpad.net/oslo.messaging/+spec/asyncio-executor13:35
tobias-urdinto run eventlet inside asyncio until everything was ported or something13:35
tobias-urdin"The greenio project allows to "plug" asyncio in eventlet" the other way around :pp13:36
hberaudgood catch13:38
hberaudinteresting docuemtns13:38
yoctozeptohttps://github.com/miguelgrinberg/greenletio13:46
yoctozeptoalso, it looks like SQLAlchemy is using greenlet to do asyncio13:46
yoctozeptohttps://docs.sqlalchemy.org/en/14/orm/extensions/asyncio.html13:47
yoctozeptoI think I am confused now :S13:47
tobias-urdinme too, what a spider web13:54
hberaud+113:55
hberaudFrom the doc: "Where above, an API call always starts as asyncio, flows through the synchronous API, and ends as asyncio, before results are propagated through this same chain in the opposite direction. In between, the message is adapted first into sync-style API use, and then back out to async style. Event hooks then by their nature occur in the middle of the “sync-style13:57
hberaudAPI use”. From this it follows that the API presented within event hooks occurs inside the process by which asyncio API requests have been adapted to sync, and outgoing messages to the database API will be converted to asyncio transparently."13:57
hberaudhttps://docs.sqlalchemy.org/en/14/orm/extensions/asyncio.html (looks for the table under "asyncio and events, two opposites")13:58
hberaudI think sqlalchemy do the magical abstraction by calling: AsyncSession.run_sync() 14:01
* hberaud goes to the release meeting14:01
yoctozeptoo/14:02
hberaudin other words I think sqlalchemy is doing something similar to greenio under its hood14:03
yoctozeptomaybe, though that part is only about the events, not the general flow14:06
hberaudindeed14:06
tobias-urdinlink so that i don't forget https://lists.openstack.org/pipermail/openstack-dev/2013-May/009784.html14:24
hberaudcould be worth to put all of these links into the review too, to centralize things.14:28
fricklerI was just about to suggest an etherpad14:28
hberaudalso, why not14:29
yoctozeptoI have been adding some more to the review so that they stay longer14:29
hberaudthx14:30
yoctozeptobtw, regarding the pthread/eventlet issue with heartbeats - was there a debugging conclusion? or only that we revert because it behaves oddly?14:33
hberaudwe just reverted the default value to avoid breaking nova and neutron parts by default. The option should be turned on manually to run under pthread.14:35
hberaudFrom an oslo view point we don't have lot of options here. We are stuck with this eventlet/uwsgi constraints from the heartbeat view point14:36
yoctozeptoI mean if anyone discovered *why* nova-compute and friends were failing with hearbeats being on pthread14:37
hberaudbecause nova-compute is not running under uwsgi so this flag should be set to false and turn off by default.14:40
hberauds/turn off/turned off/14:42
yoctozeptobut pthread is not wsgi-specific; I mean you are talking about the symptoms and I mean if anyone run a Root Cause Analysis14:45
yoctozeptobecause I can't find it in the bug report14:45
hberaudI'm not aware of an existing RCA14:46
yoctozeptoack14:52
yoctozeptothe context to my queries is that the greenletio seems to be using threads to support both styles (unless that was too quick of an analysis)14:57
yoctozeptothat said, it's greenlet only, without eventlet14:58
hberaudgreenletio? You means greenio?15:00
hberaudor greenlet?15:00
yoctozeptohttps://github.com/miguelgrinberg/greenletio15:01
yoctozeptowe have been discussing this already15:01
hberaudoh I missed this one15:03
hberaudI was thinking about https://github.com/1st1/greenio/ 15:03
yoctozeptook15:16
gibiheads up, I'm in the middle of debugging https://bugs.launchpad.net/nova/+bug/1988311/ oslo.concurrency fair lock + eventlet is broken. https://gist.github.com/gibizer/9051369e67fd46a20d52963dac534852 This is probably the same issue melwitt reported in https://github.com/eventlet/eventlet/issues/731 when we fixed a test only break in nova https://review.opendev.org/c/openstack/nova/+/81311416:54
gibiI wanted to say: I think ... broken16:55
gibiI summarized what we know so far in https://bugs.launchpad.net/oslo.concurrency/+bug/1988311/comments/417:39
yoctozeptoanother reason to drop eventlet :S19:06

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!