| *** jamesmcarthur has joined #openstack-tc | 00:05 | |
| *** jamesmcarthur has quit IRC | 00:10 | |
| *** jamesmcarthur has joined #openstack-tc | 00:12 | |
| *** jamesmcarthur has quit IRC | 00:16 | |
| *** jamesmcarthur has joined #openstack-tc | 00:18 | |
| *** jamesmcarthur has quit IRC | 00:21 | |
| *** jamesmcarthur has joined #openstack-tc | 00:21 | |
| *** tetsuro has joined #openstack-tc | 00:35 | |
| *** tetsuro_ has joined #openstack-tc | 00:36 | |
| *** tetsuro has quit IRC | 00:40 | |
| *** tkajinam has quit IRC | 01:55 | |
| *** tkajinam has joined #openstack-tc | 01:55 | |
| *** jamesmcarthur has quit IRC | 02:11 | |
| *** jamesmcarthur has joined #openstack-tc | 02:11 | |
| *** jamesmcarthur has quit IRC | 02:16 | |
| *** tetsuro_ has quit IRC | 03:02 | |
| *** tetsuro has joined #openstack-tc | 03:03 | |
| *** tetsuro has quit IRC | 03:07 | |
| *** ricolin_ has joined #openstack-tc | 03:25 | |
| *** jamesmcarthur has joined #openstack-tc | 03:29 | |
| *** jamesmcarthur has quit IRC | 03:32 | |
| *** jamesmcarthur has joined #openstack-tc | 03:32 | |
| *** jamesmcarthur has quit IRC | 04:08 | |
| *** jamesmcarthur has joined #openstack-tc | 04:08 | |
| *** jamesmcarthur has quit IRC | 04:09 | |
| *** jamesmcarthur_ has joined #openstack-tc | 04:09 | |
| *** jamesmcarthur_ has quit IRC | 04:11 | |
| *** tetsuro has joined #openstack-tc | 04:39 | |
| *** tetsuro has quit IRC | 04:39 | |
| *** cloudnull has quit IRC | 04:52 | |
| *** cloudnull has joined #openstack-tc | 04:53 | |
| *** tetsuro has joined #openstack-tc | 05:00 | |
| *** tetsuro has quit IRC | 05:30 | |
| *** tetsuro has joined #openstack-tc | 06:08 | |
| *** tetsuro has quit IRC | 06:25 | |
| *** tetsuro has joined #openstack-tc | 06:25 | |
| *** tetsuro has quit IRC | 06:27 | |
| *** dklyle has quit IRC | 06:36 | |
| *** tetsuro has joined #openstack-tc | 06:42 | |
| *** tetsuro has quit IRC | 06:47 | |
| *** tetsuro has joined #openstack-tc | 06:48 | |
| *** tetsuro has quit IRC | 06:52 | |
| *** ralonsoh has joined #openstack-tc | 07:33 | |
| *** tosky has joined #openstack-tc | 08:02 | |
| *** bnemec has quit IRC | 08:14 | |
| *** tosky has quit IRC | 08:16 | |
| *** bnemec has joined #openstack-tc | 08:17 | |
| *** e0ne has joined #openstack-tc | 08:21 | |
| *** tosky has joined #openstack-tc | 08:33 | |
| *** ricolin_ has quit IRC | 09:45 | |
| *** dmellado has quit IRC | 09:59 | |
| *** dmellado has joined #openstack-tc | 10:07 | |
| *** ricolin_ has joined #openstack-tc | 11:23 | |
| cloudnull | o/ | 12:10 |
|---|---|---|
| ShadowJonathan | not much activity going on here | 12:12 |
| *** ijolliffe has joined #openstack-tc | 12:13 | |
| njohnston | o/ | 12:21 |
| fungi | ShadowJonathan: that is often the case for office hours, but folks try to be around in case someone in the community has something they want to bring up with the tc in ways the mailing list doesn't satisfy | 12:48 |
| fungi | oh, though today's office hour starts a little over two hours from now anyway | 12:49 |
| ShadowJonathan | heh, in that case, i'd like to bring up if mailing posts like mine aren't getting any attention simply due to the lack of genuine interest (in the subjects), or it is normal that discussions like these take a while | 12:49 |
| fungi | but yeah, the tc tries to conduct most of its business via mailing list and code review | 12:50 |
| ShadowJonathan | maybe its a part of my social anxiety taking over, but i kinda started to frown after i hadn't gotten any acknowledgement after 2 days | 12:50 |
| ShadowJonathan | "the tc tries to conduct most of its business via mailing list and code review", ah, alright | 12:51 |
| evrardjp | ShadowJonathan: You shouldn't stress, things might take a while indeed | 13:14 |
| evrardjp | :) | 13:14 |
| ShadowJonathan | Yeah, I'm not used to the community or know the ways "go" around here, so I don't have much base to go off of | 13:15 |
| ShadowJonathan | know how* the ways "go" | 13:15 |
| mnaser | tc-members: please review the proposal to the ideas repo by ShadowJonathan alongside the ML posts ^ | 13:20 |
| ShadowJonathan | oh shit, I didn't wanna ask for more attention, just read the room | 13:20 |
| ShadowJonathan | But thanks | 13:20 |
| mnaser | ShadowJonathan: no worries -- our community operates on pinging people to raise attention and shopping ideas around :) | 13:21 |
| ShadowJonathan | Ah, alright | 13:22 |
| knikolla | o/ | 13:38 |
| *** ianychoi has joined #openstack-tc | 14:05 | |
| gmann | o/ | 14:17 |
| *** dklyle has joined #openstack-tc | 14:46 | |
| *** tkajinam has quit IRC | 14:47 | |
| *** irclogbot_3 has quit IRC | 14:55 | |
| *** gibi has quit IRC | 14:55 | |
| *** irclogbot_3 has joined #openstack-tc | 14:57 | |
| *** gibi has joined #openstack-tc | 14:57 | |
| *** johnthetubaguy has quit IRC | 15:16 | |
| *** johnthetubaguy has joined #openstack-tc | 15:20 | |
| *** ricolin_ is now known as ricolin | 15:30 | |
| ricolin | o/ | 15:30 |
| ShadowJonathan | i think imma pick project dew to talk about | 15:32 |
| ShadowJonathan | who here has some interest in looking through https://review.opendev.org/#/c/741008/ to get up to speed with one of my ideas real quick? | 15:32 |
| ShadowJonathan | in short: provide openstack to hobbyists in their homelabs | 15:33 |
| *** johnthetubaguy has quit IRC | 16:17 | |
| knikolla | ShadowJonathan: i did give it a read yesterday. i wish our services individually weren't as heavy. | 16:19 |
| ShadowJonathan | that's the thing that surprised me the most, though i do also feel a little bit cliched because i know exactly how python trades away performance for developing and runtime flexibility | 16:20 |
| *** johnthetubaguy has joined #openstack-tc | 16:20 | |
| ShadowJonathan | that's why one of my suggestions was to develop alongside (not mainly, a pure python implementation should always exist) a native extension version that can hold critical functions and data much more efficiently | 16:21 |
| ShadowJonathan | knikolla: (whoops, didnt ping) | 16:21 |
| knikolla | we also usually optimise the code for readability, rather than performance. | 16:21 |
| knikolla | ShadowJonathan: but that native extension part would lead to rewrites and duplication. | 16:23 |
| ShadowJonathan | agreed, i'm still not entirely "all" for that, as its (again) a devil's triangle | 16:24 |
| fungi | also if you're going to optimize anything, that should start with profiling. putting effort into something which runs infrequently is often a total waste | 16:24 |
| knikolla | ++ | 16:24 |
| fungi | find the parts which run constantly over and over and optimize those first | 16:25 |
| ShadowJonathan | i have yet to actually dive and profile much of anything, but these were just based off of my initial impressions (when running devstack) and requirements i saw on the wikis | 16:25 |
| ShadowJonathan | actually dive into the code* | 16:25 |
| ShadowJonathan | where i saw python memory usage being relatively high for a no-status minimal configuration, which gave me that impression of this being more geared towards heavy-duty servers | 16:26 |
| ShadowJonathan | but that optimization can give benefits regardless of this project's goal, as it would require less resources, and in turn be more cost-effective | 16:27 |
| knikolla | but i think the proposal touches on a good subject, you can't really run openstack for your own pleasure at home unless you by expensive or cumbersome hardware | 16:27 |
| knikolla | buy* | 16:27 |
| knikolla | i live in a tiny studio, so a NUC is the most I can do. | 16:27 |
| *** ijolliffe has quit IRC | 16:27 | |
| clarkb | worth noting the memory pressure is an issue for CI | 16:29 |
| fungi | mtreinish did a great presentation on his closet cloud racked into an ikea end table | 16:30 |
| knikolla | i was thinking exactly about him | 16:30 |
| clarkb | I think a fair bit of our background fail rate noise is related to swap usage which has a dominoe effect as you start using up all the disk iops | 16:30 |
| clarkb | I don't think this is intentional because its for big servers, its a side effect of using a language where udnerstanding memroy use can be difficult so it is often ignored | 16:30 |
| clarkb | and because many users do have relatively large servers the issue isn't a priority | 16:31 |
| fungi | well, also python's gc doesn't free allocations | 16:31 |
| knikolla | could we implement some automatic profiling in the CI? | 16:31 |
| clarkb | knikolla: we have that | 16:31 |
| clarkb | its not very granular though | 16:31 |
| fungi | so if you have a long-running python process, it's never going to reduce its memory utilization, at best it will stop increasing it after some point | 16:31 |
| clarkb | fungi: sort of, the magic of virtual memory means you can free physical memory use | 16:31 |
| clarkb | but ya the vmem won't decrease | 16:31 |
| fungi | true | 16:31 |
| clarkb | the trick is to ensure that pages can be paged out | 16:32 |
| knikolla | clarkb: interesting. can you point me in the right direction? | 16:32 |
| clarkb | knikolla: ya let me see if I can find some links | 16:32 |
| fungi | knikolla: it's in the dstat logs | 16:32 |
| clarkb | we have dstat and some others but ya ^ | 16:32 |
| fungi | devstack has dstat defined as one of its services | 16:32 |
| ShadowJonathan | i think a lot can definitely be optimized in following general areas (talking from a uneducated perspective (with the actual code)): | 16:33 |
| ShadowJonathan | - limiting one-second objects to a minimum, dont create and destroy elaborate JSON objects if its not needed, use list comprehensions and such | 16:33 |
| ShadowJonathan | - only kick off background jobs and threads when they're actually needed, have them idle in a block context out of scope of many objects that were needed, so they're released | 16:33 |
| ShadowJonathan | - scale up memory only to the actual required virtual entities in use for the service, if there's 0 virtual entities, there should only be a bedrock of memory in use by basic application code and config values | 16:33 |
| ShadowJonathan | (now lemme read back) | 16:33 |
| knikolla | awesome, thanks, i'll look into it. | 16:33 |
| *** e0ne has quit IRC | 16:33 | |
| fungi | ShadowJonathan: as for threads, in openstack api services a lot of that is going to depend on the eventlet library most of them use | 16:34 |
| clarkb | knikolla: https://zuul.opendev.org/t/openstack/build/0b050c2a04b24b6a845df91a061a17d0/log/controller/logs/dstat-csv_log.txt can be fed into something like https://lamada.eu/dstat-graph/ that gives you overall system stats | 16:34 |
| ShadowJonathan | i think i'll use some of my python knowledge to kinda see where i can better memory usage to "scale on entity increase, bedrock at 0", but also to kinda get used to the code itself | 16:34 |
| ShadowJonathan | how much of openstack uses threads, and how much uses asyncio? | 16:34 |
| clarkb | knikolla: there is also https://zuul.opendev.org/t/openstack/build/0b050c2a04b24b6a845df91a061a17d0/log/controller/logs/screen-dstat.txt which may be more readable for identifying memory hogs | 16:34 |
| clarkb | knikolla: once specific users of memory are identified you'll need targetted profiling of that process and we don't really have anything like that yet | 16:35 |
| clarkb | ShadowJonathan: much of it is eventlet | 16:35 |
| knikolla | clarkb: appreciate it, that gives me a great starting point. | 16:35 |
| ShadowJonathan | don't know much in-depth about that | 16:35 |
| fungi | ShadowJonathan: last i checked almost none of it uses asyncio because that was hard to make compatible across python 2.x and 3.x, but now that python 3.5 is the minimum openstack's supporting more of that may become possible | 16:35 |
| ShadowJonathan | yeah | 16:36 |
| clarkb | knikolla: cinder backup was a big memory user in the past but it looks like we've removed it from our default jobs | 16:36 |
| ShadowJonathan | i think that porting much code to use asyncio will help, threads are (from a performance perspective) functionally non-distinguishable, as they constantly pre-emptively rob the GIL from eachother, but cannot run at the same time | 16:36 |
| clarkb | (I'm guessing it may still have memory issues though) | 16:36 |
| clarkb | ShadowJonathan: eventlet is cooperative multithreading like asyncio | 16:37 |
| ShadowJonathan | meanwhile asyncio is coorperative, which means memory usage could only "bounce" between induvidual runs, when objects come in and out of scope | 16:37 |
| ShadowJonathan | hmmmm | 16:37 |
| clarkb | ShadowJonathan: just done outside of stdlib and requiring certain hacks | 16:37 |
| ShadowJonathan | yeah, i really need to take a deeper look at it before i can say anything about it | 16:37 |
| fungi | ShadowJonathan: to put it into perspective, we only just emerged from our first coordinated release without python 2.x support | 16:37 |
| ShadowJonathan | oh | 16:37 |
| ShadowJonathan | hmmm yeah | 16:38 |
| ShadowJonathan | i think i'll focus on contributing patches for performance when spitting through services to see how i can better memory usage | 16:38 |
| ShadowJonathan | solve my own problem, so to speak | 16:38 |
| knikolla | fungi: almost, though, right? isn't swift still py2 also? | 16:38 |
| clarkb | ya there are a small number of exceptions still | 16:39 |
| knikolla | https://governance.openstack.org/tc/goals/selected/ussuri/drop-py27.html | 16:39 |
| clarkb | swift and then also some supporting libs | 16:39 |
| fungi | knikolla: fair, we said last release was the first one where teams *could* drop existing python 3 support | 16:39 |
| fungi | er, drop existing python 2 support i mean | 16:39 |
| knikolla | it did go much smoother than i expected though | 16:40 |
| ShadowJonathan | at the very least i think my experience with going extremely low-level with asyncio, and looking at how the GIL operates (as of 3.8), would help with analyzing potential optimizations | 16:40 |
| fungi | but anyway, widescale changes implementing py3k-only functionality are possible now, they'll be slow to implement across such a large and diverse software ecosystem, and we should try to make sure anything we do along those lines is as consistent an approach as we can manage everywhere | 16:40 |
| ShadowJonathan | im not saying im the best at it, just that i think i can help | 16:40 |
| knikolla | ........ type hinting...... /me runs away | 16:41 |
| ShadowJonathan | not sure whats controversion about type hinting? | 16:41 |
| ShadowJonathan | controversial* | 16:41 |
| fungi | it's just a massive undertaking if you want to do it everywhere | 16:43 |
| ShadowJonathan | ive been spitting through google search results for 3 minutes, and i *just* got a vague idea of what eventlet is supposed to be, not a good first impression | 16:43 |
| ShadowJonathan | if the top 3 results all don't speak regarding what it actually *is*, then there's something seriously wrong with your messaging | 16:43 |
| ShadowJonathan | (sorry) | 16:43 |
| clarkb | ShadowJonathan: yes, its one of those legacy thorns that I think many would be happy to move away from, but also I'm not sure it would be a major difference in memory consumption | 16:43 |
| fungi | eventlet is used in a similar manner to greenthreads | 16:43 |
| clarkb | ShadowJonathan: its green threads for python | 16:43 |
| ShadowJonathan | > `greenthreads` | 16:44 |
| ShadowJonathan | which is another thing i dont know about | 16:44 |
| clarkb | but preemption tends to be implicitly explicit | 16:44 |
| ShadowJonathan | i have to step away from my pc right now, thanks for the explainations, i'll be back in an hour or so | 16:44 |
| clarkb | (you can't preempt at any point only during specific actions) | 16:44 |
| *** ijolliffe has joined #openstack-tc | 16:48 | |
| ShadowJonathan | fungi: would there be will and intent in the dev community to move to a more "modern" framework, though? (eventlets to asyncio) | 16:49 |
| ShadowJonathan | Just asking in case whatever efforts I'll have to move elements of a library to async-based code would be denied based on some agreement or pre-existing consensus | 16:50 |
| ShadowJonathan | I want to contribute, I just don't wanna step on any toes | 16:50 |
| fungi | ShadowJonathan: there has been repeated interest, but it would be a large, multi-year effort most likely | 17:03 |
| fungi | maybe not as hard as it was when we needed to maintain compatibility with python 2 though | 17:03 |
| ShadowJonathan | Agreed, I think I'll be able to contribute some of my accumulated experience to that effort then | 17:03 |
| ShadowJonathan | Once I get comfortable with submitting and switching between change proposals, and get generally comfortable with Gerrit | 17:04 |
| ShadowJonathan | For now I think I'll simply get some residual ideas as to how individual components of openstack look like, and learn rust this vacation :D | 17:05 |
| *** ShadowJonathan has quit IRC | 17:06 | |
| *** ShadowJonathan has joined #openstack-tc | 17:06 | |
| *** ralonsoh has quit IRC | 17:32 | |
| clarkb | knikolla: oh there is also https://zuul.opendev.org/t/openstack/build/0b050c2a04b24b6a845df91a061a17d0/log/controller/logs/screen-memory_tracker.txt | 17:44 |
| clarkb | knikolla: that shows that the privsep helpers when looked at in aggregate are a decent chunk of memory | 17:45 |
| clarkb | almost half a gig maybe? | 17:46 |
| clarkb | we also ahve rootwrap daemons and privsep helpers for neutron. Maybe we can consolidate a bit there and end up with some easy memory wins (no idea how feasible that is) | 17:53 |
| fungi | it's a cycle goal, or at least was | 17:59 |
| fungi | ahh, yeah, it's proposed but not yet selected: https://governance.openstack.org/tc/goals/index.html#community-goals | 18:00 |
| clarkb | one upside to improving privsep memory consumption is that it would likely help for any service that uses it | 18:01 |
| *** ijolliffe has quit IRC | 19:28 | |
| *** ricolin has quit IRC | 19:36 | |
| mnaser | ShadowJonathan: i think implementing nd moving to asyncio would be very welcome | 19:51 |
| mnaser | as an operator, i am getting hit by a few annoyances of eventlet often in high load envs | 19:51 |
| ShadowJonathan | alright, thanks :D | 19:51 |
| mnaser | ShadowJonathan: i'd personally help you chase down reviews :) | 19:52 |
| ShadowJonathan | ? | 19:53 |
| ShadowJonathan | what do you mean by that? sorry | 19:53 |
| mnaser | ShadowJonathan: as in, find cores and ask them to help merge those changes :) | 19:53 |
| ShadowJonathan | ahhh okay | 19:53 |
| ShadowJonathan | "cores" as in "core devs"? | 19:53 |
| mnaser | ah, yes | 19:54 |
| fungi | more precisely, "core reviewers" | 19:54 |
| mnaser | even more accurately :) | 19:54 |
| fungi | our development workflow has a heavy emphasis on code review | 19:54 |
| *** e0ne has joined #openstack-tc | 20:02 | |
| ShadowJonathan | mnaser: in that case, thanks for the offer, it'll probably take a while before i'll get so far as to actually propose such a change, though | 20:04 |
| ShadowJonathan | in the meantime: where on the opendev review website can i see a simple list of open changes per project? | 20:04 |
| ShadowJonathan | does it have to be via selection operators in the search bar? | 20:04 |
| clarkb | ShadowJonathan: that is one way to do it. We also link to them via the gitea mirrors. | 20:05 |
| clarkb | https://opendev.org/openstack/nova/ forexample has a proposed changes link | 20:06 |
| ShadowJonathan | ah yeah, that just shows the list with the same search operators, but prefilled | 20:06 |
| ShadowJonathan | thanks | 20:06 |
| *** e0ne has quit IRC | 20:22 | |
| *** markvoelker has joined #openstack-tc | 21:23 | |
| *** markvoelker has quit IRC | 21:26 | |
| *** markvoelker has joined #openstack-tc | 21:33 | |
| *** markvoelker has quit IRC | 21:38 | |
| *** tkajinam has joined #openstack-tc | 23:02 | |
| *** tosky has quit IRC | 23:04 | |
| *** irclogbot_3 has quit IRC | 23:16 | |
| *** gibi has quit IRC | 23:16 | |
| *** irclogbot_3 has joined #openstack-tc | 23:23 | |
| *** gibi has joined #openstack-tc | 23:23 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!