| *** dklyle has quit IRC | 01:18 | |
| *** takashin has left #openstack-placement | 03:02 | |
| *** e0ne has joined #openstack-placement | 05:13 | |
| *** e0ne has quit IRC | 05:27 | |
| *** belmoreira has joined #openstack-placement | 06:15 | |
| *** belmoreira has quit IRC | 06:16 | |
| *** e0ne has joined #openstack-placement | 07:52 | |
| openstackgerrit | Tetsuro Nakamura proposed openstack/placement master: Add tests demonstrating overlapping same_subtreeZ https://review.opendev.org/676530 | 08:00 |
|---|---|---|
| openstackgerrit | Merged openstack/placement master: Add tests demonstrating overlapping same_subtreeZ https://review.opendev.org/676530 | 08:37 |
| *** cdent has joined #openstack-placement | 09:46 | |
| openstackgerrit | Chris Dent proposed openstack/placement master: gabbi test for same_subtree with an ancestry hole https://review.opendev.org/676491 | 09:52 |
| openstackgerrit | Chris Dent proposed openstack/placement master: Add gabbits using a DeepNUMANetworkFixture https://review.opendev.org/676204 | 09:52 |
| openstackgerrit | Chris Dent proposed openstack/placement master: Add rw_ctx.parent_uuid_by_rp_uuid, for clarity https://review.opendev.org/676233 | 09:52 |
| openstackgerrit | Chris Dent proposed openstack/placement master: Use rp.id instead of uuid in _rp_rc_key https://review.opendev.org/676234 | 09:52 |
| openstackgerrit | Chris Dent proposed openstack/placement master: Add a rw_ctx.psum_res_by_rp_rc, for clarity https://review.opendev.org/676235 | 09:52 |
| openstackgerrit | Chris Dent proposed openstack/placement master: Avoid duplicate ProviderSummary in _merge_candidates https://review.opendev.org/675606 | 09:52 |
| openstackgerrit | Chris Dent proposed openstack/placement master: WIP: Use orjson in python3 for allocation candidate dump https://review.opendev.org/674661 | 09:52 |
| cdent | rebase for merge conflict ^ | 09:52 |
| gibi | cdent: if you can chose one patch what would be the most priority to review in placement for me? I'm a bit lost having both the unioned same_subtree and the holed ancestry bug in picture | 10:02 |
| cdent | gibi: there's really only one stack now. after the work that eric did to make tests for both of those things, and some tuneups tetsuro did overnight, what's left are my optimizations (above) | 10:03 |
| cdent | that stack beings with the a same subtree hole test, but that's just a protection test | 10:04 |
| cdent | (to make sure we don't break existing behavior in my stack) | 10:04 |
| cdent | s/beings/begins/ | 10:04 |
| gibi | cdent: OK, then I will start with that test. thanks for the guidance | 10:04 |
| cdent | great, thank you | 10:05 |
| *** cdent has quit IRC | 11:25 | |
| *** artom has joined #openstack-placement | 11:42 | |
| *** belmoreira has joined #openstack-placement | 11:53 | |
| *** cdent has joined #openstack-placement | 12:10 | |
| openstackgerrit | Merged openstack/placement master: gabbi test for same_subtree with an ancestry hole https://review.opendev.org/676491 | 12:19 |
| openstackgerrit | Merged openstack/placement master: Add gabbits using a DeepNUMANetworkFixture https://review.opendev.org/676204 | 12:21 |
| belmoreira | cdent in your placement performance analysis which apache mpm are you using? | 12:27 |
| cdent | not using apache | 12:27 |
| cdent | uwsgi, standalone | 12:27 |
| cdent | however, in the past when I've done it with apache I had to make some tweaks | 12:28 |
| cdent | let me look for that info, one sec | 12:28 |
| cdent | belmoreira: there's a bit of discussion about apache in this thing from a year and a half ago: https://anticdent.org/placement-scale-fun.html but mostly that's me saying "don't use apache when you just need a reverse proxy" | 12:29 |
| cdent | I know the tripleo people have done some experiments with various mpm setups | 12:29 |
| cdent | belmoreira: i use a little script called pusgi to start the uwsgi server direclty in the code base: | 12:31 |
| cdent | http://paste.openstack.org/show/757963/ | 12:31 |
| cdent | that way I also don't need a configuration | 12:31 |
| belmoreira | I'm running apache since I deployed placement. At beginning it required some tweaking... but has been running fine | 12:32 |
| cdent | it's not broken or anything like that, just that it is heavy | 12:32 |
| belmoreira | today I upgraded one region to Stein and placement gave me some work | 12:32 |
| cdent | (I'm not suggesting you use uwsgi standalone, in production, just that it's handy for testing) | 12:33 |
| belmoreira | cdent, yes I understand | 12:33 |
| cdent | I think in production I'd probably do nginx + uwsgi. | 12:33 |
| cdent | placement in nova or placement extracted? what kind of trouble? | 12:33 |
| belmoreira | placement in nova | 12:34 |
| belmoreira | It starts to log the following and then stalls | 12:34 |
| belmoreira | https://www.codepile.net/pile/YzAn63Qn | 12:34 |
| cdent | yeah, that's a known bug | 12:35 |
| cdent | let me find the info | 12:35 |
| belmoreira | the solution/mitigation was to reduce the number of threads | 12:35 |
| cdent | there's a pretty easy workaround | 12:35 |
| cdent | better than that one | 12:35 |
| cdent | (although that can work too) | 12:36 |
| cdent | in the process running placement you want to set OS_NOVA_DISABLE_EVENTLET_PATCHING to 1, true, or yes | 12:37 |
| cdent | placement doesn't need or use eventlet | 12:38 |
| cdent | but the version that is in the nova code repo automatically monkey patches | 12:38 |
| cdent | leading to that problem you're seeing | 12:38 |
| cdent | so the workarounds are either to set thread=1 in the wsgidaemon set up | 12:38 |
| cdent | or set the env variable | 12:38 |
| cdent | the latter is _much_preferred | 12:39 |
| cdent | the reason the version in nova automaticaly monkey patches is because somewhere in the package hierarchy (only implicitly used by placement) the call to monkey patch happens | 12:39 |
| cdent | that env variable short circuits out of it | 12:39 |
| cdent | what I would do in your shoes is: | 12:39 |
| cdent | 1. set the env variable | 12:39 |
| cdent | 2. as soon as possible move to extracted placement (which a) doesn't have the problem, b) has other improvements | 12:40 |
| cdent | 3. and then as soon as possible after that move to train placement, even if staying on stein with everything else, because it is _way_ faster | 12:40 |
| belmoreira | thanks for all the info | 12:41 |
| belmoreira | Yes, I plan to do 2. asap | 12:41 |
| cdent | great | 12:41 |
| belmoreira | Is this issue documented somewhere? (release notes?) I wasn't aware of it. I imagine others can suffer from it | 12:42 |
| cdent | I _thought_ it was. let me dig a bit harder | 12:42 |
| cdent | hmmm. doesn't look like it i s | 12:44 |
| cdent | Where do you think it should go? It's sort of hard to choose because of the extraction and the fact that the problem goes away when you start using the extracted code. | 12:45 |
| belmoreira | In my option it should be in nova release notes | 12:45 |
| cdent | efried: thoughts on that ^. Perhaps somewhere in the stein-related nova docs/notes? | 12:45 |
| belmoreira | but also doesn't hurt if included in placement release notes | 12:46 |
| cdent | yeah | 12:47 |
| belmoreira | at least I use the release notes as the source of what I should be aware for the upgrade | 12:48 |
| cdent | that's what they're there for | 12:48 |
| cdent | so yeah. | 12:48 |
| efried | one sec | 12:48 |
| cdent | I was just getting read to say "when efried surfaces, he'll have some thoughts" | 12:48 |
| efried | I'm doing my morning battle with sdk | 12:49 |
| cdent | I'm writing a pupdate, but need more coffee in order to be able to continue | 12:51 |
| efried | Am I reading backscroll from :27 ish? | 12:52 |
| cdent | efried: yes | 12:52 |
| efried | on it | 12:52 |
| cdent | brb | 12:52 |
| efried | cdent, belmoreira: Yes, I agree nova release notes is a good place for this info. Remember when proposing release notes to stable, they must be proposed only to the stable branch. There is no backporting. | 12:55 |
| efried | (I don't think it makes sense to have anything in placement's release notes on this, because if you're on extracted placement, the issue is n/a) | 12:56 |
| cdent | is there a section that's appropriate for known bugs/workarounds? I can't remember the names of all the sections | 13:01 |
| * efried creates reno template... | 13:02 | |
| efried | prelude: > | 13:02 |
| efried | features: | 13:02 |
| efried | issues: | 13:02 |
| efried | upgrade: | 13:02 |
| efried | deprecations: | 13:02 |
| efried | critical: | 13:02 |
| efried | security: | 13:02 |
| efried | fixes: | 13:02 |
| efried | other: | 13:02 |
| cdent | that's a nice list but I'm no closer to the answer | 13:03 |
| efried | "issues" sounds appropriate to me. | 13:04 |
| * cdent will cook something up before end of day | 13:05 | |
| belmoreira | cdent efried thank you | 13:05 |
| cdent | thanks for reminding us of a thing we should have already done :) | 13:06 |
| openstackgerrit | Merged openstack/placement master: Add rw_ctx.parent_uuid_by_rp_uuid, for clarity https://review.opendev.org/676233 | 13:24 |
| openstackgerrit | Merged openstack/placement master: Use rp.id instead of uuid in _rp_rc_key https://review.opendev.org/676234 | 13:31 |
| openstackgerrit | Merged openstack/placement master: Add a rw_ctx.psum_res_by_rp_rc, for clarity https://review.opendev.org/676235 | 13:39 |
| openstackgerrit | Chris Dent proposed openstack/placement master: WIP: Use orjson in python3 for allocation candidate dump https://review.opendev.org/674661 | 13:39 |
| openstackgerrit | Merged openstack/placement master: Avoid duplicate ProviderSummary in _merge_candidates https://review.opendev.org/675606 | 13:48 |
| *** belmoreira has quit IRC | 14:36 | |
| cdent | latest pupdate: https://anticdent.org/placement-update-19-32.html | 14:36 |
| cdent | more narration than normal | 14:36 |
| cdent | plus announcing me being gone next week | 14:36 |
| gibi | cdent: have a nice time off next week | 14:50 |
| cdent | thanks. hoping to do some long overnight walks | 14:50 |
| gibi | I will be off Monday, Tuesday due to national holiday | 14:51 |
| gibi | cdent: do you walk at night because it is more relaxing? | 14:55 |
| cdent | gibi: I meant walking over two days, including the overnight. so backpacking, essentially | 14:56 |
| cdent | but I probaby won't take much kit | 14:56 |
| gibi | ahh I see | 14:56 |
| cdent | depends on the weather forecast, I suppose | 14:58 |
| cdent | but I'd prefer to not take tent, but in england that's generally a recipe for being very damp | 14:58 |
| gibi | :) sound interesting. I always liked the nigth sky but I never have never slept under it directly (yet) | 15:01 |
| cdent | definitely worth doing | 15:01 |
| efried | I do that almost never. Last time I can remember was 1995, West Texas, and there wasn't actually any sleeping. | 15:03 |
| cdent | I camped near Hueco Tanks once, and nearly froze (inside of tent was coated in ice) | 15:03 |
| *** dklyle has joined #openstack-placement | 15:05 | |
| *** efried is now known as efried_afk | 15:56 | |
| *** e0ne has quit IRC | 15:58 | |
| openstackgerrit | Chris Dent proposed openstack/placement master: Merge request log and request id middlewares https://review.opendev.org/676982 | 16:21 |
| cdent | efried_afk: that ^ is attempt at what I described in a comment on your version of similar. | 16:21 |
| cdent | and that's it for me | 16:22 |
| cdent | see you all in a bit more than a week | 16:22 |
| *** cdent has quit IRC | 16:22 | |
| *** dklyle has quit IRC | 17:40 | |
| *** efried_afk is now known as efried | 18:52 | |
| *** spatel has joined #openstack-placement | 20:53 | |
| *** spatel has quit IRC | 20:57 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!