| *** dklyle has quit IRC | 03:29 | |
| *** e0ne has joined #openstack-placement | 06:09 | |
| *** e0ne has quit IRC | 06:18 | |
| *** tetsuro has joined #openstack-placement | 06:39 | |
| *** tetsuro has quit IRC | 07:06 | |
| *** tetsuro has joined #openstack-placement | 07:08 | |
| *** tetsuro has quit IRC | 07:57 | |
| *** takashin has left #openstack-placement | 08:00 | |
| *** cdent has joined #openstack-placement | 08:09 | |
| *** e0ne has joined #openstack-placement | 08:19 | |
| *** tetsuro has joined #openstack-placement | 08:41 | |
| *** tetsuro has quit IRC | 08:45 | |
| *** tssurya has joined #openstack-placement | 09:22 | |
| cdent | tssurya: thanks for the comment on the consumer types stuff. All it really needs at this point is some review to make sure I didn't miss anything, and to make sure it is right. I hope you had a nice vacation. | 09:31 |
|---|---|---|
| tssurya | cdent: vacation was good :) thanks for asking, I'll review it as soon as I start recollecting things | 09:32 |
| cdent | great, thanks | 09:33 |
| cdent | sean-k-mooney: is there any more to do on this: https://storyboard.openstack.org/#!/story/2005447 | 09:35 |
| *** tetsuro has joined #openstack-placement | 10:58 | |
| *** tetsuro has quit IRC | 10:59 | |
| sean-k-mooney | we punted merging the request filter to make room to land the sev and pcpu in placement work | 11:09 |
| sean-k-mooney | you can mark it closed form an os-traits/placement point of view | 11:10 |
| sean-k-mooney | the nova code should merge early in Ussuri | 11:10 |
| cdent | thanks sean-k-mooney | 11:12 |
| cdent | doing a bit of a story cleanup/verification | 11:13 |
| *** mriedem has joined #openstack-placement | 12:57 | |
| *** e0ne has quit IRC | 13:19 | |
| mriedem | cdent: thanks for making that list, i'll take a whack at https://storyboard.openstack.org/#!/story/2005171 | 13:25 |
| cdent | cool, glad it is useful, and thank you | 13:25 |
| mriedem | the tricky part is detecting if the oslo policy_file is being used or the placement one, since they both provide a default value, though both are different (placement's defaults to policy.yaml while oslo's defaults to policy.json), i'm thinking just checking to see if the file exists? | 13:39 |
| *** e0ne has joined #openstack-placement | 13:50 | |
| cdent | mriedem: I would go with your instincts on this one mriedem. I'm not going to be much help because most of my policy interaction has been with placement, where the defaults are good, why mess with em (kinda thing). | 13:51 |
| mriedem | will talk to the oslo people to get recommendations | 14:00 |
| *** belmoreira has joined #openstack-placement | 14:27 | |
| *** dklyle has joined #openstack-placement | 14:29 | |
| *** belmoreira has quit IRC | 14:50 | |
| *** belmoreira has joined #openstack-placement | 14:56 | |
| *** jroll has joined #openstack-placement | 15:00 | |
| openstackgerrit | Matt Riedemann proposed openstack/placement master: WIP: Deprecate [placement]/policy_file config option https://review.opendev.org/681047 | 15:22 |
| mriedem | if this looks sane ^ i'll proceed with the todos | 15:22 |
| mriedem | will also ask bnemec and lbragstad to look once the oslo meeting is over | 15:22 |
| efried | cdent: nova.exception.AllocationDeleteFailed: Failed to delete allocations for consumer 23bbec78-ebc2-4304-bdf9-0166667e6a72. Error: {"errors": [{"status": 500, "title": "Internal Server Error", "detail": "The server has either erred or is incapable of performing the requested operation.\n\n 'NoneType' object has no attribute 'authorize' ", "code": "placement.undefined_code", "request_id": "req-48b4bab3-0c22-4380-974e-9de2a | 15:23 |
| efried | This happened in a nova functional test, so using the placement fixture, so not sure if it's possible irl, but lmk if you want a closer look. | 15:23 |
| mriedem | that would be some policy thing | 15:24 |
| mriedem | by the looks of it | 15:24 |
| mriedem | what a coincidence | 15:24 |
| cdent | yeah, what mriedem said | 15:24 |
| cdent | efried: do you have a link to the code the spawned that error. It's seems like it ought to be impossible, but stranger things have happened | 15:25 |
| efried | https://review.opendev.org/#/c/674895/28/nova/tests/functional/libvirt/test_numa_servers.py@563 if you comment out the delete_server lines it happens. | 15:25 |
| efried | which is weird, because that's the end of the test case and nothing else should be happening at that point | 15:26 |
| efried | must be a tearDown from a superclass or fixture or something. | 15:26 |
| efried | it doesn't cause the test to fail though | 15:26 |
| efried | stephenfin: fyi ^ | 15:26 |
| mriedem | just an ugly traceback or what? | 15:26 |
| efried | yeah | 15:26 |
| efried | and that 500 is only a piece of the craziness | 15:27 |
| efried | there's like six tracebacks | 15:27 |
| cdent | seems like we got at least two bugs in play | 15:27 |
| cdent | efried: if you already have the entire traceback suite can you paste it somewhere | 15:27 |
| efried | ... | 15:28 |
| cdent | otherwise I'll spin up some stuff locally to make my own | 15:28 |
| efried | http://paste.openstack.org/show/774331/ | 15:28 |
| stephenfin | efried: You've clearly looked into it more than I did :) | 15:28 |
| efried | stephenfin: it's because I didn't believe you. | 15:29 |
| stephenfin | I asked gibi, he didn't know. I removed that, it fixed it. Done. | 15:29 |
| efried | nothing personal | 15:29 |
| efried | and again, it doesn't fail the test case | 15:29 |
| efried | I'm responding in the review. | 15:29 |
| stephenfin | I wouldn't trust me either, in fairness | 15:29 |
| efried | tldr, happy to leave it in place since it reduces output ugliness. | 15:29 |
| *** tssurya has quit IRC | 15:30 | |
| cdent | efried: from my standpoint the thing that matters here is that placement is producing a 500, but there seems to more than just that at work | 15:31 |
| efried | agree | 15:31 |
| cdent | lemme finish up this storyboard comment... on sec | 15:31 |
| efried | I suspect it's somewhere around the compute service fixture trying to clean itself up; maybe it's trying to delete providers but not cascading. | 15:32 |
| efried | but as you say there's several things at play here because I can't think of any one fault that would cause this much chaos by itself. | 15:33 |
| mriedem | i think it just means that placement.policy.init isn't getting called from the PlacementFixture | 15:35 |
| mriedem | normally that happens via the get_enforcer extension point thing in setup.cfg | 15:36 |
| mriedem | although the PlacementFixture is using the PolicyFixture which calls placement.policy.init directly | 15:37 |
| mriedem | weird | 15:37 |
| mriedem | my guess is the PlacementFixture and placement PolicyFixture are tearing down before that nova code is tearing down | 15:38 |
| cdent | mriedem: if that were the case I wouldn't have thought the request to placement would get that far along | 15:39 |
| mriedem | it's something with _ensure_console_log_for_instance not being stubbed properly in nova - i've seen something like this before | 15:41 |
| mriedem | you hit that StopIteration and all hell breaks loose | 15:41 |
| mriedem | meaning i think there is a stub in nova that mocks that out and expects it to only be called like twice, but the test is using more than 2 servers | 15:43 |
| mriedem | stephenfin: ^ | 15:43 |
| mriedem | yup, found it | 15:44 |
| gibi | mriedem: nice catch! | 15:44 |
| mriedem | i've had to figure it out before | 15:44 |
| cdent | that seems rather fragile ... | 15:45 |
| mriedem | stephenfin: right here https://github.com/openstack/nova/blob/ba3147420c0a6f8b17a46b1a493b89bcd67af6f1/nova/tests/functional/libvirt/base.py#L63 | 15:45 |
| mriedem | cdent: it's extremely fragile | 15:45 |
| mriedem | "using this fixture, you'd better only ever spawn at most 2 servers" | 15:45 |
| mriedem | "you're welcome" | 15:45 |
| cdent | or to be more accurate: the way the cascade of failure happens is scary | 15:46 |
| efried | agree | 15:46 |
| stephenfin | mriedem: I literally (re-)hit that an hour ago and in true three days to feature freeze fashion, just restubbed the thing | 15:46 |
| mriedem | efried: i left a comment https://review.opendev.org/#/c/674895/28/nova/tests/functional/libvirt/test_numa_servers.py@563 | 15:46 |
| efried | nice | 15:47 |
| efried | stephenfin: that ^ can be done in a separate patch afaiac | 15:47 |
| efried | unless you've already done it, which may be what you just said. | 15:48 |
| stephenfin | efried: Didn't you have a patch up to remove 'file_open' entirely? | 15:49 |
| efried | doesn't sound familiar. Are you talking about aspiers' patch_open for mocking? | 15:49 |
| efried | or this? https://review.opendev.org/#/c/645087/ | 15:51 |
| mriedem | efried: he did this https://review.opendev.org/#/c/674895/31/nova/tests/functional/libvirt/test_numa_servers.py@551 | 15:52 |
| mriedem | mocked around the bad stub in the functional test case that was suffering from it | 15:52 |
| *** belmoreira has quit IRC | 16:04 | |
| *** e0ne has quit IRC | 16:10 | |
| cdent | mriedem: not that you probably needed it, but for my own sanity I traced how those those fixtures were unwinding | 16:12 |
| cdent | and where the placement 500 was coming into it | 16:12 |
| mriedem | i also noticed that the functional test in questoin is setting up the placement fixture twice | 16:13 |
| cdent | it _might_ make sense to change things so that the placement fixture was cleaned up later/last | 16:13 |
| mriedem | once early and once late | 16:13 |
| cdent | rad! | 16:13 |
| mriedem | i think the problem is the duplicate | 16:13 |
| mriedem | so it cleans up too early and then shits the bed | 16:13 |
| mriedem | lemme clean that up quick | 16:14 |
| cdent | mriedem: yeah, just tried it, that fixes it | 16:14 |
| cdent | (getting rid of the placement turn on in libvirt/base.py | 16:15 |
| mriedem | sweet | 16:15 |
| mriedem | push that if you want and i'll +2 | 16:15 |
| mriedem | +W even | 16:15 |
| cdent | rad, one sec | 16:15 |
| cdent | that answers my one remaining question which was "if the fixture is gone how come it can still wsgi-intercept" | 16:15 |
| mriedem | _IntegratedTestBase at the top sets up placement | 16:16 |
| mriedem | so yeah | 16:16 |
| cdent | aye | 16:16 |
| *** N3l1x has joined #openstack-placement | 16:33 | |
| *** cdent has quit IRC | 16:50 | |
| *** e0ne has joined #openstack-placement | 18:12 | |
| *** e0ne has quit IRC | 18:34 | |
| *** e0ne has joined #openstack-placement | 19:03 | |
| *** e0ne has quit IRC | 19:09 | |
| *** e0ne has joined #openstack-placement | 19:10 | |
| *** e0ne has quit IRC | 21:40 | |
| *** mriedem has quit IRC | 22:51 | |
| *** takashin has joined #openstack-placement | 23:32 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!