| *** larainema has joined #openstack-oslo | 01:00 | |
| *** threestrands has joined #openstack-oslo | 01:39 | |
| *** ianychoi has quit IRC | 03:29 | |
| *** ianychoi has joined #openstack-oslo | 03:30 | |
| *** njohnston has quit IRC | 04:35 | |
| *** dave-mccowan has quit IRC | 04:40 | |
| *** jhesketh has quit IRC | 05:34 | |
| *** jhesketh has joined #openstack-oslo | 05:42 | |
| *** Luzi has joined #openstack-oslo | 05:46 | |
| *** lpetrut has joined #openstack-oslo | 06:00 | |
| *** trident has quit IRC | 07:08 | |
| *** rcernin has quit IRC | 07:12 | |
| *** trident has joined #openstack-oslo | 07:17 | |
| *** tosky has joined #openstack-oslo | 07:40 | |
| *** ralonsoh has joined #openstack-oslo | 08:24 | |
| *** jaosorior has quit IRC | 08:28 | |
| *** jaosorior has joined #openstack-oslo | 08:29 | |
| *** jaosorior has quit IRC | 08:46 | |
| *** jaosorior has joined #openstack-oslo | 09:31 | |
| *** tesseract has joined #openstack-oslo | 11:12 | |
| *** boden has joined #openstack-oslo | 11:42 | |
| *** jaosorior has quit IRC | 11:58 | |
| openstackgerrit | Merged openstack/oslo.utils master: Add digestmod when using hmac https://review.opendev.org/677770 | 12:09 |
|---|---|---|
| *** jaosorior has joined #openstack-oslo | 12:11 | |
| openstackgerrit | Merged openstack/oslo.service master: Move doc related modules to doc/requirements.txt https://review.opendev.org/669408 | 12:19 |
| *** larainema has quit IRC | 12:30 | |
| *** kgiusti has joined #openstack-oslo | 12:44 | |
| *** dkehn has quit IRC | 12:53 | |
| *** boden has quit IRC | 12:58 | |
| *** dave-mccowan has joined #openstack-oslo | 13:10 | |
| *** boden has joined #openstack-oslo | 13:20 | |
| *** tosky has quit IRC | 13:30 | |
| *** tosky has joined #openstack-oslo | 13:30 | |
| *** openstackgerrit has quit IRC | 14:22 | |
| *** amotoki_ has quit IRC | 14:34 | |
| *** tosky_ has joined #openstack-oslo | 14:34 | |
| *** tosky has quit IRC | 14:34 | |
| *** tosky_ is now known as tosky | 14:35 | |
| *** amotoki has joined #openstack-oslo | 14:35 | |
| *** Luzi has quit IRC | 14:40 | |
| *** lpetrut has quit IRC | 14:52 | |
| *** lpetrut has joined #openstack-oslo | 15:15 | |
| efried | bnemec: Okay, so what is the expected behavior if I'm blasting a ton of messages to the privsep daemon and it shuts down? | 15:17 |
| efried | I would think | 15:18 |
| efried | a) any messages I had already sent, but had not yet been processed by the time the daemon is gone, should legitimately raise this IOError; and | 15:18 |
| efried | b) any messages I send after that should trigger the new behavior and restart the daemon. | 15:18 |
| efried | In other words, seeing that IOError for a) is legit as currently written... no? | 15:19 |
| * efried has idea... | 15:19 | |
| bnemec | efried: I agree, if the daemon dies while a message is being processed the client wil have to get an error back. | 15:21 |
| bnemec | Although in the SIGHUP case it might be nice to have a graceful shutdown that stops accepting new connections and finishes any existing work. | 15:21 |
| efried | bnemec: I'm thinking ^ ...unless our _wrap traps IOError and retries, recreating the daemon. | 15:22 |
| efried | well *that* didn't work at all. | 15:23 |
| bnemec | Yeah, that was kind of my intent with the retry in https://review.opendev.org/#/c/678653/4/oslo_privsep/priv_context.py | 15:23 |
| efried | bnemec: but you're retrying on OSError -- what were you trying to catch there, the EPIPE? | 15:24 |
| bnemec | I think I saw two different cases, one plain OSError and something that inherited from OSError. | 15:24 |
| efried | I was thinking we should make a wrapper around IOError (so we can tell it apart from an IOError raised by the actual priv'd function) and raise that instead of plain IOError in ClientChannel._reader_main when the loop breaks. | 15:25 |
| bnemec | But I think it would be reasonable to expand what we retry on. There's little harm in making another call. | 15:25 |
| efried | at that point are we guaranteed that we didn't actually attempt the privileged function? | 15:25 |
| efried | ...yeah, looks like it. | 15:26 |
| bnemec | Just for kicks, I tried to change the except OSError to an except Exception and it seems to make the functional tests pass in parallel. | 15:28 |
| efried | heh | 15:29 |
| efried | bnemec: won't that also mask any exception from the privileged command itself? | 15:30 |
| bnemec | Yeah, we don't want to actually do that. | 15:30 |
| efried | um, worse than mask, cause us to retry it indefinitely | 15:30 |
| bnemec | I just wanted to see if it would work. | 15:30 |
| efried | In fact, even the OSError could conceivably do that, no? | 15:30 |
| bnemec | The retry isn't a loop though, so it will only ever retry once. | 15:30 |
| efried | mm | 15:31 |
| efried | bnemec: Looking at this guy https://opendev.org/openstack/oslo.privsep/src/branch/master/oslo_privsep/daemon.py#L467-L489 | 15:37 |
| efried | ...we can't tell the difference between IOError from channel.send() and IOError from result.result(). I think that's bad. | 15:37 |
| efried | what I'm trying to work towards is a way for the privsep framework itself to raise unique exceptions when things like comm go wrong, so that we can do your retry thing in all and only those, without worrying about accidentally retrying a privileged command that actually ran already. | 15:39 |
| *** niceplace_ has joined #openstack-oslo | 15:39 | |
| bnemec | Bleh | 15:42 |
| efried | bleh which? | 15:44 |
| efried | bleh "can't tell the difference" | 15:44 |
| efried | or are you hatin on my unique-exceptions idea? | 15:44 |
| *** openstackgerrit has joined #openstack-oslo | 15:46 | |
| openstackgerrit | Ben Nemec proposed openstack/oslo.i18n master: Allow Message.translate to handle unhashable inputs https://review.opendev.org/679092 | 15:46 |
| efried | bnemec: this https://opendev.org/openstack/oslo.privsep/src/branch/master/oslo_privsep/daemon.py#L502-L511 | 15:49 |
| efried | looks to me like it's possible we *did* execute the privileged function, and just have no way to send the result back. | 15:49 |
| efried | So if the client channel dies between sending a command and receiving the result, the caller can't actually know whether their thing was run or not. | 15:49 |
| efried | Which I guess is an acceptable limitation. | 15:49 |
| efried | Right now we're worried about the daemon dying, not the client channel | 15:50 |
| efried | BUT | 15:50 |
| efried | we need to be careful not to shut down the client channel before we've drained the message queue from the daemon. | 15:50 |
| bnemec | (the bleh was for the can't tell the difference problem) | 16:00 |
| bnemec | Does the socket buffer? Because if it doesn't then there's no chance of reading anything from it after the daemon dies anyway. | 16:00 |
| *** tesseract has quit IRC | 16:08 | |
| *** bobh has joined #openstack-oslo | 16:11 | |
| efried | like the socket socket? or the message queue in oslo.privsep code itself? The latter yes; the former... no idea. | 16:12 |
| bnemec | Hmm, actually except Exception in the retry didn't fix the problem. | 16:12 |
| * bnemec hates heisenbugs | 16:12 | |
| efried | bnemec: I need to run for a bit, but I'd like to put this thought in your head: | 16:14 |
| efried | My patch as written fixes the SIGHUP problem in an actual running nova. | 16:14 |
| efried | And I can't see how it could make any issues with high concurrency/parallelism *worse* than they already are | 16:14 |
| efried | so | 16:14 |
| efried | Could we consider merging the fix, and trying to nail down some of these races "later"? | 16:14 |
| *** bobh has quit IRC | 16:15 | |
| *** efried is now known as efried_rollin | 16:16 | |
| *** bobh has joined #openstack-oslo | 16:16 | |
| *** bobh has quit IRC | 16:20 | |
| bnemec | efried_rollin: Yeah, perfect being the enemy of good and all that. | 16:21 |
| *** tosky_ has joined #openstack-oslo | 17:45 | |
| *** tosky has quit IRC | 17:45 | |
| *** tosky_ is now known as tosky | 17:45 | |
| *** mmethot has quit IRC | 17:58 | |
| *** zbitter has joined #openstack-oslo | 17:58 | |
| *** mmethot has joined #openstack-oslo | 17:58 | |
| *** zaneb has quit IRC | 17:58 | |
| openstackgerrit | Ben Nemec proposed openstack/oslo.cache stable/rocky: Use upper constraints from opendev.org https://review.opendev.org/655637 | 18:13 |
| sean-k-mooney | bnemec: o/ i noticed a bug in oslo utils last night. i still need to write it up but can that be fixed fater friday is the feature freeze well just for features | 18:38 |
| bnemec | sean-k-mooney: Yeah, Friday is just for features. We have a bit more time for bugfixes. | 18:39 |
| sean-k-mooney | ok. oslo_utiles.imageutils is parsing some direct output of the qemu image info command | 18:39 |
| sean-k-mooney | i think that output has chagned or we just had a latent bug | 18:39 |
| sean-k-mooney | in any case its trying to do int('12.1') which is not allowed | 18:40 |
| sean-k-mooney | so i need to figure out if we shoudl do int(float(raw_vaule)) | 18:40 |
| sean-k-mooney | or maybe do ceil(float(raw_value)) | 18:41 |
| sean-k-mooney | to round up | 18:41 |
| sean-k-mooney | but its fairly trivial once i confirm what the correct behaivor is | 18:41 |
| sean-k-mooney | the corect behavior might just be use float instead of int | 18:42 |
| sean-k-mooney | ill file the bug when i get time and propose a patch this week | 18:42 |
| bnemec | Sounds good, thanks. | 18:45 |
| *** tosky has quit IRC | 18:56 | |
| *** lpetrut has quit IRC | 19:02 | |
| *** efried_rollin is now known as efried | 19:22 | |
| *** ralonsoh has quit IRC | 19:42 | |
| *** kgiusti has left #openstack-oslo | 20:35 | |
| openstackgerrit | Eric Fried proposed openstack/oslo.privsep master: Self-resetting PrivContext https://review.opendev.org/678323 | 20:35 |
| efried | bnemec: Tweaked and un WIPed ^ | 20:37 |
| *** boden has quit IRC | 21:08 | |
| efried | thanks bnemec. Now, who's oslo-core-ish that we can pester to get both of those merged so we can put this bug to bed? | 21:11 |
| bnemec | efried: stephenfin is probably your man, given that this is relevant to his interests on both sides. | 21:12 |
| efried | Heh, just yesterday for an unrelated reason stephenfin disclaimed understanding of oslo.service and oslo.privsep | 21:13 |
| bnemec | And FTR I would consider zane's +2 on https://review.opendev.org/#/c/641907/ still valid since there was only a commit message change since then. | 21:13 |
| efried | cool. Does zaneb review privsep? | 21:13 |
| bnemec | Nobody understands them, that's no excuse. :-P | 21:13 |
| efried | <chuckle> | 21:13 |
| bnemec | Oh, he's zbitter here. | 21:14 |
| bnemec | But yeah, he's full oslo-core now too. | 21:14 |
| bnemec | Although he mostly sticks to oslo.service and related stuff. | 21:14 |
| efried | hey look, dhellmann is specifically an oslo-privsep-core... | 21:14 |
| bnemec | Yeah, he's not doing much oslo these days though. :-( | 21:15 |
| bnemec | Unfortunately we only have about 3 active cores in oslo right now and one of them is on extended PTO. | 21:18 |
| *** jbadiapa has quit IRC | 21:52 | |
| *** adriant has quit IRC | 23:17 | |
| *** adriant has joined #openstack-oslo | 23:18 | |
| *** adriant3 has joined #openstack-oslo | 23:19 | |
| *** rcernin has joined #openstack-oslo | 23:21 | |
| *** adriant3 has quit IRC | 23:21 | |
| *** adriant4 has joined #openstack-oslo | 23:22 | |
| *** adriant has quit IRC | 23:23 | |
| *** adriant49 has joined #openstack-oslo | 23:24 | |
| *** adriant49 has quit IRC | 23:24 | |
| *** adriant4 has quit IRC | 23:28 | |
| *** adriant has joined #openstack-oslo | 23:48 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!