Wednesday, 2018-02-21

*** yamamoto has joined #openstack-oslo00:02
*** yamamoto has quit IRC00:07
*** openstackstatus has quit IRC00:12
*** openstackstatus has joined #openstack-oslo00:14
*** ChanServ sets mode: +v openstackstatus00:14
*** AlexeyAbashkin has joined #openstack-oslo00:23
*** AlexeyAbashkin has quit IRC00:27
*** lbragstad has quit IRC00:32
*** ihrachys has quit IRC00:35
*** d0ugal_ has joined #openstack-oslo00:48
*** daidv has joined #openstack-oslo00:51
*** tovin07_ has joined #openstack-oslo00:51
*** yamamoto has joined #openstack-oslo01:03
*** aspiers has quit IRC01:05
*** dave-mccowan has joined #openstack-oslo01:06
*** yamamoto has quit IRC01:08
*** rcernin has quit IRC01:36
*** rcernin has joined #openstack-oslo01:36
*** rcernin has quit IRC01:42
*** rcernin has joined #openstack-oslo01:43
*** rcernin has quit IRC01:53
*** rcernin has joined #openstack-oslo01:53
*** ansmith has quit IRC01:55
*** salv-orl_ has joined #openstack-oslo02:03
*** yamamoto has joined #openstack-oslo02:04
*** salv-orlando has quit IRC02:07
*** yamamoto has quit IRC02:09
*** tovin07_ has quit IRC02:23
*** tovin07_ has joined #openstack-oslo02:27
*** harlowja_ has quit IRC02:28
*** links has joined #openstack-oslo02:49
*** yamamoto has joined #openstack-oslo03:06
*** prashkre has joined #openstack-oslo03:09
*** yamamoto has quit IRC03:11
*** prashkre_ has joined #openstack-oslo03:13
*** prashkre has quit IRC03:13
*** prashkre has joined #openstack-oslo03:15
*** prashkre_ has quit IRC03:15
*** prashkre_ has joined #openstack-oslo03:17
*** prashkre has quit IRC03:17
*** AlexeyAbashkin has joined #openstack-oslo03:23
*** AlexeyAbashkin has quit IRC03:27
*** prashkre_ has quit IRC03:39
*** lbragstad has joined #openstack-oslo03:43
*** prashkre_ has joined #openstack-oslo03:57
*** yamamoto has joined #openstack-oslo04:07
*** d0ugal__ has joined #openstack-oslo04:13
*** yamamoto has quit IRC04:13
*** d0ugal_ has quit IRC04:15
*** prashkre_ has quit IRC04:19
*** AlexeyAbashkin has joined #openstack-oslo04:22
*** tovin07_ has quit IRC04:23
*** tovin07_ has joined #openstack-oslo04:23
*** AlexeyAbashkin has quit IRC04:26
*** dave-mccowan has quit IRC04:34
*** yamamoto has joined #openstack-oslo05:04
*** openstackgerrit has joined #openstack-oslo05:11
openstackgerritJoshua Harlow proposed openstack/futurist master: Fix alignment of last finished strategy  https://review.openstack.org/54649005:11
openstackgerritJoshua Harlow proposed openstack/futurist master: Fix alignment of last finished strategy  https://review.openstack.org/54649005:19
*** lbragstad has quit IRC05:28
openstackgerritJoshua Harlow proposed openstack/futurist master: Add last finished strategy + fast forward  https://review.openstack.org/54649005:28
openstackgerritJoshua Harlow proposed openstack/futurist master: Add last finished strategy + fast forward  https://review.openstack.org/54649005:29
*** yamamoto has quit IRC05:51
*** janzian has quit IRC06:00
*** yamamoto has joined #openstack-oslo06:02
*** aspiers has joined #openstack-oslo06:17
*** d0ugal__ has quit IRC06:27
*** d0ugal__ has joined #openstack-oslo06:33
*** msimonin has joined #openstack-oslo06:33
*** msimonin has quit IRC06:34
*** msimonin has joined #openstack-oslo06:34
*** msimonin has quit IRC06:35
*** msimonin has joined #openstack-oslo06:35
*** msimonin has quit IRC06:36
*** msimonin has joined #openstack-oslo06:36
*** msimonin has quit IRC06:36
*** d0ugal__ has quit IRC06:52
*** rcernin has quit IRC07:21
*** msimonin has joined #openstack-oslo07:25
*** msimonin has quit IRC07:32
*** hoonetorg has quit IRC07:46
*** pcaruana has joined #openstack-oslo07:47
*** e0ne has joined #openstack-oslo07:52
*** d0ugal has joined #openstack-oslo07:55
*** d0ugal has quit IRC07:55
*** d0ugal has joined #openstack-oslo07:55
*** hoonetorg has joined #openstack-oslo08:00
*** d0ugal has quit IRC08:00
*** d0ugal has joined #openstack-oslo08:01
*** e0ne has quit IRC08:04
*** d0ugal has quit IRC08:06
*** d0ugal has joined #openstack-oslo08:15
*** e0ne has joined #openstack-oslo08:16
*** msimonin has joined #openstack-oslo08:20
*** AlexeyAbashkin has joined #openstack-oslo08:22
*** d0ugal has quit IRC08:29
*** tesseract has joined #openstack-oslo08:34
*** e0ne has quit IRC08:47
*** d0ugal has joined #openstack-oslo08:48
*** d0ugal has joined #openstack-oslo08:48
*** d0ugal has quit IRC08:58
*** msimonin has quit IRC09:00
*** d0ugal has joined #openstack-oslo09:08
*** shardy has joined #openstack-oslo09:12
*** vipuls has quit IRC09:31
*** vipul has joined #openstack-oslo09:33
*** shardy has quit IRC09:43
*** msimonin has joined #openstack-oslo09:49
*** daidv has quit IRC09:58
*** pcaruana has quit IRC10:05
*** d0ugal_ has joined #openstack-oslo10:06
*** d0ugal has quit IRC10:06
*** tovin07_ has quit IRC10:10
*** d0ugal_ has quit IRC10:11
*** d0ugal__ has joined #openstack-oslo10:17
*** pcaruana has joined #openstack-oslo10:20
*** e0ne has joined #openstack-oslo10:21
*** shardy has joined #openstack-oslo10:27
*** d0ugal__ has quit IRC10:54
*** d0ugal has joined #openstack-oslo10:54
*** d0ugal has quit IRC10:54
*** d0ugal has joined #openstack-oslo10:54
*** pcaruana has quit IRC11:19
*** d0ugal has quit IRC11:23
*** d0ugal has joined #openstack-oslo11:31
*** pcaruana has joined #openstack-oslo11:33
*** pbourke has quit IRC11:40
*** yamamoto has quit IRC11:41
*** pbourke has joined #openstack-oslo11:42
*** ansmith has joined #openstack-oslo12:03
*** d0ugal has quit IRC12:06
*** d0ugal has joined #openstack-oslo12:10
*** raildo has joined #openstack-oslo12:13
*** ansmith has quit IRC12:13
*** mgagne has quit IRC12:19
*** d0ugal has quit IRC12:20
*** d34dh0r53 has quit IRC12:20
*** fnordahl has quit IRC12:22
*** dhellmann has quit IRC12:22
*** d0ugal has joined #openstack-oslo12:23
*** fnordahl has joined #openstack-oslo12:23
*** d34dh0r53 has joined #openstack-oslo12:26
*** dhellmann has joined #openstack-oslo12:30
*** d0ugal has quit IRC12:34
*** mgagne has joined #openstack-oslo12:36
*** mgagne is now known as Guest2094612:36
*** yamamoto has joined #openstack-oslo12:42
*** melwitt has quit IRC12:47
*** yamamoto has quit IRC12:49
*** eck` has quit IRC12:50
*** eck`gone has joined #openstack-oslo12:50
*** eck`gone is now known as eck`12:50
*** yamamoto has joined #openstack-oslo12:51
*** melwitt has joined #openstack-oslo12:52
*** dave-mccowan has joined #openstack-oslo13:07
*** d0ugal has joined #openstack-oslo13:13
*** d0ugal_ has joined #openstack-oslo13:19
*** d0ugal has quit IRC13:20
*** d0ugal_ has quit IRC13:25
*** d0ugal_ has joined #openstack-oslo13:38
*** kgiusti has joined #openstack-oslo13:40
*** ansmith has joined #openstack-oslo13:46
openstackgerritJoshua Harlow proposed openstack/futurist master: Add last finished strategy + fast forward  https://review.openstack.org/54649013:47
*** d0ugal_ has quit IRC13:48
*** d0ugal has joined #openstack-oslo13:48
*** d0ugal has quit IRC13:48
*** d0ugal has joined #openstack-oslo13:48
*** d0ugal has quit IRC13:58
*** d0ugal has joined #openstack-oslo13:59
*** ihrachys has joined #openstack-oslo14:03
*** gordc has joined #openstack-oslo14:03
*** d0ugal has quit IRC14:04
*** lbragstad has joined #openstack-oslo14:15
*** d0ugal has joined #openstack-oslo14:22
*** links has quit IRC14:29
*** dmellado has quit IRC14:37
*** dmellado has joined #openstack-oslo14:42
*** gordc has quit IRC14:49
*** d0ugal has quit IRC14:49
*** janzian has joined #openstack-oslo14:51
*** d0ugal has joined #openstack-oslo14:58
*** crushil_ has joined #openstack-oslo15:00
*** crushil has joined #openstack-oslo15:00
*** crushil_ has quit IRC15:00
*** spilla has joined #openstack-oslo15:03
*** trevormc has joined #openstack-oslo15:26
*** gordc has joined #openstack-oslo15:43
*** tobasco has joined #openstack-oslo15:46
tobascois there any support matrix for oslo.messaging supported versions for each os release? in my case rabbitmq15:46
*** yamamoto has quit IRC15:55
kgiustitobasco: hi - I'm not aware of any such document.  As far as a release is concerned most testing is against whichever version of rabbitmq is provided by the platform (e.g. Ubuntu Xenial, Centos 7) at the time of release.16:02
tobascokgiusti: ok thx16:03
*** tobasco has left #openstack-oslo16:04
*** tobasco has joined #openstack-oslo16:04
*** crushil has quit IRC16:06
*** crushil has joined #openstack-oslo16:06
*** trevormc has quit IRC16:12
*** pcaruana has quit IRC16:16
*** openstackgerrit has quit IRC16:19
*** AlexeyAbashkin has quit IRC16:35
*** john51 has quit IRC16:45
*** john51 has joined #openstack-oslo16:49
*** salv-orl_ has quit IRC16:49
*** salv-orlando has joined #openstack-oslo16:50
*** salv-orlando has quit IRC16:54
*** yamamoto has joined #openstack-oslo16:56
*** yamamoto has quit IRC17:02
*** e0ne has quit IRC17:25
*** AlexeyAbashkin has joined #openstack-oslo17:57
*** yamamoto has joined #openstack-oslo17:58
*** AlexeyAbashkin has quit IRC18:01
*** yamamoto has quit IRC18:04
*** msimonin has quit IRC18:05
*** crushil has quit IRC18:13
*** crushil has joined #openstack-oslo18:13
*** tesseract has quit IRC18:25
*** e0ne has joined #openstack-oslo18:59
*** yamamoto has joined #openstack-oslo19:00
*** e0ne has quit IRC19:00
dansmithsileht: are you still around? and are you still the go-to person for oslo.messaging?19:02
*** tbarron has quit IRC19:02
*** salv-orlando has joined #openstack-oslo19:03
*** yamamoto has quit IRC19:06
*** tbarron has joined #openstack-oslo19:13
*** AlexeyAbashkin has joined #openstack-oslo19:21
*** AlexeyAbashkin has quit IRC19:25
*** openstackgerrit has joined #openstack-oslo19:29
openstackgerritprashkre proposed openstack/oslo.concurrency master: Mask passwords only when command execution fails  https://review.openstack.org/53782319:29
*** salv-orlando has quit IRC19:38
*** salv-orlando has joined #openstack-oslo19:39
*** lbragstad has quit IRC19:40
openstackgerritprashkre proposed openstack/oslo.concurrency master: Mask passwords only when command execution fails  https://review.openstack.org/53782319:42
*** salv-orlando has quit IRC19:43
kgiustidansmith: hey - anything I can help you with?19:45
dansmithkgiusti: I just wanted to discuss a thing I was thinking about proposing to o.msg19:46
dansmithjust in case it's a "hell no" kind of situation :)19:46
kgiustidansmith: kewl - are you going to PTG?19:46
dansmithyeah19:46
kgiustidansmith: I'm all ears - they'll be more people at PTG if you want to discuss it more on Tuesday19:47
dansmithkgiusti: well, I'd like a little pre-sanity-checking and then I can get my code into a little closer to reviewable shape19:47
kgiustidansmith: you've got code?  Man you are serious :)19:47
dansmithoh yeah, working PoC and everything :)19:48
dansmithso, we have some long-running operations that can vary from quick to very long, that we access via rpc call19:48
dansmithwe don't really want to increase the global rpc timeout to the longest one of those could be, but thats what users end up doing, sometimes to extreme values like an hour or more19:49
dansmithso I've had this idea in my head for a while and just banged out a PoC,19:49
dansmithwhich is to heartbeat from the server to the client that the thing is still running, so the client could have a soft and hard timeout kind of behavior19:50
dansmithlike "if it's still running, I'll wait up to ten minutes, but if you stop responding (or never did), then I'm timing out in 30 seconds19:50
*** e0ne has joined #openstack-oslo19:51
kgiustidansmith: interesting...19:51
dansmiththe alternative is for us to convert things that are fundamentally calls to casts, and set up a rendezvous type thing at the higher layer to be able to continue processing a thing when the remote side casts back to us,19:51
dansmithbut that's just us duplicating what is already there at the lower layer in o.msg19:52
dansmithso I figure if the server can indicate to the client that the thing is still running, the client can continue to wait up to the hard timeout (which may indicate something is hung) and that's a much cleaner way to go19:52
kgiustidansmith: so there'd be some application to application heart beat that runs "in the background" ?19:52
dansmithit's all in o.msg, but yeah, separate from the connection heartbeating that we do with rabbit to know that rabbit is still alive19:53
dansmithI was thinking that if we pass what our timeout is to the server side, the server can heartbeat every timeout/2 seconds, so we're really not generating a ton of traffic like we would if we just did it every ten seconds or something19:53
kgiustidansmith: are you trying to distinguish between an RPC call that is just taking extra time to execute, or a backed up queue, or a missing server?19:55
kgiustidansmith: the broker in the middle adds a bit of uncertainty since the client and server are decoupled.19:55
dansmiththat's the thing, right now we can't tell the difference between "the service isn't running" and "this is running, but taking a long time"19:56
kgiustidansmith: right.  So one thing to consider is the queues.  your rpc request may be sitting on the queue for a while if no or slow server.19:56
kgiustidansmith: won't the heartbeat messages also be queued up?19:57
dansmithyep, for sure, so we just reset the soft timer when we get a heartbeat,19:57
dansmithso if we get a flood of them, we don't suddenly increase the timer N times19:58
dansmiththis won't really address the case where you're lacking mq bandwidth, or where the remote server is too busy to process all the requests19:58
dansmiththat will hit the hard timeout just like it does today19:58
dansmiththis will address the fact that the server may take ten minutes to create a 4TB volume, and we time out not knowing if it was working on it or just isn't running19:59
kgiustidansmith: yeah, I'm worried that if the mq is heavily loaded and things have slowed down, then adding additional messages (heartbeat) could make things worse...19:59
kgiustidansmith: what if we had an async rpc call?19:59
dansmithalso, my code doesn't heartbeat until the dispatcher actually starts running the method, so no heartbeats while the original request is in a queue20:00
dansmithkgiusti: say more words20:00
*** lbragstad has joined #openstack-oslo20:01
*** yamamoto has joined #openstack-oslo20:02
kgiustidansmith: mmmm.... I had an idea but it may not help.  It seems you need to know that not only is the request being processed, but what happens if the server processing the message craps out.20:02
dansmithyeah, we don't care so much about the latter,20:02
dansmithalthough we have to eventually time out if that happens of course20:02
dansmithI don't want to heartbeat like every second or anything20:02
dansmithI would expect the soft timeout would still be like it is today, about 60s, but the hard timeout could be higher like ten minutes20:03
kgiustidansmith: oh.  hmmm...  well, what I meant by an async rpc call is that your client would fire off the rpc call without waiting for the response.  The call itself would block only until the server gets the message and starts processing it.  The client would get some sort of future like thingy back that would become ready when the server sends the reply.20:04
dansmithI don't think we'd add that much extra load to the mq, because we'd only heartbeat from calls that actually started in the server.. anything sitting in a queue because we're backed up won't do anything.. if we polled from the client, then we would generate extra load, which would be bad20:04
kgiustidansmith: IOW the timeout would only apply to the transfer of the request to the server.20:05
dansmithkgiusti: I don't think that helps at all, because we're not trying to parallelize extra work on the client side, we usually can't do anything until we get the reply back anyway, we just don't want to wait ten minutes if the server is down, but are willing to if that thing is running and just doing lots of work20:05
dansmithwe could also make this heartbeating thing only happen if the client requests, it, and we would only use it for things we know can take a long time like creating a volume, or scheduling a lot of instances20:07
dansmithI'd rather it just kinda work automatically, but...20:07
*** yamamoto has quit IRC20:07
dansmithif we only heartbeat at timeout/2 then that would mean today only things that took more than 30 seconds to finish would get a single heartbeat message in the interim,20:07
dansmithwhich I guess just doesn't seem like a big deal20:08
dansmithmaking calls that are running and consuming resources, but that we just give up on when we shouldn't is a much larger waste20:08
kgiustidansmith: yeah it's not so much the async aspect as the early indication that the request hit the server...20:08
kgiustidansmith: which doesn't preclude the server from heartbeating.20:09
dansmithokay, so in my model that's like one heartbeat immediately which scales the timer on the client way higher, assuming that if it hit the server it'll come back, so we wait longer than we would have normally?20:09
kgiustidansmith: roughly yes - though it doesn't necessarily eliminate heatbeats in the background.20:10
dansmithyeah, I mean that addresses part of what I want I guess.. it's certainly better than what we have today20:11
dansmithif you think about an api call which calls to the scheduler to do a very large thing, the scheduler replies once, and then OOMs because you asked for a super big thing, you sit around for a long time before realizing that systemd restarted it a while ago :)20:12
dansmithbut20:12
dansmithhalfway between would be some sort of exponential thing where we get a heartbeat when we start running, and then heartbeats after that come at increasingly long delays up to some threshold20:12
dansmithand the client expects this (or is told when the next one will be) so it plays along20:13
kgiustidansmith: yeah you'd still want that periodic "I'm still working on this" heatbeat from the server20:13
dansmithI would yeah20:14
kgiustidansmith: which isn't impossible - o.m. could send back "in process" rpc replies until the final rpc reply arrives.20:14
dansmithso the async/future thing you're talking about is really a separate thing, which could be done separately from all of this that I'm describing, and while cool, it's not something I really think we'd use at the moment, just FYI20:15
dansmithkgiusti: that's what my PoC is doing already yeah20:15
dansmiththere's already something in o.msg that seemed to anticipate this potential in the future, although it had been marked for removal "In N" which never happened20:15
kgiustidansmith: what's that?20:16
dansmiththere's an "ending" flag, which tells the waiting client whether the reply is the last one or not, and the client waits until it sees the ending one, then returns that as the reply20:16
kgiustidansmith: heh, yeah that was never used.20:17
openstackgerritDan Smith proposed openstack/oslo.messaging master: WIP: Heartbeat active calls  https://review.openstack.org/54676320:17
dansmithkgiusti: ^20:17
dansmithdon't laugh, was never meant to see the light of day20:17
kgiustidansmith: np.  I may laugh, but it will be a respectful laugh :)20:18
dansmithheh20:18
kgiustidansmith: but I'm usually too busy laughing at my own work.20:18
*** pblaho has quit IRC20:18
kgiustidansmith: I'll take a look at this, but in the meantime I'm def not saying "hell no" :D20:19
dansmithokay20:19
kgiustidansmith: so if you want to talk about it at PTG that'd be great.20:19
dansmithis there a schedule for Tuesday?20:19
dansmithfor oslo stuff I mean20:19
kgiustidansmith: not exactly a hard schedule, but there is a list of topics folks want to talk about: https://etherpad.openstack.org/p/oslo-ptg-rocky20:21
dansmithokay20:22
dansmithwell, I'll try to pop in there20:22
kgiustidansmith: feel free to raise a topic or simply add an item to my O.M. rocky goals list so I (or you) can bring it up.20:23
kgiustidansmith: I'll ping you on #messaging before we start if you want.20:23
*** crushil has quit IRC20:23
dansmithcool, I'll tag along with yours there yeah20:23
kgiustidansmith: crap - I mean #openstack-oslo20:24
*** crushil has joined #openstack-oslo20:24
dansmithack, cool20:24
*** crushil has quit IRC20:25
*** crushil has joined #openstack-oslo20:26
kgiustidansmith: plan.20:26
dansmiththanks!20:26
*** lbragstad has quit IRC20:29
*** lbragstad has joined #openstack-oslo20:30
*** lbragstad has quit IRC20:32
*** lbragstad has joined #openstack-oslo20:33
*** salv-orlando has joined #openstack-oslo20:39
*** salv-orlando has quit IRC20:44
*** crushil has quit IRC20:45
*** rmascena has joined #openstack-oslo20:57
*** dave-mccowan has quit IRC21:00
*** raildo has quit IRC21:00
*** rmascena__ has joined #openstack-oslo21:01
*** kgiusti has left #openstack-oslo21:02
*** salv-orlando has joined #openstack-oslo21:03
*** openstackgerrit has quit IRC21:03
*** yamamoto has joined #openstack-oslo21:04
*** rmascena has quit IRC21:05
*** crushil has joined #openstack-oslo21:09
*** yamamoto has quit IRC21:09
*** ansmith has quit IRC21:16
*** msimonin has joined #openstack-oslo21:31
*** pooja_jadhav has quit IRC21:35
*** bhagyashris has quit IRC21:44
*** e0ne has quit IRC21:46
*** rmascena__ has quit IRC21:53
*** dmellado has quit IRC22:00
*** crushil has quit IRC22:05
*** yamamoto has joined #openstack-oslo22:05
*** yamamoto has quit IRC22:10
*** ansmith has joined #openstack-oslo22:17
*** spilla has quit IRC22:24
*** rcernin has joined #openstack-oslo22:32
*** salv-orlando has quit IRC22:34
*** salv-orlando has joined #openstack-oslo22:34
*** yamamoto has joined #openstack-oslo22:35
*** salv-orlando has quit IRC22:39
*** lbragstad has quit IRC22:47
*** lbragstad has joined #openstack-oslo22:48
*** salv-orlando has joined #openstack-oslo22:53
*** gordc has quit IRC22:54
*** yamamoto has quit IRC23:08
*** msimonin has quit IRC23:27
*** msimonin has joined #openstack-oslo23:27
*** msimonin has quit IRC23:28
*** msimonin has joined #openstack-oslo23:28
*** msimonin has quit IRC23:29
*** msimonin has joined #openstack-oslo23:29
*** msimonin has quit IRC23:30
*** msimonin has joined #openstack-oslo23:30
*** msimonin has quit IRC23:30
*** msimonin has joined #openstack-oslo23:31
*** msimonin has quit IRC23:31
*** salv-orlando has quit IRC23:32
*** salv-orlando has joined #openstack-oslo23:33
*** salv-orlando has quit IRC23:37

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!