| *** mustapha has joined #openstack-zaqar | 01:51 | |
| mustapha | Hey! peps | 01:51 |
|---|---|---|
| *** kgriffs|afk is now known as kgriffs | 02:02 | |
| *** kgriffs is now known as kgriffs|afk | 02:07 | |
| *** cpallares has quit IRC | 03:03 | |
| *** chandankumar has joined #openstack-zaqar | 05:00 | |
| *** AAzza_afk is now known as AAzza | 06:54 | |
| *** AAzza is now known as AAzza_afk | 07:56 | |
| *** flwang1 has joined #openstack-zaqar | 08:30 | |
| *** julienvey has joined #openstack-zaqar | 09:03 | |
| *** flwang1 has quit IRC | 09:34 | |
| *** flwang1 has joined #openstack-zaqar | 09:35 | |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack/zaqar-specs: Fix .gitreview due to the repo rename/move https://review.openstack.org/114779 | 10:29 |
| *** flwang1 has quit IRC | 10:37 | |
| *** flwang1 has joined #openstack-zaqar | 10:38 | |
| *** flwang1 has quit IRC | 10:49 | |
| *** flwang1 has joined #openstack-zaqar | 11:06 | |
| *** vkmc has joined #openstack-zaqar | 11:10 | |
| *** vkmc has joined #openstack-zaqar | 11:10 | |
| *** fifieldt has joined #openstack-zaqar | 11:23 | |
| *** mpanetta has joined #openstack-zaqar | 11:56 | |
| openstackgerrit | Fei Long Wang proposed a change to openstack/zaqar: Zaqar is depending on python-keystoneclient https://review.openstack.org/114944 | 12:04 |
| *** mpanetta has quit IRC | 12:06 | |
| *** mpanetta has joined #openstack-zaqar | 12:07 | |
| *** mpee|away has joined #openstack-zaqar | 12:20 | |
| vkmc | malini|afk, hi! I was trying to spot why is the tempest-dsvm gate failing... the renaming in https://github.com/openstack-infra/config/blob/master/modules/openstack_project/files/jenkins_job_builder/config/devstack-gate.yaml looks fine, what am I missing? | 12:20 |
| *** flwang1 has quit IRC | 12:37 | |
| *** AAzza_afk is now known as AAzza | 12:38 | |
| *** sriram has joined #openstack-zaqar | 12:43 | |
| *** itisit has joined #openstack-zaqar | 13:26 | |
| *** malini has joined #openstack-zaqar | 13:26 | |
| *** amitgandhinz has joined #openstack-zaqar | 13:40 | |
| *** kgriffs|afk is now known as kgriffs | 13:41 | |
| *** kgriffs is now known as kgriffs|afk | 13:42 | |
| *** chandankumar has quit IRC | 13:42 | |
| *** ametts has joined #openstack-zaqar | 13:47 | |
| malini | hello!! | 13:58 |
| vkmc | hi malini! | 13:58 |
| malini | heyyy vkmc! how are you? | 13:59 |
| vkmc | all good and you? | 13:59 |
| malini | good :) | 13:59 |
| malini | I just submitted a new patchset to update devstack - marconi —>zaqar | 13:59 |
| vkmc | oh cool :) | 14:00 |
| malini | Will get the tempest one out shortly | 14:00 |
| vkmc | awesome | 14:01 |
| vkmc | I was checking that earlier today | 14:01 |
| vkmc | we also have to perform the rename in Tempest | 14:02 |
| *** mpanetta_ has joined #openstack-zaqar | 14:03 | |
| openstackgerrit | Nataliia Uvarova proposed a change to openstack/zaqar: Improve benchmarking tool https://review.openstack.org/112236 | 14:06 |
| *** mpanetta has quit IRC | 14:11 | |
| *** itisit has left #openstack-zaqar | 14:36 | |
| *** itisit has joined #openstack-zaqar | 14:41 | |
| *** kgriffs|afk is now known as kgriffs | 14:42 | |
| prashanthr_|afk | good morning ! :) | 14:42 |
| *** prashanthr_|afk is now known as prashanthr_ | 14:43 | |
| vkmc | hi prashanthr_! | 14:44 |
| prashanthr_ | vkmc: Hello ! :) | 14:44 |
| prashanthr_ | pencils down day of gsoc today | 14:44 |
| vkmc | yuuup | 14:45 |
| vkmc | we have to submit the final evaluations tomorrow | 14:45 |
| vkmc | and then the code sample | 14:45 |
| prashanthr_ | vkmc: True :) | 14:47 |
| *** cpallares has joined #openstack-zaqar | 14:47 | |
| kgriffs | hi everyone. sorry for being MIA friday... lots of meetings | 14:52 |
| vkmc | AAzza, hey! did flaper87|afk address your comments in https://review.openstack.org/#/c/106346/? | 14:53 |
| vkmc | hi kgriffs! | 14:53 |
| kgriffs | what's all this pencil's down business? | 14:57 |
| kgriffs | gsoc is coming to a close? | 14:57 |
| *** alcabrera|afk is now known as alcabrera | 14:57 | |
| vkmc | kgriffs, yes... today is the last day of GSoC | 14:57 |
| kgriffs | oh! | 14:58 |
| alcabrera | good morning, all. :) | 14:59 |
| alcabrera | yeah, GSoC/OPW both end today. Firm "pencil's down" date is today. | 14:59 |
| alcabrera | "end" as in - time to start the final eval. | 14:59 |
| kgriffs | I hope everyone learned some cool stuff this summer that will be useful in future adventures. :D | 14:59 |
| alcabrera | oh yeah! I hope so, too. :) | 15:00 |
| vkmc | thanks kgriffs, alcabrera! | 15:01 |
| vkmc | I feel I learned a lot although I know I still have a long way to go | 15:02 |
| vkmc | and I'm happy I finally could get officially involved with the Marconi/Zaqar project | 15:02 |
| alcabrera | I have so much to learn, as well. It never ends! | 15:02 |
| alcabrera | hurray, vkmc! | 15:03 |
| vkmc | true that... it never ends, and I'm glad it's like that in our profession | 15:04 |
| vkmc | we won't ever get bored | 15:04 |
| vkmc | hope I can keep learning from you guys, you are amazing developers with an incredible experience | 15:05 |
| kgriffs | You cease to be wise when you think you have learned everything there is to know. | 15:05 |
| AAzza | kgriffs: not sure :) i think no, we agreead to use sorting by name, but after he updated patch, he removes sorting totally, so it is better to clarify with him tomorrow. | 15:07 |
| kgriffs | kk | 15:07 |
| prashanthr_ | True. Gsoc was one of the greatest experiences :) | 15:08 |
| vkmc | AAzza, cool :) thanks | 15:09 |
| vkmc | AAzza, could you leave record of that in the tracker? | 15:12 |
| vkmc | AAzza, well spotted btw | 15:12 |
| *** mpanetta_ is now known as mpanetta | 15:14 | |
| *** chandankumar has joined #openstack-zaqar | 15:20 | |
| kgriffs | prashanthr_: btw, how's the redis work going? | 15:27 |
| prashanthr_ | kgriffs: I was not well for two days :( | 15:27 |
| prashanthr_ | recovering today | 15:27 |
| prashanthr_ | Will resume work tomorrow | 15:27 |
| kgriffs | oh, sorry to hear that. get well soon! | 15:29 |
| prashanthr_ | thank you kgriffs !! | 15:29 |
| *** julienvey has quit IRC | 16:01 | |
| *** chandankumar has quit IRC | 16:10 | |
| *** AAzza is now known as AAzza_afk | 16:11 | |
| *** kgriffs is now known as kgriffs|afk | 16:29 | |
| *** reed has joined #openstack-zaqar | 16:30 | |
| *** kgriffs|afk is now known as kgriffs | 17:06 | |
| kgriffs | malini: is there a tempest patch or something that I can +1 ? | 17:35 |
| malini | kgriffs: yes..let me get u the link | 17:35 |
| malini | kgriffs: devtstack needs to go first https://review.openstack.org/#/c/111399/ | 17:36 |
| malini | tempest https://review.openstack.org/#/c/114980/ | 17:36 |
| * kgriffs clicks | 17:37 | |
| kgriffs | ok, I +1's the devstack patch | 17:39 |
| kgriffs | once that lands, I'll do tempest | 17:39 |
| malini | thanks kgriffs! | 17:39 |
| kgriffs | thanks for submitting those! | 17:39 |
| malini | I just pinged the devstack core reviewers | 17:39 |
| malini | yw :) | 17:40 |
| vkmc | :) and once those get merged we have to ninja approve https://review.openstack.org/#/c/114776/ | 17:40 |
| kgriffs | sure thing | 17:41 |
| vkmc | oh kgriffs, how can I regenerate the conf file? | 18:06 |
| vkmc | the transport options are missing | 18:06 |
| kgriffs | there is a script for doing that | 18:08 |
| kgriffs | tools/config/generate_sample.sh | 18:08 |
| kgriffs | it is sort of broken on Mac (or used to be) FYI | 18:08 |
| vkmc | thanks | 18:09 |
| kgriffs | should really be ported to python, but nobody has taken the time to do it yet. :p | 18:09 |
| kgriffs | malini, vkmc: I learned something interesting last week | 18:13 |
| vkmc | kgriffs, what did you learn? | 18:14 |
| malini | I am curious too.. | 18:14 |
| kgriffs | if you use a tiered NTP setup, meaning You run your own NTP box close to your servers, and then have that NTP server, in turn, sync with the public pool, you are supposed to get clock sync to ~500 microseconds | 18:15 |
| malini | I am sure mpanetta wud apprciate tht info | 18:16 |
| kgriffs | that is pretty cool, because it means we may be able to switch to using timestamp-based IDs after all. The trick is, you artificically delay each queue by V variance | 18:16 |
| vkmc | and what would happen if the delay isn't added? | 18:17 |
| kgriffs | where V = max_expected(clock_drift + db_latency + context_switch_overhead + fudge_room) | 18:18 |
| kgriffs | vkmc: what might happen is that in between calculating the timestamp-based ID and persisting to the DB, you can have a temporal inversion. Let me see if I can make a little drawing to explain... | 18:19 |
| kgriffs | (in any case, it can ultimately lead to a message getting missed by a client) | 18:19 |
| kgriffs | the delay-queue scheme isn't perfect... there is always the chance that something crazy will happen that will cause a message to take 10 seconds to persist, but it seems like that should be manageable by ops just ensuring there is always enough headroom/capacity on the deployment to prevent overload. | 18:21 |
| kgriffs | lots of notes here, but they are a bit scatter-brained at the moment: https://gist.github.com/kgriffs/2f83c10654f1ace06fc3 | 18:22 |
| vkmc | it's good because we count with more information, but there is always the chance of something going wrong | 18:23 |
| vkmc | time is one of the more complex problems in computing | 18:23 |
| vkmc | s/more/most | 18:23 |
| kgriffs | yep, sure is | 18:23 |
| * vkmc clicks | 18:23 | |
| kgriffs | I'll be back in a few minutes | 18:24 |
| vkmc | thanks for sharing :) | 18:24 |
| *** kgriffs is now known as kgriffs|afk | 18:24 | |
| vkmc | hm the generate_sample.sh is not working http://paste.openstack.org/show/96956/ | 18:28 |
| vkmc | as kgriffs|afk warned | 18:29 |
| *** alcabrera is now known as alcabrera|afk | 18:33 | |
| *** kgriffs|afk is now known as kgriffs | 18:43 | |
| kgriffs | here is the explanation for the race condition and need for a "variance" window: http://www.educreations.com/lesson/view/delayed-queues/23270365/?ref=appemail | 18:53 |
| vkmc | cool | 18:54 |
| vkmc | offtopic, how we call the service Zaqar provides? Message Queing service, Message Queueing service, Queues service, Messaging service | 18:56 |
| vkmc | ? | 18:56 |
| vkmc | I remember we chatted about this in a meeting | 18:56 |
| kgriffs | good question. iirc the idea was to put the focus on "messages" | 18:57 |
| kgriffs | program name ought to be changed to something like | 18:57 |
| kgriffs | OpenStack Messaging and Notifications Service | 18:57 |
| kgriffs | brb (meeting) | 18:58 |
| *** kgriffs is now known as kgriffs|afk | 18:58 | |
| vkmc | right, let's add that to the etherpad so we all follow that convention | 19:02 |
| *** kgriffs|afk is now known as kgriffs | 19:25 | |
| *** malini has quit IRC | 19:32 | |
| *** AAzza_afk is now known as AAzza | 19:58 | |
| *** vkmc has quit IRC | 19:58 | |
| *** vkmc has joined #openstack-zaqar | 19:58 | |
| *** vkmc has joined #openstack-zaqar | 19:58 | |
| *** alcabrera|afk is now known as alcabrera | 20:04 | |
| vkmc | kgriffs, btw I saw the variance window explanation :) it makes sense | 20:22 |
| vkmc | I wonder though which is the criteria on choosing the length of the window | 20:23 |
| kgriffs | there are several factors | 20:23 |
| kgriffs | first is the expected clock drift between webheads (boxes running uwsgi or gunicorn) | 20:23 |
| kgriffs | then you have to decide how long a given worker thread (probably a green thread) may ever stall | 20:24 |
| kgriffs | then you have to think about how long it might take to persist the message in the DB | 20:25 |
| kgriffs | those last two basically determine the time window between assigning a timestamp-based ID, and then actually writing it to the DB, after which is can be read back out by a subsequent client request | 20:26 |
| kgriffs | does this make any sense? :p | 20:26 |
| vkmc | that is part of the variation formula you detailed before | 20:27 |
| vkmc | yes :) | 20:27 |
| kgriffs | yeah | 20:27 |
| vkmc | if the variation is enough then this will work as expected | 20:28 |
| vkmc | but what if the variation is not enough | 20:28 |
| kgriffs | then you have the potential to miss messages | 20:28 |
| kgriffs | the potential will increase as the client request frequency increases | 20:29 |
| alcabrera | that variation factor would also need to be recalculated for every update in the software, every update in the hardware | 20:29 |
| kgriffs | (and as the number of outliers increases due to insufficient V) | 20:29 |
| kgriffs | alcabrera: adding a fudge factor can mitigate the risk there, also provide some wiggle-room to keep ahead of system load by adding capacity so things to slow down to the point that outliers become more common. | 20:30 |
| kgriffs | but yeah, you would also need to be careful about software not slowing things down, and hardware changes | 20:31 |
| alcabrera | risk can be reduced, I agree | 20:31 |
| alcabrera | and risk can be further reduced by enforcing homogeneous hardware on deployments | 20:31 |
| alcabrera | the unfortunate part is that no guarantee can be made | 20:31 |
| kgriffs | it will always be there, however. On the upside, you get better scaling options since you don't need to rely on a centralized counter mechanism thingy | 20:32 |
| alcabrera | as long as time is used as the consistency mechanism | 20:32 |
| kgriffs | yep. This is the fundamental tradeoff in a stateless message feed design. Atom Hopper has the same problem. | 20:32 |
| kgriffs | it may also provide some insight into why some queuing systems do not guarantee message delivery, ordering | 20:34 |
| alcabrera | agreed ^ | 20:34 |
| alcabrera | if you guarantee FIFO, you have a CP system | 20:34 |
| kgriffs | speaking of ordering, as long as your clock granularity is below expected throughput, ordering is virtually guaranteed. otherwise not. :p | 20:34 |
| alcabrera | FIFO == linearizable | 20:34 |
| kgriffs | how does google solve it? | 20:35 |
| alcabrera | there's the hardware clocks approach, in one paper | 20:35 |
| kgriffs | yep, that's what I was about to say | 20:35 |
| kgriffs | #cheaters | 20:35 |
| alcabrera | :P | 20:35 |
| mpanetta | #bigmoney :P | 20:35 |
| kgriffs | #notopenstack | 20:36 |
| * kgriffs needs a hug | 20:36 | |
| * mpanetta gives kgriffs a GPS clock source instead. | 20:36 | |
| * kgriffs goes off to find a turing machine to play with | 20:36 | |
| alcabrera | :) | 20:37 |
| alcabrera | as long as zaqar clearly advertises that it does not guarantee FIFO, I think it's okay to move away from the atomic counter scheme | 20:37 |
| * alcabrera offers hugs to all | 20:38 | |
| kgriffs | alcabrera: we would also have to rescind the delivery guarantee. "In extenuating circumstances a client may miss a message. But here is how you make the risk of that virtually zero..." | 20:39 |
| alcabrera | yup! | 20:39 |
| alcabrera | "at least once" and "at most once" properties are also lost without a mechanism to enforce consistency | 20:39 |
| alcabrera | **linear consistency | 20:40 |
| kgriffs | hmmm, I'm trying to decide if you can still guarantee "at most once". I think you can? | 20:40 |
| kgriffs | oh, I guess it would depend on not having a collision in the timestamp value. so you have to granular enough to support expected throughput. | 20:41 |
| alcabrera | if shards diverge because of a network partition, that might also cause problems with "at most once" | 20:41 |
| kgriffs | 100-nanoseconds should be enough. ;) | 20:41 |
| alcabrera | well, once we get to shard pools | 20:41 |
| alcabrera | as it is, a network partition would just result in "not found" | 20:41 |
| alcabrera | or 500 | 20:41 |
| kgriffs | yeah | 20:42 |
| kgriffs | anyway, lots of juicy CS problems in this "humble" project of ours. :p | 20:42 |
| alcabrera | oh yeah! :D | 20:42 |
| alcabrera | you chew on distributed systems, you | 20:42 |
| alcabrera | you're forced to the forefront of CS | 20:42 |
| alcabrera | :) | 20:42 |
| kgriffs | srsly. | 20:42 |
| alcabrera | curiously, the same thing happens with program correctness | 20:42 |
| alcabrera | and performance | 20:43 |
| alcabrera | and.. | 20:43 |
| alcabrera | naming! | 20:43 |
| alcabrera | >.> | 20:43 |
| kgriffs | meeting in 15 min | 20:45 |
| kgriffs | alcabrera: NAME ALL THE THINGS! | 20:46 |
| * kgriffs dies trying | 20:46 | |
| alcabrera | D: | 20:46 |
| * alcabrera revives kgriffs | 20:46 | |
| kgriffs | The three hardest things in CS | 20:46 |
| kgriffs | 1. Time-based algorithms | 20:46 |
| kgriffs | 2. Naming stuff | 20:47 |
| kgriffs | 3. Responding to ALL Hacker News comments | 20:47 |
| alcabrera | that third one is just painful. >.> | 20:47 |
| alcabrera | I don't even look at HN any more | 20:47 |
| kgriffs | it's the new slashdot. :p | 20:47 |
| kgriffs | a victim of the dehumanizing nature of the internets | 20:48 |
| kgriffs | (p.s. - read this! http://www.amazon.com/Leaders-Eat-Last-Together-Others-ebook/dp/B00G3L0ZTQ) | 20:48 |
| kgriffs | alcabrera: that reminds me... | 20:49 |
| kgriffs | http://trololololololololololo.com/ | 20:49 |
| alcabrera | :P | 20:50 |
| alcabrera | i wasn't aware of the history behind tro(lo * 10) 'til just now | 20:51 |
| alcabrera | TIL: http://knowyourmeme.com/memes/trololo-russian-rickroll | 20:51 |
| vkmc | trololololo | 20:51 |
| vkmc | lololo | 20:51 |
| * vkmc sings | 20:51 | |
| *** AAzza is now known as AAzza_afk | 20:54 | |
| vkmc | meeeeting | 21:01 |
| alcabrera | heading out. you all have an awesome meeting! :) | 21:01 |
| vkmc | alcabrera, babai o/ | 21:02 |
| *** alcabrera is now known as alcabrera|afk | 21:05 | |
| vkmc | kgriffs, meeting? | 21:06 |
| *** itisit has quit IRC | 21:08 | |
| kgriffs | oh, crap | 21:11 |
| kgriffs | lost track of time | 21:11 |
| kgriffs | vkmc: nobody around? | 21:12 |
| vkmc | here :) | 21:13 |
| vkmc | let's ping people | 21:13 |
| flwang | meeting? | 21:14 |
| vkmc | ! | 21:14 |
| *** mpanetta has quit IRC | 21:26 | |
| *** sriram has quit IRC | 21:27 | |
| *** malini has joined #openstack-zaqar | 21:31 | |
| *** malini has quit IRC | 21:33 | |
| *** malini has joined #openstack-zaqar | 21:33 | |
| *** itisit has joined #openstack-zaqar | 21:51 | |
| *** cpallares has quit IRC | 21:51 | |
| *** vkmc_ has joined #openstack-zaqar | 22:10 | |
| *** vkmc has quit IRC | 22:11 | |
| vkmc_ | :( darn | 22:13 |
| *** vkmc_ is now known as vkmc | 22:13 | |
| *** vkmc has quit IRC | 22:13 | |
| *** vkmc has joined #openstack-zaqar | 22:13 | |
| vkmc | my connection died | 22:14 |
| *** flwang1 has joined #openstack-zaqar | 22:14 | |
| vkmc | kgriffs, so... I asked, how do you feel about the graduation | 22:14 |
| *** amitgandhinz has quit IRC | 22:15 | |
| kgriffs | I was feeling optimistic until that thread blew up | 22:15 |
| kgriffs | TBH, I don't see why a project has to be "perfect" before it graduates, but that seems to be the prevailing wind and I am honestly too burned out to fight it any more | 22:16 |
| vkmc | I think that the barrier is high | 22:17 |
| vkmc | but we have what it takes | 22:17 |
| vkmc | and Zaqar team already faced the graduation process before | 22:17 |
| kgriffs | for us to have a good chance, we'll have to start socializing each of the "concerns raised" on the ML or something | 22:23 |
| kgriffs | (see relevant section of https://etherpad.openstack.org/p/marconi-graduation) | 22:23 |
| kgriffs | do one at a time, in order of least controversial to most. | 22:24 |
| *** malini has quit IRC | 22:26 | |
| vkmc | yeah | 22:27 |
| kgriffs | vkmc: different topic | 22:27 |
| vkmc | sure | 22:27 |
| kgriffs | it just occurred to me that you would only have to delay queues when listing messages. claiming messages doesn't have that race condition problem | 22:27 |
| kgriffs | (since you don't provide a marker) | 22:28 |
| kgriffs | I'm really tempted to do a timestamp-ID patch for mongo driver and benchmark it | 22:28 |
| vkmc | yeah with claims you access with the id | 22:29 |
| vkmc | the problem is on accesing the collection | 22:29 |
| vkmc | right? | 22:29 |
| kgriffs | it's when you list messages - page through them with a marker | 22:30 |
| kgriffs | the problem occurs because you use a marker that would be based on the timestamp ID | 22:30 |
| * kgriffs is glad to tell someone all this in case he is hit by a bus tomorrow | 22:30 | |
| vkmc | haha | 22:30 |
| kgriffs | ROFL - https://twitter.com/flaper87/status/501490742955892737/photo/1 | 22:32 |
| vkmc | oh pls no | 22:32 |
| vkmc | lol | 22:32 |
| *** staphae has joined #openstack-zaqar | 22:40 | |
| staphae | authenticating with keystone? | 22:42 |
| *** malini has joined #openstack-zaqar | 22:44 | |
| *** kgriffs is now known as kgriffs|afk | 22:46 | |
| *** itisit has quit IRC | 22:51 | |
| malini | ,,,,,lop[ | 22:51 |
| malini | []=p-ko ./,;'\]] | 22:52 |
| *** malini has left #openstack-zaqar | 22:53 | |
| *** itisit has joined #openstack-zaqar | 22:54 | |
| *** sriram has joined #openstack-zaqar | 23:06 | |
| *** staphae has quit IRC | 23:09 | |
| *** amitgandhinz has joined #openstack-zaqar | 23:16 | |
| *** amitgandhinz has quit IRC | 23:20 | |
| *** itisit has quit IRC | 23:28 | |
| *** sriram has quit IRC | 23:47 | |
| *** itisit has joined #openstack-zaqar | 23:55 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!