Thursday, 2015-01-29

*** openstackgerrit has quit IRC00:21
*** openstackgerrit has joined #openstack-zaqar00:21
*** kgriffs is now known as kgriffs|afk00:35
*** achanda has quit IRC00:56
*** achanda has joined #openstack-zaqar01:00
*** openstackgerrit has quit IRC01:05
*** openstackgerrit has joined #openstack-zaqar01:05
*** amalagon_ has quit IRC01:19
*** amalagon has joined #openstack-zaqar02:06
*** achanda has quit IRC02:09
*** achanda has joined #openstack-zaqar02:10
*** openstackgerrit has quit IRC02:20
*** openstackgerrit has joined #openstack-zaqar02:21
*** wpf has quit IRC02:36
*** wpf has joined #openstack-zaqar02:37
*** openstackgerrit has quit IRC03:20
*** openstackgerrit has joined #openstack-zaqar03:20
*** achanda has quit IRC03:32
*** wpf has quit IRC03:43
*** achanda has joined #openstack-zaqar03:56
*** wpf has joined #openstack-zaqar04:00
*** achanda has quit IRC04:06
*** wpf has quit IRC04:10
*** boris-42 has quit IRC04:13
*** wpf has joined #openstack-zaqar04:23
*** flwang has quit IRC04:27
*** amalagon has quit IRC04:30
*** achanda has joined #openstack-zaqar04:45
*** achanda has quit IRC04:46
*** achanda has joined #openstack-zaqar04:57
*** echevemaster has quit IRC05:31
*** wpf has quit IRC06:19
*** wpf has joined #openstack-zaqar06:21
*** reed has quit IRC06:26
*** achanda has quit IRC06:53
*** achanda has joined #openstack-zaqar06:58
*** achanda has quit IRC07:19
*** dynarro has joined #openstack-zaqar08:12
*** dynarro has quit IRC09:01
*** dynarro has joined #openstack-zaqar09:07
*** flwang has joined #openstack-zaqar09:14
flwangflaper87: poing09:17
flaper87flwang: poing ?09:17
flaper87:P09:17
flaper87or I should say09:17
flaper87piong ?09:17
flwangflaper87: you're smart :)09:17
flwangare u available now?09:17
flaper87yes09:17
flwangas for your comments, https://review.openstack.org/#/c/142432/8/zaqar/transport/validation.py09:19
flwangthe TTL of subscription09:19
flaper87ah yeah09:20
flaper87shoot09:20
*** achanda has joined #openstack-zaqar09:21
flwanghttp://specs.openstack.org/openstack/zaqar-specs/specs/kilo/notification-api.html09:21
flwangI implemented TTL based on the spec09:21
flaper87yes, I'm now questioning the spec09:22
flaper87:P09:22
flwangflaper87: TBH, i was wondering it when I coded it09:23
flwangbut I saw SNS has the similar design, iirc09:23
flwanglet me check09:23
flaper87flwang: nice, that means probably the right thing to do is either remove ttl or make it infinite by default09:23
flaper87flwang: I think they don't have ttl09:23
flaper87or at least it's not the default09:23
flaper87flwang: brb09:26
flwangflaper87: checking sns doc...09:26
flaper87back09:34
flaper87flwang: anything interesting?09:34
flwangflaper87: nope, seeems my memory is not correct09:36
flwangarn:aws:sns:us-west-2:429486775985:alarmsqsarn:aws:sqs:us-west-2:429486775985:beijingarn:aws:sns:us-west-2:429486775985:alarm:77be0caf-126d-43a7-ab71-229f4e9f652309:36
flwangTopic    Protocol    Endpoint      Subscription ID09:36
flwangthis is the attributes of SNS's subscription09:37
flaper87right, no ttl09:38
flaper87so, I think ttl is a nice thing to have09:38
flaper87but the default should be no ttl09:38
flwangflaper87: agree09:39
flwangI will drop another patch to fix that09:39
flaper87ok09:39
flaper87I think you could fix the API in this one and then fix the storage later09:40
flwangflaper87: yep, that's what I want to do09:40
flwangflaper87: any status about the graduation?09:42
flaper87no graduation needed09:44
flaper87Zaqar is now an official openstack project09:44
flaper87since we all live under the same big tent now09:44
flaper87I'm waiting for the tags thing to be resolved so we can apply to that09:44
flaper87I gotta remove all the "incubated" tags now from everywhere09:44
flwangflaper87: sorry ?09:48
flwangflaper87: so there is no graduation thing in OpenStack, is it?09:48
*** exploreshaifali has joined #openstack-zaqar09:48
flaper87flwang: nope, no more graduations09:48
flwangis there an official announcement from TC?09:49
flwangI didn't see that from the mail list09:49
flaper87yeah, there's an active discussion on the tags09:52
flaper87plus all the governance reviews for the big tent thing09:52
flaper87but I think the final official announcement hasn't been made yet09:53
flwangflaper87: ok, so we won't be care about the incubated/graduation stuff anymore, right?09:54
flaper87flwang: exactly09:56
flaper87we're officially in09:56
flwangcool09:56
flaper87now we need to get people to use Zaqar :/09:56
*** dynarro has quit IRC09:58
flwangflaper87: as for the subscriber driver, any thoughts about where to put those code?10:21
flwangflaper87: how about like zaqar/common/subscriber/webhook.py?10:22
flaper87mmh, is it something that should go in common ?10:23
flaper87what about under notifications ?10:23
flaper87or a "notifiers" package10:23
*** exploreshaifali has quit IRC10:28
flwangflaper87: did you mean create a new notifications package under zaqar? at the same level with transport, storage?10:30
flaper87flwang: yup, probably notifiers10:33
flaper87mmh10:33
flaper87I'm really bad at naming things10:33
flaper87so :P10:33
flwangflaper87: see https://github.com/openstack/zaqar/tree/master/zaqar/common10:34
flwangwe're putting transport, storage stuff under common10:34
flwangand you know, in glance10:34
flwangwe also put the async task driver and those scripts under common10:35
flaper87yeah but those are helpers10:35
flwangflaper87: yep10:35
flaper87what you're working on now are the actual drivers that will push stuff to clients10:35
flwangi can see your point10:35
flaper87and they won't be used throughout zaqar but just the notification feature10:35
flwangflaper87: so how about just keep a new subscriber package under zaqar?10:36
flwangjust like storage/transport10:36
flwangand we can refactor it in the future if it's necessary10:36
flaper87flwang: sounds good10:37
*** exploreshaifali has joined #openstack-zaqar10:39
flwangcool10:40
flaper87flwang: can I get some reviews from you here? https://review.openstack.org/#/q/status:open+project:openstack/zaqar,n,z10:43
flaper87we need to move some patches forward10:43
flwangflaper87: it's on my todo list :)10:43
flwangI will to that tonight10:43
flwangto/do10:43
flaper87flwang: thank you very much10:44
flwangflaper87: sorry, i have a lot of internal work to do recently10:44
flaper87flwang: no worries at all10:45
flaper87that's why I'm here, to make sure you get fired for taking care of things you probably shouldn't10:45
flaper87:)10:46
openstackgerritFlavio Percoco proposed openstack/zaqar: Use pool's capabilities for flavors in v2  https://review.openstack.org/14856610:46
flwang:P10:46
flaper87exploreshaifali: looking at your patch in more depth now10:47
*** exploreshaifali has quit IRC11:08
*** achanda has quit IRC11:22
*** exploreshaifali has joined #openstack-zaqar11:23
exploreshaifaliflaper87: Thanks for that :)11:24
vkmcmorniiiiiiiiing11:36
flwangflaper87: still around?11:39
exploreshaifalivkmc: morning!!!!!!!1111:40
vkmchow are you all?11:43
vkmchey flwang!11:43
flwangvkmc: morning11:43
vkmcflwang, +1 to the subscription package11:45
flwangthanks :)11:46
vkmcalthough12:06
vkmcregarding the level12:06
vkmcsubscriptions could be considered as part of transport12:06
*** exploreshaifali has quit IRC12:13
*** dynarro has joined #openstack-zaqar12:15
*** dynarro has quit IRC12:20
flaper87flwang: I'm now12:26
*** exploreshaifal has joined #openstack-zaqar12:32
exploreshaifalvkmc: if you have a bit of time please can we disucss about the issue of message_controller which I am facing right now ?12:33
vkmcexploreshaifal, sure, could you please update me with the current situation?12:36
exploreshaifalvkmc: yes, current task is moving QueueController storage cofigurations into Control plane from data plane12:37
exploreshaifalthat is basicallly shifting queue_controller method https://github.com/openstack/zaqar/blob/master/zaqar/storage/base.py#L213 from DataDriver to ControllerDriver12:39
exploreshaifalokay12:39
exploreshaifalso what I did is rather than shifting queue_controller method, I changed it defination12:39
exploreshaifalsuch that it will access QueueController from ControlDriver12:40
vkmcexploreshaifal, cool, I follow that12:40
exploreshaifalso now QueueController is instantiated througn ControlDriver and not through DataDriver12:40
exploreshaifalvkmc: here is my current patch https://review.openstack.org/#/c/144340/12:41
exploreshaifalsee storage/mongodb/driver.py file12:41
exploreshaifalvkmc: so now the issue is that ControlDriver is instantiating QueueController, ControlDriver does not have message_controller12:43
exploreshaifalbut few method of QueueController need message_controller12:43
exploreshaifallike https://github.com/openstack/zaqar/blob/master/zaqar/storage/mongodb/queues.py#L28112:43
exploreshaifalpreviously it was working well because DataDriver was instantialing QueueController, and DataDriver have messageController12:44
exploreshaifalnow we can't add message_controller to ControlDriver becaues it won't be correct12:44
exploreshaifalso how to make this work?12:44
vkmcohhhh I see12:45
vkmcso definitely we will have to change a little the design12:45
vkmcI mean, we changed our POV about the place queues have in Zaqar12:45
vkmcso we should also change how we manage other entities in Zaqar12:45
flaper87vkmc: any thoughts on that?12:46
flaper87this one be an easy thing to model12:46
flaper87FWIW, I really hate that the queue controller depends on the message_controller12:46
vkmcyeah12:46
flaper87we could probably move stats down there12:46
vkmcI think we should remove that dependency12:46
vkmcthat is what we wanted to do in the first place12:46
vkmcso... for what I see, delete and stats use it12:46
flaper87redis uses it in other places12:47
flaper87IIRC12:47
vkmcoh12:47
vkmclet's check redis12:47
vkmcno.. better12:48
vkmclets focus on mongo first12:48
vkmcone thing we should figure it out is how we make this change in a restful way12:50
flaper87well, not much will change from what we have12:50
flaper87I mean, if you delete a queue, the existing behavior should remain12:50
flaper87the queue and its messages should be deleted12:50
flaper87I think we should have some event based pipeline12:51
vkmcyeah, I was thinking on... what happens if we delegate the remove messages to the message controller directly12:51
flaper87Right now, we've a pipeline implementation that calls a method for each object in the pipeline12:51
flaper87vkmc: mmh, not sure I follow. The only reason a queue would call message deletion is because it's to drop the queue12:52
vkmcflaper87, in our lazy way of viewing things, a queue exist only if it has messages12:53
flaper87right12:53
vkmcso if you delete all the messages == you dropped the queue12:53
flaper87unless the queue has been created explicitly12:54
flaper87as in, there's a metadata12:54
vkmchmm12:54
vkmcgood point12:54
vkmcso, we depend on  the queue controller knowing the message controller12:55
vkmcbecause, in the case of a deletion, if the queue is empty and only has metadata... ok, just drop it... but if it has messages, then it has to delete every message there12:56
vkmcand that is something the message controller does12:56
vkmcand that is because we are considering the queue as a container of message, not as an attribute only12:56
vkmcflaper87, what is your plan with the pipeline?12:57
exploreshaifalcan we shift the delete and stats to message_controller12:58
exploreshaifalbrb in 10 min13:00
*** exploreshaifal has quit IRC13:04
flaper87vkmc: sooooooooooooooooooooo13:11
flaper87this is just a fresh thought13:11
flaper87We've been (IIRC) talking about *always* using a pipeline13:12
flaper87this would be useful for the notifications thing13:12
flaper87so, what if we would use this same pipeline for something more than that13:13
* flaper87 working out the idea as he writes13:13
* vkmc tries to follow flaper87's train of thoughts13:13
flaper87What if we add the message controller to the queue's pipeline so it's called whenever a queue is deleted13:14
flaper87That means we need to rename the `delete` method to `delete_queue` or something like that13:14
* flaper87 checks the code13:14
vkmchmm13:15
flaper87so, yeah, we could do something really nasty here13:16
flaper87We can creare a MessageQueuecontroller13:16
flaper87that has a `_delete` method with the same signature as the QueueController13:16
flaper87and we'd register it in the queue's pipeline13:16
flaper87if only I remembered how the pipeline works13:17
flaper872s, checking code again13:17
flaper87so yeah13:20
flaper87I think we've most of what we need to do that13:21
flaper87the pipeline loading process may need some polishment13:21
flaper87vkmc: ^ thoughts ?13:22
vkmcflaper87, I'm thinking what is the advantage of doing that13:24
vkmcinstead of having the dataplane and controlplane tightly couple (aka keeping references to the controllers we need)13:24
flaper87It'll be really hard to keep the references because now we have a circular dependency13:25
flaper87FWIW, the data driver will still have a reference to the control driver13:26
flaper87but not the other way around13:26
flaper87and the other benefit is that when changes like this one need to happen, it's easier to do them13:26
*** exploreshaifali has joined #openstack-zaqar13:27
* vkmc checks the pipeline code13:28
*** boris-42 has joined #openstack-zaqar13:31
flaper87flwang: 'x<=0' <- SERIOUSLY ? >.>13:33
flaper87:P13:33
flaper87let me add the docstring13:33
vkmcok so13:34
vkmcwhen the enqueing in the pipeline ocurrs?13:34
vkmcwe are using the pipeline in the storage layer13:34
flaper87vkmc: but we're using it for pooling13:35
flaper87not for the general case13:35
flaper87the pipeline loads entry_points when creating instances for the many controllers13:35
flaper87if we add an entry point for the queue_pipeline that delets messages, then we should be good to go13:35
vkmcflaper87, I'm checking out this pipeline over here https://github.com/openstack/zaqar/blob/master/zaqar/storage/pipeline.py13:38
openstackgerritFlavio Percoco proposed openstack/zaqar: Forbid adding stores with mismatching capabilities  https://review.openstack.org/14578313:38
openstackgerritFlavio Percoco proposed openstack/zaqar: Use pool's capabilities for flavors in v2  https://review.openstack.org/14856613:38
openstackgerritFlavio Percoco proposed openstack/zaqar: Split MongoDB's default implementation from FIFO's  https://review.openstack.org/14829713:38
flaper87vkmc: flwang ^13:38
flaper87vkmc: yup13:38
flaper87vkmc: https://github.com/openstack/zaqar/blob/master/zaqar/storage/pipeline.py#L44-L8113:39
flaper87we should probably move that to `common/pipeline.py`13:39
flaper87so we can use it in other places13:39
flaper87That said13:39
flaper87vkmc: https://github.com/openstack/zaqar/blob/master/zaqar/storage/pipeline.py#L7113:39
flaper87vkmc: https://github.com/openstack/zaqar/blob/master/zaqar/storage/pipeline.py#L11013:39
flaper87See how the pipeline for queue's is loaded?13:39
flaper87at the very end of the "dynamic" pipeline, the queue_controller is appended13:40
flaper87vkmc: https://github.com/openstack/zaqar/blob/master/zaqar/storage/pipeline.py#L11113:40
vkmcflaper87, yeah13:40
flaper87if we add entrypoints for other things we want to call on each pipeline13:40
vkmcflaper87, so... what I was trying to say, we are not using the pipe for pooling only13:40
flaper87vkmc: oh really?13:40
flaper87NIIIIIIIIIICE13:41
* vkmc tries to detect sarcarsm13:41
flaper87https://github.com/openstack/zaqar/blob/master/zaqar/bootstrap.py#L96-L10813:41
flaper87no no, I really didn't know that13:41
vkmcaahahahaha13:41
flaper87I had forgotten we weren't using it just for pooling13:41
flaper87that makes this even easier13:41
vkmcyeah, that's the point13:41
vkmcbut but ... we add the message controller endpoint in the pipeline13:42
vkmcalready13:42
vkmcso adding the delete method should be enough *I think*13:42
flaper87nope because that's the *message* pipeline13:42
flaper87we need to add a "message controller" to the queue's pipeline13:42
vkmcwe use the words message and queue too much13:46
* flaper87 reminds vkmc we're working on a messaging system that happens to have queues13:47
flaper87ok that was a joke13:48
flaper87I couldn't hold it back13:48
flaper87:P13:48
vkmcaahahaha I know but13:48
vkmcre read this sentence 'nope because that's the *message* pipeline -  we need to add a "message controller" to the queue's pipeline'13:49
vkmcit sounded like this in my brain 'thats the message we need to add the message to the queue'13:49
flaper87vkmc: if you're confused, that was the goal13:50
flaper87jokes apart13:50
vkmcok, seriously... in bootstrap.py we are returning a pipeline with the DataDriver https://github.com/openstack/zaqar/blob/master/zaqar/bootstrap.py#L10813:50
flaper87We need to create in mongodb.messages a `QueueMessageHandlerThingy`13:50
flaper87then we need to add that to the entry-points and let the pipeline do its job13:50
vkmcwe want to do something similar for control()13:51
vkmcok exploreshaifali, seems like there is some consensus on what to do13:54
exploreshaifalivkmc, yeah, I am following whole discussion here13:54
vkmcflaper87, in the case of stats... we could proceed the same way?13:55
flaper87vkmc: I'd assume so, yes.13:56
flaper87not sure how we can "merge" the results, though.13:56
flaper87Unlike deletes, for stats we need to unify the results of each call13:56
flaper87I think this should be hanlded by the transport itself13:57
flaper87not sure13:57
vkmcstats are generated on the fly so... we could delegate that to the transport13:57
vkmcexploreshaifali, where is the etherpad you were working on? let's write some notes there14:02
exploreshaifalivkmc, https://etherpad.openstack.org/p/exploreshaifali-opw-split-layers14:03
*** sriram has joined #openstack-zaqar14:13
*** exploreshaifali has quit IRC14:32
*** mpanetta has joined #openstack-zaqar14:33
*** jasondot_ has joined #openstack-zaqar14:34
*** jasondot_ has quit IRC14:45
openstackgerritMerged openstack/zaqar: Refactor Api classes to enhance consistency  https://review.openstack.org/14826515:00
*** kgriffs|afk is now known as kgriffs15:04
*** amitgandhinz has joined #openstack-zaqar15:26
*** achanda has joined #openstack-zaqar15:31
*** achanda has quit IRC15:41
*** diga has joined #openstack-zaqar15:42
*** diga has quit IRC16:09
*** amalagon has joined #openstack-zaqar16:16
*** ametts has joined #openstack-zaqar16:40
*** exploreshaifali has joined #openstack-zaqar16:48
*** cpallares has joined #openstack-zaqar17:23
exploreshaifalivkmc, around?17:30
*** bradjones has quit IRC17:38
*** bradjones_ has joined #openstack-zaqar17:39
*** bradjones_ is now known as bradjones17:39
*** bradjones has joined #openstack-zaqar17:39
*** reed has joined #openstack-zaqar18:00
*** ff has joined #openstack-zaqar18:11
*** achanda has joined #openstack-zaqar18:29
*** exploreshaifali has quit IRC18:34
*** openstackgerrit has quit IRC18:50
*** openstackgerrit has joined #openstack-zaqar18:51
*** kgriffs is now known as kgriffs|afk18:53
*** kgriffs|afk is now known as kgriffs18:53
*** flwang has quit IRC18:56
*** achanda has quit IRC19:10
*** achanda has joined #openstack-zaqar19:18
*** reed has quit IRC19:27
*** amitgandhinz has quit IRC19:44
*** amitgandhinz has joined #openstack-zaqar19:45
*** amitgandhinz has quit IRC19:52
*** amitgandhinz has joined #openstack-zaqar19:53
*** flwang has joined #openstack-zaqar19:56
* vkmc lurks19:58
*** openstackgerrit has quit IRC20:04
*** openstackgerrit has joined #openstack-zaqar20:05
*** kgriffs is now known as kgriffs|afk20:05
*** amitgandhinz has quit IRC20:07
*** exploreshaifali has joined #openstack-zaqar20:14
*** fifieldt has quit IRC20:24
*** fifieldt has joined #openstack-zaqar20:24
*** achanda has quit IRC20:55
*** miqui has quit IRC21:18
*** achanda has joined #openstack-zaqar21:19
*** miqui has joined #openstack-zaqar21:21
flwangflaper87: ping21:31
*** flwang has quit IRC21:41
*** flwang has joined #openstack-zaqar21:41
*** achanda has quit IRC22:03
*** kgriffs|afk is now known as kgriffs22:04
*** kgriffs is now known as kgriffs|afk22:10
*** achanda has joined #openstack-zaqar22:14
*** achanda has quit IRC22:14
ffflwang: pong22:15
*** ff has quit IRC22:15
flaper87flwang: pong22:16
*** kgriffs|afk is now known as kgriffs22:17
*** achanda has joined #openstack-zaqar22:17
*** sriram has quit IRC22:20
flwangflaper87: as for your comments https://review.openstack.org/#/c/142432/8/zaqar/tests/unit/transport/wsgi/v2_0/test_subscriptions.py22:23
flwangabout the db uri22:23
flaper87flwang: sup ?22:26
flwangnow messages test is used the same way like I did22:26
flwangand there is any other test using pool for test(pls correct me)22:26
flwangand I also tried use sqlite memory db, but it will fail22:27
*** mpanetta has quit IRC22:27
flwangI haven't find the root cause22:27
flaper87np, what I meant to say there is that we should probably have a generic way to determine what db should be loaded22:28
flaper87That test exists just in test_mongo_impl22:28
flaper87I mean, it's subclassed there22:28
*** kgriffs is now known as kgriffs|afk22:34
flwangflaper87: https://github.com/openstack/zaqar/blob/master/zaqar/tests/unit/transport/wsgi/v2_0/test_messages.py#L4222:44
flwangI think you're talking about two points. #1 is we shouldn't allow duplicated db uri for pools22:45
flaper87yeah, forget about the duplicate uris for now22:45
flwang#2 is remove the mongodb specific stuff22:46
flaper87mmh, but that test isn't a base test22:46
flaper87mhh, I'd like to remove mongodb specific things22:46
flaper87if we can move that to base.V2Base22:46
flaper87and make loading a db uri dynamic, it'd be better22:47
flaper87can you file a bug for that, at least ?22:47
flaper87and put a comment there22:47
flwangflaper87: sure22:47
flwangflaper87: i just want to point out the db uri can't be replace with sqlite memory db at here22:48
flwangflaper87: we may need another patch to address that22:48
flaper87yeah, we'll fix this in a follow up patch22:49
flaper87vkmc: will do it22:49
flaper87:P22:49
flwangflaper87: haha22:49
flwangflaper87: I will file 2 bugs to address above issues separately22:50
*** openstackgerrit has quit IRC22:51
*** openstackgerrit has joined #openstack-zaqar22:51
flwangflaper87: as for the notification sending, last time you suggested add a post_message method in the subscriber driver, right?22:53
flaper87flwang: yeah, well. it should have the same name and signature as the message controller `post` method22:54
flwangflaper87: but I'm still not very clear when to trigger the notification sending22:54
flwangflaper87: i mean22:54
flwangwhen user post a message to a queue, then transport layer will get the message at https://github.com/openstack/zaqar/blob/master/zaqar/transport/wsgi/v2_0/messages.py#L14822:55
flwangthen the it will call the post method of message controller22:56
flaper87flwang: right but, what the transport actually does is call the *pipeline* which will the call the message_controller22:57
flaper87we want to put the subscribers controller in there so we can catch the `post_message` call and send the notification22:58
flaper87does that make sense?22:58
flwangflaper87: so, you mean we need to add subscriber to the pipeline of message controller, is it?22:59
flaper87yes22:59
flaper87:)22:59
flaper87somehow, in a nice way :D22:59
flwangok, it may works, but just a little big ugly from the  semantics view :D23:00
flwangthanks for your comment, Fla23:03
flaper87flwang: np :)23:03
*** cpallares has quit IRC23:18
*** echevemaster has joined #openstack-zaqar23:30
*** exploreshaifali has quit IRC23:37
*** ametts has quit IRC23:39

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!