| *** malini1 has joined #openstack-zaqar | 00:08 | |
| *** achanda has quit IRC | 00:08 | |
| *** malini has quit IRC | 00:09 | |
| *** achanda has joined #openstack-zaqar | 00:26 | |
| *** achanda has quit IRC | 00:37 | |
| *** kgriffs is now known as kgriffs|afk | 00:44 | |
| *** davideagnello has quit IRC | 00:50 | |
| *** achanda has joined #openstack-zaqar | 00:57 | |
| *** davideagnello has joined #openstack-zaqar | 01:05 | |
| *** amalagon has joined #openstack-zaqar | 01:06 | |
| *** achanda has quit IRC | 01:22 | |
| *** achanda has joined #openstack-zaqar | 01:31 | |
| *** achanda has quit IRC | 01:35 | |
| *** X019 has joined #openstack-zaqar | 01:41 | |
| *** malini1 has left #openstack-zaqar | 01:42 | |
| *** kgriffs|afk is now known as kgriffs | 01:45 | |
| *** X019 has quit IRC | 01:52 | |
| *** kgriffs is now known as kgriffs|afk | 01:56 | |
| *** vipul has quit IRC | 02:33 | |
| *** vipul has joined #openstack-zaqar | 02:37 | |
| *** kgriffs|afk is now known as kgriffs | 02:59 | |
| *** fifieldt has joined #openstack-zaqar | 03:01 | |
| *** vkmc has quit IRC | 03:12 | |
| *** echevemaster has joined #openstack-zaqar | 03:48 | |
| *** sgotliv has joined #openstack-zaqar | 04:07 | |
| *** achanda has joined #openstack-zaqar | 04:12 | |
| *** sgotliv has quit IRC | 04:16 | |
| *** flwang1 has quit IRC | 04:27 | |
| *** achanda has quit IRC | 04:30 | |
| *** amalagon has quit IRC | 05:43 | |
| *** sgotliv has joined #openstack-zaqar | 06:03 | |
| *** amalagon has joined #openstack-zaqar | 06:07 | |
| *** echevemaster has quit IRC | 06:33 | |
| *** sgotliv has quit IRC | 06:50 | |
| *** sgotliv has joined #openstack-zaqar | 06:50 | |
| *** fifieldt has quit IRC | 07:16 | |
| *** exploreshaifali has joined #openstack-zaqar | 07:18 | |
| *** achanda has joined #openstack-zaqar | 07:31 | |
| *** achanda has quit IRC | 07:36 | |
| openstackgerrit | Flavio Percoco proposed openstack/zaqar-specs: Define a wire protocol for non-REST APIs https://review.openstack.org/122425 | 07:45 |
|---|---|---|
| flaper87 | flwang: pon | 07:49 |
| flaper87 | flwang: pong | 07:49 |
| openstackgerrit | Merged openstack/zaqar-specs: Define a wire protocol for non-REST APIs https://review.openstack.org/122425 | 07:51 |
| exploreshaifali | hey flaper87 goood morning :) | 07:54 |
| flaper87 | exploreshaifali: hey hey | 07:56 |
| flaper87 | exploreshaifali: how are you doing? | 07:56 |
| openstackgerrit | Merged openstack/zaqar-specs: Add a persistent transport alternative https://review.openstack.org/134567 | 07:56 |
| openstackgerrit | Merged openstack/zaqar-specs: Spec for notification service https://review.openstack.org/129192 | 07:56 |
| exploreshaifali | flaper87, trying to figure out reason for `AssertionError: True is not false` | 07:56 |
| exploreshaifali | occuring in latest patch :) | 07:57 |
| flaper87 | exploreshaifali: let me take a look | 07:59 |
| exploreshaifali | flaper87, https://review.openstack.org/#/c/134910/ | 07:59 |
| flaper87 | exploreshaifali: https://git.openstack.org/cgit/openstack/zaqar/tree/zaqar/queues/storage/utils.py#n46 | 08:04 |
| flaper87 | you need to fix that | 08:04 |
| flaper87 | FWIW, we're going to get rid of that function | 08:04 |
| flaper87 | but you'll have to fix it until we do | 08:05 |
| flaper87 | :( | 08:05 |
| exploreshaifali | flaper87, okay ! Thanks :) | 08:08 |
| exploreshaifali | flaper87, btw how you caught that the problem is with storage/utils.py ? | 08:08 |
| exploreshaifali | because the problem is in *tests/unit/common/storage/test_utils.py* that is why ? | 08:10 |
| flaper87 | exploreshaifali: because the error is in this line: self.assertFalse(utils.can_connect(uri, conf=self.conf)) | 08:11 |
| flaper87 | `can_connect` is defined in storage/utils.py | 08:11 |
| exploreshaifali | flaper87, all right, got it :) | 08:12 |
| exploreshaifali | flaper87, Thanks a looot !! | 08:12 |
| exploreshaifali | :) | 08:13 |
| flaper87 | exploreshaifali: hehe, np girl! I like the way the patch is moving. Great work. | 08:13 |
| exploreshaifali | flaper87, yo yo!!! | 08:14 |
| *** achanda has joined #openstack-zaqar | 08:33 | |
| *** achanda has quit IRC | 08:48 | |
| flaper87 | exploreshaifali: heeeeeey | 08:49 |
| flaper87 | :D | 08:49 |
| exploreshaifali | xD | 08:49 |
| *** flwang1 has joined #openstack-zaqar | 09:28 | |
| openstackgerrit | Shaifali Agrawal proposed openstack/zaqar: Split Control and Data planes of Storage layer https://review.openstack.org/134910 | 11:08 |
| *** X019 has joined #openstack-zaqar | 11:09 | |
| *** exploreshaifali has quit IRC | 11:20 | |
| *** malini has joined #openstack-zaqar | 11:30 | |
| *** vkmc has joined #openstack-zaqar | 11:34 | |
| vkmc | good morniing | 11:37 |
| kragniz | vkmc: morning! | 11:40 |
| kragniz | \o/ | 11:40 |
| vkmc | heeeeey kragniz \o/ | 11:40 |
| *** flwang1 has quit IRC | 11:56 | |
| *** malini1 has joined #openstack-zaqar | 12:12 | |
| *** malini has quit IRC | 12:14 | |
| *** X019 has quit IRC | 12:23 | |
| flaper87 | vkmc: goooooooood morning :) | 12:24 |
| vkmc | :) | 12:40 |
| * vkmc sees specs approved | 12:40 | |
| vkmc | time to code! | 12:41 |
| flaper87 | vkmc: yeah, they all landed | 12:42 |
| flaper87 | w000t | 12:42 |
| flaper87 | I gotta update mines and keep coding | 12:42 |
| vkmc | flaper87, what are you working on right now? :) | 12:43 |
| vkmc | I have to dig on the cross api spec, without that we don't have ws >.> | 12:43 |
| * flaper87 is working on making others work | 12:43 | |
| vkmc | that sounds like you are a pm | 12:45 |
| flaper87 | vkmc: OMFG, you just called me hipster-pm..... | 12:47 |
| flaper87 | that's insulting | 12:47 |
| flaper87 | :D | 12:47 |
| vkmc | flaper87, you are calling yourself that | 12:47 |
| vkmc | hipster... yes you are | 12:47 |
| flaper87 | no no no no no no | 12:48 |
| flaper87 | NOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOO | 12:48 |
| vkmc | hahaha | 12:49 |
| vkmc | but its cool fla, a lot of folks try hard to be hipster | 12:49 |
| vkmc | you are a natural one | 12:49 |
| vkmc | don't fight the mustache | 12:49 |
| vkmc | embrace the mustache | 12:49 |
| vkmc | HAHA | 12:49 |
| flaper87 | http://s.pikabu.ru/images/previews_comm/2013-04_4/13661252013664.jpg | 12:49 |
| vkmc | I need a tshirt with that | 12:49 |
| flaper87 | LOL | 12:49 |
| * flaper87 pictures a dude embracing a mustache | 12:50 | |
| vkmc | lool | 12:50 |
| vkmc | did you see this blog? http://www.artofmanliness.com/ | 12:50 |
| vkmc | ah internet <3 | 12:51 |
| kragniz | vkmc: haha | 12:56 |
| vkmc | kragniz, you knew it? :) haha | 12:56 |
| *** exploreshaifali has joined #openstack-zaqar | 13:04 | |
| exploreshaifali | hi guyz | 13:07 |
| exploreshaifali | vkmc, \o | 13:07 |
| *** X019 has joined #openstack-zaqar | 13:07 | |
| vkmc | hiiiiiiiii exploreshaifali o/ | 13:08 |
| vkmc | how are you today? | 13:08 |
| exploreshaifali | \o/ | 13:08 |
| exploreshaifali | vkmc, eating loots of chocolates ;) | 13:09 |
| exploreshaifali | 5 star at present | 13:10 |
| * vkmc looks exploreshaifali with crazy eyes | 13:10 | |
| vkmc | chocolates you said? | 13:10 |
| vkmc | nom nom | 13:11 |
| exploreshaifali | yes... vkmc :D | 13:11 |
| kragniz | I ate all the chocolate I was hiding under my desk :( | 13:11 |
| exploreshaifali | http://www.google.co.in/imgres?imgurl=&imgrefurl=https%3A%2F%2Fgypsyfoods.wordpress.com%2F2012%2F06%2F22%2Frecipe-29-5-star-cake%2F&h=0&w=0&tbnid=ZAIB-LXVXgCPLM&zoom=1&tbnh=194&tbnw=259&docid=t5UpQpve3ITYjM&tbm=isch&client=ubuntu&ei=jtF1VL6XLNfluQTN9YCgCg&ved=0CA0QsCUoAw | 13:12 |
| exploreshaifali | tooo big url.. didn't realized :P | 13:12 |
| vkmc | that looks so good | 13:12 |
| vkmc | np | 13:12 |
| exploreshaifali | kragniz, no worries we will have more | 13:13 |
| exploreshaifali | vkmc, how are you ? | 13:14 |
| exploreshaifali | and kragniz what about you ? | 13:14 |
| kragniz | exploreshaifali: I'm rather good! | 13:14 |
| vkmc | I'm doing good, reading some white papers and setting up RedStack | 13:14 |
| vkmc | and you? | 13:14 |
| flaper87 | kragniz: you didn't hid it well enough | 13:14 |
| kragniz | exploreshaifali: currently annoying flaper87 | 13:14 |
| flaper87 | A LOT | 13:15 |
| flaper87 | just FYI | 13:15 |
| flaper87 | :P | 13:15 |
| exploreshaifali | kragniz, hahaha yes on glance channel | 13:15 |
| kragniz | flaper87: FOR THE GREATER GOOD! | 13:15 |
| flaper87 | yeah, yeah.... who told you making users happy is the greater good | 13:15 |
| flaper87 | ??? | 13:15 |
| flaper87 | money is, once you sell this shit, you can run away. | 13:15 |
| flaper87 | :P | 13:15 |
| * flaper87 hides | 13:15 | |
| exploreshaifali | xD | 13:16 |
| flaper87 | jokes apart, I do agree with your and jokke's POV now | 13:16 |
| kragniz | haha | 13:16 |
| flaper87 | it makes sense | 13:16 |
| kragniz | flaper87: <3 | 13:16 |
| *** dynarro has joined #openstack-zaqar | 13:26 | |
| *** dynarro_ has joined #openstack-zaqar | 13:26 | |
| *** tmu has quit IRC | 13:41 | |
| *** tmu has joined #openstack-zaqar | 13:48 | |
| *** jchai has joined #openstack-zaqar | 13:52 | |
| vkmc | flaper87, can you update me on the status of https://blueprints.launchpad.net/zaqar/+spec/cross-transport-api-spec? | 13:59 |
| vkmc | when you have a moment | 13:59 |
| *** sriram has joined #openstack-zaqar | 14:04 | |
| *** sriram has joined #openstack-zaqar | 14:04 | |
| *** exploreshaifali has quit IRC | 14:15 | |
| *** mpanetta has joined #openstack-zaqar | 14:24 | |
| flaper87 | vkmc: mmh, you should have rights to do that | 14:27 |
| flaper87 | don't you have them? | 14:27 |
| vkmc | I didn't mean that | 14:27 |
| vkmc | I was asking you to give me details on the implementation of that | 14:28 |
| vkmc | I cannot find reviews or something else apart from an etherpad | 14:28 |
| flaper87 | vkmc: ah damn, sorry | 14:29 |
| flaper87 | :( | 14:29 |
| flaper87 | I misunderstood | 14:29 |
| flaper87 | ok | 14:29 |
| vkmc | flaper87, no worries :) | 14:29 |
| flaper87 | so, there's actually not much done | 14:30 |
| * flaper87 gets some links | 14:30 | |
| vkmc | first of all, do we really need that? | 14:30 |
| vkmc | the PoC is based on Zaqar as is | 14:30 |
| flaper87 | vkmc: you mean the cross-api thing? | 14:31 |
| vkmc | yep | 14:31 |
| flaper87 | vkmc: I think we do | 14:33 |
| flaper87 | the point of that bp is to create something like we have in the client but for the server | 14:33 |
| flaper87 | https://github.com/openstack/zaqar/blob/master/zaqar/queues/api/v1_1/request.py | 14:33 |
| flaper87 | just a way to get requests in a dict-like object, validate them and process them | 14:33 |
| flaper87 | this acts like as the validator https://github.com/openstack/zaqar/blob/master/zaqar/common/api.py | 14:34 |
| flaper87 | vkmc: does that make sense? | 14:35 |
| vkmc | flaper87, it does yeah | 14:36 |
| vkmc | which was the blockers you find on implementation? | 14:36 |
| vkmc | maybe cpallares could give me more details on that so I can (try to) carry on with it | 14:36 |
| vkmc | I'm reading this https://etherpad.openstack.org/p/cross-transport-api-spec | 14:37 |
| flaper87 | vkmc: there were no real blockers, TBH | 14:39 |
| flaper87 | I mean, the main blocker was with the original idea | 14:39 |
| flaper87 | the original idea was to replace *all* transports with this cross-api thing | 14:39 |
| flaper87 | but it'd have been aweful to migrate the wsgi transport | 14:39 |
| flaper87 | because we'd have had to duplicate exceptions raising etc | 14:39 |
| vkmc | k | 14:40 |
| vkmc | yeah | 14:40 |
| flaper87 | if we leave the wsgi transport as is, as a reference, we're good to make this more specific to lower level protocols | 14:40 |
| vkmc | the idea of the cross-api spec is to avoid duplication by setting a common api for all transports to communicate with the storage | 14:42 |
| vkmc | why you say that migrating wsgi would make us to duplicate code? | 14:42 |
| vkmc | I mean, I know it will be painful to do that | 14:42 |
| vkmc | but I'm not sure why duplication is going to happen | 14:43 |
| *** ametts has joined #openstack-zaqar | 14:44 | |
| flaper87 | vkmc: it;d make us duplicate exceptions | 14:46 |
| flaper87 | for example | 14:46 |
| flaper87 | in that cross-api thing we'll have to have "common" exceptions and we'd raise something like: exception.QueueDoesNotExist | 14:46 |
| flaper87 | however, we'll still have to have views in the wsgi transport that catch these exceptions and re-raise the proper HTTP exception | 14:47 |
| flaper87 | that catching-re-raising is actually part of the REST api definition | 14:47 |
| flaper87 | which means we wouldn't be able to completely rely on the cross-api thing | 14:47 |
| flaper87 | not for the wsgi transport | 14:47 |
| vkmc | makes sense | 14:47 |
| flaper87 | in the case of other transports, we can just serialize the exception | 14:47 |
| flaper87 | but it can be done in the cross-api implementation | 14:48 |
| vkmc | all right | 14:49 |
| vkmc | thanks fla :) | 14:49 |
| *** exploreshaifali has joined #openstack-zaqar | 15:19 | |
| *** sriram has quit IRC | 15:54 | |
| *** sriram has joined #openstack-zaqar | 15:54 | |
| *** cpallares has joined #openstack-zaqar | 16:16 | |
| openstackgerrit | Doraly Navarro proposed openstack/python-zaqarclient: Gets 'pool' data if the resourse exists https://review.openstack.org/137398 | 16:40 |
| openstackgerrit | Flavio Percoco proposed openstack/zaqar-specs: Improve storage capabilities https://review.openstack.org/126531 | 16:43 |
| flaper87 | vkmc: ^ | 16:43 |
| flaper87 | could you sanity check the Pool and Flavors section | 16:43 |
| flaper87 | I updated it based on our discussion at the meeting | 16:44 |
| *** amalagon has quit IRC | 16:44 | |
| openstackgerrit | Flavio Percoco proposed openstack/zaqar-specs: Make FIFO guarantee optional https://review.openstack.org/125986 | 16:45 |
| flaper87 | vkmc: and ^ | 16:46 |
| vkmc | :o | 16:51 |
| openstackgerrit | Doraly Navarro proposed openstack/python-zaqarclient: Gets 'pool' data if the resource exists https://review.openstack.org/137398 | 16:51 |
| vkmc | I'll review after lunch :) | 16:52 |
| flaper87 | vkmc: enjoy | 16:53 |
| *** echevemaster has joined #openstack-zaqar | 16:54 | |
| kgriffs | reviewed | 17:17 |
| * kgriffs goes back to lurking | 17:17 | |
| flaper87 | kgriffs: hey man :D | 17:34 |
| flaper87 | thank you | 17:34 |
| flaper87 | kgriffs: https://review.openstack.org/#/c/126531/3/specs/kilo/storage-capabilities.rst,cm | 17:34 |
| flaper87 | what do you mean with "What should happen when a driver specified for the flavor does not support the desired capabilities"? | 17:35 |
| flaper87 | The driver is not specified for the flavor | 17:35 |
| flaper87 | I mean, drivers are set per pool | 17:36 |
| kgriffs | oh, right | 17:36 |
| kgriffs | let me see... | 17:36 |
| kgriffs | so when I create a flavor, I associate it with one or more pools, right? | 17:36 |
| * kgriffs needs to go back and RTFM | 17:36 | |
| flaper87 | kgriffs: with 1 pool group | 17:37 |
| *** echevemaster has quit IRC | 17:37 | |
| flaper87 | kgriffs: one more question re DELETE_MSG and CLAIM_MSG | 17:37 |
| flaper87 | I didn't add thsoe because they have a direct impact on the API | 17:38 |
| flaper87 | What should the API do if the store doesn't support message deletion ? | 17:38 |
| flaper87 | no-op ? | 17:38 |
| flaper87 | How do we comunicate that back to the user? | 17:38 |
| flaper87 | etc | 17:38 |
| kgriffs | so, my thought was | 17:38 |
| flaper87 | Do you think that's something we should revisit later? or should we do it now? | 17:39 |
| kgriffs | first, you don't expose those endpoints in JSON-home (or swagger or whatever else we use now or in the future) | 17:39 |
| kgriffs | and second, the API layer returns a graceful error | 17:39 |
| kgriffs | how about this | 17:39 |
| *** jchai has quit IRC | 17:40 | |
| flaper87 | wait, how can you do that in advance? | 17:40 |
| flaper87 | that's a per-flavor thing | 17:40 |
| kgriffs | oh, good point | 17:41 |
| kgriffs | hmmm | 17:41 |
| kgriffs | well, one thing we said at the lunch meeting with Keith was that perhaps we don't need to bend over backwards to support this, but at a minimum the process shouldn't crash and burn with an unhandled or hard-to-diagnose error | 17:41 |
| kgriffs | so what I was thinking is... | 17:42 |
| kgriffs | is there an error that, say, a Kafka driver could raise when an operation is not supported that would be translated to a reasonable response by the API layer? | 17:42 |
| kgriffs | and that we could log to make it easy for an ops person to figure out what went wrong if someone sends in a support ticket? | 17:43 |
| flaper87 | kgriffs: I think we can come up with something | 17:43 |
| flaper87 | I'm more worried about the API than the storage layer | 17:43 |
| flaper87 | other than that, I think returning a nice error back to the user is fine | 17:44 |
| flaper87 | as long as the client is smart enough to make user's life simple | 17:44 |
| flaper87 | as in, Client(..., ..., **dict(ignore_interoperability=True)) | 17:44 |
| flaper87 | That'd make the client catch missing features and just move on | 17:45 |
| kgriffs | right | 17:45 |
| flaper87 | Queue('my-kafka-queue').claim() <- this won't raise any exception | 17:46 |
| flaper87 | or it probably should | 17:46 |
| flaper87 | I mean, if you're creating an instance of a kafka queue, you should probably know what's supported | 17:46 |
| kgriffs | hmm | 17:46 |
| *** sriram has quit IRC | 17:47 | |
| kgriffs | yeah, the trouble is the client can't know what is supported unless it can query the supported operations/endpoints for a given flavor | 17:47 |
| kgriffs | i think we have a couple stepping stones at least | 17:48 |
| flaper87 | if we raise a specific error for unsupported features, we can know whether the feature is supported or not | 17:48 |
| flaper87 | We could also have a per-queue home | 17:49 |
| flaper87 | :/ | 17:49 |
| kgriffs | yeah, that could be a first step | 17:49 |
| kgriffs | (raise a specific error) | 17:49 |
| kgriffs | second step would be to figure out a way to make what is supported discoverable | 17:49 |
| flaper87 | right | 17:49 |
| kgriffs | i mean, with capabilities you have that in some sense | 17:49 |
| flaper87 | yeah | 17:50 |
| kgriffs | but we haven't really been thinking of capabilities in terms of this operation is supported or not | 17:50 |
| flaper87 | right | 17:50 |
| flaper87 | ok, lets do that | 17:50 |
| *** dynarro has quit IRC | 17:50 | |
| *** dynarro_ has quit IRC | 17:50 | |
| flaper87 | I'll add claims as a capability, I think deletion can be a no-op for cases like kafka | 17:51 |
| kgriffs | if we want to be RESTful then the API needs to be self-describing somehow. so a second iteration on this would be to figure out how to do that | 17:51 |
| kgriffs | you do need a way to probe operations to see if they are supported | 17:51 |
| flaper87 | yup | 17:51 |
| kgriffs | whether that is just trying it and getting an error status code, or some kind of hypermedia approach | 17:51 |
| *** mpanetta has quit IRC | 17:53 | |
| flaper87 | kgriffs: btw, I think we have a no so nice bug in flavors | 17:54 |
| flaper87 | kgriffs: Queue's are lazy, right? | 17:54 |
| flaper87 | now | 17:54 |
| flaper87 | If I post a message to a queue that doesn't exist, a new queue will be created on the default pool. If I then decide to use a flavor for that queue and I set the flavor by updating queue's metadata, I'll change the storage for that queue | 17:55 |
| flaper87 | and messages posted on the other pool will be lost | 17:55 |
| *** amalagon has joined #openstack-zaqar | 17:55 | |
| openstackgerrit | Flavio Percoco proposed openstack/zaqar-specs: Improve storage capabilities https://review.openstack.org/126531 | 17:58 |
| flaper87 | kgriffs: ^ updated... Thanks for the reviews | 17:58 |
| *** amalagon has quit IRC | 18:00 | |
| *** sgotliv has quit IRC | 18:01 | |
| openstackgerrit | Merged openstack/zaqar-specs: Make FIFO guarantee optional https://review.openstack.org/125986 | 18:02 |
| *** exploreshaifali has quit IRC | 18:13 | |
| kgriffs | back | 18:35 |
| kgriffs | flaper87: btw, I would looking at the v1.1 api spec | 18:35 |
| kgriffs | I think the flavors docs could be improved. it doesn't say anything about a | 18:36 |
| kgriffs | pool group | 18:36 |
| kgriffs | am I understanding correctly? | 18:36 |
| kgriffs | flavor -> pool group -> N pools | 18:36 |
| openstackgerrit | Merged openstack/zaqar-specs: Improve storage capabilities https://review.openstack.org/126531 | 18:36 |
| kgriffs | flaper87: also, looking at the docs, it seems that the operator is supposed to set capabilities when PUTing a new flavor | 18:40 |
| kgriffs | so my question was, what happens if they set capabilities there that are not offered by the pool group? | 18:40 |
| kgriffs | alternatively, should a flavor's capabilities simply bubble up from what the underlying pool group can do rather than being set by the operator manually? | 18:41 |
| *** X019 has quit IRC | 18:41 | |
| * vkmc lurks | 18:56 | |
| vkmc | IIRC if the operator selects capabilities that are not offered by the pool group then an error should raise | 18:57 |
| vkmc | and we cannot afford the second one because there are some capabilities that may be offered by the pool group that doesn't fit the operator needs | 18:57 |
| vkmc | as in.. FIFO or not FIFO | 18:57 |
| *** reed has quit IRC | 19:02 | |
| kgriffs | suppose we allow a capability overlay/mask to be provided when defining a flavor | 19:03 |
| *** cpallares has quit IRC | 19:03 | |
| kgriffs | In this design, I can have multiple flavors that share the same pools behind the scenes, but perhaps expose them with different capabilities to the user. | 19:05 |
| *** ametts has quit IRC | 19:05 | |
| vkmc | that would be possible if flavors were associated to a queue/topic | 19:06 |
| vkmc | in this design | 19:06 |
| kgriffs | alternatively, I can have, say two pools | 19:07 |
| kgriffs | they are identical, with the same capabilities | 19:07 |
| kgriffs | then I create two flavors, one for each pool | 19:07 |
| kgriffs | but in one flavor, I specify different capabilities | 19:07 |
| kgriffs | in this case, I might as well just configure the pool in the first place with the capabilities I want to expose through the flavor | 19:08 |
| *** amalagon has joined #openstack-zaqar | 19:09 | |
| kgriffs | what are some other (potential) benefits of setting capabilities when PUTing a new flavor? | 19:10 |
| kgriffs | (how would this be used?) | 19:10 |
| vkmc | the thing is... you select the flavor according to the capabilities | 19:11 |
| vkmc | the idea of the capabilities happened because of flavors | 19:12 |
| vkmc | we needed a way to know if a flavor could be assigned to a pool | 19:12 |
| vkmc | in the use case you mentioned... you have two pools with the same capabilities | 19:13 |
| vkmc | you may want that one pool behaves in some way, different from the other | 19:14 |
| vkmc | or maybe you want both pools to behave the same way | 19:14 |
| kgriffs | it seems to me that capabilities should be owned by the pool then, not the flavor | 19:14 |
| vkmc | you adjust the flavors according to the use cases you want to cover | 19:14 |
| kgriffs | like, I configure a pool | 19:14 |
| kgriffs | I say I want it to have these capabilities | 19:14 |
| kgriffs | then when I add nodes to the pool, the service checks to see if the driver configs I am giving can support the capabilities for that pool | 19:15 |
| kgriffs | then when I add a flavor, i am really just saying "when someone wants a topic/queue, it has such and such a flavor, which is just a grouping of pools which some capabilities" | 19:15 |
| kgriffs | a flavor is just a way to say "use this particular group of pools" right? | 19:16 |
| kgriffs | and I can ask what capabilities that flavor has | 19:16 |
| kgriffs | therefore, it doesn't make sense to me for the operator to specify capabilities when creating a flavor. They tell the flavor what pools it maps to, and the capabilities are derived from that. | 19:17 |
| flaper87 | very quick because I've got to go | 19:17 |
| flaper87 | The idea is to disable the PUT on flavors | 19:18 |
| flaper87 | now that the storage exposes the capabilities | 19:18 |
| flaper87 | not sure if that makes sense or not | 19:18 |
| flaper87 | If we enable it, there won't be a way to prevent the operator to set random capabilities | 19:19 |
| kgriffs | yeah, that makes more sense. It wasn't clear from reading the spec. | 19:19 |
| kgriffs | but | 19:19 |
| flaper87 | my thinking is that the storage knows best about what it can/cannot do | 19:19 |
| kgriffs | what is a flavor then | 19:19 |
| kgriffs | is it a group of pools? | 19:19 |
| flaper87 | mmh, good question | 19:19 |
| kgriffs | a mapping that says, for messages using this topic/queue they should go to this group of pools? | 19:20 |
| flaper87 | it's not a group of pools | 19:20 |
| flaper87 | because falvors can have just 1 pool | 19:20 |
| flaper87 | Basically, flavors are user-facing pools | 19:20 |
| vkmc | flavors define the behaviour of a pool | 19:20 |
| flaper87 | a group of pools | 19:21 |
| flaper87 | * | 19:21 |
| flaper87 | but yeah | 19:21 |
| flaper87 | Flavors don't contain sensible information like pool's configs, etc | 19:21 |
| flaper87 | they just contain the pool group and capabilities | 19:21 |
| kgriffs | i thought you just said a flavor can have just one pool, not a group? | 19:21 |
| vkmc | <flaper87> because falvors can have just 1 pool <- | 19:21 |
| vkmc | maybe he meant just one group | 19:22 |
| flaper87 | vkmc: sorry, just 1 group of pools | 19:22 |
| kgriffs | lol | 19:22 |
| vkmc | cool | 19:22 |
| vkmc | :) | 19:22 |
| flaper87 | :D | 19:22 |
| kgriffs | ok, so is that what we are saying a flavor is now | 19:22 |
| flaper87 | sorry about that | 19:22 |
| kgriffs | a pool group | 19:22 |
| flaper87 | That's my point | 19:22 |
| vkmc | no problem dud | 19:22 |
| flaper87 | the flavor doesn't group the pools | 19:22 |
| flaper87 | The pools already have a group | 19:22 |
| flaper87 | the flavor just exposes the group to the user | 19:22 |
| *** reed has joined #openstack-zaqar | 19:23 | |
| vkmc | flavors just say... ok, this group here will be able to do this and that | 19:23 |
| kgriffs | ok | 19:23 |
| kgriffs | so let me see if I understand | 19:23 |
| flaper87 | Ultimately, I think flavors will do more than that | 19:23 |
| flaper87 | sorry, got to go | 19:23 |
| flaper87 | bbib | 19:23 |
| flaper87 | :( | 19:23 |
| kgriffs | ok | 19:23 |
| kgriffs | I am out the next two days | 19:23 |
| vkmc | ttfn flaper87! | 19:23 |
| kgriffs | so we may have to pick this up on monday | 19:23 |
| kgriffs | once we sort this out, we should make some pretty pictures and a nice wiki page | 19:24 |
| vkmc | yeah :) | 19:24 |
| kgriffs | better yet, a nice addition to the operator docs | 19:24 |
| kgriffs | ok, so here is where I am right now. we can pick up the conversation when flaper87 returns | 19:24 |
| kgriffs | suppose I am an operator | 19:24 |
| vkmc | sure thing | 19:24 |
| kgriffs | i want to deploy Zaqar for much win | 19:24 |
| kgriffs | so I set up, say, 3 pools | 19:25 |
| kgriffs | 2 pools are my "high-throughput" pools | 19:25 |
| kgriffs | I configure those with shorter max TTL and turn off FIFO/AOD | 19:25 |
| kgriffs | then I add 1 more pool | 19:27 |
| kgriffs | this is my "durable" pool | 19:27 |
| kgriffs | I let people keep things around longer, use larger message sizes, enable FIFO | 19:28 |
| kgriffs | next, I suppose I would create two pool groups | 19:28 |
| kgriffs | group A and group B | 19:28 |
| kgriffs | put the first two pools in A, the other in B | 19:28 |
| kgriffs | now, I want to expose this to my users | 19:28 |
| kgriffs | rather than the service simply exposing those groups directly, I have to create flavors that map 1:1 to group A and B? | 19:29 |
| kgriffs | so, my (devil's advocate) question: why make me go through the extra layer of indirection? | 19:29 |
| kgriffs | (brb - need to grab food) | 19:30 |
| kgriffs | (before the food truck leaves) | 19:30 |
| vkmc | go go go :) | 19:31 |
| *** kgriffs is now known as kgriffs|afk | 19:31 | |
| * vkmc thinks about Kurt's case | 19:32 | |
| *** exploreshaifali has joined #openstack-zaqar | 19:37 | |
| *** sgotliv has joined #openstack-zaqar | 19:39 | |
| *** amalagon has quit IRC | 19:47 | |
| *** kgriffs|afk is now known as kgriffs | 19:51 | |
| exploreshaifali | flaper87, around ? | 19:55 |
| flaper87 | kgriffs: around? | 20:12 |
| flaper87 | exploreshaifali: yeah | 20:12 |
| kgriffs | here | 20:12 |
| flaper87 | kgriffs: I'd like to clarify this now to merge the spec | 20:12 |
| flaper87 | :D | 20:12 |
| exploreshaifali | flaper87, I am suppose to change https://github.com/openstack/zaqar/blob/master/zaqar/queues/storage/utils.py#L46 | 20:12 |
| exploreshaifali | but that will affect other data stored too | 20:13 |
| exploreshaifali | redis and sql-alchemy | 20:13 |
| flaper87 | exploreshaifali: right | 20:13 |
| exploreshaifali | so should I modify options.py and drivers and rest test cases for two storage also | 20:13 |
| flaper87 | ah mmh | 20:13 |
| flaper87 | yeah, you may need to do that in this patch too | 20:13 |
| flaper87 | :/ | 20:14 |
| exploreshaifali | or any other option is there by which we can use if else statement | 20:14 |
| exploreshaifali | to identify | 20:14 |
| exploreshaifali | if it is mongodb or not | 20:14 |
| flaper87 | exploreshaifali: it'd be better to just do everything now | 20:14 |
| flaper87 | it's clearer | 20:14 |
| flaper87 | I think | 20:14 |
| flaper87 | kgriffs: so, flavors | 20:14 |
| kgriffs | yeah | 20:14 |
| flaper87 | kgriffs: we should disable PUT there | 20:14 |
| flaper87 | to get capabilities from pools | 20:15 |
| flaper87 | then your question | 20:15 |
| kgriffs | ok, that makes sense | 20:15 |
| flaper87 | what are flavors then ? | 20:15 |
| kgriffs | right | 20:15 |
| kgriffs | you saw my thought experiment above? | 20:15 |
| flaper87 | ah mmh, lemme read | 20:15 |
| kgriffs | just trying to think of this from a user perspective | 20:15 |
| vkmc | in reply to your thoughts kgriffs | 20:17 |
| flaper87 | the extra level of indirection is because you don't want your users to see your pools url, etc | 20:17 |
| flaper87 | also | 20:17 |
| vkmc | you have the three pools, but you have to define which one will be the faster and which one will be the persistent | 20:17 |
| flaper87 | In the pools you have extra things you can do | 20:17 |
| flaper87 | for example, a pool group could be a per-region cluster of nodes | 20:17 |
| flaper87 | Say you have your us-1-pool and emea-1-pool group | 20:18 |
| flaper87 | You don't want users to know that | 20:18 |
| flaper87 | So you put an envelop on top of your group and say: This is your flavor high-throughput and those are its capabilities | 20:18 |
| kgriffs | ok, so it is really a way to give a friendly name to your group? | 20:18 |
| flaper87 | We originally created flavors to let ops set the capabilities, TBH | 20:19 |
| flaper87 | the other things I just said are based on some thoughts I had while working on flavors | 20:19 |
| flaper87 | think about pools like you'd think about nova's compute nodes | 20:21 |
| flaper87 | you've flavors on top of those explaining what kind of compute you need | 20:21 |
| flaper87 | but the you've your compute nodes registered | 20:21 |
| flaper87 | and compute hosts are balanced by the scheduler and whatnot | 20:21 |
| flaper87 | Lets imagine we delete flavors | 20:22 |
| flaper87 | The pool group field would become what the flavor name is now | 20:23 |
| flaper87 | and capabilities would be picked automagically from the stores | 20:23 |
| flaper87 | If we'd like to migrate a queue from one group to another, we'd need to change the pool group in the queue metadata | 20:24 |
| flaper87 | kgriffs: ^ | 20:26 |
| flaper87 | vkmc: ^ | 20:26 |
| kgriffs | do you think that there will always be a 1:1 ratio between flavors and groups? | 20:26 |
| flaper87 | I've been thinking a bit about that, I think at one point we'll want to have more groups per flavor | 20:27 |
| flaper87 | I'm not sure | 20:27 |
| flaper87 | really | 20:27 |
| kgriffs | ok | 20:28 |
| kgriffs | the reason i ask | 20:28 |
| kgriffs | is that one way we could think about flavors is like an encapsulation of the "human" attributes of a group | 20:29 |
| kgriffs | meaning, a group is a thing | 20:29 |
| kgriffs | and that group has a flavor document | 20:29 |
| kgriffs | that flavor document has a human-friendly name, description, etc. - stuff that is useful for horizon to display, or a CLI tool for example. | 20:29 |
| flaper87 | kgriffs: right | 20:30 |
| *** sgotliv has quit IRC | 20:30 | |
| vkmc | that is metadata | 20:32 |
| vkmc | flavors describe what a group can or cannot do | 20:32 |
| kgriffs | mmm | 20:32 |
| kgriffs | vkmc: that conflates capabilities with flavors | 20:32 |
| kgriffs | actually | 20:33 |
| vkmc | capabilities is what the group can do | 20:33 |
| kgriffs | it is sort of the human-readable description of capabilities | 20:33 |
| vkmc | flavors is what the operators want the group to do | 20:33 |
| kgriffs | well, kind of | 20:33 |
| kgriffs | I see it as a way for the operator to describe two things | 20:33 |
| kgriffs | actually, three | 20:33 |
| kgriffs | first, give a group a friendly name, like "high-throughput" | 20:34 |
| flaper87 | kgriffs: https://soundcloud.com/kaleidamusic?utm_source=soundcloud&utm_campaign=share&utm_medium=twitter <- you should totally listen to "Archive" | 20:34 |
| *** flwang1 has joined #openstack-zaqar | 20:34 | |
| vkmc | flaper87, recommend me another band, different from Archive | 20:34 |
| kgriffs | second, explain some things that can't really be expressed programmatically with capabilities | 20:34 |
| kgriffs | like "this flavor is on our super-awesome Redis boxes that have been tuned for maximum magic" | 20:35 |
| vkmc | much magic, wow | 20:35 |
| kgriffs | third, maybe put a prose description there about the capabilities | 20:35 |
| kgriffs | "we don't guarantee FIFO and AOD here, so this is probably only good for that twitter firehose where you are ok if you miss one out of every million tweets | 20:36 |
| flaper87 | vkmc: did you like it ? | 20:36 |
| * kgriffs goes looking for Archive | 20:36 | |
| vkmc | flaper87, yeah, I'm just digging into your progressive rock knowledge | 20:36 |
| *** exploreshaifali has quit IRC | 20:37 | |
| vkmc | kgriffs, where do you think this descriptions should go? | 20:38 |
| flaper87 | vkmc: well, not exactly progressive knowledge but have you listened to Pink Floyd's new and last album? | 20:38 |
| kgriffs | vkmc: I think it needs a distinct home | 20:38 |
| kgriffs | whether it is a standalone resource or just some additional fields for a group, i'm not sure yet | 20:39 |
| flaper87 | kgriffs: when you get a chance, it'd be great to get your thoughts on this spec too https://review.openstack.org/#/c/134015/ | 20:39 |
| kgriffs | in either case, I feel like this is the direction flavors should evolve in 2.0 | 20:39 |
| * flaper87 reads backlog | 20:39 | |
| vkmc | flaper87, I did yeah | 20:40 |
| flaper87 | kgriffs: what do you think is missing in the current flavor implementation? | 20:40 |
| flaper87 | kgriffs: AFAICT, we can do what you described by combining capabilities and a good name | 20:41 |
| flaper87 | we could add more fields to it | 20:41 |
| kgriffs | so, what is missing | 20:41 |
| flaper87 | vkmc: did you like it ? | 20:41 |
| kgriffs | this is how I envision flavors evolving | 20:41 |
| kgriffs | first, 1.1 capabilities in flavors evolves to just be some well-named fields that a UX (horizon, CLI) tool can pick up and show to the user | 20:42 |
| kgriffs | name, description... | 20:42 |
| kgriffs | then we have a newer notion of capabilities, but these are actually standardized as has been proposed, and actually live with drivers and pools | 20:43 |
| vkmc | flaper87, more or less... it made me remember The Division Bell but I feel it is missing something | 20:43 |
| kgriffs | these capabilities bubble up and the user can see them as part of the flavor resource | 20:43 |
| kgriffs | but really they aren't hard-coded in the resource - they are sourced dynamically from the associated pool group | 20:44 |
| kragniz | A++ would listen to soundcloud links from flaper87 again | 20:44 |
| flaper87 | vkmc: right? I had the same feeling. | 20:45 |
| flaper87 | kragniz: haha, awesome. | 20:45 |
| kgriffs | not sure if that makes sense. I have a picture in my head that is hard to convert to text. | 20:45 |
| flaper87 | kragniz: well, to be fair, I got that soundcloud link from kgriffs's twitter | 20:45 |
| flaper87 | kgriffs: It kinda does | 20:46 |
| flaper87 | I think I got what you mean | 20:46 |
| * kgriffs has this morph animation going on in his head from flavors in 1.1 to 2.0 | 20:46 | |
| flaper87 | IIUC, you're saying we should expand the current flavor adding it some fields that would make them more like a descriptive resource than a logical resource | 20:47 |
| vkmc | kgriffs, draw it! | 20:48 |
| flaper87 | BTW, Archive is coming to Milan in March 2015 | 20:48 |
| flaper87 | Guess who's going? | 20:48 |
| flaper87 | :) | 20:48 |
| vkmc | da flaper! | 20:49 |
| kgriffs | https://gist.github.com/anonymous/d47069aa05a1d46b914f | 20:52 |
| vkmc | I was listening TMV, but I think I'll check another of Archive | 20:52 |
| kgriffs | so the biggest change going to 2.0 is conceptual. The way we think about flavors and document them. | 20:52 |
| vkmc | kgriffs, when would you set the title and description? on creation? | 20:53 |
| kgriffs | yes | 20:53 |
| vkmc | can you later change it? | 20:54 |
| kgriffs | capabilities can't be set by the operator though when creating the flavor. They are determined dynamically according to the group the operator associates the flavor with | 20:54 |
| kgriffs | when I create a flavor, I give some initial metadata, like title and group it is associated with | 20:55 |
| kgriffs | when an admin later lists those flavors, they can see everything, including the group it was associated with | 20:56 |
| kgriffs | but a user listing flavors probably shouldn't see the underlying pool group name | 20:56 |
| kgriffs | I've got a meeting in a few minutes, but can chat some more after that | 20:56 |
| kgriffs | (btw) | 20:56 |
| *** amalagon has joined #openstack-zaqar | 20:57 | |
| vkmc | ok :) | 20:58 |
| kgriffs | brb | 20:58 |
| *** kgriffs is now known as kgriffs|afk | 21:01 | |
| *** amalagon has quit IRC | 21:02 | |
| *** achanda has joined #openstack-zaqar | 21:15 | |
| *** achanda has quit IRC | 21:20 | |
| *** echevemaster has joined #openstack-zaqar | 21:33 | |
| *** achanda has joined #openstack-zaqar | 21:48 | |
| *** achanda has quit IRC | 21:49 | |
| *** amalagon has joined #openstack-zaqar | 21:50 | |
| *** kgriffs|afk is now known as kgriffs | 22:07 | |
| kgriffs | back | 22:07 |
| flwang | kgriffs: around? | 22:37 |
| *** achanda has joined #openstack-zaqar | 22:50 | |
| *** achanda has quit IRC | 22:55 | |
| kgriffs | yes | 23:06 |
| flwang | kgriffs: i have a question about the code structure of zaqar | 23:09 |
| kgriffs | kk | 23:09 |
| flwang | kgriffs: given we're going to support notification service | 23:09 |
| flwang | after discussed with flaper87, he suggest we keep using one process to deliver both queue and notification service | 23:10 |
| flwang | but you know, current code structure can't support that. | 23:11 |
| flwang | kgriffs: now I'm working on a proof of concept, see https://github.com/OpenStacker/zaqar/tree/notification | 23:11 |
| flwang | but I would like get some advices from your view :) | 23:12 |
| flwang | 1. should we keep two separate wsgi app for queues and notifications? so that operators can deploy them separately | 23:12 |
| flwang | 2. any suggestion about how organize the common code of the two wsgi apps ? given wsgi is just one of the transport implement | 23:14 |
| kgriffs | hmm | 23:15 |
| kgriffs | it might help to think about the notifications feature in pieces | 23:16 |
| kgriffs | first, there is the piece where you manage your subscription | 23:16 |
| flwang | kgriffs: yep | 23:17 |
| kgriffs | that will be part of the regular API | 23:18 |
| flwang | kgriffs: agree | 23:18 |
| kgriffs | so, it seems like it would just live with the WSGI and websocket transport code as do all the other operations | 23:18 |
| flwang | kgriffs: so do you mean we should not keep the notification service in a separate package? | 23:18 |
| kgriffs | that part of it doesn't seem like it should be separate. but let's think about the other piece(s) | 23:19 |
| flwang | kgriffs: or maybe we should make the transport layer be more common so that to be shared by the two services | 23:19 |
| flwang | kgriffs: I know it's not a 2 minutes job :) | 23:20 |
| kgriffs | hmm | 23:20 |
| kgriffs | well, i think the subscription management piece is just another operation and not a separate service. | 23:20 |
| flwang | so please help review my current silly POC code and we can talk later | 23:20 |
| flwang | kgriffs: oh | 23:21 |
| flwang | yes, we can say that | 23:21 |
| kgriffs | what may be a different service/process/daemon is the worker pool that delivers the messages | 23:21 |
| flwang | so is it necessary to keep it in a separate package? | 23:21 |
| flwang | I mean same level like queues? | 23:21 |
| * kgriffs thinking | 23:22 | |
| flwang | kgriffs: I have to join another meeting. would you mind me catching you tomorrow? :) | 23:23 |
| kgriffs | I'm having trouble coming up with any benefits for having a separate package for notifications unless we have a separate daemon or something | 23:23 |
| kgriffs | i mean, it is an integral part of the service | 23:24 |
| flwang | and it would be nice if we can talk with flaper87 together | 23:24 |
| kgriffs | flwang: actually, I will be on holiday thursday and friday | 23:24 |
| kgriffs | I would be happy to discuss this more on monday | 23:24 |
| flwang | kgriffs: ah, ok, sure | 23:24 |
| flwang | kgriffs: but I think i have got your opinion | 23:24 |
| flwang | kgriffs: thank you so much | 23:24 |
| kgriffs | sure. | 23:24 |
| flwang | kgriffs: have a good holiday ;) happy thanksgiving day! | 23:25 |
| kgriffs | thanks! | 23:25 |
| flwang | we don't have it in New Zealand :( | 23:25 |
| kgriffs | ah, too bad. | 23:25 |
| flwang | see you next Mon :) | 23:25 |
| kgriffs | kk, take care! | 23:25 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!
