| *** malini has joined #openstack-marconi | 00:18 | |
| *** rossk has quit IRC | 00:33 | |
| *** prashanthr_ has quit IRC | 00:39 | |
| *** prashanthr_ has joined #openstack-marconi | 00:42 | |
| *** prashanthr_ has quit IRC | 00:50 | |
| *** prashanthr_ has joined #openstack-marconi | 00:52 | |
| *** keith_newstadt has quit IRC | 01:01 | |
| *** keith_newstadt has joined #openstack-marconi | 01:01 | |
| *** malini has left #openstack-marconi | 01:06 | |
| *** flwang_ has joined #openstack-marconi | 01:19 | |
| *** flwang_ has quit IRC | 01:23 | |
| *** ametts has quit IRC | 01:27 | |
| *** vkmc has quit IRC | 01:27 | |
| *** rwsu has quit IRC | 01:27 | |
| *** rektide has quit IRC | 01:27 | |
| *** pquerna has quit IRC | 01:27 | |
| *** haomaiw__ has quit IRC | 01:27 | |
| *** keith_newstadt has quit IRC | 01:27 | |
| *** jraim has quit IRC | 01:27 | |
| *** ciypro|afk has quit IRC | 01:27 | |
| *** jay-atl has quit IRC | 01:27 | |
| *** openstackgerrit has quit IRC | 01:27 | |
| *** ekarlso has quit IRC | 01:27 | |
| *** boris-42 has quit IRC | 01:27 | |
| *** reed has quit IRC | 01:27 | |
| *** dmitryme has quit IRC | 01:27 | |
| *** whenry has quit IRC | 01:27 | |
| *** malini|afk has quit IRC | 01:27 | |
| *** peoplemerge has quit IRC | 01:27 | |
| *** AAzza_afk has quit IRC | 01:27 | |
| *** tmu_ has quit IRC | 01:27 | |
| *** sebasmagri has quit IRC | 01:27 | |
| *** fifieldt has quit IRC | 01:27 | |
| *** prashanthr_ has quit IRC | 01:27 | |
| *** Ephur has quit IRC | 01:27 | |
| *** megan_w has quit IRC | 01:27 | |
| *** wpf has quit IRC | 01:27 | |
| *** kgriffs|afk has quit IRC | 01:27 | |
| *** VeggieMeat has quit IRC | 01:27 | |
| *** lvh has quit IRC | 01:27 | |
| *** alcabrera|afk has quit IRC | 01:27 | |
| *** flaper87|afk has quit IRC | 01:27 | |
| *** ChanServ has quit IRC | 01:27 | |
| *** rektide has joined #openstack-marconi | 01:30 | |
| *** pquerna has joined #openstack-marconi | 01:30 | |
| *** rektide has quit IRC | 01:33 | |
| *** nosnos has joined #openstack-marconi | 01:45 | |
| *** fifieldt has joined #openstack-marconi | 01:58 | |
| *** kgriffs|afk has joined #openstack-marconi | 01:58 | |
| *** wpf has joined #openstack-marconi | 01:58 | |
| *** megan_w has joined #openstack-marconi | 01:58 | |
| *** Ephur has joined #openstack-marconi | 01:58 | |
| *** prashanthr_ has joined #openstack-marconi | 01:58 | |
| *** flaper87|afk has joined #openstack-marconi | 01:58 | |
| *** alcabrera|afk has joined #openstack-marconi | 01:58 | |
| *** lvh has joined #openstack-marconi | 01:58 | |
| *** VeggieMeat has joined #openstack-marconi | 01:58 | |
| *** ciypro|afk has joined #openstack-marconi | 01:58 | |
| *** jraim has joined #openstack-marconi | 01:58 | |
| *** ekarlso has joined #openstack-marconi | 01:58 | |
| *** openstackgerrit has joined #openstack-marconi | 01:58 | |
| *** jay-atl has joined #openstack-marconi | 01:58 | |
| *** rektide has joined #openstack-marconi | 01:58 | |
| *** haomaiw__ has joined #openstack-marconi | 01:58 | |
| *** dickson.freenode.net sets mode: +oo kgriffs|afk flaper87|afk | 01:58 | |
| *** haomaiw__ has quit IRC | 01:58 | |
| *** haomaiwa_ has joined #openstack-marconi | 01:59 | |
| *** ametts has joined #openstack-marconi | 02:04 | |
| *** reed has joined #openstack-marconi | 02:05 | |
| *** dmitryme has joined #openstack-marconi | 02:05 | |
| *** whenry has joined #openstack-marconi | 02:05 | |
| *** malini|afk has joined #openstack-marconi | 02:05 | |
| *** peoplemerge has joined #openstack-marconi | 02:05 | |
| *** AAzza_afk has joined #openstack-marconi | 02:05 | |
| *** tmu_ has joined #openstack-marconi | 02:05 | |
| *** sebasmagri has joined #openstack-marconi | 02:05 | |
| *** vkmc has joined #openstack-marconi | 02:06 | |
| *** rwsu has joined #openstack-marconi | 02:08 | |
| *** ChanServ has joined #openstack-marconi | 02:20 | |
| *** dickson.freenode.net sets mode: +o ChanServ | 02:20 | |
| *** oz_akan has joined #openstack-marconi | 02:20 | |
| *** haomaiw__ has joined #openstack-marconi | 02:35 | |
| openstackgerrit | Malini Kamalambal proposed a change to openstack/marconi: Implement POP in v1.1 API https://review.openstack.org/90202 | 02:37 |
|---|---|---|
| *** haomaiwa_ has quit IRC | 02:38 | |
| *** alcabrera|afk is now known as alcabrera | 02:48 | |
| *** boris-42 has joined #openstack-marconi | 02:49 | |
| *** flwang_ has joined #openstack-marconi | 03:20 | |
| *** flwang_ has quit IRC | 03:24 | |
| *** boris-42 has quit IRC | 03:38 | |
| *** keith_newstadt has joined #openstack-marconi | 03:42 | |
| *** boris-42 has joined #openstack-marconi | 03:42 | |
| *** nosnos has quit IRC | 03:43 | |
| *** haomaiw__ has quit IRC | 03:47 | |
| *** keith_newstadt has quit IRC | 03:47 | |
| *** haomaiwang has joined #openstack-marconi | 03:47 | |
| *** keith_newstadt has joined #openstack-marconi | 03:48 | |
| *** chandankumar has joined #openstack-marconi | 03:50 | |
| *** whenry has quit IRC | 03:59 | |
| *** nosnos has joined #openstack-marconi | 04:09 | |
| *** whenry has joined #openstack-marconi | 04:11 | |
| *** haomai___ has joined #openstack-marconi | 04:12 | |
| *** alcabrera is now known as alcabrera|afk | 04:14 | |
| *** haomaiwang has quit IRC | 04:14 | |
| *** chandankumar has quit IRC | 04:48 | |
| *** keith_newstadt has quit IRC | 04:54 | |
| vkmc | flwang, hey! | 04:56 |
| *** mkoderer has joined #openstack-marconi | 04:58 | |
| *** chandankumar has joined #openstack-marconi | 05:01 | |
| vkmc | next meeting is at 21UTC, just in case you didn't see it :) | 05:01 |
| *** whenry has quit IRC | 05:17 | |
| *** k4n0 has joined #openstack-marconi | 05:17 | |
| *** oz_akan has quit IRC | 05:26 | |
| openstackgerrit | Victoria Martínez de la Cruz proposed a change to openstack/marconi: Drop pylint due to the huge amount of false positives https://review.openstack.org/106272 | 05:30 |
| flwang | vkmc: got it, thanks | 05:30 |
| flwang | vkmc: btw, IIRC, you asked some questions to me, but I was busy. can I help you now? | 05:31 |
| vkmc | flwang, fortunately I could figure it out | 05:32 |
| flwang | vkmc: awesome | 05:32 |
| vkmc | flwang, thanks! | 05:32 |
| flwang | sorry I wish I can help you | 05:32 |
| vkmc | flwang, oh it's ok! you always help me when you have some time :) | 05:34 |
| flwang | hehe, glad to know that, let's rock on :) | 05:35 |
| flwang | recently, I was busy to work on an internal billing system | 05:35 |
| flwang | and the code is not very good, sigh | 05:36 |
| openstackgerrit | Victoria Martínez de la Cruz proposed a change to openstack/marconi: Drop pylint due to the huge amount of false positives https://review.openstack.org/106272 | 05:36 |
| vkmc | boo | 05:36 |
| vkmc | legacy code? | 05:36 |
| flwang | not really, you know, there is no official billing system for openstack, so we have to do home-cooking | 05:37 |
| vkmc | yeah.. it's not a trivial system | 05:37 |
| flwang | but the original author is not really a python guy and not familiar with OpenStack, so... | 05:38 |
| vkmc | do you know what happened with billingstack? | 05:38 |
| flwang | vkmc: AFAIK, there is no progress | 05:38 |
| flwang | https://github.com/billingstack | 05:39 |
| vkmc | ah well, that's a bummer | 05:39 |
| flwang | https://github.com/stacksherpa/billingstack | 05:39 |
| flwang | the last commit was 2 years ago | 05:39 |
| vkmc | yeah it looks quiet | 05:39 |
| flwang | so maybe I need to create a new wheel :) | 05:40 |
| vkmc | it looks like it... hope you have some other devs with you | 05:41 |
| *** prashanthr_ has quit IRC | 05:41 | |
| flwang | maybe, but seems community are not really interested in billing | 05:41 |
| flwang | though there is a real strong requirement from a customer perspective | 05:42 |
| *** prashanthr_ has joined #openstack-marconi | 05:42 | |
| vkmc | flwang, yeah it's understandable... billing is not that opensource | 05:47 |
| vkmc | but if it's a need, then we should cover it | 05:47 |
| vkmc | somehow | 05:47 |
| flwang | operating system can be opensourced, why not a billing system? :D | 05:48 |
| flwang | i'm kidding | 05:48 |
| vkmc | haha | 05:48 |
| vkmc | free doesn't mean it doesn't cost money! | 05:49 |
| flwang | obviously, a billing system should be a plugable, consumer can define their own billing strategy | 05:49 |
| flwang | just think alound :) | 05:49 |
| flwang | /alound/aloud | 05:49 |
| vkmc | yeah, most of the cloud services I tested charge you by data transfer, compute hours and standard io | 05:54 |
| vkmc | an equation that kills your pocket :') | 05:54 |
| vkmc | but you have a shiny vm somewhere over the cloud | 05:55 |
| vkmc | I recently get a free subscription to Azure to test some AMQP related stuff... it costs $900 per month | 05:56 |
| vkmc | its crazy | 05:57 |
| vkmc | flaper87|afk, test it before they charge me ^^ lol | 05:57 |
| vkmc | well, I'm heading off for the day o/ | 05:59 |
| vkmc | ttfn! | 05:59 |
| *** vkmc has quit IRC | 05:59 | |
| *** nosnos has quit IRC | 06:08 | |
| *** reed has quit IRC | 06:24 | |
| *** flaper87|afk is now known as flaper87 | 07:13 | |
| flaper87 | flwang: yo! | 07:14 |
| flaper87 | flwang: I really hope you're having a great time there because I miss talking to you and it's all your fault >.> | 07:15 |
| *** flwang_ has joined #openstack-marconi | 07:21 | |
| *** flwang_ has quit IRC | 07:26 | |
| *** oz_akan_ has joined #openstack-marconi | 07:28 | |
| *** oz_akan_ has quit IRC | 07:33 | |
| *** openstack has joined #openstack-marconi | 07:42 | |
| *** mkoderer has quit IRC | 07:52 | |
| *** mkoderer has joined #openstack-marconi | 08:19 | |
| *** oz_akan_ has joined #openstack-marconi | 08:29 | |
| *** oz_akan_ has quit IRC | 08:33 | |
| *** oz_akan_ has joined #openstack-marconi | 09:00 | |
| *** oz_akan_ has quit IRC | 09:05 | |
| *** haomai___ has quit IRC | 09:24 | |
| *** haomaiwang has joined #openstack-marconi | 09:24 | |
| *** flwang_ has joined #openstack-marconi | 09:39 | |
| *** AAzza_afk is now known as AAzza | 09:43 | |
| *** prashanthr_ has quit IRC | 09:57 | |
| *** denis_makogon has joined #openstack-marconi | 09:57 | |
| *** oz_akan_ has joined #openstack-marconi | 10:01 | |
| *** oz_akan_ has quit IRC | 10:05 | |
| *** AAzza is now known as AAzza_afk | 10:25 | |
| *** ajayaa has joined #openstack-marconi | 10:27 | |
| *** oz_akan_ has joined #openstack-marconi | 11:02 | |
| *** oz_akan_ has quit IRC | 11:06 | |
| *** openstackgerrit has quit IRC | 11:16 | |
| *** openstackgerrit has joined #openstack-marconi | 11:18 | |
| *** tedross has joined #openstack-marconi | 11:21 | |
| *** keith_newstadt has joined #openstack-marconi | 11:45 | |
| *** keith_newstadt has quit IRC | 11:49 | |
| *** keith_newstadt has joined #openstack-marconi | 11:50 | |
| *** flwang_ has quit IRC | 11:54 | |
| *** keith_newstadt has quit IRC | 11:57 | |
| openstackgerrit | Flavio Percoco proposed a change to openstack/marconi: Add flavors support to mongodb https://review.openstack.org/98793 | 11:59 |
| openstackgerrit | Flavio Percoco proposed a change to openstack/marconi: Add API support for flavors https://review.openstack.org/106346 | 11:59 |
| *** k4n0 has quit IRC | 12:00 | |
| *** vkmc has joined #openstack-marconi | 12:12 | |
| vkmc | hi all! | 12:16 |
| flaper87 | GO GO GO Germany | 12:18 |
| flaper87 | ops | 12:18 |
| flaper87 | :P | 12:18 |
| flaper87 | vkmc: gooood morning :) | 12:18 |
| * vkmc pulls flaper87's chair | 12:19 | |
| * flaper87 falls and hears vkmc's evil laugh | 12:20 | |
| vkmc | mueehehehe | 12:20 |
| vkmc | morning flaper87 :) | 12:20 |
| *** haomaiw__ has joined #openstack-marconi | 12:35 | |
| *** haomaiwang has quit IRC | 12:37 | |
| *** sriram has joined #openstack-marconi | 12:45 | |
| *** mpanetta has joined #openstack-marconi | 12:59 | |
| *** AAzza_afk is now known as AAzza | 13:00 | |
| *** oz_akan_ has joined #openstack-marconi | 13:02 | |
| *** ajayaa has quit IRC | 13:15 | |
| *** oz_akan_ has quit IRC | 13:18 | |
| *** mwagner_lap has joined #openstack-marconi | 13:18 | |
| *** mwagner_lap is now known as mwagner_rdu | 13:19 | |
| *** whenry has joined #openstack-marconi | 13:19 | |
| *** ajayaa has joined #openstack-marconi | 13:28 | |
| *** Obulpathi has joined #openstack-marconi | 13:35 | |
| *** Obulpathi has quit IRC | 13:46 | |
| *** Obulpathi has joined #openstack-marconi | 13:47 | |
| *** oz_akan_ has joined #openstack-marconi | 13:50 | |
| *** AAzza is now known as AAzza_afk | 13:57 | |
| *** amitgandhi has joined #openstack-marconi | 14:00 | |
| *** malini has joined #openstack-marconi | 14:01 | |
| *** ametts has quit IRC | 14:02 | |
| *** ametts has joined #openstack-marconi | 14:15 | |
| *** kgriffs|afk is now known as kgriffs | 14:26 | |
| *** Hao has joined #openstack-marconi | 14:26 | |
| *** mwagner_rdu has quit IRC | 14:26 | |
| flaper87 | kgriffs: good morning | 14:31 |
| flaper87 | kgriffs: lemme know when you've a min to discuss flavors | 14:31 |
| flaper87 | got a couple of things I'd like to share and brainstorm on | 14:32 |
| flaper87 | vkmc: malini ^ | 14:32 |
| *** alcabrera|afk is now known as alcabrera | 14:32 | |
| malini | o/ | 14:32 |
| malini | I was just reading through https://wiki.openstack.org/wiki/Marconi/specs/api/v1.1#List_Messages | 14:33 |
| malini | "limit specifies up to 20 messages (configurable) to return. If not specified, limit defaults to 10." | 14:33 |
| malini | This sounds wrong.. | 14:33 |
| malini | Shouldn't we default to max_messages_per_page? | 14:33 |
| *** ajayaa has quit IRC | 14:34 | |
| malini | (which happens to be 20) | 14:34 |
| flaper87 | malini: yup, that's wrong | 14:34 |
| vkmc | o/ | 14:34 |
| malini | flaper87: I am going to update tht, ok? | 14:34 |
| flaper87 | malini: perfect, thanks | 14:34 |
| kgriffs | malini: better check the code and submit a patch too. :D | 14:35 |
| malini | kgriffs: tht's what I was going to do | 14:35 |
| kgriffs | excellent | 14:35 |
| malini | my POP test is failing because of this | 14:35 |
| malini | So I'll need this fixed to get through | 14:35 |
| *** alcabrera is now known as alcabrera|afk | 14:37 | |
| kgriffs | flaper87: re flavors, what's up? | 14:37 |
| flaper87 | Next step is to make the storage data API aware of flavors. That means, when creating a queue allow people to select a flavor and more importantly make the storage API use that flavor. | 14:38 |
| flaper87 | the thing is we don't know what flavor the data should be stored in until we get to the storage layer | 14:39 |
| flaper87 | (as it's planned right now) | 14:39 |
| flaper87 | This is wrong in many ways. The worst part is that in order to post a message to the right store, we need to get the flavor for the queue | 14:40 |
| flaper87 | but without knowing where the queue was created - having the flavor, that is - we can't get the queue | 14:40 |
| flaper87 | this had me thinking that maybe queues are more part of the control API than they are of the data API | 14:41 |
| flaper87 | If we think about queues, when are they actually used? | 14:41 |
| flaper87 | As it is now (after the lazy queue's patch landed) they're basically never used | 14:41 |
| flaper87 | The, perhaps crazy, proposal is to move queues away from the data API to the control API. This means that whatever store being used to keep pools will be used for queues as well | 14:42 |
| flaper87 | another approach would be to not store flavors in the queues | 14:42 |
| *** Hao has quit IRC | 14:42 | |
| flaper87 | I probably like the second better, although both proposals bring me back to the "Why are we keeping queues around?" question | 14:43 |
| flaper87 | kgriffs: malini vkmc ^ | 14:43 |
| kgriffs | great thoughts | 14:43 |
| kgriffs | if only we had a whiteboard handy at the moment. :p | 14:44 |
| flaper87 | LOL | 14:44 |
| flaper87 | indeed, I drew crazy things on mine that I don't understand anymore | 14:44 |
| * malini reading..I am little slow today | 14:44 | |
| * flaper87 gives malini a redbull | 14:45 | |
| kgriffs | let me think out loud for a bit | 14:45 |
| flaper87 | kgriffs: sure thing | 14:45 |
| vkmc | I assume that it's possible to have more than one queue with the same flavor | 14:45 |
| flaper87 | but just you, I want malini's answer NOW! | 14:45 |
| kgriffs | rhetorical question: what *is* a queue? | 14:45 |
| flaper87 | vkmc: correct | 14:45 |
| vkmc | so, apart from the flavor, a queue has other important attributes (or should have) | 14:45 |
| kgriffs | first, it is an orginizational concept. A way to group related messages together | 14:46 |
| flaper87 | My favorite way to describe queue's so far is: "A way to group messages under the same box" | 14:46 |
| flaper87 | >.> | 14:46 |
| vkmc | so we are keeping queues in order to classify messages and make it easy for clients to find what they are looking for | 14:46 |
| kgriffs | second, it is a way to associate common attributes with those messages | 14:46 |
| kgriffs | basically, a way to group them for querying, and also for DRYing up common attributes | 14:47 |
| flaper87 | vkmc: re queue having other important attributes: It could, yes. We just don't have that use case yet. | 14:47 |
| vkmc | cool | 14:47 |
| flaper87 | flavors seemed to be the first one | 14:47 |
| vkmc | what about specifying a queue as a message attribute? | 14:48 |
| vkmc | (this is the topics idea once again) | 14:48 |
| flaper87 | right | 14:48 |
| kgriffs | for example, a Barbican key ID to use to sign the messages | 14:48 |
| kgriffs | or whatever | 14:48 |
| flaper87 | kgriffs: +1 | 14:48 |
| kgriffs | default TTL, things like that | 14:48 |
| kgriffs | now, does it make sense to also have a flavor there? | 14:48 |
| kgriffs | otherwise, you have to specify the flavor for each message | 14:49 |
| flaper87 | I pretty much see queues for messages as I see heat templates for infrastructure | 14:49 |
| flaper87 | the described how the distribution of message should look like | 14:49 |
| kgriffs | and now I have a problem if I want to query a set of related messages, because I have to check all the flavors unless there is something grouping them together | 14:49 |
| kgriffs | like... | 14:49 |
| kgriffs | a "queue | 14:49 |
| *** tedross has quit IRC | 14:49 | |
| flaper87 | but per-message flavors imply that you can use more flavors for 1 queue | 14:50 |
| flaper87 | which is not what we want | 14:50 |
| malini | we also had someone mention using queues to RBAC etc. in the summit | 14:50 |
| flaper87 | and we loose FIFO | 14:50 |
| flaper87 | malini: right, I had forgotten | 14:50 |
| flaper87 | s/the described/they describe/ | 14:50 |
| kgriffs | yeah. I think it is unnecessary complexity. When people ask for sliders on durability vs. performance, for example, they are thinking in coarse-grained terms afaict | 14:51 |
| kgriffs | Q.E.D. It makes sense to use queues to group messages, and that is a natural place to store the flavor ID | 14:52 |
| kgriffs | next point | 14:52 |
| kgriffs | should a queue be in the control plane or data plane? | 14:52 |
| kgriffs | I posit that it should be in the control plane | 14:53 |
| vkmc | I think that the control plane fits better our queue concept | 14:53 |
| kgriffs | with the caveat that you will also need to have the queue name as part of each message record in the data plane, so you can associate the two record types | 14:53 |
| * flaper87 can't stop thinking about this title for the current kgriffs messages: "Come and join us on a trip into how kgriffs brain works" | 14:53 | |
| kgriffs | :D | 14:54 |
| * kgriffs could do even better if he had a whiteboard to draw on | 14:54 | |
| kgriffs | let's see how this might work | 14:54 |
| kgriffs | 1. message is posted to the server | 14:54 |
| malini | the whiteboard thing shud be our next project | 14:55 |
| flaper87 | So, I put some thoughts on what would happen if queues were in the control plane | 14:55 |
| flaper87 | but lets go through kgriffs thoughts first | 14:55 |
| kgriffs | 2. server looks up queue record in cache, if not in cache gets from control driver and caches for next time | 14:55 |
| kgriffs | 3. server looks up flavor from queue record which tells it which data driver instance to use to store the message | 14:56 |
| kgriffs | 4. server may also look up default TTL or whatever (TBD, maybe in a future API version) | 14:57 |
| kgriffs | 5. server uses info to store message | 14:57 |
| kgriffs | (meanwhile, in the lower-east side of Manhattan) | 14:57 |
| flaper87 | ROFL | 14:57 |
| kgriffs | 1. client lists messages, sending a specific queue name to the server | 14:58 |
| kgriffs | 2. server looks up queue record as before | 14:58 |
| kgriffs | 3. server uses flavor ID in queue record to figure out which storage driver to use. Also takes into consideration which pool to query. | 14:59 |
| kgriffs | hmmm | 15:00 |
| flaper87 | exactly, 1 thing there | 15:00 |
| kgriffs | actually, I guess the server just needs to know which pool the queue belongs to | 15:00 |
| flaper87 | flavors imply that pools are enabled | 15:00 |
| kgriffs | yeah. | 15:00 |
| flaper87 | if pools are not enabled then flavors can't be used | 15:00 |
| kgriffs | let me try again | 15:01 |
| * kgriffs scratches out previous #3 | 15:01 | |
| flaper87 | but we'll still need to go through the control API to get the queue's info for other things, say barbican | 15:01 |
| * flaper87 invents a new command: #undo #3 | 15:01 | |
| flaper87 | :P | 15:01 |
| kgriffs | #undo #2 | 15:01 |
| kgriffs | hmmm | 15:02 |
| kgriffs | 2. server looks up queue record as before, but only if it needs queue metadata (the pool can be looked up simply by queue name, not needing the full queue record) | 15:02 |
| kgriffs | 3. server uses pool info to figure out which storage driver to query for messages | 15:03 |
| kgriffs | 4. server queries appropriate storage driver and returns the results to the client | 15:03 |
| kgriffs | fin. | 15:03 |
| kgriffs | something like that, anyway. | 15:03 |
| flaper87 | yeah, that sounds about right to me | 15:03 |
| kgriffs | wait a minute | 15:05 |
| *** cpallares has joined #openstack-marconi | 15:05 | |
| * flaper87 waits a min | 15:05 | |
| kgriffs | when a queue is created, I think the server would assign it's pool at that moment | 15:05 |
| kgriffs | so for an incoming message | 15:06 |
| kgriffs | things would happen as they do now | 15:06 |
| kgriffs | the server looks up the assigned pool for the queue name specified in the POST | 15:06 |
| flaper87 | it also depends on the flavor, though. | 15:06 |
| flaper87 | oh shit | 15:06 |
| kgriffs | yes | 15:06 |
| flaper87 | what happens if there's a pool assigned to a queue and then the queue gets created with a flavor that points to another pool ? | 15:07 |
| kgriffs | so, when a queue is created, the server would say "what flavor is this?" and then pick from a subset of the pools one that supports that flavor | 15:07 |
| flaper87 | and here comes another crazy thought, sit tight | 15:07 |
| kgriffs | wait, how would a queue get reassigned a different flavor? | 15:08 |
| flaper87 | you mean pool ? | 15:08 |
| kgriffs | I was thinking that the flavor and pool is determined when the queue is first created, but does not change after that... except when the operator wants to rebalance queues behind the scenes. | 15:08 |
| flaper87 | yeah but what happens if: | 15:09 |
| flaper87 | 1. create flavor A that points to store A | 15:09 |
| kgriffs | I guess I view flavor as just being a filter on the set of pool | 15:09 |
| flaper87 | 2. assign pool B to queue Q | 15:09 |
| flaper87 | 3. Create queue Q w/ flavor A | 15:10 |
| flaper87 | which pool should we pick ? | 15:10 |
| flaper87 | another way to do this is to re-think pool catalogs | 15:10 |
| flaper87 | can we make those catalogs flavors ? | 15:10 |
| kgriffs | wait, help me understand | 15:10 |
| kgriffs | how do you assign pool B to queue Q before it is created? | 15:10 |
| * flaper87 is giving kgriffs things to think over the weekend so he doesn't get bored | 15:11 | |
| * kgriffs chuckles | 15:11 | |
| flaper87 | to assign a pool you don't need to create the queue | 15:11 |
| flaper87 | AFAIK | 15:11 |
| *** whenry has quit IRC | 15:11 | |
| * flaper87 checks the code | 15:11 | |
| kgriffs | WAH? | 15:12 |
| flaper87 | kgriffs: https://github.com/openstack/marconi/blob/master/marconi/queues/storage/base.py#L503 | 15:12 |
| flaper87 | mmh, wait, so, basically the catalog is what I want to do w/ flavors | 15:13 |
| flaper87 | something's fishy here | 15:13 |
| kgriffs | ihttps://github.com/openstack/marconi/blob/master/marconi/queues/storage/pooling.py#L168 | 15:14 |
| kgriffs | https://github.com/openstack/marconi/blob/master/marconi/queues/storage/pooling.py#L168 | 15:14 |
| *** chandankumar has quit IRC | 15:15 | |
| *** reed has joined #openstack-marconi | 15:16 | |
| flaper87 | kgriffs: oh mmh | 15:17 |
| flaper87 | you're right | 15:17 |
| kgriffs | <phew> | 15:18 |
| flaper87 | would it be fair to say that whenever a queue is created with a flavor, it'd be enough to create a new catalog pointing to that pool | 15:18 |
| flaper87 | ? | 15:18 |
| flaper87 | I think there's an overlap between catalog and flavors | 15:18 |
| flaper87 | kgriffs: ^ | 15:21 |
| kgriffs | I think the flavor determines a subset of the pools in the catalog to choose. So, the catalog needs to be updated to store a flavor ID with each pool record. Then when registering a queue, only those pools supporting the requested flavor would be listed. | 15:22 |
| kgriffs | https://github.com/openstack/marconi/blob/master/marconi/queues/storage/pooling.py#L397 | 15:22 |
| kgriffs | e.g. | 15:22 |
| kgriffs | self._pools_ctrl.list(limit=0, flavor=X) | 15:22 |
| kgriffs | anyway, that's they way I was thinking of it | 15:23 |
| * kgriffs goes off to draw some pictures | 15:23 | |
| *** whenry has joined #openstack-marconi | 15:25 | |
| flaper87 | ah damn, right. I had forgotten the flavor has a pool not a URI | 15:25 |
| flaper87 | ok, nvm. all clear now | 15:26 |
| flaper87 | create queue->get flavor->select store from pool->register catalog | 15:26 |
| flaper87 | does that sound right? | 15:26 |
| flaper87 | we still need to move queues to the control API, though | 15:26 |
| kgriffs | yeah, sounds about right | 15:27 |
| kgriffs | 1. create queue | 15:28 |
| kgriffs | 2. use flavor from create queue call to filter list of pools | 15:28 |
| kgriffs | 3. pick from resulting list using weighted algorithm | 15:28 |
| kgriffs | 4. register mapping between queue and pool in the catalog | 15:29 |
| kgriffs | two things | 15:29 |
| kgriffs | first is the question of moving queues to the control API | 15:29 |
| kgriffs | second is how to handle lazy queue creation | 15:29 |
| kgriffs | latter one first | 15:29 |
| kgriffs | seems like we need to have a way to define a "default" flavor per project | 15:30 |
| kgriffs | what do you think about that? | 15:30 |
| malini | kgriffs: '"default" flavor per project —> does tht imply projects cannot have more than one flavor? | 15:30 |
| kgriffs | no, just that this is the flavor to use when automatically creating a queue if it doesn't already exist (when a message is first posted with that queue name) | 15:31 |
| flaper87 | mmh, I thought about using the store defined in the config as the default one | 15:32 |
| kgriffs | or, when the client does not specify a flavor when it explicitly creates the queue (we can allow that to be optional) | 15:32 |
| kgriffs | flaper87: I think that is the first step | 15:32 |
| flaper87 | actually, that's probably a bad idea | 15:32 |
| kgriffs | then we can decide whether to allow the user to override it | 15:32 |
| kgriffs | I mean, we don't have a "create project" call | 15:32 |
| kgriffs | so we have to have a default for the default if you know what I mean. | 15:33 |
| flaper87 | because then we need to have 2 options: 1 to have the default store for the control API and one for the data api | 15:33 |
| flaper87 | here's a thing, for the sake of making things more complex but consistent: | 15:33 |
| flaper87 | What if we move to just use pools | 15:33 |
| flaper87 | this means, the store defined in the config is the one used for the control API. Then admins are required to create a default pool to use | 15:34 |
| flaper87 | if no flavor is passed along with the queue, then the default pool will be used | 15:35 |
| flaper87 | no need to create a default flavor | 15:35 |
| *** AAzza_afk is now known as AAzza | 15:36 | |
| * flaper87 is full of crazy and radical thoughts today | 15:36 | |
| kgriffs | just to clarify, are you saying to do away with the separate concept of a flavor? | 15:36 |
| flaper87 | Nope, I'm proposing this: | 15:36 |
| flaper87 | 1. Get rid of the non-pooled way to deploy marconi | 15:36 |
| flaper87 | 2. Have a default pool | 15:37 |
| flaper87 | 3. Use the default pool when a queue is created w/o flavor | 15:37 |
| flaper87 | this probably makes deploying marconi a bit more complex but consistent | 15:37 |
| kgriffs | hmmm | 15:38 |
| kgriffs | so, first off, I like the idea of saying "there are always pools" and for small deployments you just have a single pool in your catalog | 15:39 |
| flaper87 | right | 15:39 |
| kgriffs | cons are perhaps a little performance degradation because I have to look up the pool for the queue for each operation | 15:40 |
| kgriffs | we can mitigate that with caching and do some profiling to tune the code | 15:40 |
| kgriffs | other con is that, as you say, setup is now a little more complicated. | 15:40 |
| flaper87 | the first point concerns me more than the second one | 15:41 |
| kgriffs | I think we will need to get pretty sophisticated with our caching | 15:41 |
| kgriffs | to minimize the cost of the extra lookup, we need a tiered caching system | 15:42 |
| kgriffs | so that most lookups happen from local RAM | 15:42 |
| kgriffs | anyway, something to think more on another time | 15:42 |
| flaper87 | agreed | 15:43 |
| kgriffs | Ideally, we could find a way to have shared RAM between processes. uwsgi supports this but we can't be tightly coupled to uwsgi | 15:43 |
| flaper87 | now, regardless on whether we go "always pooled" or not, the things to do on the queues and flavors side won't change | 15:43 |
| flaper87 | I'd like to reach a consensus here and decide what to do with flavors, queues and where queues should live | 15:44 |
| kgriffs | let me just touch on one more thing regarding defaults, then we can talk about where queues live | 15:44 |
| kgriffs | I think the first, simplest thing to do is have a way for the administrator to designate a default flavor for all projects that will be used when a queue is created lazily | 15:45 |
| kgriffs | we can decide later whether to allow the user to customize that default per-project | 15:46 |
| kgriffs | does that make sense? | 15:46 |
| flaper87 | yup, it does | 15:46 |
| kgriffs | ok | 15:46 |
| * kgriffs is starting to wonder if we will need a v1.2 API to add in per-project and per-queue defaults | 15:47 | |
| kgriffs | ok, so let's move on to the control plane thing | 15:47 |
| kgriffs | first, I want to ask something | 15:47 |
| kgriffs | s/ask/clarify | 15:48 |
| kgriffs | https://blueprints.launchpad.net/marconi/+spec/split-storage-drivers | 15:48 |
| kgriffs | if we split the drivers, then is pooling only applicable to message store, not control plane? | 15:48 |
| flaper87 | correct | 15:49 |
| kgriffs | ok | 15:49 |
| kgriffs | the fact that we can use caching in the control plane means that I don't think we need to scale out nearly as much as the message store | 15:50 |
| kgriffs | ok, next thought | 15:50 |
| kgriffs | I think it makes sense to store the queue records in the data plane storage | 15:50 |
| kgriffs | but... | 15:51 |
| kgriffs | (and this is a bit of a paradox) | 15:51 |
| kgriffs | from the user's perspective, they control the queue creation, not the administrator | 15:51 |
| kgriffs | so, it seems like we have three parts of our API now | 15:51 |
| kgriffs | 1. data plane, controlled by the user | 15:51 |
| kgriffs | 2. control plane subset A, controlled by the user (queue management) | 15:52 |
| kgriffs | 3. control plane subset B, controlled by the cloud operator (pools, health, flavor catalog, etc.) | 15:52 |
| flaper87 | But the API should remain the same, IMHO. What should change is the way we do things under-the-hood | 15:53 |
| kgriffs | that was sort of what I was thinking too | 15:53 |
| kgriffs | we have a "user" portion of the API which contains data plane and control pane endpoints (but the user doesn't really care which is which) | 15:54 |
| kgriffs | (and each element is implemented by a corresponding driver) | 15:54 |
| kgriffs | and we have an "admin" portion of the API that the user doesn't see/care about | 15:54 |
| kgriffs | (and that is also protected through access controls) | 15:55 |
| flaper87 | right | 15:55 |
| kgriffs | then, we move queue records to be stored in the control plane driver | 15:55 |
| flaper87 | one thing that worries me (even with a fancy caching system) is that we'll be storing lots of queues in the control database | 15:55 |
| *** flwang_ has joined #openstack-marconi | 15:56 | |
| flaper87 | I'm not sure how many folks will use non-default flavors for queues but we have to cover the case where there are millions of those records | 15:56 |
| kgriffs | suppose an operator used maria for control and mongo for data | 15:56 |
| flaper87 | that's exactly my concern | 15:56 |
| kgriffs | hmm | 15:58 |
| flaper87 | supossely, the store used for the data plane, it's the fastest and most scalable one | 15:59 |
| flaper87 | the control plan store is suppose to keep things controlled by the admin | 15:59 |
| flaper87 | which means the growth can be monitored | 15:59 |
| flaper87 | and controlled | 15:59 |
| kgriffs | good point | 16:00 |
| kgriffs | let's see | 16:01 |
| flaper87 | supposedly* | 16:01 |
| *** flwang_ has quit IRC | 16:01 | |
| kgriffs | what constitutes the control plane | 16:01 |
| kgriffs | 1. list of pools | 16:01 |
| kgriffs | 2. list of flavors | 16:01 |
| kgriffs | 3. mapping between pools and queues | 16:01 |
| kgriffs | 4. list of queues | 16:01 |
| *** chandankumar has joined #openstack-marconi | 16:03 | |
| * flaper87 thinking | 16:04 | |
| flaper87 | sounds right | 16:04 |
| * flaper87 guesses kgriffs is drawing rainbows on the whiteboard | 16:10 | |
| malini | :D | 16:10 |
| flaper87 | my concern is that I don't think we should go out there and say: "You need 2 super scalable databases to deploy marconi" | 16:10 |
| flaper87 | I think it'd be very common to use the same db for both control and data plans | 16:11 |
| flaper87 | planes | 16:11 |
| kgriffs | sorry about that. stupid wifi | 16:12 |
| kgriffs | flaper87: so, it seems we already have this issue - see #3 | 16:13 |
| kgriffs | #3 and #4 could get really big | 16:13 |
| flaper87 | yeah | 16:15 |
| * flaper87 is thinking | 16:20 | |
| kgriffs | mysql can scale reads pretty easily by having read-only slaves | 16:24 |
| kgriffs | to scale writes, we would have to recommend people either scale up their boxes or use something like mongo that has sharding | 16:25 |
| flaper87 | I was more worried about scaling space and replications and stuff | 16:25 |
| kgriffs | question is, what is the % reads vs. writes? | 16:25 |
| flaper87 | I don't expect queues to be created/destroyed that frequent (at least for those using flavors) | 16:25 |
| flaper87 | but I should stop assuming shuch things | 16:26 |
| flaper87 | we need real data | 16:26 |
| flaper87 | Although, I think it'd be very common to use the same db for both control and data planes | 16:26 |
| kgriffs | yeah, probably. | 16:26 |
| kgriffs | I think what we support for the control plane would be a superset of the data plane | 16:27 |
| kgriffs | well, actually, maybe not... depends on if we end up doing brokers and backends at some point | 16:27 |
| flaper87 | right | 16:27 |
| kgriffs | anyway, I agree that for the most part it will be easier for the operator to run the same thing, so they will tend to do that | 16:28 |
| kgriffs | (if we give them that option) | 16:28 |
| flaper87 | ok | 16:28 |
| flaper87 | soooo | 16:28 |
| flaper87 | man, lots of things we've agreed on today | 16:29 |
| * flaper87 needs to write all this down | 16:29 | |
| flaper87 | anyway, one more time, should we move queues to the control plane ? | 16:29 |
| kgriffs | yes | 16:29 |
| flaper87 | ok, lets do that first and then figure out the "always pooled" thing | 16:30 |
| kgriffs | would it be easier to do that before or after splitting the storage drivers | 16:30 |
| kgriffs | https://blueprints.launchpad.net/marconi/+spec/split-storage-drivers | 16:30 |
| kgriffs | second question | 16:31 |
| kgriffs | actually, a thought | 16:31 |
| flaper87 | oh mmh, probably first since after we'll have to fix imports | 16:31 |
| flaper87 | I think | 16:31 |
| flaper87 | not sure | 16:31 |
| flaper87 | it's probably the same | 16:31 |
| kgriffs | ok, feel free to move to j-2 if needed | 16:31 |
| kgriffs | although, we only have a couple weeks left. :p | 16:31 |
| kgriffs | also, do we need a blueprint for having "always-on" pooling ? | 16:32 |
| flaper87 | kgriffs: yup, I was going to work on that before I forget | 16:32 |
| flaper87 | actually, a spec | 16:32 |
| flaper87 | :P | 16:32 |
| kgriffs | btw, I think we could get away with not adding a queue record unless it does not use the default flavor | 16:32 |
| kgriffs | and we could get away with not mapping queues to pools if there is only one pool defined | 16:33 |
| flaper87 | yup, pretty much as it works today | 16:33 |
| flaper87 | mmh, well | 16:33 |
| flaper87 | what happens if then I add another pool ? | 16:33 |
| kgriffs | oh, good point | 16:33 |
| kgriffs | :p | 16:33 |
| flaper87 | :P | 16:33 |
| kgriffs | premature optimization | 16:34 |
| kgriffs | let's just get it working, we can worry about employing some tricks after that | 16:34 |
| flaper87 | +! | 16:34 |
| flaper87 | +1 | 16:34 |
| flaper87 | can I haz a pop-tart? | 16:34 |
| openstackgerrit | Nataliia Uvarova proposed a change to openstack/marconi: Make storage/pooling reflect storage/base https://review.openstack.org/102457 | 16:34 |
| openstackgerrit | Nataliia Uvarova proposed a change to openstack/marconi: Run storage unit tests in pooled context https://review.openstack.org/105848 | 16:34 |
| flaper87 | AFAIK, you're the one that just got back from vacations | 16:35 |
| * kgriffs gives flaper87 a delicious Pop-Tart™ | 16:35 | |
| *** AAzza is now known as AAzza_afk | 16:46 | |
| *** amitgandhi has quit IRC | 16:57 | |
| *** amitgandhi has joined #openstack-marconi | 17:01 | |
| openstackgerrit | Malini Kamalambal proposed a change to openstack/marconi: Implement POP in v1.1 API https://review.openstack.org/90202 | 17:05 |
| *** kgriffs is now known as kgriffs|afk | 17:12 | |
| *** rossk has joined #openstack-marconi | 17:21 | |
| *** alcabrera|afk is now known as alcabrera | 17:21 | |
| peoplemerge | vkmc: flwang: you guys were talking about billing, I remember from Atlanta that Redhat released theirs. | 17:27 |
| *** Obulpathi has quit IRC | 17:32 | |
| *** oz_akan__ has joined #openstack-marconi | 17:33 | |
| peoplemerge | g'morning all | 17:34 |
| * peoplemerge is looking at review 104385 | 17:35 | |
| *** malini1 has joined #openstack-marconi | 17:37 | |
| flaper87 | peoplemerge: goood morning to you too | 17:38 |
| peoplemerge | flaper87: :) | 17:38 |
| *** malini has quit IRC | 17:41 | |
| *** oz_akan_ has quit IRC | 17:41 | |
| *** mpanetta has quit IRC | 17:41 | |
| *** rektide has quit IRC | 17:41 | |
| *** jay-atl has quit IRC | 17:41 | |
| *** ekarlso has quit IRC | 17:41 | |
| *** jay-atl has joined #openstack-marconi | 17:42 | |
| *** rektide has joined #openstack-marconi | 17:46 | |
| *** ekarlso has joined #openstack-marconi | 17:48 | |
| *** mpanetta has joined #openstack-marconi | 17:49 | |
| openstackgerrit | Dave Thomas proposed a change to openstack/marconi: Updated from global requirements https://review.openstack.org/104385 | 17:49 |
| *** reed has quit IRC | 17:52 | |
| *** randallburt has joined #openstack-marconi | 17:52 | |
| randallburt | exit | 17:53 |
| randallburt | lol | 17:53 |
| *** randallburt has left #openstack-marconi | 17:53 | |
| peoplemerge | that's a simple rebase, could use a extra set of eyes, tho. We don't have a policy to always merge and not rebase right? | 17:53 |
| peoplemerge | hm shoot, another patch landed | 17:56 |
| flaper87 | peoplemerge: if the patch is not going to conflict with yours, then that's fine | 17:56 |
| flaper87 | gerrit will take care of the rest | 17:56 |
| flaper87 | it normally lets you know as soon as one hits +A | 17:56 |
| peoplemerge | I didn't see patchset 3 when I rebased | 17:57 |
| *** AAzza_afk is now known as AAzza | 17:58 | |
| peoplemerge | flaper87: patch 2 couldn't automerge, complaining: 'Please rebase your change and upload...' | 18:00 |
| * peoplemerge is also slow this morning | 18:00 | |
| flaper87 | peoplemerge: ah ok | 18:00 |
| flaper87 | peoplemerge: btw, we normally leave the requirements thing to the infra bot | 18:00 |
| flaper87 | :P | 18:00 |
| flaper87 | It's an outsourcing era | 18:00 |
| peoplemerge | flaper87: fancy ! | 18:00 |
| peoplemerge | flaper87: so does that mean that this review can be left open indefinitely? | 18:01 |
| peoplemerge | ... or at least until j2? | 18:01 |
| flaper87 | yup | 18:01 |
| flaper87 | and it'll be updated by the bot itself | 18:02 |
| flaper87 | once it's updated and tests pass we'll approve it | 18:02 |
| flaper87 | if tests keep failing then we dig into why they're failing and fix things | 18:02 |
| malini1 | flaper87: https://review.openstack.org/#/c/90202/ review plzzzzz! | 18:05 |
| * flaper87 clicks | 18:05 | |
| *** Obulpathi has joined #openstack-marconi | 18:15 | |
| *** AAzza is now known as AAzza_afk | 18:42 | |
| *** kgriffs|afk is now known as kgriffs | 18:45 | |
| *** AAzza_afk is now known as AAzza | 18:54 | |
| *** reed has joined #openstack-marconi | 18:54 | |
| *** mwagner_lap has joined #openstack-marconi | 19:14 | |
| *** alcabrera is now known as alcabrera|afk | 19:18 | |
| kgriffs | AAzza, malini1, flaper87: seems like pools aren't getting tested, or we would have caught this a long time ago? | 19:23 |
| kgriffs | https://review.openstack.org/#/c/102457/4/marconi/queues/storage/pooling.py | 19:23 |
| malini1 | kgriffs: yes | 19:24 |
| malini1 | The tempest failure for tht patch has probably nothing to do with marconi | 19:25 |
| kgriffs | yeah, I was ignoring that tempest failure for the moment | 19:25 |
| kgriffs | i mean, take a look at this | 19:26 |
| kgriffs | https://github.com/openstack/marconi/blob/master/marconi/tests/queues/storage/base.py#L263 | 19:26 |
| kgriffs | seems like if that were calling the pooling message controller python would complain about a TypeError | 19:27 |
| kgriffs | here is MessageController.post (before that patch above) | 19:27 |
| kgriffs | def post(self, queue, project, messages, client_uuid) | 19:27 |
| *** malini1 has left #openstack-marconi | 19:28 | |
| kgriffs | https://gist.github.com/anonymous/351ac5ec3c5fa60469e8 | 19:30 |
| *** chandankumar has quit IRC | 19:30 | |
| openstackgerrit | Victoria Martínez de la Cruz proposed a change to openstack/marconi: Drop pylint due to the huge amount of false positives https://review.openstack.org/106272 | 19:38 |
| kgriffs | flaper87: doesn't look like we are testing pooling. I run this | 19:43 |
| kgriffs | MARCONI_TEST_MONGODB=1 tox -e py27 -- test_message_lifecycle | 19:43 |
| kgriffs | and it passes, which it shouldn't based on what I saw above. | 19:44 |
| kgriffs | Looking at https://github.com/openstack/marconi/tree/master/tests/unit/queues/storage | 19:45 |
| kgriffs | doesn't seem there is a test for a pooled config? | 19:45 |
| AAzza | kgriffs: the some tests were run for pooling, all on transport layer, no for storage itself | 19:45 |
| kgriffs | hmmm | 19:45 |
| AAzza | i tried to run them https://review.openstack.org/#/c/105848/ | 19:46 |
| AAzza | and that's how find all this incompatibilites | 19:46 |
| kgriffs | ah, that makes sense | 19:47 |
| kgriffs | hmmm | 19:47 |
| kgriffs | but I don't understand how the transport/functional tests were succeeding? | 19:48 |
| AAzza | i gues cause they call controllers with all keyword arguments, not relying on defaults or positions, but better to look in code | 19:48 |
| kgriffs | message_ids = self.message_controller.post( | 19:50 |
| kgriffs | queue_name, | 19:50 |
| kgriffs | messages=messages, | 19:50 |
| kgriffs | project=project_id, | 19:50 |
| kgriffs | client_uuid=client_uuid) | 19:50 |
| kgriffs | bingo, you are right | 19:50 |
| kgriffs | AAzza: nice catch! | 19:51 |
| AAzza | kgriffs: thanks) | 19:55 |
| AAzza | it's late, i'm going away to sleep | 19:56 |
| AAzza | good weekend to all) | 19:57 |
| kgriffs | take care! | 19:57 |
| *** mkoderer has quit IRC | 20:02 | |
| *** AAzza is now known as AAzza_afk | 20:05 | |
| *** cpallares has quit IRC | 20:14 | |
| *** fifieldt has quit IRC | 20:15 | |
| kgriffs | flwang: if you are around, this needs another +2 here: https://review.openstack.org/#/c/102457/ | 20:18 |
| kgriffs | flaper87: ^^^ | 20:18 |
| *** jay-atl has quit IRC | 20:20 | |
| openstackgerrit | A change was merged to openstack/marconi: Updated from global requirements https://review.openstack.org/104385 | 20:24 |
| openstackgerrit | A change was merged to openstack/marconi: Primary key for pool in catalogue table is unreasonable https://review.openstack.org/105645 | 20:24 |
| *** fifieldt has joined #openstack-marconi | 20:28 | |
| *** sriram has quit IRC | 20:35 | |
| peoplemerge | kgriffs: can you take a quick look at https://review.openstack.org/#/c/105830/ | 21:05 |
| peoplemerge | really just https://review.openstack.org/#/c/105830/1/marconi/tests/queues/transport/wsgi/v1_1/test_messages.py,unified | 21:06 |
| kgriffs | yeah, just a sec | 21:07 |
| peoplemerge | If it's on the right track I'll do the implementation | 21:07 |
| peoplemerge | kgriffs: thx | 21:07 |
| kgriffs | looking now... | 21:12 |
| *** Obulpathi has quit IRC | 21:15 | |
| * peoplemerge 's patch got spanked by pep8 | 21:15 | |
| kgriffs | heh, pep8 tends to do that. | 21:16 |
| kgriffs | takes a little getting used to | 21:17 |
| *** mpanetta has quit IRC | 21:17 | |
| peoplemerge | I also realize my commit message was suboptimal | 21:18 |
| *** malini has joined #openstack-marconi | 21:20 | |
| *** keith_newstadt has joined #openstack-marconi | 21:21 | |
| *** keith_newstadt has quit IRC | 21:25 | |
| *** amitgand_ has joined #openstack-marconi | 21:32 | |
| *** amitgandhi has quit IRC | 21:34 | |
| peoplemerge | I tossed around the idea of putting the packed message in code before deciding not to. | 21:37 |
| kgriffs | peoplemerge: ok, I submitted some inline comments | 21:45 |
| peoplemerge | thx, reading... | 21:45 |
| kgriffs | one thing I didn't comment on in gerrit was the overall testing strategy | 21:45 |
| kgriffs | I think there is a more elegant way we could do it | 21:46 |
| kgriffs | basically, have a base class and define a class variable to toggle whether to use json or msgpack | 21:46 |
| peoplemerge | you mean the if | 21:46 |
| peoplemerge | ah | 21:46 |
| peoplemerge | I was thinking the same thing | 21:46 |
| kgriffs | then, make the base class setup method call create_marconi_headers with that flag | 21:47 |
| kgriffs | and also helper methods could just use that class variable to switch | 21:48 |
| peoplemerge | or strategy | 21:48 |
| kgriffs | for example, _test_post could do that instead of making all the callers pass it in | 21:48 |
| peoplemerge | yes | 21:48 |
| kgriffs | that way, you mostly get msgpack testing for free - just inherit from the base class twice and override the class variable | 21:49 |
| kgriffs | anyway, I just suggest that because it's been used in other marconi tests | 21:49 |
| kgriffs | if you have an even better approach, that's cool too. | 21:49 |
| peoplemerge | I'll do that, I just wanted to get the simplest thing that illustrated the code paths. Thanks for your comments | 21:49 |
| kgriffs | sure thing | 21:50 |
| openstackgerrit | OpenStack Proposal Bot proposed a change to openstack/marconi: Updated from global requirements https://review.openstack.org/106506 | 21:50 |
| *** amitgand_ has quit IRC | 21:51 | |
| kgriffs | peoplemerge: btw, little thing I tested | 21:52 |
| kgriffs | msgpack.packb() is slower than creating Packer once | 21:53 |
| kgriffs | as in, if you call msgpack.packb() N times | 21:53 |
| kgriffs | vs. | 21:53 |
| kgriffs | packer = msgpack.Packer() | 21:53 |
| kgriffs | then | 21:53 |
| kgriffs | packer.pack() N times | 21:53 |
| kgriffs | the later is faster | 21:53 |
| kgriffs | my guess is, msgpack.packb instantiates packer behind the scenes each time, so by factoring that out you get a performance boost | 21:54 |
| peoplemerge | Yeah there's probably a lot mor object creation / GC | 21:54 |
| peoplemerge | agreed | 21:54 |
| kgriffs | btw, let me know what you figure out about needing to set encoding for Unpacker | 21:55 |
| peoplemerge | will do :) | 21:56 |
| kgriffs | seems like I had to do that to get unicode strings that were packed as such to decode back to the unicode type correctly or something | 21:56 |
| kgriffs | but i may be wrong, so it would be cool if you could look into that | 21:56 |
| peoplemerge | kgriffs: I'll check it out | 21:56 |
| kgriffs | msgpack didn't used to have unicode strings as a first-class type, so when that changed things got sort of ugly | 21:57 |
| kgriffs | add into that the way python 2 deals with strings vs python 3 and things get rather...interesting. :D | 21:58 |
| peoplemerge | I'll have fun with it :) | 21:58 |
| kgriffs | thanks man | 21:58 |
| kgriffs | I catch you monday | 21:58 |
| kgriffs | have a good weekend! | 21:58 |
| peoplemerge | you too | 21:58 |
| kgriffs | o/ | 21:58 |
| *** flwang_ has joined #openstack-marconi | 21:59 | |
| *** kgriffs is now known as kgriffs|afk | 21:59 | |
| *** malini has quit IRC | 22:00 | |
| *** whenry has quit IRC | 22:02 | |
| *** flwang_ has quit IRC | 22:04 | |
| *** oz_akan__ has quit IRC | 22:10 | |
| *** flwang_ has joined #openstack-marconi | 22:11 | |
| *** whenry has joined #openstack-marconi | 22:31 | |
| *** ametts has quit IRC | 22:45 | |
| *** reed has quit IRC | 23:02 | |
| *** flwang_ has quit IRC | 23:14 | |
| *** whenry has quit IRC | 23:22 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!