Monday, 2014-07-21

*** peopleme1ge has joined #openstack-marconi00:11
*** peoplemerge has quit IRC00:12
*** amitgandhi has joined #openstack-marconi00:33
*** amitgandhi has quit IRC00:37
wpfpeoplemerge_:  py33 works after tox --recreate00:48
*** oz_akan has joined #openstack-marconi00:52
*** oz_akan has quit IRC00:57
*** amitgandhi has joined #openstack-marconi01:34
*** amitgandhi has quit IRC01:38
*** renlt has joined #openstack-marconi01:49
*** oz_akan has joined #openstack-marconi02:07
*** renlt has quit IRC02:11
*** jeffreycoho has joined #openstack-marconi02:21
*** amitgandhi has joined #openstack-marconi02:34
*** amitgandhi has quit IRC02:39
*** wirehead_ has joined #openstack-marconi03:27
*** amitgandhi has joined #openstack-marconi03:35
*** oz_akan has quit IRC03:36
*** amitgandhi has quit IRC03:40
*** amitgandhi has joined #openstack-marconi04:36
*** amitgandhi has quit IRC04:40
*** chandankumar has joined #openstack-marconi04:50
*** chandankumar has quit IRC05:12
*** prashanthr_ has joined #openstack-marconi05:15
*** prashanthr_ has quit IRC05:19
*** prashanthr_ has joined #openstack-marconi05:21
*** amitgandhi has joined #openstack-marconi05:37
*** amitgandhi has quit IRC05:41
*** flaper87|afk is now known as flaper8705:46
*** flaper87 is now known as flaper87|afk06:12
*** k4n0 has joined #openstack-marconi06:16
*** chandankumar has joined #openstack-marconi06:34
*** amitgandhi has joined #openstack-marconi06:37
*** amitgandhi has quit IRC06:42
*** Catherine has joined #openstack-marconi06:54
*** Catherine has quit IRC06:59
*** chandankumar has quit IRC07:33
*** amitgandhi has joined #openstack-marconi07:38
*** amitgandhi has quit IRC07:43
*** prashanthr_ has quit IRC08:27
*** prashanthr_ has joined #openstack-marconi08:31
*** chandankumar has joined #openstack-marconi08:32
*** amitgandhi has joined #openstack-marconi08:39
*** amitgandhi has quit IRC08:43
*** amitgandhi has joined #openstack-marconi09:40
*** amitgandhi has quit IRC09:44
*** chandankumar has quit IRC09:46
*** chandankumar has joined #openstack-marconi10:10
*** chandankumar has quit IRC10:11
*** chandankumar has joined #openstack-marconi10:11
*** amitgandhi has joined #openstack-marconi10:41
*** amitgandhi has quit IRC10:45
*** mkoderer has joined #openstack-marconi10:47
*** malini has joined #openstack-marconi11:27
*** malini has quit IRC11:27
*** malini has joined #openstack-marconi11:28
*** mwagner_lap has quit IRC11:29
*** tongli has joined #openstack-marconi11:29
*** denis_makogon has joined #openstack-marconi11:32
*** amitgandhi has joined #openstack-marconi11:41
*** amitgandhi has quit IRC11:45
*** vkmc has joined #openstack-marconi11:49
*** vkmc has quit IRC11:49
*** vkmc has joined #openstack-marconi11:49
vkmcmorning all12:09
*** abettadapur has joined #openstack-marconi12:13
*** k4n0 has quit IRC12:28
prashanthr_vkmc: Good morning!12:34
prashanthr_thanks for the review12:34
vkmcprashanthr_, hey :)12:34
vkmcgood evening for you!12:35
vkmcnp, those are just style nits, no need to block the queue for that12:36
vkmcit looks great!12:36
prashanthr_anyways wil do it and submit it :)12:38
prashanthr_let all the reviews be in12:38
prashanthr_vkmc: Did the __init__ realignment12:41
prashanthr_great catch actually12:41
prashanthr_somehow i forget such small things12:41
*** amitgandhi has joined #openstack-marconi12:42
vkmcthanks prashanthr_!12:45
*** sriram has joined #openstack-marconi12:46
*** amitgandhi has quit IRC12:46
*** Catherin_ has joined #openstack-marconi12:59
*** mpanetta has joined #openstack-marconi13:00
*** Catherin_ has quit IRC13:00
*** Catherin_ has joined #openstack-marconi13:01
*** jmckind has joined #openstack-marconi13:06
*** mpanetta has quit IRC13:10
*** mpanetta_ has joined #openstack-marconi13:10
*** mpanetta has joined #openstack-marconi13:11
*** mpanetta_ has quit IRC13:11
*** mpanetta_ has joined #openstack-marconi13:13
*** mpanetta has quit IRC13:13
vkmcdarn python26 gate is messing with us13:14
*** keith_newstadt has joined #openstack-marconi13:22
*** mpanetta_ has quit IRC13:25
*** mpanetta has joined #openstack-marconi13:25
*** mpanetta has quit IRC13:26
*** mpanetta has joined #openstack-marconi13:27
*** oz_akan has joined #openstack-marconi13:28
*** mpanetta_ has joined #openstack-marconi13:31
*** mwagner_lap has joined #openstack-marconi13:33
*** Obulpathi has joined #openstack-marconi13:34
*** mpanetta has quit IRC13:34
abettadapurvkmc: I responded to some of your comments13:38
*** Catherin_ has left #openstack-marconi13:39
*** amitgandhi has joined #openstack-marconi13:43
*** ametts has joined #openstack-marconi13:43
*** jay-atl has joined #openstack-marconi13:43
*** amitgandhi has quit IRC13:57
*** itisit has joined #openstack-marconi13:59
peoplemerge_g'morning all!14:02
sriramgood morning :)14:07
*** amitgandhi has joined #openstack-marconi14:18
*** abettadapur has quit IRC14:19
mpanetta_morning :)14:30
*** mpanetta_ is now known as mpanetta14:30
malinimorning !!14:31
malinilooks like our gate is broken - all patches have an import error from falcon14:32
*** rwsu has joined #openstack-marconi14:32
*** kgriffs|afk is now known as kgriffs14:33
kgriffsmalini: yeah, I just saw that14:34
maliniit is just the 2.614:34
kgriffsthere is some bug in the way packages get installed, to make falcon think it is being installed with python 2.7 and not 2.614:34
kgriffsso it doesn't require ordered dict14:35
kgriffslet me see what I can do14:35
*** cpallares has joined #openstack-marconi14:40
*** prashanthr_1 has joined #openstack-marconi14:42
*** prashanthr_ has quit IRC14:43
prashanthr_1good morning all. Can you please review when free ?14:43
sriramwas digging around, Entropy does kinda look like notifications.
sriramsure prashanthr_1, will give it a go soon :)14:44
malinisure prashanthr_114:44
prashanthr_1thanks malini and sriram :)14:45
sriramObulpathi, malini : ^14:45
Obulpathilet me look into it14:45
Obulpathithanks sriram :)14:46
*** prashanthr_1 has quit IRC14:46
maliniWe should redirect the entropy Join Us to Marconi Join Us :-P14:46
sriramhaha lol :P14:46
*** AAzza has joined #openstack-marconi14:48
*** tonytan4ever has joined #openstack-marconi14:48
Obulpathiwow .. on the top level .. what they are doing and what we did for notifications is very similar ..14:52
Obulpathiexcept that use cases are different14:52
sriramyeah, thats why it peaked my interest.14:53
sriramwrong peaked..14:53
* sriram facepalm14:53
Obulpathilet me talk to them and see if we can have them collaborate with us14:53
Obulpathihahah .. lol sriram ..14:54
Obulpathithans buddy tough ... :)14:54
*** AAzza has quit IRC14:58
*** chandankumar has quit IRC15:00
*** openstackgerrit has joined #openstack-marconi15:03
*** tonytan4ever has quit IRC15:06
*** tonytan4ever has joined #openstack-marconi15:16
peoplemerge_malini: glad to hear it's not just my falcon15:39
peoplemerge_kgriffs: I made progress this weekend on msgpack15:39
peoplemerge_I want to get all of the inherited tests passing and hope to get POST method into review today15:40
*** shakamunyi has joined #openstack-marconi15:54
srirambbl lunch15:58
*** Obulpathi has quit IRC16:01
*** chandankumar has joined #openstack-marconi16:07
*** chandankumar has quit IRC16:19
openstackgerritOpenStack Proposal Bot proposed a change to openstack/marconi: Updated from global requirements
*** tonytan4ever has quit IRC16:24
* peoplemerge_ is leaving for the office16:25
*** chandankumar has joined #openstack-marconi16:33
*** celttechie has joined #openstack-marconi16:35
*** chandankumar has quit IRC16:39
*** kgriffs is now known as kgriffs|afk16:41
*** chandankumar has joined #openstack-marconi16:53
*** mkoderer has quit IRC17:02
*** abettadapur has joined #openstack-marconi17:08
*** cpallares has quit IRC17:14
*** Obulpathi has joined #openstack-marconi17:16
*** tonytan4ever has joined #openstack-marconi17:26
*** tonytan4ever has quit IRC17:31
*** kgriffs|afk is now known as kgriffs17:32
peopleme1gekgriffs: Remember our discussion on msgpack instance optimization? msgpack.packb()x5 is slower than packer = msgpack.Packer() + packer.pack()xn (obvious object creation/GC)17:36
peopleme1geI have something I'm not sure how to implement regarding the unpacker17:36
*** peopleme1ge is now known as peoplemerge17:40
*** kgriffs is now known as kgriffs|afk17:41
peoplemergeit can wait for review I guess17:42
peoplemergeI'm just not sure how to avoid instantiating the unpacker each time if I make it a singleton.  It might be possible to make a pool or attach an instance per wsgi thread.17:44
peoplemergeThat's a little beyond my abilities to produce, certainly in time for j2 close/j3 start.17:45
*** shakamunyi_ has joined #openstack-marconi17:45
peoplemerge(in the singleton case, we'd face race conditions with multiple threads feeding a single instance)17:46
*** shakamunyi has quit IRC17:49
*** Catherin_ has joined #openstack-marconi17:51
*** Catherin_ has quit IRC17:51
*** Catherin_ has joined #openstack-marconi17:52
*** Catherin_ is now known as Catherine_18:08
*** kgriffs|afk is now known as kgriffs18:18
*** vkmc has quit IRC18:24
*** vkmc has joined #openstack-marconi18:24
kgriffspeoplemerge: hey, just wanted to follow up on instantiating the unpacker18:29
kgriffswe don't have to worry about thread-safety IMHO, since people typically deploy python web services using multiple separate worker procs, and use green threads to multiplex I/O18:31
kgriffspeoplemerge: anyway, I would say, first make it work, then we will worry about making it fast. :)18:33
kgriffsjust use msgpack.unpackb for now, and put a TODO(peoplemerge) comment on there18:33
kgriffsthat's my $0.02 anyway18:34
*** Catherine_ has left #openstack-marconi18:36
*** tonytan4ever has joined #openstack-marconi18:38
vkmckgriffs, hey :) how are you? if you have a moment I would like your opinion about the some implementation details of the AMQP transport and the client18:45
vkmcs/the some/some18:46
kgriffssure. what's up?18:47
vkmcwell, in the case of the AMQP transport18:48
vkmcwithout entering in details, AMQP has a special type of message that allows it to be super flexible between implementations18:49
vkmcso...  we should respect it as much as possible18:49
*** itisit has quit IRC18:49
vkmcthing is, sounds like a good idea to store it marshalled in the storage backend?18:49
vkmcor do you think on a better way to save it so it's compatible with our API and also keep the AMQP message format?18:50
*** chandankumar has quit IRC18:52
*** chandankumar has joined #openstack-marconi18:53
kgriffsok, so let me make sure I understand18:54
kgriffsare you saying every message that comes in has the potential of including a bunch of metadata or something that we need to treat a opaque in terms of our backend message store?18:56
vkmc(sorry if I'm not giving you enough context, I don't want to flood you with details about AMQP)18:56
vkmccurrently we are handling messages in the storage backend considering a couple of parameters, the body and the ttl18:58
*** denis_makogon has quit IRC18:59
vkmcthe transport pass an iterable list of messages to the storage with those params so the storage process them accordingly18:59
vkmcin AMQP we receive an AMQP message, with its own attributes18:59
kgriffsok. BTW, how are your dealing with TTL19:00
*** alcabrera|afk is now known as alcabrera19:00
vkmcright now... I just setted a default ttl19:01
vkmcbut I could use AMQP message ttl directly19:01
vkmc(although it could be not initialized)19:02
kgriffsok, so the question is, how to preserve those AMQP-specific message attributes?19:02
kgriffsso, supposing we were to just msgpack.packb all that and put it in the body field19:04
kgriffsso, client A submits a message via AMQP19:04
kgriffsthen client H retrieves the message using HTTP19:04
kgriffswhat would happen then?19:05
vkmcgood point19:05
vkmcit wouldn't work19:05
kgriffsvkmc: I'm assuming an AMQP message has a well-defined "payload" part, separate from the other attributes?19:06
vkmcmy first approach was to 'parse them'19:06
vkmcin order to avoid that situation19:06
vkmcyes it does19:06
*** vkmc has left #openstack-marconi19:09
*** vkmc has joined #openstack-marconi19:09
vkmcbut if we parse them we don't know for sure which attributes has an assigned value... so we can store them in the backend19:11
*** alcabrera is now known as alcabrera|afk19:14
kgriffssorry, just a sec19:16
vkmcsure :)19:17
*** mwagner_lap has quit IRC19:18
kgriffstrying something new19:20
kgriffsso, is this picture kinda sorta what we are after?19:21
kgriffsbasically, store all attributes that were inserted19:22
kgriffsonly give back basic attributes for HTTP requests19:22
kgriffsbut return "extended" attrs as well for AMQP requests19:22
vkmcthat's exactly the situation yeah19:24
kgriffswhat do you think about adding an "extended attributes" field to our message store schema19:24
kgriffsthen you can msgpack whatever, slam it into that field19:24
kgriffsprobably should namespace it19:25
vkmcthat could work19:25
kgriffs{"amqp": { ... } } ===> extended_attrs or meta or something19:25
vkmcI'm only worried about what will happen when an AMQP retrieves that message19:25
vkmcit should get an AMQP message19:25
kgriffswhat else do you need in order to reconstruct the original AMQP message?19:26
vkmcwell, basically... this are the properties we should preserve in order to reconstruct it for a consumer
vkmcin the Message class, id, user_id, subject, reply_to, ...19:29
kgriffsso, if we just marshal everything to the body field in the messag estore19:30
kgriffsthe HTTP API would return a body that is like19:30
vkmcit would return an instance of the Message object19:31
kgriffs{"subject": "cats vs. dogs",  "body": ...}19:32
kgriffswhereas a message POSTed with HTTP would be19:32
kgriffs(the body directly)19:33
kgriffsI feel like we should normalize things in the message store19:33
kgriffsso that body is body, regardless of the transport19:33
kgriffsI wonder if it would be interesting to expose those other attributes to HTTP clients too19:34
vkmcwell, the message has a content-type19:36
vkmcit looks good19:37
kgriffsso, assume that we add a field called extattrs or something19:39
* kgriffs is channelling file system design19:39
kgriffsor just19:39
kgriffsthen, as I said before, you can drop namespaced extended attributes in there19:40
kgriffsdifferent transports can expose those differently19:40
vkmcyeah... some extra field to storage everything that is just for AMQP (in this case)19:40
kgriffsfor HTTP, we may not allow you to POST them, but we will return them in a listing using a well-known algorithm19:40
kgriffsbasically, by taking each top-level key in xattrs and putting that in as a top-level key in the message document that is returned to the client19:41
kgriffsso if you have this for xattrs:19:41
kgriffstry again19:41
kgriffs{"proton": { ... }, "susans_cool_plugin_thing": {...}}19:42
kgriffsyou would get a message doc like19:42
kgriffs{"id": "50b68a50d6f5b8c8a7c62b01", "ttl": 120, "age": 53, "proton": {...}, "susans_cool_plugin_thing": {...}, "body": {...}}19:43
vkmcit makes sense19:44
vkmcthat if you perform a GET with an HTTP client19:44
vkmcbut if you do the same with an AMQP client, we should reconstruct the Proton message with the attributes in proton: { ... }19:44
vkmcI like it :)19:46
kgriffsI wonder if we should use some dotted convention or something for the key names19:46
kgriffsprobably YAGNI19:46
kgriffsare those attributes standardized in AMQP 1.0 ?19:48
vkmcyes, Proton follows the AMQP 1.0 standard strictly19:50
vkmcso, we could use those attribute names for Marconi19:50
kgriffsI'm just trying to figure out whether we need to define a schema for the xattr key names19:53
kgriffs{"amqp": {...}}19:54
*** chandankumar has quit IRC19:56
kgriffs{"vnd.amqp-v1.0": {...}}19:57
*** whenry has joined #openstack-marconi19:57
kgriffs{"amqp-v1.0": {...}}19:57
kgriffsI think we should for sure include a version19:58
vkmcI prefer the latter19:59
kgriffsok, let's run with this20:00
kgriffs(the latter)20:00
vkmcI don't know if it's better if we leave the creation of new attributes for third-party drivers20:00
vkmcor if we provide a general, unique one20:00
kgriffswell, by definition, we can't anticipate what 3rd-party drivers will want/need20:01
vkmcyes... that's true20:01
kgriffsso, best to provide a naming guideline and then some kind of "registry" of types to avoid name collisions20:01
kgriffskind of like the way internet media types work20:01
vkmcmakes sense20:02
kgriffsvkmc: we can start a wiki page for this right now, but we will need a documentation blueprint for creating a "3rd-party driver guide" that also mentioned this20:02
kgriffsin other words20:02
vkmcI started a wiki for AMQP
vkmcI could mention about this and link it to a future 3rd party driver guide20:03
kgriffswill you start a wiki page that is a general 3rd-party driver dev guide, and create a blueprint for moving that over to the user docs at some point?20:03
kgriffsyou can link to that from the home page20:03
vkmcsure, I could do that :)20:03
kgriffsexcellent, thanks!20:03
kgriffswe should get our Redis d00d to contribute to that page too20:04
vkmcthanks for guided brainstorming, as always20:04
vkmcsure, I'll ping prashanthr20:04
vkmcand a one extra question kgriffs20:07
vkmcregarding the client20:07
vkmcright now we have CRUD for queues20:07
vkmcand I added CRUD for messages20:07
vkmcdoes it make sense to add CRUD for claims?20:08
kgriffsoh, you mean command line client, not the library20:08
vkmcin the spec you didn't mentioned it so I wasn't sure
kgriffsyeah, I'd say anything the API supports, you should be able to do from the CLI20:09
kgriffsthanks for working on the client, BTW!20:09
vkmcnp! it's fun20:09
kgriffsI'd love to get a queue "tail" command, by the way. :D20:09
vkmcnoted :)20:10
kgriffswith regex highlighting as a bonus20:10
kgriffsconsole colors FTW!20:10
kgriffssrsly, though, that is super helpful when diagnosing a problem in a distributed system20:11
kgriffswe have a tool like that that we use for Marconi's predecessor, used in Rackspace Cloud Backup20:11
kgriffsI probably submitted a bp on it... aaaaanyway20:11
kgriffskeep on rockin'20:12
kgriffsone more thing20:12
vkmcyeah I can't imagine that... I haven't worked with such huge deployments but going through logs can be tedious20:13
vkmcin any scale20:13
kgriffsso, on the subject of marshaling20:14
kgriffsI have been wanted for a long time to migrate the mongodb driver to store bodies as snappy-compressed msgpacked blobs20:15
kgriffsin the sqlalchemy driver, this isn't such a concern since that driver is just for development20:15
kgriffsso, JSON is fine20:15
kgriffsnow, we are going to add another opaque field, xattrs20:15
kgriffsI guess there are a few questions to "unpack" here. ;)20:16
kgriffsactually... hmmm20:16
kgriffsfirst thing to do is design the interface20:17
kgriffshow will the storage base class be modified to support arbitrary xattrs20:17
vkmchmm, so by storing blobs we don't care too much about the actual content of the message20:17
kgriffswhere, a single xattr is defined as name: value pair20:17
vkmcright now we only support strings, right?20:17
kgriffslet's see.20:18
kgriffsright now everything is JSON20:18
kgriffsso we have only coded with JSON types in mind20:18
kgriffshowever, we are adding msgpack support to the HTTP driver20:18
kgriffsso storage drivers have to allow for all msgpack types20:18
kgriffswell, actually, the driver has to allow for whatever msgpack types translate to in Python when they are deserialized20:19
kgriffsin SQL we JSON-encode the body before storing it20:20
kgriffspeoplemerge may have to change that as part of his msgpack work20:20
kgriffsin the mongo driver we just try to write the dict, and as long as BSON supports it, we are cool20:21
kgriffsso, I suspect msgpack --> dict --> bson will work OK20:21
kgriffsso that is the situation with how we store the body today20:22
kgriffsnow we need to store xattrs20:22
kgriffshow shall we go about that?20:22
*** alcabrera|afk is now known as alcabrera20:22
vkmcif we add attributes on demand, like an amqp-v1.0 attr, then we should be able to manage them for different clients20:24
vkmcwhen a HTTP client request a message with that attribute, it should ignore it and get only what it requires... the body, ttl and age20:25
vkmcbut if an AMQP client request a message with that attribute, Marconi should reconstruct a Proton message using those attributes accordingly and the client should get that message20:26
kgriffswe could also just dump out all the message attributes as I mentioned before, to HTTP. Not sure how useful that is for clients to see20:26
kgriffsmay be helpful for diagnostics?20:26
kgriffsare any of those xattrs something that can't be expressed as JSON?20:27
kgriffsI guess that is the rub20:28
kgriffsoh crumbs20:28
kgriffsjust thought of something20:28
vkmcin this case... yeah, could be useful as extra information but HTTP cannot do anything with it20:28
kgriffsif someone posts binary payload via msgpack or amqp, and you want to retrieve using JSON... we will have to detect that and base64-encode I guess20:28
kgriffspeoplemerge: ^^^20:29
*** mpanetta has quit IRC20:29
vkmcthat's why I thought about marshalling20:32
vkmcthe body at least20:32
kgriffsAMQP body can be binary, right?20:38
*** abettadapur has quit IRC20:38
*** sriram has quit IRC20:40
vkmckgriffs, yeah it can20:41
vkmchere is the type system spec if you wanna check it out
kgriffsok, I am playing with pymongo to see how it handles binary strings20:45
kgriffsso, this fails (just tried it)20:51{'thing': b'\x81\x82\x83\x84\x85'})20:51
kgriffsthis worked20:52{'thing': bson.Binary(b'\x81\x82\x83\x84\x85')})20:52
vkmcso we have to check if the payload is binary20:53
kgriffsbut then when you do a find20:53
kgriffsyou get20:53
kgriffs{u'_id': ObjectId('53cd7d7911cfa134e670dde9'),20:53
kgriffs u'thing': Binary('\x81\x82\x83\x84\x85', 0)}20:53
kgriffsassume that is assigned to a var called "doc"20:54
kgriffsjson.dumps(doc) crashes and burns20:54
kgriffslooks like Binary class implements __str__20:55
kgriffsso json calls str on this thing20:55
kgriffswhich yields '\x81\x82\x83\x84\x85'20:55
kgriffsand then json chokes because it can't convert to utf-820:55
kgriffsI think we will have to mention this in the driver dev guide20:57
vkmcor make mongo store msgpack20:58
vkmcor that's not an option?20:58
* vkmc got confused20:58
kgriffsyeah, there are several related things we need to sort out20:59
vkmcI wonder if Redis can store binary21:00
vkmcif so, we can enforce choosing Redis if AMQP is enabled21:00
kgriffsI'd rather not couple the transport with the backend if we can avoid it21:01
vkmcnot the best option but it could work just until we migrate to msgpack or we find another solution21:01
kgriffslet's walk through this21:01
kgriffsfirst, http21:01
kgriffsactually, http+json21:01
kgriffssomeone submits a JSON document. We decode it, assuming all the strings are UTF-8 encoded.21:02
kgriffsnow we have a python dict21:03
kgriffsdoes that dict contain unicode strings?21:03
kgriffslooks like we do21:04
kgriffsstrutils.safe_decode(, 'utf-8')21:04
kgriffsok, that gives a six.text_type string21:05
kgriffsthen we do json.loads21:05
kgriffslooks like that decodes to a dict which contains u'' stirngs21:06
kgriffsOK, so everything that is submitted using JSON via HTTP will be persisted as u'' strings21:07
kgriffsso then pymongo will UTF-8 encode those and store as BSON strings21:07
kgriffssqlalchemy driver will dump to a JSON string, utf-8 encoded21:08
kgriffsnow, let's walk through what happens with http+msgpack21:08
kgriffsmsgpack document comes in21:09
kgriffswe deserialize that to a dict21:09
kgriffsthat dict may contain a mixture of b'' and u''21:09
kgriffslet me verify21:09
kgriffsyeah, so it will contain ttl and body fields. the value of body may be binary, or one of it's sub-fields may be.21:12
vkmcbinary or unicode21:12
kgriffswell, technically they could both be unicode - unicode has multiple encodings, utf-16, utf-8, etc.21:12
kgriffsbut let's say they use the binary msgpack field type21:13
kgriffsthen it should unpack to str in py2 and binary in py321:13
kgriffsspecifying the binary msgpack type when the client encodes is important21:14
kgriffsm = msgpack.packb(value, use_bin_type=True)21:14
kgriffsotherwise it is stored using a byte array "string"21:14
*** cpallares has joined #openstack-marconi21:14
kgriffsand when decoding, you don't know whether to decode as a binary string or decode from utf-8 to utf-1621:15
kgriffsin any case, assuming the client did flag bin fields as such21:15
kgriffsthen we decode to a b'' string in py221:15
kgriffsbut then pymongo is going to barf unless we wrap with Binary21:15
kgriffsI don't want to have to crawl the dict looking for strings to wrap21:16
kgriffs(I'm assuming this is only necessary on py2, but still boooh)21:16
vkmcno... it's too expensive21:16
kgriffsman, peoplemerge really should be here21:16
kgriffswe will have to have him read today's log. :)21:17
kgriffsBUT, if we just tell msgpack21:18
kgriffsmsgpack.packb(value, use_bin_type=True)21:18
vkmcpeoplemerge, peoplemerge_ ^^ (all the pings!)21:18
kgriffsthen it will store b'' as-is, and utf-8 encode six.text_type21:18
kgriffsso that would solve storing binary strings21:18
kgriffsbut I think we have a problem getting those out the other end21:19
kgriffslet's walk through that next21:19
kgriffsno, suppose we store the body using msgpack21:19
kgriffsalong comes a client21:20
kgriffssays "gimme some 'a that there JSON'21:20
kgriffsso we read the mspack blob from storage21:20
kgriffsdeserialize to JSON21:20
kgriffsactually, sorry21:21
kgriffsnot JSON21:21
kgriffsfirst, a python dict21:21
kgriffsthat dict will have some b'this-is-not-utf-8'21:21
kgriffsso, we can't simply encode that to JSON, because our JSON must only contain valid UTF-8 bytes21:22
kgriffsat this point, we don't have any information that tells us that the original string was binary21:23
kgriffsso... what do we do?21:23
*** shakamunyi_ has quit IRC21:24
vkmcwe should add some kind of flag21:24
vkmcwhen storing that message21:24
vkmcso when the retrieval happens we know how to reconstruct it21:24
vkmcthat could be extra information in the xattrs field21:25
kgriffsin the case of AMQP, that would work. The body is a blob21:26
vkmceach driver has to make sure that adds the required information in xattrs in order to be able to retrieve that information later21:26
kgriffshowever, if the message was submitted with msgpack, we could end up with a mixed scenario21:26
kgriffsmsgpack comes in, we deserialize, and ideally we have utf-8 encoded strings turn into six.text_type21:27
kgriffsand client specified binary fields turn into six.binary_type21:28
*** Obulpathi has quit IRC21:29
kgriffsthen when we serialize back to msgpack for storing, those two field types (both of which may be present in the body) will be flagged as binary and string types, respectively, in the msgpack format21:29
kgriffsbut here is what has be concerned21:29
kgriffswe may be ok21:29
kgriffsa unicode string we be output back as six.text_type I think21:30
kgriffsand a binary will be six.binary_type21:30
kgriffsnow if we received these via JSON21:30
kgriffsthen serialize to msgpack, they will be str type21:30
kgriffsand when we get them back out they will only have six.text_type strings21:30
kgriffs(need to verify)21:31
flwanghey guys, sorry for the rush in21:31
kgriffsso... we could say, any field that is six.binary_type will need to be base64-encoded if the client only accepts JSON21:31
flwangso what's the meeting time of this week?21:31
*** amitgandhi has quit IRC21:32
vkmcflwang, it looks like we will stick one more time to the regular time21:32
flwangyep, I saw kgriffs's mail, just wanna confirm21:33
vkmcoh cool21:33
flwangvkmc: thanks :)21:33
vkmcflwang, np!21:33
flwangkgriffs: the deadline of j-2 is 24 Jul, right?21:33
kgriffsI'd like to get as much as possible merged a day early21:34
flwangkgriffs: I assume my /health bp is on your list :D21:34
kgriffsyeah... we are going to have to slip a few things, but let's get as much done as we can21:35
flwangbtw, I have addressed all your comments21:35
kgriffscool, I'll take another look21:35
flwangpls revisit it when you're most convenience, cheers21:35
kgriffssure thing21:35
kgriffsvkmc: re unicode madness21:35
kgriffsnote for posterity21:36
kgriffsIn [62]: msgpack.unpackb(m, encoding='utf-8')21:36
kgriffsOut[62]: {u'thing': u'thing2', 'value': '\x81\x82\x83\x84\x85'}21:36
kgriffssetting the encoding is important21:36
kgriffsotherwise you get21:36
kgriffsIn [60]: msgpack.unpackb(m)21:36
kgriffsOut[60]: {'thing': 'thing2', 'value': '\x81\x82\x83\x84\x85'}21:36
kgriffsthat was assuming m came from:21:37
kgriffsIn [58]: m = msgpack.packb({u'thing':u'thing2', 'value': value}, use_bin_type=True)21:37
kgriffsvalue = b'\x81\x82\x83\x84\x85'21:37
kgriffsok, so here is the thing that really stinks21:37
kgriffshow do we efficiently base64-encode stuff on the response?21:37
kgriffscan we register our own serializer for six.binary_type?21:38
*** celttechie has quit IRC21:39
*** jmckind has quit IRC21:39
vkmcdoesn't look like :/21:39
vkmcit should work for py2 y py321:47
kgriffsyes, but we need a way to b64 encode instances of six.binary_type when serializing to JSON21:47
kgriffsin the case of AMQP we can have any of a number of content types for the body21:49
kgriffslet's see21:50
kgriffssuppose we add a content-type field to messages21:50
vkmcwe can have primitive types or more complex structures in the payload21:50
vkmcAMQP messages has a content-type fields we can use21:51
* kgriffs is thinking21:52
*** oz_akan has quit IRC21:55
*** tonytan4ever has quit IRC21:55
kgriffsvkmc: are those AMQP content types unique to AMQP or are they internet media types?21:55
vkmckgriffs, internet media types21:57
kgriffscan you include encoding21:58
vkmcI'm checking the RFC21:58
kgriffs"application/xml; encoding=utf-16"21:58
vkmcthis one
vkmcit only mentions text/plain or text/enriched21:58
kgriffsoops, I mean21:59
kgriffs"application/xml; charset=utf-16"21:59
vkmcthe content type of AMQP is a MIME content type21:59
vkmcand yeah, there is another field21:59
kgriffslots of RFCs22:00
kgriffsvkmc: oic....22:00
peoplemergekgriffs: vkmc: just checked scrollback, I think we're good on all those cases22:01
kgriffsso, it has been proposed in the past by a couple different people to not use JSON at all and move stuff like TTL into X-headers22:01
kgriffsbut then it becomes a pain when you want to get more than one message at a time22:01
peoplemergeWe unpack msgpack on reciept so it's just a dict to store to mongo etc after22:02
peoplemergeeverything is set to msgpck binary22:02
*** rwsu has quit IRC22:02
*** keith_newstadt has quit IRC22:02
* peoplemerge is in meetings, be back this eve22:03
kgriffspeoplemerge: the issue is that after you unpack and send to pymongo, pymongo will barf on six.binary_type if they contain non-unicode chars22:03
kgriffsso I think we need to change the mongo driver to store as msgpack rather than BSON22:04
kgriffsthere isn't any reason for mongo to "see" the body fields in any case (they can be opaque)22:04
peoplemergekgriffs: as part of this story22:04
kgriffspeoplemerge: it will have to be, otherwise you can't get tests that post binary field types in msgpack to marconi22:05
kgriffspeoplemerge: I've actually been wanting to do this for a while anyway22:05
peoplemergekgriffs: ok22:05
peoplemergecan't mongo intuit json structs for querying tho?22:05
kgriffsbecause it lets us also use lz4 or snappy22:05
peoplemergeinteresting, hadn't thought of that22:06
kgriffspeoplemerge: not automatically22:06
kgriffsyou have to wrap the string in a bson.Binary object22:06
peoplemergeah nice22:06
kgriffsbut... then you have to take the pain to detect what fields to wrap22:06
kgriffsand it seems like we might as well just persist back to msgpack22:07
kgriffsthen wrap that whole mspack in a single bson.Binary()22:07
peoplemergeI must got to this meeting but if anyone wants to play with msgpack POST and my ugly unrefactored code instructions
peoplemergeit works tho :)22:07
kgriffsah, thanks for sharing22:08
vkmcthanks peoplemerge!22:10
vkmckgriffs, would you like to start a discussion about this in the ml?22:11
vkmcoops phone, brb again22:14
*** mwagner_lap has joined #openstack-marconi22:31
*** keith_newstadt has joined #openstack-marconi22:34
*** Obulpathi has joined #openstack-marconi22:47
*** Obulpathi has quit IRC22:52
*** Obulpathi has joined #openstack-marconi22:53
*** shakamunyi_ has joined #openstack-marconi22:53
*** Obulpathi has quit IRC22:55
*** ametts has quit IRC22:56
*** Catherine has joined #openstack-marconi22:56
peoplemergekgriffs: So sorry about that!22:57
* peoplemerge is bac22:57
*** Catherine has quit IRC23:01
kgriffsyeah, we need to discuss this with the team. I will try to write up some specs and put them out for discussion.23:06
kgriffstwo things I am pretty sure about though23:07
kgriffs1. we need to store stuff as msgpack rather than JSON/BSON23:07
kgriffs2. we should store the content type with the message (regardless of whether we expose that to the clients in message listings)23:08
kgriffsby content type, I mean the type that the body was received as (json, msgpack, or in the case of amqp, whatever the content_type field is)23:08
kgriffsit's getting late for me...23:09
kgriffsso I will have to work on this more tomorrow23:09
kgriffsI started to write up this bp:23:09
vkmcthanks kgriffs23:10
peoplemergekgriffs: deadline for j2 is here, should I attempt to finish what's on the translation layer today / tomorrow?23:10
kgriffsI think we will need another bp or something for storing content_type with the message as well - or that may be part of an existing bp, not sure yet23:10
peoplemergej3 starts wed, no?23:10
vkmcyeah peoplemerge!23:10
kgriffspeoplemerge: mmm, I think we will need to slip this to j3 to get it done right23:10
kgriffs(to be honest)23:10
kgriffsbecause I think it will depend on the bp I linked above23:11
kgriffsbut we need to try and get all v1.1 stuff that slips done at the front end of j323:11
peoplemergekgriffs: reading store-message-as-binary bp23:11
peoplemergethere's lots of work to do on python-marconiclient23:12
peoplemergeI didn't know where to start becasuse there is no 1.1 there23:12
kgriffsyep, there sure is a lot of work to do23:12
peoplemergeflaper87 commented on that last night23:13
kgriffsthat's another reason we need to get 1.1 done ASAP on the server side, so we can update the client and test against the new API23:13
peoplemergekgriffs: the bp makes sense to me.  I assume we'll want compression be something admin-tunable, right?  So if they're already sending compressed data, anything we try to do will make it worse23:16
kgriffsthat is a good point.23:21
kgriffsquestion is, what is the overhead for lz4 or snappy23:22
kgriffsin two cases23:22
kgriffsa base64-encoded jpeg23:22
kgriffsand a raw binary string with the same jpeg23:23
kgriffsideally, it would just be "always on"23:23
kgriffsor, if we know the content type, we could be smart about it... we could check magic bytes at the beginning of the string23:23
kgriffsmaybe this is something that should wait for K23:23
peoplemergecompression is just a small part of the bp23:24
peoplemergeDo people really use messaging for large binaries like images and vid?23:26
peoplemergeWhen we architected Verifi's batch platform, I argued against gi-normous binaries in a mq23:27
peoplemergekgriffs: always-on might be a better idea, or at least a tunable queue-level thing you can turn off23:29
kgriffspeoplemerge: well, what do you know, I lz4'd a jpg23:29
peoplemergekgriffs: did it get smaller or larger?23:29
kgriffsit actually got smaller23:29
peoplemergetry doing it with an h.265 vid :)23:29
kgriffsand for the record I agree with you re embedding binary objects23:29
kgriffsgenerally to be avoided23:30
kgriffsand I don't think it is the most common use case for messaging23:30
kgriffsbefore size: 1912623:30
kgriffsafter: 1812823:30
peoplemergeit's not nothing :)23:31
kgriffstook 15 microseconds to do it23:31
kgriffs(on my MBP)23:31
peoplemergeIMO a domain specific *good* algo will outsmart a general purpose one23:31
kgriffsyep, for sure23:32
*** cpallares has quit IRC23:32
peoplemergebut in messaging do we care?23:32
*** shakamunyi_ has quit IRC23:33
kgriffsidk, maybe. We'd have to do some benchmarks, also insert tons of messages see how much we can fit in a reasonable amount of RAM w/o compression23:33
peoplemergeA wise man once said "thou shalt not build hopelessly general apps for thine unknown intended market"23:33
kgriffsthat's why I'm starting to think this should go in it's own bp... needs some experimentation23:33
kgriffsOK, I updated the description on the bp to note that we won't do compression at this time, but the bp does pave the way for it if we want to do it later23:36
peoplemergeversion field++23:37
kgriffsflwang: ping23:45
flwangkgriffs: pong23:49
kgriffsflwang: hey, I just wanted to let you know I'm reviewing the health patch23:50
flwangmy baby is waiting for bless23:50
kgriffsoh, no problem23:50
kgriffson the case23:50
flwangkgriffs: I mean my baby /health patch :D23:50
kgriffsat first you had me there, then I realized what you meant23:50
flwanghope we can make it in j-223:51
kgriffsI was at church yesterday and we were talking about baby blessings for real. So I guess i had it on my mind. :p23:51
flwangthen in j-3, we can fix bugs and adding more KPI23:51
flwangkgriffs: haha, wow23:51
kgriffsso far I just found a little IMO, but I don't think it should block the patch23:51
flwangkgriffs: sure, I can fix it right now and give you a new patch set, sir23:52
peoplemergehaha love it!23:52
kgriffsflwang: I like the _handle_status23:54
flwangit is what it looks based on the valuable comments from you guys :)23:56

Generated by 2.14.0 by Marius Gedminas - find it at!