Monday, 2013-10-14

*** vkmc has joined #openstack-marconi00:10
*** fifieldt_ has joined #openstack-marconi00:24
*** vkmc has quit IRC00:39
*** oz_akan_ has joined #openstack-marconi00:44
*** oz_akan_ has quit IRC01:06
*** oz_akan_ has joined #openstack-marconi01:06
*** oz_akan_ has joined #openstack-marconi01:07
*** oz_akan_ has quit IRC01:13
*** oz_akan_ has joined #openstack-marconi01:20
*** oz_akan_ has quit IRC02:27
*** oz_akan_ has joined #openstack-marconi02:28
*** oz_akan_ has quit IRC02:32
*** oz_akan_ has joined #openstack-marconi02:41
*** oz_akan_ has quit IRC02:43
*** oz_akan_ has joined #openstack-marconi03:54
*** oz_akan_ has quit IRC03:58
*** dafter has joined #openstack-marconi05:34
*** dafter has quit IRC05:40
*** dafter has joined #openstack-marconi05:41
*** dafter has quit IRC05:45
*** ykaplan has joined #openstack-marconi06:40
*** dafter has joined #openstack-marconi07:46
*** ykaplan has quit IRC08:04
*** yassine has joined #openstack-marconi08:14
openstackgerritFlavio Percoco proposed a change to openstack/marconi: 'Persist' __getattr__ results
openstackgerritFlavio Percoco proposed a change to openstack/marconi: Use stevedore instead of importutils
openstackgerritFlavio Percoco proposed a change to openstack/marconi: Return a consumer function instead of consuming
*** ykaplan has joined #openstack-marconi08:38
*** ykaplan has quit IRC09:03
*** ykaplan has joined #openstack-marconi10:38
*** dafter has quit IRC11:04
*** tedross has joined #openstack-marconi11:18
*** fifieldt_ has quit IRC11:21
*** dafter has joined #openstack-marconi12:06
*** dafter has quit IRC12:12
*** tvb|afk has joined #openstack-marconi12:14
*** tvb|afk has quit IRC12:14
*** tvb|afk has joined #openstack-marconi12:14
*** tedross has quit IRC12:23
*** malini_afk is now known as malini12:33
*** mpanetta has joined #openstack-marconi12:40
*** tvb|afk has quit IRC12:41
*** tedross has joined #openstack-marconi12:41
*** mpanetta has quit IRC12:42
*** mpanetta has joined #openstack-marconi12:42
*** dafter has joined #openstack-marconi12:44
*** dafter has quit IRC12:44
*** dafter has joined #openstack-marconi12:44
*** oz_akan_ has joined #openstack-marconi13:05
*** oz_akan_ has quit IRC13:07
*** oz_akan_ has joined #openstack-marconi13:08
*** rustlebee is now known as russellb13:10
*** amitgandhi has joined #openstack-marconi13:26
*** amitgandhi has quit IRC13:26
*** amitgandhi has joined #openstack-marconi13:27
*** vkmc has joined #openstack-marconi13:27
*** kgriffs_afk is now known as kgriffs13:38
*** kgriffs is now known as kgriffs_afk13:40
*** ametts has joined #openstack-marconi13:44
*** jcru has joined #openstack-marconi14:08
*** alcabrera has joined #openstack-marconi14:21
*** dafter has quit IRC14:33
*** dafter has joined #openstack-marconi14:38
*** tvb|afk has joined #openstack-marconi14:38
*** tvb|afk has quit IRC14:38
*** tvb|afk has joined #openstack-marconi14:38
*** dafter has quit IRC14:38
*** kgriffs_afk is now known as kgriffs14:41
kgriffsalcabrera: morning!14:41
alcabrerakgriffs: morning! :)14:41
kgriffsI kicked over an ant pile when I started working on getting rid of global conf14:42
alcabreraoh? :x14:42
alcabreraWhat'd you find that cause a stir?14:42
kgriffswell, it turns out there are tons of things that break when you try to do that14:42
kgriffsso, I am putting that it it's own patch14:42
kgriffsand then the sharding will depend on it14:43
zyuankgriffs: i remembered there was a negative ttl problem reported to cloudqueues@, is there any further information?14:43
kgriffszyuan: ah, that one's been fixed14:43
kgriffsalcabrera: so, i am getting close to having that done, stand by14:44
alcabrerakgriffs: yikes. well, the separate patch approach works for me, esp. considering the scope of the change.14:44
alcabreraMarconi meeting in an hour, kgriffs? :P14:44
* alcabrera goes to check agenda14:44
kgriffsoh, yeah. forgot to send out the email again14:44
alcabrerait was a frantic week, this last one. Seems like one of us should start sending out that email Fridays.14:45
kgriffsyeah. Fridays would be good14:45
kgriffsanyway, yeah, we are meeting14:46
alcabreraheh. :P14:47
zyuankgriffs: which patch?14:47
*** tvb|afk has quit IRC14:50
kgriffsalcabrera: just registered this bug, fwiw -
alcabrerakgriffs: thanks! There's this other very related one here:
alcabreraactually... the title is 90% identical. :P14:53
alcabreraMaybe dup?14:53
alcabreraSince you're working on eliminating the global config, I'm going to merge the two tickets into yours and leave you assigned to it.14:54
kgriffsoh, ok14:54
kgriffsI saw that one but it was specific to proxy and the proxie's future is TBD14:55
kgriffsnot sure if it is worth merging yet14:55
*** flaper87|afk is now known as flaper8714:57
alcabrerakgriffs: fair enough.14:57
alcabreraflaper87: o/14:57
alcabrerakgriffs: I updated the title instead (prefix: [queues])14:57
zyuankgriffs: not this one, this one fixes negative age. serveral days ago, someone mentioned there is a negative ttl15:01
kgriffshmm, I didn't hear about that15:02
kgriffsis there a bug registered?15:02
*** yassine has quit IRC15:03
*** tedross has quit IRC15:03
alcabrerakgriffs, zyuan: there is no open bug registered for negative TTL/age at this time.15:04
kgriffsmalini: ping15:05
flaper87yo yo yooooo15:10
flaper87how are y'all doiiiing???15:11
zyuankgriffs: ah ok, it is the one you fixed15:11
alcabreraflaper87: alright. How'd PyconIE go? (is it still going?) :)15:11
flaper87it went pretty well, loved it! The talk was good as well,at least that's what people said :D15:13
* flaper87 talked about the new partition stuff15:14
flaper87folks were very happy about it15:14
*** yassine has joined #openstack-marconi15:16
flaper87I'm about to jump on my flight back, I guess I'll miss the meeting this week15:17
flaper87did you guys saw my comments on gerrit?15:17
flaper87(and the new patches) "D15:17
alcabreraflaper87: yup, I've been checking them out. :)15:17
flaper87kgriffs: btw, if you get a chance, could you go through the client queue ?15:18
kgriffsyeah, sorry I've been behind on that15:18
kgriffshopefully later today15:18
flaper87alcabrera: awesome, thanks a lot!15:18
alcabreraflaper87: cached_getattr is cool. Nice decorator!15:18
kgriffsI am getting rid of global config in queues at the moment15:19
flaper87I'll be back full-time tomorrow15:19
kgriffsit's blocking the sharding work15:19
*** yassine has quit IRC15:19
flaper87alcabrera: :D15:19
*** yassine has joined #openstack-marconi15:19
alcabreraflaper87: looking forward to your return, but glad to hear the trip has been going well. :D15:19
flaper87kgriffs: awesome, that will definitely help with the global config patch15:19
flaper87yeah! It's my first time in Dublin, very cool city15:20
flaper87I don't like the wheather, though. :P15:20
flaper87but it seems that it's like 'the best it gets' here15:20
*** tedross has joined #openstack-marconi15:20
flaper87just like london!15:20
flaper87but london gets better, though15:20
kgriffsI'm thinking maybe we skip today's meeting15:23
kgriffsThere is a bunch of stuff on the agenda that I'd like flaper87 here for15:23
kgriffsand the remaining stuff would make for a very short meeting15:24
alcabrerakgriffs: sounds fair. I'm double-checking the agenda to be sure.15:24
amettsOkay with me.15:24
flaper87okey with me! Sorry for not being around today!15:24
alcabreraaction review, bug update, graduation status, API feedback, API versioning strategy, open discussion15:24
alcabreraSo of those items, I wouldn't want flaper87 to miss out on graduation/API discussions. :P15:25
alcabreraPlus, we can start making it a routine to announce our meeting on the ML every Friday.15:26
kgriffsI'll do that15:26
alcabrerakgriffs: my thoughts - I'm in favor of skipping this meeting.15:26
kgriffsanyway, I'll pop into #openstack-meeting-alt and let folks know15:26
alcabreracool, thanks!15:27
flaper87awesome, thanks guys, A LOT!15:27
malinisorry just got back from the appt15:31
maliniWe had a negative TTL bug (for message & claims) a whole back tht alcabrera fixed15:31
maliniThe other bugs which kgriffs fixed recently is the negative age for messages from the stats endpoint15:32
alcabreramalini: o/15:32
kgriffs# Maximum Content-Length allowed for metadata updating and15:34
kgriffs# message posting.15:34
kgriffs;metadata_max_length = 6553615:34
kgriffs;content_max_length = 26214415:34
kgriffsit looks like if I have, say, queue metadata with whitespace in it, i could exceed 64 K15:35
kgriffsand the request would fail15:35
kgriffs…even though15:36
kgriffswe say they can have up to 64 K compacted (without whitespace)15:36
zyuankgriffs: wait15:36
kgriffsam I misunderstanding?15:36
kgriffs        # Place JSON size restriction before parsing15:37
kgriffs        if req.content_length > CFG.metadata_max_length:15:37
zyuanwhat's the problem?15:37
kgriffsthe problem is that we aren't making good on our promise15:37
kgriffsat least, with the default settings15:37
zyuanthose are just content_length limits15:38
kgriffsyes, but if they are both 64 K15:38
kgriffsso, whitespace isn't really ignored, in effect15:38
zyuanit is ignored if you condigured the limit to ignore to a smaller value15:39
zyuanit's just not observable when the configuration is 64k/64k15:39
alcabrerakgriffs: I'm of the opinion that we should take back our promise on compaction, since that necessarily implies that somewhere in the processing pipeline, we'd be doing some form of compaction.15:40
alcabreraOn whitespace.15:40
kgriffsalcabrera: we are15:40
kgriffsthe validator compacts the json before checking length15:41
alcabrerakgriffs: ah15:41
zyuanstorage don;t receive request15:41
kgriffswe thought it would be easier for the user15:41
kgriffseasier/more predictable15:41
kgriffssay I set the uplimit to 1115:41
alcabrera"length = _compact_json_length(metadata)"15:41
alcabreraI see it now15:41
kgriffsand max content length to 1115:42
kgriffsand I pass {'thing':1}15:42
kgriffsthat works fine15:42
kgriffs{'thing': 1}15:42
kgriffswould fail15:42
kgriffsthat's all I'm saying15:42
zyuanif you do want to observe the effect of checking15:42
zyuanset them to different values15:42
zyuanthe fact is15:42
kgriffsthat is the whole point15:42
kgriffsof the compaction15:42
zyuandon't change *_length15:43
*** dafter has joined #openstack-marconi15:43
zyuanuser are not expected to change *_max_length15:43
* kgriffs is confused15:43
kgriffswhy not15:43
kgriffsand for that matter, what is the point of _max_length?15:43
zyuanthe limit of content_length is not so useful to user15:43
zyuanit's just a defender, that's it15:43
* alcabrera just learned about the separators option in json.dumps15:44
kgriffsthe web server already has limits like that15:44
zyuanthen why you want to change it?15:44
kgriffswhat do we gain by doing it in our app?15:44
kgriffsi mean, the content length limits?15:44
alcabrerakgriffs: good point15:45
zyuanbecause you said web server may not have it15:45
* alcabrera does some research15:45
zyuanand marconi should do it by itself15:45
kgriffsif I did, I mispoke15:45
alcabrera (nginx)15:45
kgriffsevery real web server restricts body size15:45
zyuanand i agree it's ok to do it byitself15:45
alcabrera (Microsoft IIS)15:45
zyuanbecause it looks wired if you need to configure things in two places15:46
zyuanmarconi don't know how you configured your web server15:46
zyuanand don't know the limit15:46
kgriffsI think we should just let the web server do it's thing, and in the app we only check compact json, since that has storage ramifications, and the webserver isn't smart enough to parse the json15:47
zyuanto take advantage of not to performan the whitespace checking15:47
zyuani decided to inline the content_length checking15:47
*** flaper87 is now known as flaper87|afk15:47
zyuankgriffs: my suggestion is we can keep this config var15:47
kgriffsalcabrera: does json schema act on the raw text?15:47
zyuanbut perform no checking15:48
zyuanjust ask user to fill those config var with what their server have15:48
zyuansounds better?15:48
alcabrerakgriffs: lemme double check that jsonschema behavior15:48
alcabrera (Apache httpd)15:48
kgriffsi don't see any point in requiring the operator to configure that in two places15:48
zyuankgriffs: otherwise you have to perform real whitespace checking in deployment15:49
kgriffsbasically, we need to verify JSON, and we need to verify length of compact json for the user. Fudging it is a bad UX15:49
zyuani need an approach to workaround it15:49
kgriffsI was under the impression we were doing real checking already15:49
zyuankgriffs: no15:50
zyuankgriffs: not in deployment, i belive15:50
zyuanthis is deployment-dependent15:50
kgriffsthen what we are telling users and what we are checking doesn't match15:50
alcabrerakgriffs: jsonschema works off of the already-decoded JSON, so... dict objects.15:51
zyuanthis behavior is not observable, so we kept the promise15:51
kgriffsit is observable15:51
kgriffssee my example above15:51
alcabrerajsonschema.validate(schema: dict, data: dict)15:51
zyuanno, it's not, it violated one of the limit15:51
kgriffsa user will submit metadata with whitespace, expecting it to be valid, because we promise that we will strip whitespace before checking the limit15:52
kgriffswe check it against content length and return 40015:52
kgriffsso, we are violating the contract15:52
kgriffsor am I missing something?15:52
zyuansetting _max_length is equivilient to set it in server15:52
zyuanso user should understand this as "my request is totally wrong and not reached marconi yet"15:53
zyuanuser violated the precondition first15:53
zyuanso we just done the right thing15:53
kgriffs_max_length would need to be large enough to account for potential whitespace difference between the two checks15:53
zyuankgriffs: so you need to set it to a large enough value!15:54
zyuannot 1115:54
zyuanfor example, 64K15:54
kgriffsthe default config it is equivalent, and it isn't obvious that this is the case15:54
kgriffsand in any case, we are duplicating what the web server already does for us15:54
zyuankgriffs: we can alwasy change the other value15:54
zyuankgriffs: i agree, but i also want to say we'd better to keep this value15:55
kgriffsI'm still not convinced we need _max_length at all15:55
kgriffsit's redundant15:55
alcabreragiven that IIS, apache httpd, and nginx all support content-length limitation, I'm in agreement, kgriffs.15:55
kgriffs(since the server already checks this for us)15:55
zyuankgriffs: i know! but how marconi get server setting?15:55
kgriffsmarconi doesn't care15:55
zyuankgriffs: i has to15:56
zyuanit has to15:56
zyuanto skip the deserilization checking at all15:56
zyuanwithout breaking any promise15:56
kgriffsfirst of all, I think it is safe to assume that the server's max content length will always be greater than marconi's15:57
zyuanthey shall be same15:57
zyuanthey'd better be same15:57
kgriffsi mean15:57
kgriffsgreater than marconi's uplimit15:57
zyuanuplimit is fine, different15:57
kgriffsassume we remove marconi's metadata_max_length15:57
zyuanmax_length shall be as same as server setting15:58
kgriffsI'm saying, let's remove it15:58
zyuanif we remove it, then for each request we need to do deserialization15:58
kgriffsand also assume that server's max length is like 2x metadata_size_uplimit or whatever15:58
zyuankgriffs: *_uplimit is fine, it can be anyX15:59
alcabrerakgriffs: that kind of assumption should be documented in the *yet to be born* marconi operator's manual.15:59
kgriffszyuan: for each request that is within a sane limit, yes we need to deserialize15:59
kgriffsthat happens anyway15:59
alcabreraIt's going to be fun working on marconi docs when we get to that point. :)15:59
zyuanserver's max length and metadata_size_uplimit are just not related, i actually don't know that you mean15:59
kgriffshere's the thing16:00
kgriffsright now there is16:00
kgriffsif the raw, unparsed json exceeds that length, we return 40016:01
*** whenry has joined #openstack-marconi16:01
kgriffsotherwise we keep going16:01
kgriffsnext we check compact json16:01
kgriffsbut only16:01
kgriffsif metadata_size_uplimit < metadata_max_length16:01
kgriffsso, assuming that the operator would always set metadata_max_length to something greater than metadata_size_uplimit16:02
kgriffs(to account for possible whitespace)16:02
zyuanit's ok, and it's right, just slow16:02
kgriffsthat short-circuit, in practice, will never happen16:03
zyuanit must16:03
zyuanSQS does it16:03
kgriffsif we are always skipping the compact JSON check then there is not point in having it in the first place16:03
zyuankgriffs: there is a point, i want this behavior to be opauqe16:04
zyuanoh, you mean, first place16:04
zyuanok. i don't know.16:04
kgriffsseems like we either need to decide to take the hit and validate this for real, or tell users that we validate with whitespace included16:05
*** alcabrera has quit IRC16:06
zyuani just provided things work for both case16:06
zyuan(it sounds like sophistry but i belive that's key point of design...)16:08
*** dafter has quit IRC16:08
kgriffs%timeit _compact_json_length({})16:09
kgriffs10.6 us16:09
*** reed has joined #openstack-marconi16:09
*** alcabrera has joined #openstack-marconi16:09
kgriffslet me think on this. that is pretty slow. we either need to speed it up or abandon the idea altogether16:09
alcabreraback - internet had a disconnect. :P16:10
* alcabrera catches up16:10
zyuanand there is always an idea around which to use a much faster and native and (fill whatever you want) json parser16:10
kgriffsyeah, the original discussion is coming back to me16:13
*** yassine has quit IRC16:13
kgriffsthe main slowdown was reserializing the body of each message16:13
*** ykaplan has quit IRC16:15
kgriffsbasically, we would need a way to very quickly check a subdocument's length, with whitespace stripped16:17
kgriffswhether we compact or not, we need to verify the size of each message body16:19
zyuankgriffs: i want to ask a question16:21
zyuanif i send a piece of JSON in UTF-816:22
zyuanwhich contains { "key": "this is one unicode char represented in 3 bytes" }16:22
zyuanwhat's the expected "docoment length with whitespace stripped"?16:23
zyuan{"key": "1"} or {"key": "xxx"}?16:24
kgriffsthe spec says "charactes" not "bytes"16:24
kgriffsso we should be counting chars I guess16:24
kgriffsor we can change the spec16:25
zyuani'm ok with chars16:25
zyuanbecause "whitespace" only make sense on chars16:25
kgriffslooks like _compact_json_length is counting chars16:25
zyuankgriffs: yes16:25
zyuani ask this just for measure how hard to write a size-only parser.16:26
alcabrera```len(filter(lambda x: x not in string.whitespace, document)``` is crazy fast on Python 3 (~300 ns), and about 3us on Python 2.716:27
alcabrerafor a small document, I should say.16:28
zyuanalcabrera: but obviously that does not work because JSON string can also contains whitespace16:28
kgriffsif we could avoid even parsing messages['body'] in the first place, keeping it a blob, then we could just strip whitespace and do len()16:29
zyuankgriffs: a more realistic problem16:30
zyuanwhat is the length of : 3.1416?16:30
zyuanthe storage presure of 3.1416 is as same as .016:31
kgriffscome on, I bet we could contribute a patch to simplejson to support that16:31
kgriffswhere's your sense of adventure?16:31
zyuankgriffs: i have a feeling that "size without whitespace" have different meaning to different usecase, can the term "size" itself can also have different meanings16:32
kgriffswell, we have already established that "size" means "number of characters"16:32
kgriffswhether or not we strip whitespace, we still need to validate number of characters for each body16:33
zyuanbut as i shown it does not mean too much for storage16:33
kgriffsunless we want to tell users something different than we've been telling them16:33
zyuani agree it's quite clear to JSON16:34
kgriffsthought experiment16:34
kgriffswhat if we were to allow application/msgpack16:34
kgriffs(assuming that were a real media type16:34
zyuanmsgpack has only 1 representation16:34
kgriffsnow, we end up defining limits differently16:35
zyuanso, no problem16:35
zyuanthat's true16:35
kgriffsi think it is ok to define limits based on media type16:35
kgriffsthe main thing is the user knows what to expect16:35
zyuanhowever, things become tricky because16:35
zyuanmsgpack size = representation size = bytes16:35
zyuanjson size (in our definition) != json chars sent by user != bytes16:36
zyuanJSON has two more layers here, when comparing to msgpack16:37
zyuanone for encoding (utf-8), one for whitespace16:37
zyuanand if storage comes in, things become furtherly tricky16:38
zyuandoes mongo support insert BSON directly?16:39
kgriffsmongo only speaks bson, so I assume you mean pymongo16:39
zyuan(who cares the server api)16:40
zyuananyway, it simplfies mongo but not for others...16:41
zyuanneed to revisit the sizing concept.  if needed, i'm ok with doing anything in C/C++16:48
kgriffsOK. I think the best bet would be to have a JSON validator that can check the size of subdocuments16:49
zyuan(simplejson the one we are using is also in C)16:49
zyuankgriffs: hard to say at this point of time16:50
alcabreraInvalidDocument: BSON document too large (1000000098 bytes) - the connected server supports BSON document sizes up to 16777216 bytes  - given a large enough project-id16:51
kgriffswe have three options16:51
kgriffs1. verify size before serialization16:51
kgriffs2. verify size after serialization, which would require re-serialization16:51
kgriffs3. don't serialize the body in the first place16:51
kgriffsnumber 3 will cause us problems if we ever want to support non-JSON media types16:52
kgriffsunless we tell users that body has to be a string blob16:52
kgriffsbut that sucks16:52
kgriffsif body is structured as the user submits it to us, we need to give it back to them structured16:52
kgriffs2. is what we are doing now16:53
zyuani have one more option in mind16:53
kgriffsI'm all ears16:54
zyuanfirst, define size differently, for each json types16:54
zyuanfor example, json string size as just len(s)16:54
zyuandouble size is 816:54
zyuanas well as integer16:54
zyuannull size is 816:54
zyuanbool, maybe 1?16:55
zyuanlike that16:55
zyuanthen, work through the parsed python dict, sum up the size16:55
zyuanthat's it16:55
zyuanbenefits: 1. close to what db stores16:56
zyuan2. close to user expectation16:56
zyuan3. easy to implement. slow in cpython i blieve, but ok for pypy. pypy's raw loops are fine.16:56
kgriffsI guess the con is that it is ony "close" to user expectations16:57
zyuanwhen i say "user expectation" i mean, JSON cares each chars16:57
zyuanbut user don't!16:57
zyuanif i;m user, i hope server don't care the floating point precision of my dicoment16:57
zyuanthat makes no sense16:58
zyuan3.1415345347 should be regarded to have the same size as .016:58
kgriffsthat can be confusing to some users16:58
zyuani'm a numeric anaysis user16:58
kgriffssome users would be ok, but some would be confused16:59
kgriffsthat being said16:59
kgriffsif you weren't actually emitting a serialized document string16:59
zyuanwhen you design something significant, there will be someone confused (said by Bjarne LOL)16:59
kgriffscould you make counting faster but still be accurate?16:59
kgriffsyou still have to str the numbers17:00
zyuanyou mean..?17:00
kgriffsbut bool and everything else you could just count17:00
zyuannumbers don't len17:00
zyuanthey are just constant, size 817:01
kgriffsyour would have to len(str(number))17:01
kgriffsi bet you could make it faster than simplejson, but I don't know if it would be significant17:01
zyuanwhy? in my design above they are always 817:01
kgriffsno, I mean, not having always 817:01
kgriffsbut actually counting the real text length of the number when serialized17:02
zyuanwhat i'm arguing is that counting by char is flawed17:02
zyuanit's meaningful to JSON17:02
zyuanbut JSON is just a rich representation17:02
kgriffsin the user's mind, JSON is the context17:03
zyuanand user usually care the meaning17:03
kgriffswe have to pick some neutral format, and that is JSON at the moment17:03
zyuanI don't agree. JSON is just an appraoch to represent meaning17:04
zyuanand it's an intermediate one (not close to representation in either way, JSON bytes, or DB storage)17:04
zyuanwhile my suggestion give it meaning by value17:04
zyuanand key17:05
kgriffsbut then the user will always have to be calculating in their head how many bytes they have left after serializing from their apps internal data structure to JSON17:05
kgriffssince they can't just look at the json and tell17:05
zyuankgriffs: they don't need to calc17:05
kgriffsnow they have to think about the sizes that marconi has "reserved" for each JSON type17:05
zyuanthey need to calc if we do chars counting17:06
kgriffsI guarantee you are going to get a lot of push back on this idea from users17:06
zyuanfor example17:06
kgriffsbut the calc is much simpler and more intuitive17:06
zyuan"\uaaaa" is size 817:06
zyuanin our current policy17:06
zyuanbut with my suggestion17:06
zyuanit's 317:06
zyuansame as you do s.length in our client17:07
zyuanwhatever we see in client, we have it in server17:07
zyuanno plain17:07
zyuanotherwise, you need to do repr(e) in client to get size17:07
zyuanwhich is, awful17:07
kgriffshow is "\uaaaa" is size 817:08
kgriffsif that is a single unicode char, it will count as "1"17:08
zyuanto JSON, each char counts17:08
zyuani mean, "\uaaaa" in JSON, not python17:08
kgriffsok, I see your point17:08
zyuanone more example17:08
zyuanif a user read a csv file17:09
zyuanhow he measure the size?17:09
kgriffsbut having to account for some unicode strs in my length calculation seems a lot easier than having to add up size for all the different types based on marconi's arbitrary definition of such17:09
zyuanhere is what the next thing i want to show17:09
zyuanfor scalars other than string, user may not care about the size of scalars at all!17:10
kgriffsmarconi will be deserializing17:10
zyuanif you read a csv file17:10
kgriffsnevermind my last comment17:10
zyuanyou know 1. the file size (which making no sense)17:10
kgriffszyuan: what if they are sending a huge batch of numbers17:11
zyuan2. each value's repr (which is hard to marsure)17:11
kgriffsi suppose then they will have to take array.length * marconi_size_per_float17:11
zyuanbut user know one thing: how many number i have in total17:11
zyuana client jsut need to limit on "how many numbers", which is they matrix size, that's it!17:11
zyuanforget marconi_size_per_float17:12
zyuanit's just ok to define an max matrix size in client17:12
zyuanwithout looking at marconi_size_per_float17:12
kgriffshow do I know it will fit in a single message17:13
zyuanfor example, 100 x 10017:13
zyuana client degsiner only need to test several times, then pick a number17:13
kgriffstrial by error?17:13
zyuanlike 1000 * 1000? opps, failed17:14
kgriffsseems hacky to me17:14
zyuan800 x 800 ok17:14
zyuanthen defined it as 800 x 80017:14
kgriffsand nobody knows the real limits of marconi17:14
zyuanit's client implementer's job, not user's17:14
kgriffsthe client implementor is the user17:14
zyuanso it's not quite trial by error17:14
zyuanuser only know 80017:14
kgriffsor at least, *a* user17:14
zyuanhe/she don't need to know whether the number comes from17:15
zyuanthe final interface is consist by marconi the server and client17:15
zyuanthey are not seperatale17:15
zyuanor we can say, marconi_size_per_float is something libarary implementro need to look at17:16
zyuanin any appraoch, i don't want user to care and calculate the limits.17:16
zyuanwhat they should care is how much content they have.17:17
zyuanlike matrix size, string length, etc.17:17
kgriffsthe unfortunate thing is that the user will care as soon as they get back a 400 the first time. Then they want to know *why* and then we have to tell them17:17
kgriffsand if our explaination is "well, it depends"17:18
zyuanthey should not. a domain-specific client should be alle to detect it before sending to marconi.17:18
kgriffs"you have to take into account the sizes of all these different types, blah blah"17:18
kgriffspeople are going to complain that it is "too confusing" and "not predictable"17:18
zyuanask their library implementers17:18
kgriffsnot all clients are domain-specific17:18
kgriffsyou have the generic go library17:19
kgriffssome startup uses it17:19
kgriffshits the limit17:19
kgriffsand then calls support because they are confused17:19
zyuanlastword: this algorithms is easy to implement everwhere17:19
zyuanwe can even add it to a gneric client!17:19
kgriffswell sure. we would need to run this by client implementors then to see what they think17:20
zyuani find it's reasonable, since it also solves msgpack17:20
zyuanbecause msgpack, if have the same content (dict), they get same size, and same to that JSON have17:21
zyuan... see you after lunch17:21
zyuanforgot to say, we can define marconi_size_per_float to 1, so that less confusion17:22
zyuaneach string has len(s), each scalar is 117:23
zyuanthat shows my "size as content size" better, i think17:23
*** ametts has quit IRC17:28
*** jdaggett1 has joined #openstack-marconi17:33
kgriffsmalini or alcabrera: can you verify a couple bugs?17:33
kgriffsthe second one you can see if you make content_max_length larger than message_size_uplimit17:34
kgriffsThose appear to be bugs based on looking at the code alone, but i need someone to actually test and confirm17:35
maliniI am looking at it now17:35
alcabrerakgriffs: heh, I definitely wouldn't have seen the second one, since I wipe my marconi-queues.conf of limits. :P17:35
alcabreramalini: thanks!17:35
maliniiirc we decided to go this route because as long as individual_msg_size is within limit, their sum would be too17:40
kgriffsthat doesn't sound right17:41
kgriffsif I have two messages, each with a body that is 256 KB, then the overall limit would be exceeded, even though the individual limit is OK17:41
maliniBut overall limit >= individual_msg_size_max * max_allowed_msg_count17:43
kgriffscurrently we do two things17:44
kgriffswe first check whether the entire body of the request exceeds some maximum17:45
kgriffsthat will be going away, since it is redundant with what the web server does17:45
kgriffs(web server already specifies a max request body size)17:45
kgriffsthen we go and we check the size of each message body17:45
kgriffs(not counting the envelope)17:46
kgriffs(just counting the body)17:46
kgriffsif each body is <= the limit (256 KB by default) we say OK17:46
kgriffsbut we never check whether the sum of the length of each body is greater than the overall limit17:47
maliniiow, this is not being done now 'the total size for all message bodies included in a single request is limited as well to 256 KiB by default (configurable)' ?17:47
kgriffsthe problem is, the content_length limit got conflagrated with the sum_of_bodies_limit (which setting doesn't actually exist, btw)17:47
*** fvollero is now known as fvollero|gone17:47
kgriffsthat is the bug17:48
kgriffsat least, I think it is a bug but haven't tried it17:48
kgriffsto test this, you would have to set content_max_length17:48
kgriffsto something big, like 1 MB17:48
maliniSo if I post messages with 2 messages (240KB) each, I can get a repro17:48
kgriffssince at the moment it is being (incorrectly) overloaded to mean "sum of all body lengths"17:49
kgriffsmalini: yeah, but you need to up that setting I just mentioned, since otherwise the body validation check is short-circuited17:49
maliniaah..ok..I'll add a test for this17:50
*** jdaggett2 has joined #openstack-marconi17:53
*** jdaggett1 has quit IRC17:56
*** jdaggett2 has quit IRC18:10
*** kgriffs is now known as kgriffs_afk18:13
*** vkmc has quit IRC18:15
zyuankgriffs_afk: ping18:28
*** kgriffs_afk is now known as kgriffs18:32
*** jdaggett1 has joined #openstack-marconi18:41
zyuankgriffs: how you repro the first bug?18:42
*** ametts has joined #openstack-marconi18:46
*** jdaggett1 has quit IRC18:50
*** kgriffs is now known as kgriffs_afk18:57
*** alcabrera has quit IRC19:00
*** kgriffs_afk is now known as kgriffs19:07
*** jdaggett1 has joined #openstack-marconi19:16
*** ayoung has joined #openstack-marconi19:38
*** jdaggett1 has quit IRC20:44
*** jdaggett1 has joined #openstack-marconi20:45
*** amitgandhi1 has joined #openstack-marconi20:50
*** amitgandhi has quit IRC20:51
*** malini is now known as malini_afk20:52
*** malini_afk is now known as malini20:52
*** amitgandhi1 has quit IRC20:56
*** amitgandhi has joined #openstack-marconi20:57
openstackgerritKurt Griffiths proposed a change to openstack/marconi: fix(queues): Global config used everywhere
openstackgerritKurt Griffiths proposed a change to openstack/marconi: feat: Storage sharding foundation
openstackgerritKurt Griffiths proposed a change to openstack/marconi: Setup storage pipeline in the boostrap instead of driver base
openstackgerritKurt Griffiths proposed a change to openstack/marconi: chore: Remove GC cruft from storage driver base class
*** malini is now known as malini_afk21:03
zyuanas a Chinese, i've never been to HongKong.  BTW, Chinese people need another passer to go HongKong, with a maximum stay time of 7 days -- I've heard of that HongKong is a part of China?21:19
*** mpanetta has quit IRC21:23
zyuankgriffs: more comments added
*** oz_akan_ has quit IRC21:26
*** oz_akan_ has joined #openstack-marconi21:26
openstackgerritKurt Griffiths proposed a change to openstack/marconi: fix(queues): Global config used everywhere
*** jdaggett1 has quit IRC21:28
*** jdaggett1 has joined #openstack-marconi21:29
*** oz_akan_ has quit IRC21:31
*** fifieldt_ has joined #openstack-marconi21:32
*** jdaggett1 has quit IRC21:33
openstackgerritKurt Griffiths proposed a change to openstack/marconi: feat: Storage sharding foundation
*** amitgandhi1 has joined #openstack-marconi21:35
*** amitgandhi2 has joined #openstack-marconi21:35
*** amitgandhi1 has quit IRC21:35
*** amitgandhi has quit IRC21:38
openstackgerritA change was merged to openstack/python-marconiclient: Complete refactor of
zyuanagain i found that python need a switch statement...21:50
openstackgerritA change was merged to openstack/python-marconiclient: Split get_transport into 2 different functions
*** kgriffs is now known as kgriffs_afk22:22
*** amitgandhi2 has quit IRC22:30
*** oz_akan_ has joined #openstack-marconi22:37
*** oz_akan_ has quit IRC22:41
*** kgriffs_afk is now known as kgriffs22:59
*** jcru has quit IRC23:09
*** kgriffs is now known as kgriffs_afk23:12
*** kgriffs_afk is now known as kgriffs23:14
*** kgriffs is now known as kgriffs_afk23:24

Generated by 2.14.0 by Marius Gedminas - find it at!