Wednesday, 2013-10-02

*** malini_afk is now known as malini00:28
*** oz_akan_ has joined #openstack-marconi00:45
*** oz_akan_ has quit IRC00:46
*** oz_akan_ has joined #openstack-marconi00:47
*** nosnos has joined #openstack-marconi00:49
*** amitgandhi has joined #openstack-marconi01:29
*** malini is now known as malini_afk01:34
*** amitgandhi has quit IRC02:00
*** whenry has quit IRC03:26
*** oz_akan_ has quit IRC03:43
*** oz_akan_ has joined #openstack-marconi04:54
*** oz_akan_ has quit IRC04:59
*** tvb|afk has joined #openstack-marconi05:25
*** tvb|afk has quit IRC05:30
*** tvb|afk has joined #openstack-marconi05:54
*** tvb|afk_ has joined #openstack-marconi06:16
*** tvb|afk has quit IRC06:20
*** tvb|afk_ is now known as tvb06:31
*** tvb has quit IRC06:31
*** tvb has joined #openstack-marconi06:31
*** flaper87|afk is now known as flaper8707:20
*** flaper87 is now known as flaper87|afk07:43
*** yassine has joined #openstack-marconi07:48
*** flaper87|afk is now known as flaper8707:59
*** flaper87 is now known as flaper87|afk08:09
*** flaper87|afk is now known as flaper8708:12
*** tvb has quit IRC08:26
*** flaper87 is now known as flaper87|afk08:28
*** flaper87|afk is now known as flaper8708:39
*** ykaplan has joined #openstack-marconi08:54
*** ykaplan has quit IRC09:17
*** tvb has joined #openstack-marconi09:27
*** flaper87 is now known as flaper87|afk09:31
*** tvb has quit IRC09:31
*** flaper87|afk is now known as flaper8709:42
*** ykaplan has joined #openstack-marconi09:50
*** tvb has joined #openstack-marconi10:28
*** tvb has quit IRC10:32
*** ykaplan has quit IRC10:33
*** ykaplan has joined #openstack-marconi10:48
*** ykaplan has quit IRC11:11
*** nosnos has quit IRC11:16
*** oz_akan_ has joined #openstack-marconi11:20
*** oz_akan_ has quit IRC11:22
*** tvb has joined #openstack-marconi11:23
*** ykaplan has joined #openstack-marconi11:32
*** flaper87 is now known as flaper87|afk11:53
*** tedross has joined #openstack-marconi11:59
*** fvollero-pto is now known as fvollero12:08
*** flaper87|afk is now known as flaper8712:17
*** kgriffs_afk is now known as kgriffs12:27
kgriffsflaper87: did u see my comments on this?
kgriffsoh, and good afternoon!12:29
flaper87kgriffs: gooood morning :)12:30
flaper87kgriffs: I saw them today but I've been tackling down some internal stuff today, I'll get to them in a bit12:30
flaper87I think the previous patch is ready, though12:30
kgriffsnp, just checking12:31
flaper87kgriffs: also, Gerrit is way to stupid and hides -1s under +2s12:31
kgriffstotally lame12:33
*** kgriffs is now known as kgriffs_afk12:33
openstackgerritA change was merged to openstack/python-marconiclient: Add a base transport class
*** kgriffs_afk is now known as kgriffs12:51
*** acabrera has joined #openstack-marconi12:53
kgriffsflaper87: also, it would be great if you could review Alej's latest proxy patches today. I am planning to do so as well12:53
flaper87kgriffs: will do, before working on other patches12:54
kgriffsrock on12:54
acabreraMorning! :)12:55
acabreraI get here and proxies are being mentioned. My timing is fantastic. ;)12:55
acabreraflaper87, kgriffs: Thanks!12:55
flaper87acabrera: LOOOOOL12:55
*** mpanetta has joined #openstack-marconi13:00
acabrerampanetta: o/13:01
mpanettaacabrera: morning :)13:02
mpanettaacabrera: How goes?13:03
mpanettaI did not get up as early as I wanted13:03
acabrerampanetta: I didn't get up as early as I intended, either. :P13:04
acabreraThings go well, though. :D13:04
mpanettaGood :)13:04
mpanetta6 days 22hrs 54mins and 40 secs13:04
mpanettasays the countdown timer over here13:04
acabreralol that timer...13:05
acabreraIt's fun to watch, perhaps bordering on hypnotic.13:05
acabreraI'd be distracted by it if I were on that side of the office.13:05
mpanettaIt needs to be spiced up a bit13:05
mpanettaIt is mildly distracting, but I'll get over it ;)13:05
mpanettaflaper87: countdown until GA13:06
acabreraGoogle: "Spice Up Your Timer" -> "Do you want to spice up your games and even spice up your daily life with **random timers**..."13:07
acabreraGoogle knows. :P13:07
acabreraMy email inbox is awesomely clear this morning, so I'll take a little bit to catch up on accumulated, unsorted bookmarks.13:08
mpanettaacabrera: I could send you more email if you would like :P13:10
acabrerampanetta: lol if it contains links to awesome videos/reading material, it's super welcome. :D13:11
acabrerampanetta: but yeah, I'm sure I'll have an email within 10 minutes`13:11
acabrerafrom something13:11
acabrerasome alerting system13:11
acabrerasome subscription13:11
mpanettaAh yes13:11
mpanettaI just wish every day was fruit tuesday13:13
acabrerampanetta: We could make it happen as part of the Racklanta Health initiative. I'm sure it wouldn't be too difficult to get weekly volunteers to fetch apples, bananas, and more from the nearby Kroger's.13:14
mpanettaacabrera: That would be nice.13:15
flaper87oh man!13:15
*** ayoung has joined #openstack-marconi13:16
*** amitgandhi has joined #openstack-marconi13:22
*** acabrera has quit IRC13:25
*** cppcabrera has joined #openstack-marconi13:27
*** cppcabrera has quit IRC13:27
*** alcabrera has joined #openstack-marconi13:28
alcabrerampanetta: Thanks! I just decided to swap back to using weechat after giving irssi a try for a week.13:30
mpanettaAh irssi13:31
alcabreraflaper87: Thanks for the reviews! :)13:31
flaper87alcabrera: weechat rocks. Are you using any fancy plugin that I should be using as well?13:31
flaper87alcabrera: thank you for the hard work13:31
mpanettaalcabrera: Thanks for submitting that bug last minute last night.13:31
alcabreraflaper87: no fancy plugins here - just straight weechat.13:31
flaper87I couldn't help smiling when I realized the last patch for the proxy is 260 LOC13:32
alcabrerampanetta: no problem. I ran into that bug a little earlier so it was worthwhile to triage it's status higher now that it affects more people.13:32
flaper87it was like: Ohhh, YES! We're getting there!13:32
alcabreraflaper87: hell yeah! hahaha. I'm having the same feeling. :D13:33
mpanettaThink I will have something to test soon?13:33
* mpanetta is itchy to test ;)13:33
alcabreraflaper87: it was also fun to work with jsonschema. I highly recommend it for validating transport-layer input moving forward.13:34
alcabrerampanetta: I'll start working on that bug in about 25 minutes. I hope to send you a direct, early, pre-review patch by 11am EDT.13:34
*** yassine has quit IRC13:34
*** yassine has joined #openstack-marconi13:35
mpanettaalcabrera: Sweet, that actually fits in with my schedule pretty well today.  We have a new interviewee coming in.13:35
*** malini_afk is now known as malini13:37
flaper87alcabrera: I'm glad to hear that, really!13:37
alcabreraflaper87: I took your thoughts on versioned APIs in the mirror_queues patch and turned it into a BP. :)13:39
flaper87alcabrera: awesome!13:40
flaper87alcabrera: thanks13:40
flaper87alcabrera: it's not clear to me why lookup is a function here:
alcabreraflaper87: oooh, you found some of my cruft - nice!13:43
alcabreraIt was *a lot* more useful when I had a WeightResource and a HostsResource implementing PUT on partitions in an earlier version of the patch. It was used to keep things DRY, since I used that idiom in three places.13:44
alcabreraNow it just gets in the way.13:44
alcabreraI'll remove that shortly.13:44
flaper87kk, I'll drop a comment there13:45
alcabreraflaper87: Thanks! It's also likely that on_patch will benefit from the presence of a docstring, so I'll that add in for the coming patchset.13:47
alcabreraflaper87: nevermind, it's already there, just - a little weak.13:47
* alcabrera doesn't remember what he wrote yestetday13:48
alcabreraflaper87: thanks for the reviews!13:50
flaper87alcabrera: LOL, I don't remember what I had for lunch anymore13:53
flaper87alcabrera: np, my pleasure13:53
*** yassine has quit IRC14:00
*** yassine has joined #openstack-marconi14:01
flaper87kk, reviewed everythin14:08
flaper87kk, reviewed everything14:08
*** jcru has joined #openstack-marconi14:08
alcabreraflaper87: super appreciated!14:09
alcabreraI'm almost done applying the suggested changes to the last proxy patch.14:10
* flaper87 runs away14:21
flaper87alcabrera: man, take your time!14:21
*** tvb has quit IRC14:22
alcabreraI also found some bugs in the unit test suite for test_partitions, so I'm clearing those up. tests went running. D:14:24
openstackgerritZhihao Yuan proposed a change to openstack/marconi: feat(api): Client-ID is now a real UUID
zyuanreview wanted ^^14:38
*** oz_akan_ has joined #openstack-marconi14:39
flaper87zyuan: lol @ review wanted :D14:44
openstackgerritZhihao Yuan proposed a change to openstack/marconi: feat(api): Client-ID is now a real UUID
zyuanflaper87: an interesting fact is, to make this change work, mongo driver needs to do *nothing*14:46
kgriffscool, I am looking forward to that patch14:51
kgriffsoz_akan_, mpanetta: are you currently including the Client-ID header in the uwsgi logs?14:53
mpanettaI do not believe so.14:54
kgriffsCan we add that to the "todo" list?14:54
kgriffsit would be really helpful for data mining and diagnostics14:54
mpanettaYeah, only have client addr and port.14:55
kgriffsiirc, uwsgi does not allow for logging arbitrary headers14:55
kgriffsif it does, then GREAT14:56
mpanettaI do not know.14:56
kgriffstake a look and if there doesn't seem a way to do it, I've got an EOM branch that can help14:56
kgriffsIf you need it, then ping me and i will get that code merged into the mainline14:57
mpanettakgriffs: Do you mean client ID or customer ID?14:59
kgriffsit will be a UUID string14:59
openstackgerritAlejandro Cabrera proposed a change to openstack/marconi: feat (proxy/admin): allow partition modifications
kgriffsthis is not X-Project-ID14:59
mpanettaI will take a look15:00
* kgriffs gives mpanetta a high-five15:00
alcabrerakgriffs: flaper87 reviewed all the proxy patches this morning. I've addressed all comments including those made on the most recent patch. Things are looking pretty awesome proxy-side (except for a certain memcached-keys bug that I'm going to destroy now). :)15:03
mpanettakgriffs: I do not see it as an immediate option.15:03
kgriffsok. I will add it to my list to get that EOM branch cleaned up and merged15:04
mpanettakgriffs: check other channel please15:05
openstackgerritMalini Kamalambal proposed a change to openstack/marconi: Add Tests for Queue Stats
zyuanflaper87: the transport code used str(ex) everywhere. another thing is, tis ex is generated from Python std lib, it's not very helpful to make it unicode...15:11
zyuanflaper87: one more thing, python3 has no identifier "unicode"15:11
flaper87zyuan: you can use six there, there's a bp for the unicode thing, i'll be changing the transport part soon15:22
*** jdprax has quit IRC15:23
*** tvb has joined #openstack-marconi15:23
*** tvb has quit IRC15:23
*** tvb has joined #openstack-marconi15:23
zyuanflaper87: i don't it's the case. i think for error message what we want is the default string, no explicitly unicode. python2 string has enough interfaces to make this part work, even some falcon part assume the message is unicode15:24
zyuanduck typing15:24
openstackgerritAlejandro Cabrera proposed a change to openstack/marconi: fix: encode keys before caching
zyuanflaper87: for example, if falcon want to encode the error message, just do s.encode('utf-8'), it works15:26
zyuanin python3, it also works.15:27
*** tvb has quit IRC15:30
*** flaper87 is now known as flaper87|afk15:33
zyuanalcabrera: are you working on this?
alcabrerazyuan: no15:38
alcabrerazyuan: I'm tackling the caching bug atm15:38
*** flaper87|afk is now known as flaper8715:41
flaper87zyuan: sorry, got disconnected15:41
flaper87#$%@!$%#!@%# connection15:41
flaper87dunno if you got my last messages15:41
flaper87sure, but that's falcon's issue, not marconi's. Right?15:41
flaper87so, I think we should be using unicode throughout marconi15:41
zyuani don't think so. i tend to say use unicode's interface15:42
flaper87and we should also be using unicode all around because is safer and doesn't make any assumptions15:43
*** jdprax has joined #openstack-marconi15:43
zyuanlet's say, what do you want to do with unicode ther than encode?15:46
zyuanwhen the content is an error message comes out of Python standard library?15:48
flaper87I just want to keep everything in unicode while it's in marconi-land because it's safer and doesn't make any assumption. I want to keep it consistent with the rest of the code and with what we're targeting.15:48
zyuanyou can't. at least we have bytes in marconi15:48
flaper87we want to use unicode throughout marconi and encode at the very end15:48
zyuanwe have bytes, generic string, unicde string15:49
zyuanuse them appropiately.15:49
zyuanunicode is good, but not good for everything.15:52
flaper87where are those not being used appropriately? Plus, I don't think we should be trusting blindly what comes out from python's std15:52
zyuani don't understand why people what to use it everywhere.15:52
*** yassine has quit IRC15:52
zyuani don't want to "trust" unicode "blindly" too.15:53
zyuanunicode is good for very specific things. when you know your use case, why don't use a generic string?15:53
*** kgriffs is now known as kgriffs_afk15:53
zyuanearly python3 is slower then python2, because all strings are unicode. now they made the string literal char string when possible. so here it is: do you really care about the string representation? no, the important thing is the interface.15:56
zyuanthe bad part about python2 is that it has no bytes type, it's not "it's string is not unicode".15:57
zyuanunicode the type does not save you.15:57
*** ykaplan has quit IRC16:01
flaper87zyuan: I agree with you, I also agree that unicode it's not a hammer, however, it's a practice we want to have in Marconi and in this case it won't affect performance since it's all about calling unicode on an exception that shouldn't be ocurring16:08
zyuanyes, i don't care the performance of unhappy path16:09
zyuanif you want you can use six on them later.16:10
zyuan(but i still don't see benefits here... :(16:10
flaper87zyuan: I don't think we should keep accumulating technical dept :/ you would help me if you do it in this patch, pleeeeeeeeeeassseee :D16:11
flaper87zyuan: there's no *real* benefit besides sticking to what we decided for marconi (re unicode / str )16:11
flaper87zyuan: like I said, I totally get your point and agree with it :)16:11
alcabreraflaper87: can I get a review (and maybe a +2) on this: :D16:20
alcabrerait's that patch from earlier - I got the fixes in + more tests16:21
openstackgerritZhihao Yuan proposed a change to openstack/marconi: fix: force exception content to be text_type
openstackgerritZhihao Yuan proposed a change to openstack/marconi: feat(api): Client-ID is now a real UUID
alcabreraoz_akan_, kgriffs_afk, flaper87: for whatever reason, python-memcached (cache.set) doesn't seem to work with redis (using Redis 2.6.16).16:27
alcabrerahowever, python-memcached *does* work with memcached 1.4.15 (thankfully)16:27
alcabreraThis is all in relation to oslo.cache16:27
alcabreraJust learned a few things while making sure marconi hierarchical caching is up to par16:27
flaper87alcabrera: mmh, does it throw any error?16:28
flaper87alcabrera: btw, I'll take a look at those patches16:28
flaper87zyuan: uuu, you did them all. THANKS!16:29
alcabreraflaper87: cache.set(name, thing) => False (while observing redis-cli monitor)16:30
alcabreraI've also tinkered and restarted things to make sure it wasn't a flop on my side.16:30
alcabreraIt's a *silent* failure. :x16:30
flaper87mmhh, looks like something related to redis-py16:31
alcabreraflaper87: Whenever I get a chance (if kgriffs_afk doesn't beat me to it), I'll have fun submitting a proper python-redis backend for oslo.cache. :D16:33
alcabrera(Probably not in the next week, but it's on  my TODO).16:33
openstackgerritFlavio Percoco proposed a change to openstack/python-marconiclient: Add prepare_request function to
* alcabrera enjoys working with oslo.cache16:33
alcabreraNow with a bug out of the way, I'm going to take a moment to have a desk lunch and review patches.16:33
flaper87alcabrera: AWESOME16:34
flaper87start with my last patch16:34
flaper87alcabrera: no code changed, just docstrings16:34
alcabrerasure thing. I'll then proceed to zyuan's patches, then see what else is pending16:34
alcabreraflaper87: also, beyond the proxy patch, tell me what you think of this patch (+4, -2 LOC): (bug fix)16:35
alcabreraI plan on adding unit tests to that later today, and I've tested it locally.16:35
flaper87alcabrera: perhaps we should fix that in oslo.cache directly16:37
flaper87alcabrera: I remember seeing something similar but not sure if it was the same issue16:37
alcabreraflaper87: sounds good to me. I like the idea of making the fix in oslo.cache (ensuring type(key) == bytes) since it makes it more transparent for everyone else to use.16:40
alcabreraI'll do that.16:40
* alcabrera grabs his magic six hammer16:40
flaper87alcabrera: :D16:40
alcabreragiven the existence of self._prepare_key, this seems straightforward16:41
*** tvb has joined #openstack-marconi16:41
*** whenry has joined #openstack-marconi16:42
flaper87no comments, stupid ISP16:45
flaper87ok, links to BPs added:
flaper87kgriffs_afk: ^16:45
*** tvb has quit IRC16:46
alcabreraflaper87: sweet BP linking16:46
flaper87malini: ping16:48
flaper87malini: are you already digging into marconi + tempest integration ?16:49
*** vkmc has joined #openstack-marconi16:55
*** vkmc has quit IRC16:55
*** vkmc has joined #openstack-marconi16:55
openstackgerritAlejandro Cabrera proposed a change to openstack/marconi: fix: encode keys before caching
alcabreraflaper87: updated to make the change in oslo.cache. ^^16:56
flaper87alcabrera: awesome, I'm curious to know what keys failed and why memecached tests didn't raise that16:56
alcabreracache.set(u'q._/taco', 'weee')16:58
alcabreraThat started throwing...16:58
flaper87mmmhhh, sounds like it doesn't like the '.'16:59
flaper87I guess16:59
flaper87not sure16:59
flaper87just random-stupid guess16:59
alcabrera"MemcachedStringEncodingError: Keys must be str()'s, not unicode. Convert your unicode strings using mystring.encode(charset)!"17:02
alcabreraflaper87: ^^17:02
alcabreraThe '.' is treated just fine. :)17:03
flaper87alcabrera: I said it was a random-stupid-guess!!! :D17:03
alcabreraThough it wouldn't hurt to have some form of regex that forces keys to conform to a particular character set in oslo.cache. :)17:03
flaper87Have I ever told you how much I hate memcached ?17:04
alcabreraYou have not. ;)17:04
* alcabrera listens17:04
mpanettaflaper87: No, but I am starting to hate it by proxy now heh17:04
mpanettaNo pun intended17:04
flaper87alcabrera: ok, I hate every single bit of its very stupid and silly API17:04
flaper87plus, its poor documentation and ridiculous protocol17:05
flaper87Wanna know what a good documentation looks like, take a look here:
flaper87no way you can't understand something from there!17:05
alcabreraflaper87: I *love* that page. I've been there over a hundred times since I discovered Redis.17:05
flaper87 < You f1#@$@!$#@ing kidding me ?17:06
flaper87 <- WTF is this?17:06
alcabreraflaper87: rofl @ stats17:07
alcabreraThat's just bad.17:07
flaper87"An expiration time, in seconds. Can be up to 30 days. After 30 days, is treated as a unix timestamp of an exact date." <- Yeah right, why don't we apply the same thing to strings? After 30 chars your string will be treated as a BLOB and multiplied by 517:08
flaper87^ you know, for "consistency" ^17:09
alcabreraflaper87: lol!17:09
alcabreraI noticed that in the oslo.cache:memcached docs17:09
alcabreraIt's pretty terrible.,17:09
flaper87so yeah, I hate it! :D17:09
* alcabrera takes note to avoid memcached when possible17:10
flaper87alcabrera: you mean, always, right?17:10
alcabreraI sure wish. :P17:10
torgomaticthe one saving grace is that it's 30 * 86400 seconds, not "30 days from now, including timezones and stuff"17:10
alcabreratorgomatic: +1 - though still quite the quirk17:11
torgomaticit's also a good thing memcached didn't exist in January 1970, because if your system time is near 0, there are expiration times you just can't express in that wonky format :)17:11
flaper87torgomatic: lol17:12
torgomaticalcabrera: yeah, Swift has a workaround for that quirk:
torgomaticit is not my favorite piece of code17:12
alcabreratorgomatic: looks a lot like the workaround in the upcoming oslo.cache. >.>17:15
torgomaticalcabrera: and every other project that touches memcached :|17:16
alcabreraseems like that workaround should be submitted to the python-memcached implementation.17:16
alcabreraOr better yet, in the core of memcached (if that's even possible at this point)17:16
openstackgerritZhihao Yuan proposed a change to openstack/marconi: feat(api): Client-ID is now a real UUID
*** tvb has joined #openstack-marconi17:30
zyuanalcabrera: ^^17:41
alcabrerazyuan: I checked it out. I'll have to think on it a bit. Thanks for providing the rationale for the change in the commit message!17:45
*** flaper87 is now known as flaper87|afk17:48
zyuanalcabrera: can you take a look at this change first?
alcabrerazyuan: +2 - works for me17:51
alcabrerazyuan: I'd prefer if that patch didn't depend on the UUID one. Why does it depend on the uuid one, btw?17:52
*** flaper87|afk is now known as flaper8718:01
zyuanChanServ: the uuid one depends on the text_type one18:02
zyuanalcabrera: ^^18:02
zyuan(so, you see, a habit is very hard to change)18:02
alcabreralol chanserv18:03
flaper87ChanServ: feels loved !!!!18:03
alcabreraflaper87: I appreciate you taking the time to check on marconi late into the evening. I was just thinking, "It's at least 6pm over there!". :)18:06
flaper87alcabrera: no worries! It's my pleasure!18:07
flaper87... and passion :D18:07
alcabreraI understand - totally! Still, very much appreciated. :D18:07
alcabreraflaper87: could you help me get this patch through soon (modifiable partitions): . If you'd prefer to tackle that one (and the oslo.cache fix) tomorrow morning, that works for me, too.18:09
flaper87alcabrera: no worries, I'll take a look at it while my rust code compiles :D18:09
flaper87damn, one can't use the compilation excuse nowadays anymore18:10
alcabrerayup - not in go, not in rust, and just barely in haskell18:11
alcabrerahowever - in c++, it's as valid as ever18:11
mpanettaalcabrera: "Sorry boss, I have this complex template library I need to compile, I will be off for a day." :P18:14
alcabreralol so true...18:14
flaper87alcabrera: are we sure these tests are running?
flaper87Why didn't that fail in the previous patch?18:17
flaper87I mean, because of the borked super18:17
alcabreraflaper87: They're running. I checked with jenkins just now.18:18
alcabrera (grep 'ExistingPartitionTest')18:18
flaper87alcabrera: oh ok, that's weird, though.18:18
flaper87alcabrera: LGTM18:18
alcabreraflaper87: I learned the default behavior of a borked super is to skip a test18:18
alcabrerasilent failure - yup...18:18
zyuanflaper87: ?18:19
alcabreraflaper87: thanks!18:21
flaper87zyuan: +2'd18:29
*** amitgandhi has quit IRC18:32
openstackgerritA change was merged to openstack/marconi: fix: force exception content to be text_type
*** amitgandhi has joined #openstack-marconi18:35
alcabrerazyuan: awesome!18:36
openstackgerritAlejandro Cabrera proposed a change to openstack/marconi: feat: add logging to proxy
*** kgriffs_afk is now known as kgriffs19:11
alcabrerakgriffs: o/19:13
kgriffsfinally got my passport application submitted19:15
* kgriffs is very relieved to have that done19:16
alcabrerakgriffs: awesome. I did mine a few years ago. Was the process smooth for you?19:16
* alcabrera remembers kgriffs was aiming to go to HKG19:16
malinikgriffs: right in the middle of govt shutdown :(19:16
kgriffsyeah, except i forgot to bring a photocopy of my license, so I had to run over to FedEx copy center real quick19:16
kgriffsmalini: apparently, passports are an "essential" service19:17
alcabreraclose call there19:17
zyuankgriffs: why the queue name checking need to be applied to all requests?19:17
kgriffsbecause all input must be validated. it's one of the most common way crackers break apps.19:19
zyuanhow it can break apps since queue is jue not there?19:20
kgriffsit is included in the query string, is it not?19:20
kgriffsto the DB?19:20
zyuanqueue name?19:20
zyuanthe queue name is in uri19:20
kgriffsyes, but we pull it out and then pass it directly into the storage driver19:21
kgriffswhich then uses it to query the DB19:21
zyuanall the queries will fail if there is no such queue19:21
kgriffsand there is a long, sordid history of security breaches caused by bugs and whatnot in databases19:21
* ametts hates it when I get whatnot in my databases19:22
kgriffsgiven that (a) there is a risk in NOT validating, and (b) the validation is cheap, it seems like an obvious choice19:22
alcabreraametts: lol19:22
kgriffsametts: you should try it on toast. It's delicious.19:22
zyuansorry, i can't see any risk here...19:22
zyuan...and i don't like things just because they are cheap.  i like things which are correct.19:23
kgriffscan you guarantee, 100%, that every storage driver we write, every DB library that those drivers use, and every backend system they talk to is bug free and there is no way to trigger an exploit via a well-crafted string?19:24
zyuanno matter how well-crafted they are, they have to be encoded and decoded by python...19:25
zyuanbefore goes to DB19:25
kgriffshow do you define "correct" application security?19:26
amitgandhiwhat validation are we doing on the queue_name - character limitations?19:28
kgriffscharacter limitations and length19:28
zyuanas same as its logic meaning19:28
kgriffszyuan: security is all about balancing risk with performance and usability19:28
kgriffsthere is no such thing as a perfectly secure system19:28
kgriffsunless it is one that doesn't do anything19:29
zyuanof course you can't prove a system to be secure, but i also don't want to assume some part is not secure19:29
kgriffssometimes you do have to assume things are not secure, and assign a risk level to it19:29
kgriffsparticularly, any time you cross a trust boundary19:30
zyuanyou need a reason to argue whether it's secure or not at least19:30
kgriffsand I, for one, do not trust databases19:30
zyuani trust python. it's not reached to db yet.19:31
* flaper87 doesn't trust databases either19:31
kgriffszyuan: the reason is the history of DB security breaches and programmer bugs in general. If you want to spend 6 months rigorously evaluating the security of the current version of MongoDB and MySQL, the python runtime, etc., be my guest. But it simply isn't worth trying to save a few nanoseconds.19:32
kgriffsthis is why it is an industry best practice to ALWAYS validate ALL input19:32
kgriffsMany people also advocate validating your output, although I'm not convinced that is necessary in all cases.19:32
alcabrerakgriffs: +119:33
zyuanit's equivient to being validated... because we done decoding19:33
zyuanthe input is not sent directly to db19:33
zyuanand db can accept an unicode19:34
kgriffsare you confident enough in that hypothesis to risk even one security breach?19:35
zyuani don't. but i think this is something we can't control, and performing input validation at this layer does not help, IMHO.19:36
openstackgerritFlavio Percoco proposed a change to openstack/python-marconiclient: Add prepare_request function to
kgriffsbtw, if we ever want to be PCI compliant (something users have asked for) we must rigorously validate all input.19:37
kgriffsgiven that always-on validation doesn't hurt anything, and it *may* help (TBD), it seems only logical to put it in place.19:39
kgriffsanyway, that is my $0.0219:43
* ametts thinks kgriffs writes a lot for two cents. Too bad we don't pay him by the word.19:44
alcabrerakgriffs: whenever you get a free moment, there are three proxy patches ready to merge and 3 pending review (PATCH partitions, oslo.cache bug fix, add logging).19:45
kgriffsI am typing this in between eating noodles. should be able to get into review mode shortly19:46
alcabrerakgriffs: enjoy the noodles! :D19:46
kgriffsnever had vietnamese before19:47
kgriffsso far so good19:47
alcabrerasounds delicious19:47
zyuanwhat? github down again?19:53
*** kgriffs is now known as kgriffs_afk19:53
alcabrerazyuan: likely - says issues are being investigated19:55
zyuanalcabrera: i found that proxy storage already used list; just the restful resource did something different -_-19:57
alcabrerazyuan: hehe, that sounds about right.19:57
*** kgriffs_afk is now known as kgriffs20:01
*** malini has left #openstack-marconi20:04
*** tvb has quit IRC20:06
alcabreraflaper87: lol... poor, valiant souls. Also, total_hours_wasted - nice metadata. ;)20:07
zyuanalcabrera: i just noticed we had no tech talk today20:08
alcabrerazyuan: yup. I haven't had any free time to gather together thoughts on tech cinemas or tech talks. :(20:09
alcabrerazyuan: I need a backup coordinator. :P20:10
zyuanis there any plan for next Wed?20:10
flaper87kgriffs: addressed your comments there20:17
kgriffsso, re. operation20:18
kgriffsenum vs. strings?20:18
kgriffs(what are your thoughts)20:18
alcabrerakgriffs: enums are easier to use, IMO, for defining a set of allowed operations. They're also more lightweight, AFAIK, than strings.20:20
alcabrerakgriffs: they also have type guarantees in py3 (enum == int -> TypeError)20:21
kgriffsmy other question was, should we follow the REST architectural style there? HTTP isn't required20:21
flaper87kgriffs: that's the idea so far. Re string / enum, that's a bit unclear right now, I'd prefer enums TBH but, it's hard to say right now since that code is not being used yet but required to be there already20:22
kgriffsif so, then we may want to have two params: verb and resource; but I think that may make the code a little messier20:22
flaper87dunno if that makes sense20:22
flaper87The general idea is to have all that unpacked and let the transport pack the request20:22
kgriffsflaper87: I'm fine merging this patch as-is, just want to get everyone thinking about ways to refine it later20:23
flaper87like: Put the headers where they should go, params where params belong to, etc20:23
kgriffsI kinda like the enum idea20:23
kgriffsflaper87: gotchya20:23
flaper87kgriffs: indeed, that's a alcabrera suggestion (enums, I mean)20:23
flaper87HTTP is the most fragmented protocol I can think of right now20:24
flaper87zmq will pack everything in 1 msg20:24
flaper87(most likely it'll keep things separated anyway)20:24
kgriffshave you checked out nanomsg?20:24
flaper87kgriffs: yup20:24
flaper87kgriffs: It sounds really amazing20:24
flaper87already tested on a embedded device20:24
kgriffsyeah. something to keep an eye on20:24
alcabreraflaper87, kgriffs: do we have a client issues page lying around, or do we want to start filing bug reports for ideas?20:26
flaper87alcabrera: we have bps and bugs for the client20:27
alcabreraNamely, the two that you just brought up, kgriffs, as well as the client auth session caching idea.20:27
flaper87but I think I misunderstood your question20:27
alcabreraflaper87: perfect, that's what I was asking about. :)20:27
alcabreraflaper87: could you create bps/bug reports for the ideas brought up in the client discussions just now?20:27
flaper87alcabrera: yes, I'll do that20:28
alcabreraflaper87: awesome!20:28
alcabreraWe can close them later if find better ways of doing things.20:28
alcabrera**if we find20:28
*** jdaggett has joined #openstack-marconi20:32
kgriffsflaper87, alcabrera: ^^^ Only change it from using iteritems to items per alcabrera's feedback20:49
kgriffsjust noticed that my future self snuck in hear again20:49
kgriffsit is soooo annoying to be a quantum being sometimes.20:50
kgriffsand why can't they ever speel things rite?20:50
alcabreraI'm headed out for the night, guys.20:50
kgriffstake care20:51
kgriffsor luck20:51
flaper87alcabrera: take care20:51
kgriffswhatever you want to take20:51
alcabreranight~ :D20:51
*** alcabrera has left #openstack-marconi20:51
openstackgerritA change was merged to openstack/python-marconiclient: Add prepare_request function to
openstackgerritKurt Griffiths proposed a change to openstack/marconi: fix(wsgi): Non-ASCII characters in the queue name trigger 500 status
openstackgerritAlejandro Cabrera proposed a change to openstack/marconi: feat: async forwarding
*** malini has joined #openstack-marconi21:12
openstackgerritKurt Griffiths proposed a change to openstack/marconi: fix(wsgi): Non-ASCII characters in the queue name trigger 500 status
openstackgerritA change was merged to openstack/marconi: proxy: mirror structure of marconi queues + bootstrap
openstackgerritA change was merged to openstack/marconi: feat: separate config for queues and proxy
openstackgerritA change was merged to openstack/marconi: feat: split proxy API into admin and public apps
flaper87Alej will be very happy tomorrow :D21:32
kgriffsjsonschema will have to go into global reqs then21:36
flaper87kgriffs: it should be already21:37
flaper87it's being used in glance21:37
kgriffsoh, that would be sweet21:37
* kgriffs checks21:37
flaper87plus, pep8 checks would've failed otherwise21:38
kgriffsyep, it's there21:38
flaper87or gate tests, whatever they're called21:38
kgriffsoh, right21:38
* kgriffs smacks forhead21:38
* kgriffs goes to take his meds21:38
* flaper87 hands some extra meds to kgriffs21:38
openstackgerritA change was merged to openstack/marconi: feat (proxy/admin): allow partition modifications
openstackgerritA change was merged to openstack/marconi: fix: encode keys before caching
*** jcru has quit IRC22:03
*** jdaggett1 has joined #openstack-marconi22:03
*** jdaggett has quit IRC22:06
*** flaper87 is now known as flaper87|afk22:24
*** tedross has quit IRC22:30
*** malini is now known as malini_afk22:31
*** amitgandhi has quit IRC22:34
*** mpanetta has quit IRC22:35
openstackgerritKurt Griffiths proposed a change to openstack/marconi: chore: Rename MONGODB_TEST_LIVE to MARCONI_TEST_MONGODB
*** oz_akan_ has quit IRC22:37
*** kgriffs is now known as kgriffs_afk22:49
*** jergerber has quit IRC23:31
*** oz_akan_ has joined #openstack-marconi23:48
*** oz_akan_ has quit IRC23:53

Generated by 2.14.0 by Marius Gedminas - find it at!