Friday, 2013-10-18

*** jcru has quit IRC00:08
*** mpanetta has quit IRC00:24
*** oz_akan_ has joined #openstack-marconi00:27
*** amitgandhi has joined #openstack-marconi00:28
*** fifieldt_ has joined #openstack-marconi00:52
*** fifieldt_ has quit IRC00:57
*** nosnos has joined #openstack-marconi01:08
*** reed has quit IRC01:14
*** fifieldt_ has joined #openstack-marconi01:16
*** amitgandhi has quit IRC01:26
*** fifieldt_ has quit IRC02:35
*** jergerber has quit IRC02:43
*** vkmc has quit IRC02:58
*** fifieldt has joined #openstack-marconi03:51
*** oz_akan_ has quit IRC03:52
*** oz_akan_ has joined #openstack-marconi04:13
*** oz_akan_ has quit IRC04:18
*** oz_akan_ has joined #openstack-marconi05:29
*** oz_akan_ has quit IRC05:34
*** yassine has joined #openstack-marconi06:56
*** cosss has joined #openstack-marconi08:01
*** russellb has quit IRC08:35
*** russellb has joined #openstack-marconi08:36
*** flaper87|afk is now known as flaper8709:32
openstackgerritFlavio Percoco proposed a change to openstack/marconi: Migrate from nosetest to testr
*** oz_akan_ has joined #openstack-marconi10:18
*** flaper87 is now known as flaper87|afk10:26
*** oz_akan_ has quit IRC10:28
*** flaper87|afk is now known as flaper8711:26
*** nosnos has quit IRC11:34
*** nosnos has joined #openstack-marconi11:35
*** oz_akan_ has joined #openstack-marconi11:39
*** nosnos has quit IRC11:40
*** tedross has joined #openstack-marconi11:43
*** oz_akan_ has quit IRC11:45
*** cosss has quit IRC11:47
*** fvollero|gone is now known as fvollero11:59
*** alcabrera has joined #openstack-marconi12:04
alcabreraGood morning!12:04
*** malini_afk is now known as malini12:08
*** vkmc has joined #openstack-marconi12:10
*** vkmc has quit IRC12:10
*** vkmc has joined #openstack-marconi12:10
*** wirehead_ has quit IRC12:16
*** wirehead_ has joined #openstack-marconi12:17
flaper87alcabrera: gooood morning!12:34
alcabreraflaper87: Hey! :)12:35
flaper87alcabrera: how are you ?12:38
flaper87alcabrera: btw, were you aware about these failing tests?
flaper87Somehow, nosetest didn't run them12:38
alcabreraI'm doing better today than yesterday. Today's been pretty good, so far. :D12:39
alcabreraHow're you?12:40
* alcabrera checks out failing tests12:40
alcabreraoohh, the migration to testr is going to be *awesome*.12:40
alcabreraIt's one of those little things I've been looking forward to!12:41
alcabreraI wasn't aware of any tests failing. :/12:42
alcabreraErr, not being run.12:42
alcabreraI know nose skips right past tests that have misnamed super()s.12:42
alcabreraI ran into that problem several times while adding unit tests for things like ShardDriver, CatalogueTests, etc.12:42
*** fifieldt has quit IRC12:44
flaper87alcabrera: mmh, interesting, I didn't know about it silently skipping tests with wrong super()12:54
alcabreraYeah, it'd be something silly like...12:54
alcabreraclass CatalogueTest: ... super(ClaimsTest, self).setUp()12:54
flaper87alcabrera: It hasn't been *that* straightforward, I'm working on some other improvements to that patch12:54
alcabrerathat test will just be ignored.12:55
flaper87alcabrera: for example, generate a db name (As for monogdb) for every test12:55
flaper87because they are not as isolated as I expected12:55
flaper87good to know, I'll flag that patch as WIP because I don't feel like it is ready to be merged12:56
flaper87I mean, it looks like it is12:56
flaper87but the mongodb part seems to be failing locally12:56
flaper87plus, it's a bit unstable12:56
flaper87(saw the random py26 / pypy failures ?)12:57
flaper87that's most likely related to tests being run in parallel12:57
alcabrerathose random failures... something is unstable there, and I don't know what it is. :/12:57
alcabrerapython-2.6 and pypy seem to fail periodically.12:57
alcabreraIt's weird.12:57
flaper87also, we *NEED* a way to test the mongodb code in the gate. I'll ping -infra guys today or next week12:58
alcabreragood point on the mongo tests - generating a uuid-based DB name would certainly help with isolation.12:58
alcabreraooohh, that'd be awesome.12:58
flaper87alcabrera: UUID is exaclty what I'm using ;)12:58
alcabrerahehe, uuid fixes *all* the unit test isolation problems. :D13:00
alcabreraWell, most of them, anyway. :P13:00
*** amitgandhi has joined #openstack-marconi13:01
*** amitgandhi has quit IRC13:13
*** amitgandhi has joined #openstack-marconi13:15
*** oz_akan_ has joined #openstack-marconi13:16
*** amitgandhi has quit IRC13:16
*** amitgandhi has joined #openstack-marconi13:25
*** oz_akan_ has quit IRC13:38
*** oz_akan_ has joined #openstack-marconi13:39
*** malini is now known as malini_afk13:50
*** malini_afk is now known as malini13:51
*** tedross has quit IRC13:52
alcabreraflaper87: reviwed all the pending client patches - +2.13:53
*** jcru has joined #openstack-marconi13:54
flaper87alcabrera: AWESOME!, thanks a lot! What do you think so far?13:55
*** etoews1 has joined #openstack-marconi13:57
alcabreraflaper87: it's looking good. I'd love to see the iterator support soon, but that's lower priority than getting an initial, fully-functional client impl. out, IMO. :)13:59
alcabreraIt also seems like adding a new transport won't be too bad, as it's mostly a matter of filling in the lower-level API, AFAIU.14:00
alcabreraflaper87: what do you think of the current client-side wiki page?14:01
flaper87alcabrera: yeah, adding a new transport would be awesome, I'd like to start working on the zmq one asap but we should first brainstorm a bit more on the zmq wiki page. Adding a new transport to the client is just a matter of implementing the `Transport` class for that transport. See
flaper87that file is all it takes to add support for a new transport14:04
flaper87it gets a `Request` in and has to send a `Request` out14:04
flaper87erm a `Response` out14:05
flaper87alcabrera: I like that wiki page, I first looked at it before working on the client14:05
flaper87some things changed, though.14:06
flaper87I'll update it as soon as the Queue's code lands14:06
alcabreraflaper87: awesome, thanks for helping with that page!14:06
alcabreraand if all it takes to get a new transport in is one file, niiiicee. :D14:07
* alcabrera hasn't kept up with the big picture client-side too well14:07
flaper87yeah, that's all it takes!14:07
flaper87transport's are completely isolated from the API14:07
flaper87actually, I was thinking - I'll perhaps bring this up in our next meeting - that we should do the same server-side14:07
alcabreraahh, I see...14:08
alcabreraA lower-level API14:08
flaper87We should have the API spec somewhere and let transport build their routes out of it14:08
alcabreraTo implement to higher level API14:08
*** jergerber has joined #openstack-marconi14:09
flaper87I don't like the fact that our wsgi transport has the routes, methods etc hardcoded14:09
alcabreraIt seems like the agenda needs an update.14:09
flaper87that makes it necessary to replicate that for every transport14:09
alcabreraWhich also reminds me that we should decide on that agenda today as soon as kgriffs_afk makes his way in.14:09
alcabreraTo send out the meeting reminder email on the ML14:09
flaper87alcabrera: nod!14:09
flaper87let me help you with that14:10
* alcabrera was reading over
flaper87alcabrera: added some items, anything missing?14:11
flaper87most liekly, that will take the whole hour14:12
alcabreralooks good to me, flaper8714:12
alcabreraI agree that we'll likely consume to hour with those topics, probably even before we get to versioning. :P14:13
*** flaper87 changes topic to "OpenStack Queuing and Notification Service || Smile :D || Meetings every Monday 16:00 UTC || Wiki: || Paste: || Smile: :D"14:14
flaper87alcabrera: I think versioning could be easily covered by a good API spec14:14
*** jergerber has quit IRC14:14
alcabreraOur topic now has twice as many smiles and a meeting header - nice!14:15
flaper87alcabrera: yeaaaah!! There's no concurrence nor parallelism without smiling14:15
flaper87btw, I was thinking something like:
flaper87for the API spec14:16
flaper87the later inherits from the former14:16
*** amitgandhi has quit IRC14:16
flaper87basically, jsonschema based API definition14:16
*** amitgandhi has joined #openstack-marconi14:17
flaper87readable enough, easy to use for API discovery and easier to update as well14:17
alcabreraseems reasonable. There's very little I haven't been able to express given a thorough jsonschema. Have you run into any trouble implementing the client using your current approach?14:17
* flaper87 still needs to learn how to pronounce 'through'14:18
flaper87alcabrera: I haven't yet, It allows me to add custom fields - HTTP methods, for example - optional arguments, required ones, etc14:19
alcabrerathorough ~-> tho - row14:19
flaper87alcabrera: uuu, that makes sense. Thanks :D14:19
flaper87alcabrera: I guess the missing part there are 'required headers'14:19
alcabreraflaper87: yup. If that could be filled in somehow, or represented, that'd be awesome.14:20
flaper87which, I guess, we can workaround with some custom `headers` field in the schema14:20
alcabreraThe closest thing I have is assert field in (names)14:20
alcabreraflaper87: what do you think about merging this in?
alcabrerapip install in README.rst vs. python develop14:26
* alcabrera is reviewing and assessing14:26
* flaper87 looking14:30
flaper87btw, I wanted to point this out to you guys.
alcabreraflaper87: I added one more item to the meeting agenda. It's coupled with the sharding discussion, "the future of the proxy".14:31
flaper87that's basically why I didn't start working on the remove globals right away14:31
alcabreraflaper87: ooohh, I see the word "side effects" and I know it's going to be good.14:31
flaper87alcabrera: +1 for the future of the proxy14:32
flaper87mmh, I'm a bit unsure about the pip install -e . thing, the last time I tried, it didn't get the version right14:33
flaper87I didn't had the time to look into it14:33
flaper87let me try again14:33
flaper87I do agree that's the best way to do it, though14:33
alcabreraI'm giving it a try, as well, using a venv.14:34
flaper87alcabrera: does this happen to you?
alcabreraflaper87: VersionInfo(marconi:2013.2)  # from marconi.version.version_info14:35
flaper87damn, then my venv is borked14:35
alcabrera"marconi 2013.2.a466.gd8c26e5 is already the active version in easy-install.pth"14:35
alcabreraThat's when I try to install again14:36
flaper87trying in a fresh venv14:36
flaper87alcabrera: how did you create your venv ?14:36
flaper87--no-site-packages ?14:36
alcabreravirtualenv-2.7 .marconi14:37
flaper87mmh, did you see the Successfully installed UNKNOWN thing?14:37
alcabreranot this time, though I've seen it before.14:38
alcabreratrying again14:38
alcabreranope, still looks good.14:39
alcabreravirtualenv-2,7 --version => 1.10.114:40
alcabrerapip --version => 1.4.114:40
flaper87mmh, cleaned everything and run it again14:42
flaper87it just worked14:42
flaper87:/ not sure what happened14:42
alcabreraI'm almost done triaging that review queue - keeping things moving. :)14:43
alcabreraoh, btw...  (Tulip, a.k.a, asyncio, a.k.a., PEP 3156 was merged)14:43
zyuanRan 469 tests in 65.535s14:44
alcabreraPython 3.4 is going to be amazing.14:44
openstackgerritA change was merged to openstack/marconi: doc: Recommend installing marconi via pip
alcabreraTime to do some rebasing~14:52
*** tedross has joined #openstack-marconi15:08
*** tedross has quit IRC15:08
*** tedross has joined #openstack-marconi15:08
*** whenry has joined #openstack-marconi15:08
flaper87[OT] MANNNN:
*** kgriffs_afk is now known as kgriffs15:25
alcabreraflaper87: I feel like I've listened to some stuff by Kamelot, probably via some custom dance game charts. :P15:25
alcabrerakgriffs: o/15:25
kgriffsalcabrera: hey man, good to see you15:28
alcabrerathings are better today. :)15:28
kgriffsno way, tulip will make it in after all?!15:29
alcabreraimport asyncio15:30
alcabreraexpected to drop for python 3.4a0415:30
flaper87kgriffs: it will :D15:30
flaper87alcabrera: man, I really like their music15:30
alcabreraflaper87: you going to make it to that concert? ;)15:31
flaper87alcabrera: probably, it's not near me, I've some friends in that area, I just need to find a coach I can crash into15:32
alcabrerakgriffs: I've done a little triaging of the review queue this morning, and started rebasing the admin API feature line wrt the global config patch.15:33
flaper87alcabrera: +115:33
flaper87kgriffs: it'd be really cool if we could get some other client patches merged :)15:33
flaper87I mean, reviewed15:33
kgriffsyes, that would be nice.15:34
alcabreraflaper87, kgriffs: I've updated the with a "Merged" section, as well as a "Other Patches" section. Our review queue is quite busy lately~15:35
alcabreraoh yes!15:35
alcabreraJust remembered...15:35
alcabrerakgriffs: (agenda's been updated, with help from flaper87). We should probably send out an email today.15:35
alcabreraI'll check that out. :D15:35
alcabreraThat one looks like a *great* candidate for merging.15:36
alcabreraThere's a very high chance I'll hit that "Approve" button in the next 5 minutes.15:36
alcabreraSanity checking!15:36
alcabreraHere it comes...15:38
alcabrerakgriffs: approved!15:38
openstackgerritA change was merged to openstack/marconi: feat: Storage sharding foundation
alcabrerakgriffs, flaper87: if possible, let's make it through the queues admin API side soon. We need thecatalogue storage driver from there to continue the sharding core work.15:40
alcabrera+, it helps to be able to register shards to actually test this thing. :P15:41
alcabrera[OT] flaper87: I ended up listening to that Kamelot list. Good stuff!15:41
flaper87alcabrera: man you're right. I was planning to go through those review this morning but got caught but tons of internal stuff. I'll review it in a bit15:42
flaper87alcabrera: thanks for reminding me!15:42
alcabreranp. :)15:42
flaper87alcabrera: I really really like their music. I'm glad you liked it! :)15:43
alcabrerawe're at 15 patches in the queue at this moment, 7 related to sharding.15:43
alcabreraI'm rebasing the first patch in the line, atm.15:44
alcabreraadmin/public split for queues15:44
alcabreramighty rebasing, it is.15:44
kgriffsalcabrera: regarding
kgriffsI think it may actually be a good idea to split into an admin and a queues driver15:49
kgriffs(as you suggested)15:50
*** reed has joined #openstack-marconi15:50
kgriffsmakes more sense in my mind15:50
kgriffssince they will have different users, different DBs15:50
alcabreraeven different applications launched15:51
alcabrerafair enough - for now, I'd like to keep a same-file implementation.15:51
alcabreraWith intent to figure out how best to structure admin/public storage drivers later.15:51
alcabrerathe rationale being - the refactoring will get pretty messy considering we have two storage implementations at the moment, and I worry that'll obfuscate the intent of the current patches.15:53
kgriffsjust so we don't forget to come back to it later15:57
alcabrerakgriffs: awesome!15:57
*** pasquier-s has quit IRC16:00
alcabreraflaper87: Moonlight's my favorite so far.16:05
flaper87alcabrera: I like that one a lot. I also like very very much16:12
openstackgerritAlejandro Cabrera proposed a change to openstack/marconi: feat: split queues API into public/admin
alcabreraflaper87, kgriffs: queues admin API split now rebased and ready for review. config_split really did a number on things! ;)16:26
flaper87alcabrera: I bet it did :P16:29
openstackgerritAlejandro Cabrera proposed a change to openstack/marconi: feat: add shard management resource
*** kgriffs is now known as kgriffs_afk16:36
*** yassine has quit IRC16:37
openstackgerritAlejandro Cabrera proposed a change to openstack/marconi: feat: shards storage controller interface
alcabrerathat's three rebased - winning!16:44
alcabreratwo more to go in the admin API feature line16:44
alcabreraI'll brb. I'm going to run some groceries and continue working from home.16:44
*** alcabrera has quit IRC16:46
*** etoews1 has quit IRC16:50
*** etoews1 has joined #openstack-marconi16:51
openstackgerritDirk Mueller proposed a change to openstack/marconi: Fix Pep8 indentation errors missed by Pep8 1.4.5
*** kgriffs_afk is now known as kgriffs17:28
flaper87kgriffs: very quick review:
flaper87I need that one to push an update to the testr one17:29
flaper87Heading off now!17:29
flaper87if you guys can go through client patches, it'd be really cool17:29
*** flaper87 is now known as flaper87|afk17:30
*** alcabrera has joined #openstack-marconi17:39
openstackgerritA change was merged to openstack/marconi: Fix Pep8 indentation errors missed by Pep8 1.4.5
openstackgerritAlejandro Cabrera proposed a change to openstack/marconi: feat: split queues API into public/admin
openstackgerritAlejandro Cabrera proposed a change to openstack/marconi: feat: add shard management resource
openstackgerritAlejandro Cabrera proposed a change to openstack/marconi: feat: shards storage controller interface
openstackgerritZhihao Yuan proposed a change to openstack/marconi: WIP: feat(shard): queue listing
*** tedross has quit IRC18:49
openstackgerritZhihao Yuan proposed a change to openstack/marconi: WIP: feat(shard): queue listing
openstackgerritZhihao Yuan proposed a change to openstack/marconi: WIP: feat(shard): queue listing
zyuanthe jenkins machine with py26 seems to have something different19:02
alcabreraI'm out for the day, guys.19:07
zyuansee you!19:07
alcabrerazyuan: thanks for getting started on the listing patch. :)19:07
zyuanjust able to use debugger inside .list()...19:08
alcabrerakgriffs: let me know if anything stands out to you while reviewing the admin API feature line. :D19:08
alcabrerakgriffs: I've rebased about half of it at this point. I'll finish rebasing before I call it a day. I'm still tracking our efforts here:
alcabrerahave a good weekend. :D19:09
*** alcabrera has quit IRC19:09
*** reed has quit IRC19:30
openstackgerritAlejandro Cabrera proposed a change to openstack/marconi: feat: shards mongodb driver + tests
etoews1when posting message to a queue and you get a partial response back, is it guaranteed that the n ids returned in the response coincide with the first n messages sent in the request?19:47
zyuankgriffs: ^^ i guess so, but i'm not sure19:53
*** vkmc has quit IRC20:02
kgriffsetoews1: yes, it is20:03
etoews1kgriffs zyuan: thx.20:03
kgriffsbut, to be honest, you won't ever get partial results20:03
kgriffsit used to be possible, but I recently discovered that a change made to fix a different issue actually had the nice side-effect of removing the possibility for partial updates.20:04
kgriffsthat being said, a 3rd-party driver might make use of it since it is in the v1 API20:05
kgriffsWe are considering dropping the partial result option in a future API version20:05
kgriffsanyway, that's the story. :p20:05
*** whenry has quit IRC20:26
*** whenry has joined #openstack-marconi20:29
etoews1kgriffs: is the Client-Id header only required for the messages and claims APIs?20:39
kgriffstechnically, it is only required to be there when posting and listing messages20:42
kgriffsit isn't currently used with claims20:42
kgriffsit may be some day20:42
kgriffsi mean, the spec is admittedly fuzzy there20:43
kgriffswe have been telling people to just always send it20:44
kgriffsif clients always send client-id then we can compile some interesting stats that could be added to my-queue/stats20:45
kgriffssome other uses as well20:45
kgriffsLet me add this to my API feedback wiki page. We really should define this better.20:46
russell_hnot that it makes much difference, but it'd be sort of nice if client-ID didn't have to be a UUID20:47
russell_hthere's a thing I call the "curl barrier"20:47
russell_hbasically things that make an API painful to curl, but don't affect real clients20:48
*** etoews1 has quit IRC20:48
russell_hhaving to generate a UUID raises the curl barrier20:48
*** etoews1 has joined #openstack-marconi20:52
kgriffswhat would be some alternatives?20:53
russell_hjust an opaque string20:54
kgriffswe used to let people do any arbitrary string but it seemed to cause confusion.20:55
russell_hthe error message you get if it isn't a UUID could be better too20:55
russell_hI forget what it was, but it was confusing20:55
kgriffsoh, that could be the case20:55
kgriffsI have it on my todo list to scrub all the error messages. there is a lot of improvement to be done there.20:55
russell_hkgriffs: did you end up using repose?20:56
kgriffsayway, i am open to changing back in the next API version to allowing opaque string, but I'd like to find out what other things beside a UUID people want to use.20:56
kgriffsrussell_h: Nope.20:56
kgriffsthey had some blocking bugs20:56
kgriffsand performance isn't there (yet?)20:57
russell_hthe mailgun guys are working on a thing20:57
kgriffsso, we are using some WSGI middleware for now20:57
russell_hneeds some love20:57
kgriffsyeah, I heard someone mention that earlier today. I need to check it out.20:57
russell_hbut they tried to talk to the repose people about it, and came away.. not too happy20:57
russell_hso now they want to turn it into a replacement20:58
russell_hthis, too, needs some love20:58
russell_hbut is kind of working20:59
kgriffsI'll take a look at that as well20:59
openstackgerritZhihao Yuan proposed a change to openstack/marconi: WIP: feat(shard): queue listing
*** vkmc has joined #openstack-marconi21:29
openstackgerritA change was merged to openstack/marconi: fix(bootstrap): Logging CLI options don't work
*** oz_akan_ has quit IRC22:10
*** amitgandhi has quit IRC22:10
*** jcru has quit IRC22:18
openstackgerritZhihao Yuan proposed a change to openstack/marconi: WIP: feat(shard): queue listing
openstackgerritZhihao Yuan proposed a change to openstack/marconi: WIP: feat(shard): queue listing
*** kgriffs is now known as kgriffs_afk22:35
*** etoews1 has quit IRC22:40
openstackgerritZhihao Yuan proposed a change to openstack/marconi: WIP: feat(shard): queue listing
*** openstackgerrit has quit IRC22:46
*** openstackgerrit has joined #openstack-marconi22:46
*** fifieldt has joined #openstack-marconi23:02
*** ametts has quit IRC23:03
*** etoews1 has joined #openstack-marconi23:51
*** reed has joined #openstack-marconi23:53
*** etoews1 has quit IRC23:55

Generated by 2.14.0 by Marius Gedminas - find it at!