Monday, 2014-11-17

*** macjack has quit IRC00:00
*** witlessb has quit IRC00:14
*** georgem2 has joined #openstack-sahara01:21
openstackgerritKen Chen proposed openstack/sahara: Add ZooKeeper support to CDH plugin  https://review.openstack.org/13255901:28
openstackgerritlu huichun proposed openstack/sahara: Add HBase support to CDH plugin  https://review.openstack.org/13359901:52
*** tellesnobrega has joined #openstack-sahara02:32
*** tellesnobrega has quit IRC02:41
*** georgem2 has quit IRC02:53
*** witlessb has joined #openstack-sahara02:56
*** witlessb has quit IRC03:01
*** Longgeek has joined #openstack-sahara03:28
*** Longgeek has quit IRC03:33
*** Longgeek has joined #openstack-sahara04:44
*** Longgeek has quit IRC04:48
*** chandankumar has joined #openstack-sahara05:20
*** Longgeek has joined #openstack-sahara05:37
*** hogepodge has quit IRC06:01
*** Longgeek has quit IRC06:17
*** Longgeek has joined #openstack-sahara06:18
*** Longgeek_ has joined #openstack-sahara06:18
*** miqui has quit IRC06:22
*** Longgeek has quit IRC06:22
*** hogepodge has joined #openstack-sahara06:24
*** nikunj2512 has joined #openstack-sahara06:27
nikunj2512Hi06:27
nikunj2512i am getting error while getting data from sahara service06:28
nikunj2512Error: Failed to find some config files: policy.json06:28
openstackgerritlu huichun proposed openstack/sahara: Add HBase support to CDH plugin  https://review.openstack.org/13359906:51
*** Longgeek_ has quit IRC06:56
*** Longgeek has joined #openstack-sahara06:57
*** miqui_ has quit IRC07:22
*** Longgeek has quit IRC07:27
*** Longgeek has joined #openstack-sahara07:28
*** Longgeek_ has joined #openstack-sahara07:41
*** Longgeek has quit IRC07:45
*** witlessb has joined #openstack-sahara08:54
*** IvanBerezovskiy has joined #openstack-sahara09:06
*** skolekonov has joined #openstack-sahara09:19
openstackgerritVitaly Gridnev proposed openstack/sahara: Add provision step in creating cluster with direct engine  https://review.openstack.org/13449009:35
*** chandankumar has quit IRC10:56
*** chandankumar has joined #openstack-sahara11:08
*** openstackgerrit has quit IRC11:48
*** openstackgerrit has joined #openstack-sahara11:49
openstackgerritDenis Egorenko proposed stackforge/sahara-ci-config: Fix wrong update image for CDH plugin  https://review.openstack.org/13491211:54
*** nikunj2512 has quit IRC12:04
*** witlessb has quit IRC12:19
*** witlessb has joined #openstack-sahara12:20
*** nikunj2512 has joined #openstack-sahara12:39
openstackgerritVitaly Gridnev proposed openstack/sahara: Add provision step in creating cluster with direct engine  https://review.openstack.org/13449012:40
*** georgem2 has joined #openstack-sahara12:58
openstackgerritSergey Reshetnyak proposed openstack/sahara: Add list of open ports for Spark plugin  https://review.openstack.org/13492713:09
*** Networkn3rd has joined #openstack-sahara13:33
*** k4n0_ has quit IRC13:39
*** georgem2 has quit IRC13:45
*** _crobertsrh is now known as crobertsrh13:48
openstackgerritSergey Reshetnyak proposed openstack/sahara: Add list of open ports for Spark plugin  https://review.openstack.org/13492713:52
openstackgerritVitaly Gridnev proposed openstack/sahara-specs: Added specification for event log and logs improvements  https://review.openstack.org/11905213:55
*** _mattf` is now known as mattf13:58
*** mattf has quit IRC13:58
*** mattf has joined #openstack-sahara13:58
*** tellesnobrega has joined #openstack-sahara14:05
*** miqui has joined #openstack-sahara14:10
*** georgem2 has joined #openstack-sahara14:15
*** tellesnobrega has quit IRC14:15
openstackgerritMerged stackforge/sahara-ci-config: Fix wrong update image for CDH plugin  https://review.openstack.org/13491214:28
*** tellesnobrega has joined #openstack-sahara14:28
openstackgerritMerged openstack/sahara: Add list of open ports for HDP plugin  https://review.openstack.org/13418214:31
*** tellesnobrega has quit IRC14:32
*** tmckay has joined #openstack-sahara14:32
*** egafford has joined #openstack-sahara14:40
*** georgem2 has quit IRC14:43
*** tmckay has quit IRC14:45
*** chandankumar has quit IRC14:47
openstackgerritSergey Reshetnyak proposed openstack/sahara: Add list of open ports for Spark plugin  https://review.openstack.org/13492714:48
*** Longgeek_ has quit IRC14:59
*** tmckay has joined #openstack-sahara15:07
openstackgerritAndrey Pavlov proposed openstack/sahara: Fixed bug with Hive jobs fail  https://review.openstack.org/13395215:48
*** georgem2 has joined #openstack-sahara15:51
tmckaycrobertsrh, ping15:57
crobertsrhyes?15:57
tmckaydid you say on Friday you had a spec for filtering, or did I imagine that?  My (possibly lame) search didn't turn up anything15:58
crobertsrhhttps://review.openstack.org/#/c/134319/15:59
tmckayreally??  Why didn't my search find that ... thanks15:59
crobertsrhheh15:59
tmckayoh, duh.  There is more than one page of results16:00
tmckayegafford, congrats, your spec apparently was merged :)16:03
tmckaycooking with gas now16:04
egaffordtmckay: Indeed; first OSS merge! Starting small, but that's good, really.16:05
egaffordtmckay: I'm very fond of the workflow. It's head and shoulders above any review process I've seen in private software.16:05
crobertsrhYeah, I <3 the gerrit review process16:06
tmckayegafford, I completely agree.  Sahara is some of the cleanest code I've worked on in years16:06
*** nikunj2512 has left #openstack-sahara16:06
tmckayLong live Gerrit and OpenStack!  Huzzah!16:06
* tmckay waves (red) hat wildly16:06
egaffordcrobertsh, tmckay: Oh my goodness yes. I've pointed some of my old team at the codebase, basically saying "you know all that stuff I was ranting about all year re: legibility? Look at this. Particularly the API in which every method is a 1-2 liner with 3 decorators on it."16:07
egaffordcrobertsh, tmckay: Beautiful stuff.16:08
tmckaycroberts, so do you have detail on the "query-style parameters" in search_opts for filtering?  What do they look like?16:12
tmckayI'm wondering if we need to do anything with them, or if we can just pass them through to the alembic layer stuff16:12
tmckay"== foo", "> foo", "<= foo", things like that?16:13
tmckaycrobertsrh even ^^16:13
crobertsrhI was thinking of url-style query parameters:  <uri>?<name>=<val>16:14
tmckayah16:14
crobertsrhI'm open to suggestions though.16:14
tmckayis = enough?16:14
tmckayis that what other OS projects are doing?16:14
crobertsrhI took a peek at what nova has for its filters16:14
tmckayI should take a look too16:15
crobertsrhI think SergeyLukjanov was mostly thinking about the ability to filter by name/plugin16:15
crobertsrhWe don't really have a lot of numbers to filter by16:15
tmckayhmm, true.16:16
crobertsrhIt looks like other pages really only support single field queries (UI-wise).  Although, it would be possible to specify multiple name/values via the client lib or REST16:17
tmckayon line 71 of your spec, did you mean to include a list of methods?  Or does the second sentence cover it?16:18
SergeyLukjanovcrobertsrh, ack about name/plugin/version16:19
SergeyLukjanovcrobertsrh, and I think it's done on client side for all projects16:19
SergeyLukjanovcrobertsrh, I mean filtering done in client side in the list operation of the python-NNNclient16:19
tmckaySergeyLukjanov, really, not a filter on the database get?16:20
tmckayThat would be the most efficient, in sahara-api16:21
tmckayAlthough it's more work.  But then it's an optional part of the REST API, and the client can just pass the filter parameters through16:22
elmikocrobertsrh: something else to consider, if you want more eyeballs on the api changes use the APIImpact flag in your commit message and we can get word to the API working group to weigh in16:23
tmckayelmiko, thanks for the heads up16:24
*** skolekonov has quit IRC16:24
elmikonp, i know those folks are looking for opportunities to help16:24
crobertsrhI do have the apiimpact flag in there16:28
elmikocrobertsrh: cool, i thought you did but couldn't remember16:29
*** georgem2 has quit IRC16:29
elmikoif they don't chime in before wednesday i'll make sure to bring it up at their meeting16:30
elmikoor, you can always email the openstack-dev ml with [api] in the subject to ping them16:30
elmikoi know there is a little debate currently as to how active they want to get in searching for new reviews to jump on16:30
crobertsrhIt does seem that filtering in the client only is a bit wasteful.16:32
elmikoyea, that's a lot of data coming back that might go unused16:33
tmckaycrobertsrh, looking at the nova client.  It passes search_opts to cs.xxx.list in several places16:33
tmckayThe question is, what kind of object is cs?16:33
tmckayIs it an interface to the REST API?  If so, we have a precedent16:34
tmckayIt's silly imho to filter outside of a database when that is what a database is for16:34
*** chandankumar has joined #openstack-sahara16:34
crobertsrh+1 on silly16:34
tmckaybesides, if it's behind the client, it's transparent to things consuming the client anyway16:34
elmikotmckay: agreed, seems like the params should get passed on to the rest server16:34
tmckayso it shouldn't matter16:34
crobertsrhextra bonus, any consumers of the api itself can also use filtering16:35
tmckayI just have to track down what this "cs" is.  Probably the debugger is the easiest way :)16:35
tmckaycrobertsrh, ++.  There would have to be a very compelling reason not to do it in sahara-api, I think16:35
elmikocrobertsrh: i thought that's the way it _supposed_ to be16:35
crobertsrhNova client passes it to the REST API....   return self._list("/servers%s%s" % (detail, query_string), "servers")16:35
* tmckay goes to debug nova client16:36
crobertsrhThat is from novaclient/v1_1/servers.py line 60316:36
tmckayah, there you go16:37
tmckayI was looking in v3, I think.  I'll check anyway, more info is better16:37
crobertsrhah, I suppose they could have changed things over time16:38
SergeyLukjanovtmckay, crobertsrh, I'm waiting for plain, so, unable to chat atm16:41
SergeyLukjanovtmckay, crobertsrh, and I'm on a vacation this week :)16:41
tmckaySergeyLukjanov, np, good travels :)16:41
crobertsrhNo worries.  Have fun16:41
tmckaySergeyLukjanov, don't worry, we'll make something great :)  And you can +2 or -216:41
SergeyLukjanovtmckay, crobertsrh, but re filtering - IMO it should be done first on the client side in the way to be able to implement it on server side and replace client side filtering with the new server side one16:42
SergeyLukjanovtmckay, crobertsrh, to avoid additional efforts needed to refactor server side to support filtering and concentrate on the more important stuff16:42
SergeyLukjanovmy 2 cents16:42
SergeyLukjanovheh, moving to Geneva from Paris16:43
tmckayhmm, I could see that.  It would free up UI guys sooner16:43
tmckaycrobertsrh, if we do client support first, the UI stuff can be verified and tests written, and then you are free to chase something else16:44
tmckaycertain amount of throw away, but there is a logic to it.  UI work seems heavy for kilo16:45
crobertsrhTrue, but will we ever circle back to "do it right"?16:45
tmckayI will, for sure.  Immediately.16:45
tmckayIf we know it's temporary, then it can just be a simple application of the dictionary against the list coming back.16:45
tmckayIt can even be sloppy16:46
elmikoyea, tough issue...16:46
crobertsrhI don't think we ever want to release a temporary/sloppy client, do we?16:46
tmckaynot usually a fan of temporary measures, but in this case ...16:47
* tmckay shrugs16:47
crobertsrhMaybe we can figure the effort involved in each and make a decision based on that16:47
elmikoi guess, i'd like to know more about the extent of changes needed to add server-side filtering16:47
tmckayack16:47
crobertsrhRight.  That is where I'm most uncertain16:47
tmckayalright, let's keep evaluating.  I bet I can mock something up this week on the server side, see how much effort it is16:48
tmckaycrobertsrh, on the other hand, if you write code against a modified (unreleased) client and we just verify that you pass the right stuff in, then you can go your merry way16:49
tmckaythe client can just dump it on the floor16:49
crobertsrhThat's what I figured I'd wind-up doing for starters.16:50
tmckaywe just need to know the interface works for you to be done and move on16:50
tmckaythat part at least isn't throw away16:50
tmckayk, late binding.  Let's go that way, we don't have to make a decision yet16:50
*** chandankumar has quit IRC17:03
openstackgerritAndrey Pavlov proposed openstack/python-saharaclient: Added unit tests for python bindings  https://review.openstack.org/13416717:05
openstackgerritAndrey Pavlov proposed openstack/sahara: Fixed bug with Hive jobs fail  https://review.openstack.org/13395217:10
openstackgerritSergey Reshetnyak proposed openstack/sahara: Add list of open ports for Spark plugin  https://review.openstack.org/13492717:13
*** shakamunyi has joined #openstack-sahara17:32
*** shakamunyi has quit IRC17:32
*** chandankumar has joined #openstack-sahara17:38
openstackgerritSergey Kolekonov proposed stackforge/sahara-ci-config: Fix node label for ui tests  https://review.openstack.org/13503317:44
*** stannie has quit IRC17:48
*** chandankumar has quit IRC18:25
tmckaycrobertsrh, so, playing with the /images API which supports query strings, and hacking the client, this works"20:21
tmckay'/images?tags=1.2.1&tags=vanilla'20:21
crobertsrhAwesome20:22
tmckayso we could support a multidict in the client, or a simple dict20:22
tmckaycrobertsrh, in the case of tags it's actually written to support a multidict.20:23
tmckaybut, if we're just supporting name=x and plugin=x then a simple dict will work20:23
tmckayor, we could take a list of tuples if a multidict isn't an option20:23
crobertsrhWell, then the question is:  do we also want to handle pagination, etc "while we're in there"20:23
crobertsrhI suppose pagination n/v can also just be in the simple dict20:24
elmikoi'd say start simple and build up20:24
crobertsrhjust special n/v pairs20:24
elmikodo filtering now, add pagination in another spec. imo20:24
tmckayyeah, I'm a newbie to pagination in REST.  special n/v makes sense to me, but maybe it's not done that way :)20:24
tmckayelmiko, there is some wisdom in that20:25
tmckaycrobertsrh, do we want to support a multidict scenario, as for tags?  Things where a field might be a list?  For name/plugin it's certainly not20:25
crobertsrhI can't really think of a use case for it.20:26
tmckayor, we could have the convention that we just interpret a list value for a key as key=n&key=m&key=l20:26
elmikohow do other projects do it?20:26
tmckayelmiko, don't know.  the tags thing is the only instance in Sahara, and even then a multiple value field is an outlier20:27
tmckayI suppose we can look elsewhere20:27
crobertsrhBased on the nova client lib, I don't see lists implemented there.  The REST side might tell otherwise though20:28
elmikoyea, i'm trying to think what other projcets use a list like that20:28
tmckayfor a simple dict, I just need to add a loop that tacks on ?key=value&key2=value2& ....20:28
tmckayusable for any URL20:28
tmckayI don't even think it will break Sahara to pass it in, they'll just be ignored20:29
tmckaybrb20:29
elmikoi would think in the case of tags you'd want "...?key=v1,v2,v3&anotherkey=..." but that's just a guess20:29
tmckaythe decorator for images_list is just this20:29
tmckay@rest.get('/images')20:29
tmckay@acl.enforce("images:get_all")20:29
tmckaynothing special there20:30
tmckayelmiko, based on hacking the client and running Sahara in the deugger you repeat "tags" as a key  (tags=1&tags=2)20:30
tmckaythis does the right thing20:30
elmikook20:30
tmckaynot intuitive to me, but that's how the parser works20:31
elmikoi'm thinking more about how the api call should look, less about the backend. i'm curious from a high level how it should look.20:31
tmckayand you pull it out with args.getlist('tags'), args.get('tags') will just give you one :)20:31
crobertsrhI think that's a fairly common interpretation of same name/different vals in a URL20:32
tmckaythat is how it looks in the api call20:32
tmckayThat's what flask allows, unless we parse the string ourselves and split on comma20:32
elmikocrobertsrh: not using multiple values per key?20:32
tmckaycrobertsrh, that's the way cumin did it.  Enough said :-D20:32
elmikotmckay: i'm just trying to decouple the rest api call from the wsgi framework we happen to be using20:32
crobertsrhname=myfirstval&name=mysecondval    --> name = [myfirstval, mysecondval]20:33
crobertsrhI hear that cumin was amazing20:33
elmikocrobertsrh: yea, if that's good form then cool20:33
tmckayelmiko, ack, I hear you.  I think it's standard, though.  We should verify20:33
elmikotmckay: agreed20:33
tmckayok, really brb20:33
crobertsrhIt does ring a bell of something that is "standard"20:33
elmikocool20:34
elmikoyea i don't really know what the "standard" is, i just want to avoid shaping the call based on our framework. seems like a leaky abstraction in that case.20:35
tmckayhttp://www.w3schools.com/asp/coll_querystring.asp20:43
tmckayExample 120:43
tmckayn=John&n=Susan20:44
elmikoto be fair though, that's for an asp.net application20:45
tmckaybased on a quick search, the form seems to be pretty much accepted.  What varies is how the framework returns the values.20:48
tmckayso flask has get and get_list, another framework might just return a list from get.  But it sounds like the form is expected20:49
*** openstackgerrit has quit IRC20:49
*** openstackgerrit has joined #openstack-sahara20:49
elmikotmckay: cool, i don't have any issue with the formatting, other than using the more common method(regardless of our framework)20:50
tmckaycrobertsrh, so which "list()" ops do you want to implement this for?  all of them?20:51
crobertsrhI think we should probably do all just to stay consistent20:51
elmiko+120:51
crobertsrhUnlikely that plugins really needs it, but....20:51
tmckayokay.  Anything that can be listed from the client.  Got it.  I'll start banging on a client patch.20:52
tmckayUnless you've done it already20:52
crobertsrhNo, I have not yet patched the client.20:52
tmckayk.  Do you want to, or should I?20:52
tmckayI guess it's all the same, it gets you off the task either way20:53
crobertsrhI can take care of the client part20:53
tmckayk20:53
crobertsrhSo, each client list() method gets a new param.  What should we call it?20:53
tmckaysearch_opts seems to be standard in OS20:53
crobertsrhI think the spec used "search_opts"20:53
tmckayyep20:53
crobertsrhAnd then one new method in api.base that will take the searchopts dict and convert to ?<name>=value&.....20:55
crobertsrheach of the list() client methods can just call that and form the full query string /<object_type>?<query string>20:56
crobertsrh...which will just fall on deaf ears for now until the backend is updated20:56
tmckayabsolutely20:57
tmckayjust where I was going to go.  We must be right :)20:57
crobertsrhHeh :)20:58
tmckayon an unrelated note, I heard "Dumb and Dumber To" was a big hit this weekend20:58
crobertsrhOk, I can get that up "real soon"20:58
crobertsrhI hope that's an unrelated note :)20:58
tmckayof course it is20:58
crobertsrhMy wife and her sister are pretty geeked to see it20:58
crobertsrhThey can [obnoxiously] recite the original line for line20:59
tmckayI might have to, just cause20:59
tmckayoh my20:59
tmckayI'm not worthy20:59
crobertsrhI wish I wasn't so worthy of hearing them :)20:59
elmikocrobertsrh: lol, gonna have to mention that to B., she keeps asking "who would want to see that?"21:00
crobertsrhThe original (among other things) helped build Jeff Daniels a pretty nifty compound on a lake near here.21:00
elmikonice21:00
elmikoi knew the original was like a cult-classic21:00
crobertsrhI enjoyed it....but I think I was closer to 17 at that point :)21:01
elmikolol21:01
crobertsrhI fully expect that I will eventually see the new one.21:02
crobertsrhIf only to scoff at it as an unworthy sequel, like Caddyshack 2 to Caddyshack21:02
elmikooh man...21:02
openstackgerritChad Roberts proposed openstack/python-saharaclient: Adding support for query filtering to list() calls  https://review.openstack.org/13508621:22
tmckaywoohoo, review time21:23
crobertsrhoops, tiny mistake21:23
openstackgerritChad Roberts proposed openstack/python-saharaclient: Adding support for query filtering to list() calls  https://review.openstack.org/13508621:26
tmckaynice, even  urlencode uses the & form21:29
*** miqui has quit IRC21:29
crobertsrhYeah, a lovely fit21:34
elmikolooks like that wasn't too bad at all21:37
tmckaycrobertsrh, elegant, simple21:38
tmckaynow I just have to get out the sausage maker for the backend ;-)21:39
crobertsrhIt took about 10 min....but that was with my daughter breaking down the baby gate on the stairs once to come up and visit.21:39
elmikolol21:39
tmckayand it works.  I just hacked the client shell to pass a search_opts dictionary21:41
tmckaysahara did the right thing (for images anyway)21:41
tmckayextra arguments are ignored21:42
elmikonice21:42
jodahjust noticed the conversation earlier about how to use query params for key/value pairs21:42
tmckay"GET /v1.1/5e8128971f1342a08571a5c5a0e3f9bd/clusters?goat=cheese HTTP/1.1" 200 148 0.07201021:43
jodah?tags=k1:v1,k2:v2,k3:v3&someOtherQueryParam=SomeOtherValue21:43
tmckaygoat cheese is ignored on cluster list, nice21:43
jodahas opposed to k1=v2,k2=v2,k3=v3, so as not to be confused with other possible query params21:43
elmikojodah: what about in terms of ?tags=k1:v1&tags=k2:v2....21:45
tmckayjodah, in this case, tags is the key, but may have multiple values21:45
jodahthat works too21:45
tmckayso there is only 1 key, "tags"21:45
jodahah ok21:45
tmckayit was our only example of consuming query args in Sahara, seems to be mostly unused previously21:46
tmckaybut flask and urlencode mesh nicely21:46
elmikoi feel like this is probably enshrined in an RFC somewhere...21:47
elmikosince everything seems to follow the same pattern21:47
tmckayheh, and get_images of course doesn't go to the Sahara db, it goes to the nova client21:47
tmckayso I'm breaking new ground here :)21:48
tmckayelmiko, some of the notes mentioned on RFC.  But it was silent on the multiple value thing, apparently21:48
crobertsrhI bet it will all come down to < 30 lines of code change21:49
crobertsrhand we'll ask ourselves, why didn't we already do this?21:49
tmckayheh, if I do it right21:49
elmikocrobertsrh: +221:50
tmckaypassing all the stuff down to the db layer will be easy, just a bunch of optional search_opts21:50
tmckaybut then ... but then ...21:51
* tmckay runs away waving arms21:51
tmckaywe need to have this all done and pefect before SergeyLukjanov gets back ;-) So he can keep his two cents and be richer.21:52
crobertsrhdb.magicstuff()21:52
*** tellesnobrega has joined #openstack-sahara21:52
elmikolol21:52
tmckayhmmm, ever since I did a fresh pull on Sahara the hdp plugin won't load21:53
tmckaydid you guys see that?21:53
elmikoi've had issue like that, usually i need to remove the build directory and the local sahara.egg.*, then reinstall21:54
crobertsrhI haven't come across that one yet21:57
crobertsrhI had a similar problem when the old vanilla version was removed21:57
*** crobertsrh is now known as _crobertsrh22:00
*** tmckay has quit IRC22:13
*** macjack has joined #openstack-sahara22:26
*** tellesnobrega has quit IRC22:37
*** rharwood has joined #openstack-sahara23:01
*** witlessb has quit IRC23:13
*** egafford has quit IRC23:32
*** macjack has quit IRC23:52

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!