| *** alazarev has quit IRC | 00:23 | |
| *** alazarev has joined #openstack-sahara | 00:31 | |
| *** vrovachev has quit IRC | 01:12 | |
| *** ylobankov has quit IRC | 01:12 | |
| *** IlyaE has joined #openstack-sahara | 01:15 | |
| *** IlyaE has quit IRC | 01:15 | |
| *** alazarev has quit IRC | 01:15 | |
| *** IlyaE has joined #openstack-sahara | 01:36 | |
| *** vrovachev has joined #openstack-sahara | 01:49 | |
| *** ylobankov has joined #openstack-sahara | 01:49 | |
| *** rhodgin has joined #openstack-sahara | 02:02 | |
| *** bradd1 has joined #openstack-sahara | 02:05 | |
| *** ErikB1 has joined #openstack-sahara | 02:27 | |
| *** _crobertsrh has quit IRC | 02:36 | |
| *** ErikB1 has quit IRC | 02:52 | |
| *** witlessb has joined #openstack-sahara | 02:56 | |
| *** witlessb has quit IRC | 02:56 | |
| *** bradd1 has quit IRC | 03:15 | |
| *** tmckay has quit IRC | 03:27 | |
| *** rhodgin has quit IRC | 04:35 | |
| *** Ch00k has joined #openstack-sahara | 05:26 | |
| openstackgerrit | Jenkins proposed a change to openstack/sahara: Imported Translations from Transifex https://review.openstack.org/82719 | 06:05 |
|---|---|---|
| *** IlyaE has quit IRC | 06:22 | |
| *** IlyaE has joined #openstack-sahara | 06:23 | |
| *** IlyaE has quit IRC | 06:35 | |
| *** Ch00k has quit IRC | 06:40 | |
| *** Ch00k has joined #openstack-sahara | 06:49 | |
| *** Ch00k has quit IRC | 07:25 | |
| *** Ch00k has joined #openstack-sahara | 08:06 | |
| *** tnovacik has joined #openstack-sahara | 08:26 | |
| *** Ch00k_ has joined #openstack-sahara | 08:50 | |
| *** Ch00k has quit IRC | 08:52 | |
| *** Ch00k_ has quit IRC | 08:59 | |
| *** Ch00k has joined #openstack-sahara | 08:59 | |
| *** witlessb has joined #openstack-sahara | 08:59 | |
| *** dmitryme has joined #openstack-sahara | 09:09 | |
| *** Ch00k has quit IRC | 09:11 | |
| *** Ch00k has joined #openstack-sahara | 09:20 | |
| *** sreshetnyak has quit IRC | 09:25 | |
| openstackgerrit | Sergey Lukjanov proposed a change to stackforge/puppet-sahara: Update README file https://review.openstack.org/83319 | 09:25 |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack/sahara: Remove Mirantis copyright from README https://review.openstack.org/83320 | 09:27 |
| openstackgerrit | Sergey Lukjanov proposed a change to stackforge/puppet-sahara: Update README file https://review.openstack.org/83319 | 09:28 |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack/sahara-dashboard: Remove Mirantis copyright from README https://review.openstack.org/83321 | 09:30 |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack/sahara-extra: Standardize README file header https://review.openstack.org/83322 | 09:32 |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack/sahara: Standardize README header https://review.openstack.org/83325 | 09:40 |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack/sahara: Add note about OS_TENANT_* to integration tests https://review.openstack.org/82819 | 09:46 |
| *** skolekonov has joined #openstack-sahara | 10:02 | |
| openstackgerrit | Sergey Reshetnyak proposed a change to openstack/sahara: Add EDP support for Vanilla 2 plugin https://review.openstack.org/82474 | 10:16 |
| openstackgerrit | Sergey Reshetnyak proposed a change to openstack/sahara: Add EDP integration tests for vanilla 2 plugin https://review.openstack.org/82733 | 10:16 |
| *** dmitryme has quit IRC | 10:27 | |
| *** dmitryme has joined #openstack-sahara | 10:30 | |
| *** tosky has joined #openstack-sahara | 10:36 | |
| *** Ch00k has quit IRC | 11:10 | |
| *** Ch00k has joined #openstack-sahara | 11:11 | |
| *** IvanBerezovskiy has joined #openstack-sahara | 11:38 | |
| *** ErikB1 has joined #openstack-sahara | 11:48 | |
| *** sballe has quit IRC | 12:09 | |
| *** crobertsrh has joined #openstack-sahara | 12:20 | |
| *** tmckay has joined #openstack-sahara | 12:23 | |
| *** sballe has joined #openstack-sahara | 12:25 | |
| *** ErikB1 has quit IRC | 12:30 | |
| *** tosky has quit IRC | 12:31 | |
| *** dmitryme has quit IRC | 12:36 | |
| *** tosky has joined #openstack-sahara | 12:38 | |
| *** tmckay has quit IRC | 12:42 | |
| *** dmitryme has joined #openstack-sahara | 12:43 | |
| *** tmckay has joined #openstack-sahara | 12:48 | |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack/python-saharaclient: [DO NOT MERGE] Test gating https://review.openstack.org/83364 | 12:59 |
| *** tmckay has quit IRC | 13:04 | |
| *** tmckay has joined #openstack-sahara | 13:05 | |
| *** tmckay has quit IRC | 13:07 | |
| *** tmckay has joined #openstack-sahara | 13:08 | |
| *** tmckay has quit IRC | 13:08 | |
| *** tmckay has joined #openstack-sahara | 13:08 | |
| *** Ch00k has quit IRC | 13:21 | |
| *** tmckay has quit IRC | 13:22 | |
| *** tmckay has joined #openstack-sahara | 13:26 | |
| *** Ch00k has joined #openstack-sahara | 13:27 | |
| *** tmckay has quit IRC | 13:36 | |
| openstackgerrit | Sergey Reshetnyak proposed a change to openstack/sahara: Add EDP integration tests for vanilla 2 plugin https://review.openstack.org/82733 | 13:40 |
| *** ErikB1 has joined #openstack-sahara | 13:43 | |
| openstackgerrit | Dmitry Mescheryakov proposed a change to openstack/sahara: Fix transient clusters termination https://review.openstack.org/83382 | 13:46 |
| *** elmiko has joined #openstack-sahara | 13:47 | |
| *** dmitryme has quit IRC | 13:58 | |
| openstackgerrit | A change was merged to openstack/sahara: Add integration tests for vanilla 2 plugin https://review.openstack.org/77089 | 14:03 |
| *** bradd1 has joined #openstack-sahara | 14:07 | |
| *** tmckay has joined #openstack-sahara | 14:17 | |
| *** ErikB1 has quit IRC | 14:20 | |
| *** dmitryme has joined #openstack-sahara | 14:26 | |
| *** alazarev has joined #openstack-sahara | 14:29 | |
| openstackgerrit | A change was merged to openstack/sahara: Add note about OS_TENANT_* to integration tests https://review.openstack.org/82819 | 14:35 |
| openstackgerrit | Nikita Konovalov proposed a change to openstack/sahara-dashboard: Compatibility improvement https://review.openstack.org/83402 | 14:39 |
| openstackgerrit | Nikita Konovalov proposed a change to openstack/sahara-dashboard: Horizon compatibility fix. https://review.openstack.org/83402 | 14:48 |
| openstackgerrit | Nikita Konovalov proposed a change to openstack/sahara-dashboard: Horizon compatibility fix https://review.openstack.org/83402 | 14:49 |
| openstackgerrit | Nikita Konovalov proposed a change to openstack/sahara-dashboard: Horizon compatibility fix https://review.openstack.org/83402 | 14:51 |
| openstackgerrit | Sergey Reshetnyak proposed a change to openstack/sahara: Fix parallel running integration tests with vanilla plugins https://review.openstack.org/83410 | 14:57 |
| *** skolekonov has quit IRC | 14:58 | |
| *** IlyaE has joined #openstack-sahara | 14:58 | |
| *** ErikB1 has joined #openstack-sahara | 15:02 | |
| *** alazarev has quit IRC | 15:13 | |
| *** IlyaE has quit IRC | 15:19 | |
| *** IlyaE has joined #openstack-sahara | 15:23 | |
| *** dmitryme has quit IRC | 15:25 | |
| *** Ch00k has quit IRC | 15:25 | |
| *** Ch00k has joined #openstack-sahara | 15:28 | |
| *** Ch00k has quit IRC | 15:29 | |
| *** Ch00k has joined #openstack-sahara | 15:35 | |
| *** dmitryme has joined #openstack-sahara | 15:38 | |
| openstackgerrit | Dmitry Mescheryakov proposed a change to openstack/sahara: Remove agent remote https://review.openstack.org/82757 | 15:47 |
| dmitryme | tmckay: ping, re https://review.openstack.org/#/c/75527 | 15:55 |
| dmitryme | I am not sure I understood you correctly | 15:56 |
| tmckay | dmitryme, hi, working on that now, actually | 15:56 |
| tmckay | on what? | 15:56 |
| dmitryme | are you going to make changes based on our conversation in comment, or you would prefer to commit the code as is and fix it later | 15:57 |
| dmitryme | ? | 15:57 |
| *** alazarev has joined #openstack-sahara | 15:59 | |
| dmitryme | it is not clear from your responses and you didn't post new patch sets in a couple of days, so that is why I am asking | 15:59 |
| tmckay | dmitryme, doing my best to address the comments now. For instance, I have no idea why RuntimeError is used in that print statement, I don't see the benefit. And I agree on the admin flag, it should be removed. Also, file deletion should be fixed. | 15:59 |
| tmckay | dmitryme, ack, I meant to but I got sidetracked :) | 15:59 |
| dmitryme | ok, thanks | 16:00 |
| tmckay | dmitryme, currently looking closely at Singleton wrapper -- when I run the test from tox without a config file, I get the message printiing 3 times. That doesn't seem very "singletonish" to me | 16:00 |
| tmckay | Could be from time slicing -- that Singleton wrapper is not necessarily thread safe. Although, I'm not an expert on green threads | 16:01 |
| tmckay | not sure what I'm seeing, but I'd like to understand it | 16:01 |
| tmckay | brb | 16:01 |
| dmitryme | tmckay: there is no eventlet in python-saharaclient, so there is no green threads | 16:10 |
| tmckay | ah | 16:11 |
| dmitryme | looking at the tests, frankly I don't understand how they are run | 16:12 |
| dmitryme | hmm, comparing toxin of sahara and saharaclient, seems like testr is used in both | 16:13 |
| dmitryme | I know that testr runs tests in several processes simultaneously | 16:14 |
| tmckay | dmitryme, yes, I just printed out process ids :) | 16:14 |
| tmckay | multiple processes. | 16:14 |
| tmckay | I wonder if that message should just go to a log instead of the console. Weird to see it three times. But, minor. | 16:15 |
| tmckay | dmitryme, actually, my sidetrack was splitting the cluster building out to a separate class and testing hdp. Got it working. | 16:19 |
| *** Ch00k has quit IRC | 16:20 | |
| dmitryme | tmckay: got it | 16:20 |
| tmckay | dmitryme, had a strange problem with nova, though. When a cluster is launched and the bridge is set up for floating ips, if I am using wireless I lose my connection to the outside world. Hard wired, it works. Any chance you know anything about that? :) Besides "use neutron" | 16:20 |
| dmitryme | tmckay: fortunately we got separate hardware where we run OpenStack | 16:22 |
| dmitryme | generally we don't run it on our laptops | 16:22 |
| *** Ch00k has joined #openstack-sahara | 16:24 | |
| tmckay | dmitryme, yeah... I need separate hardware :) | 16:26 |
| *** ErikB1 has quit IRC | 16:27 | |
| *** Ch00k has quit IRC | 16:34 | |
| openstackgerrit | Andrew Lazarev proposed a change to openstack/sahara: [IDH] Integration tests for IDH 3.0.2 https://review.openstack.org/79719 | 16:36 |
| openstackgerrit | Jenkins proposed a change to openstack/python-saharaclient: Updated from global requirements https://review.openstack.org/83492 | 16:40 |
| openstackgerrit | Jenkins proposed a change to openstack/sahara: Updated from global requirements https://review.openstack.org/82482 | 16:40 |
| *** tnovacik has quit IRC | 17:01 | |
| *** skostiuchenko has joined #openstack-sahara | 17:06 | |
| *** ErikB1 has joined #openstack-sahara | 17:10 | |
| *** Ch00k has joined #openstack-sahara | 17:12 | |
| *** Ch00k has quit IRC | 17:23 | |
| *** themistymay has joined #openstack-sahara | 17:30 | |
| *** dmitryme has quit IRC | 17:34 | |
| openstackgerrit | Trevor McKay proposed a change to openstack/python-saharaclient: Add cli integration test https://review.openstack.org/75527 | 17:34 |
| *** vrovachev has quit IRC | 17:36 | |
| tosky | uhm, https://wiki.openstack.org/wiki/Meetings#Savanna_team_meeting has not been renamed - does it need a blueprint for that? | 17:45 |
| *** alazarev has quit IRC | 17:47 | |
| *** alazarev has joined #openstack-sahara | 17:48 | |
| *** alazarev has quit IRC | 17:52 | |
| *** jspeidel has joined #openstack-sahara | 17:53 | |
| SergeyLukjanov | tosky, nice catch => /me updating | 17:53 |
| tosky | SergeyLukjanov: also the linked page of course | 17:53 |
| *** _mattf is now known as mattf | 17:54 | |
| ErikB1 | jspeidel, hi | 17:59 |
| jspeidel | *sigh* | 17:59 |
| aignatov | I'll join the meeting a little bi later | 18:01 |
| *** alazarev has joined #openstack-sahara | 18:04 | |
| tosky | uhm, no meeting? | 18:06 |
| mattf | SergeyLukjanov, aignatov, who's chairing the meetiing today? | 18:06 |
| SergeyLukjanov | meeting time | 18:07 |
| SergeyLukjanov | #openstack-meeting-alt | 18:08 |
| jspeidel | thanks | 18:08 |
| openstackgerrit | A change was merged to openstack/sahara: Fix running integration tests by tag https://review.openstack.org/83019 | 18:20 |
| openstackgerrit | A change was merged to openstack/sahara: Add transient tag to transient cluster test https://review.openstack.org/83022 | 18:20 |
| *** Ch00k has joined #openstack-sahara | 18:26 | |
| *** Ch00k has quit IRC | 18:29 | |
| *** dmitryme has joined #openstack-sahara | 18:30 | |
| *** Ch00k has joined #openstack-sahara | 18:33 | |
| *** Ch00k has quit IRC | 18:41 | |
| aignatov | we should postpone Heat by default | 19:00 |
| jspeidel | aignatov, +1 | 19:00 |
| aignatov | because of bug I've filed today at least | 19:01 |
| aignatov | in the heat | 19:01 |
| aignatov | https://bugs.launchpad.net/heat/+bug/1298350 | 19:01 |
| tmckay | aignatov, jvyas may have some stuff related to analyzing proteins, trying to find him but maybe he's on vacation | 19:01 |
| tmckay | that would be cool, and different | 19:01 |
| aignatov | tmckay: yes | 19:01 |
| jspeidel | mattf, can we discuss my comments on api v2? | 19:01 |
| SergeyLukjanov | elmiko, what's your launchpad id? | 19:01 |
| *** witlessb has quit IRC | 19:02 | |
| elmiko | i think mimccune | 19:03 |
| *** witlessb has joined #openstack-sahara | 19:03 | |
| elmiko | yea, that's it | 19:03 |
| tmckay | aignatov, okay, I emailed jvyas, we'll see if he pops up. | 19:04 |
| jspeidel | mattf, are you sill here? | 19:04 |
| aignatov | tmckay: thx, I still wait for ErikB1 ideas too :) | 19:05 |
| aignatov | otherwise, we should brainstorm and generate our case for demo :) | 19:05 |
| tmckay | agreed. I'll try to remember to look in the books, I left them at home this time | 19:07 |
| aignatov | tmckay: before the final decision on the example of my last demo at HK summit I got an idea from folks about calculating KLOC metric on openstack sources | 19:10 |
| aignatov | with mapreduce | 19:11 |
| aignatov | between for example havana and icehouse releases :) | 19:12 |
| mattf | jspeidel, yeah, moment | 19:12 |
| jspeidel | mattf, ok, I have 15 mins until I have a call | 19:13 |
| *** tosky has quit IRC | 19:13 | |
| tmckay | aignatov, that's a great idea | 19:13 |
| aignatov | tmckay: you think so? | 19:14 |
| aignatov | actually there is an program http://www.dwheeler.com/sloccount/ | 19:14 |
| aignatov | for linux :) | 19:14 |
| aignatov | probably we could use streaming map reduce for this | 19:15 |
| aignatov | or just java action | 19:15 |
| tmckay | aignatov, yeah, the target audience would love it, wouldn't they? Is it big enough? Enough total files across openstack? Maybe mappers do file diffs, and reducer adds up the totals? | 19:15 |
| aignatov | yep, It'll be the funniest thing to know what is the price of code of icehouse release :) | 19:16 |
| tmckay | aignatov, I like it. Wondering what's the best way to supply a mapper with files to diff. | 19:17 |
| aignatov | tmckay: today, all openstack sources has about 1GB | 19:17 |
| aignatov | in raw | 19:17 |
| tmckay | and it has to have a way to deal with files that are not in both | 19:17 |
| tmckay | aignatov, maybe we just run 2 jobs, instead of trying to diff. Total in havanna, total in icehouse | 19:18 |
| tmckay | That's much simpler | 19:18 |
| aignatov | tmckay: it's up to us :) | 19:19 |
| tmckay | aignatov, we could count comment lines as a metric, to see how well developers are documenting their work :) As a percentage of total lines of code | 19:19 |
| *** elmiko has quit IRC | 19:19 | |
| tmckay | I think diffing in the mapper is overcomplicated, firing off 2 parallel jobs at the same time is better and shows Sahara doing more than one thing at a time | 19:20 |
| *** jayunit100 has joined #openstack-sahara | 19:20 | |
| jayunit100 | hola tmckay | 19:20 |
| jayunit100 | did you miss me ? | 19:20 |
| tmckay | jayunit100, I did! We're looking for cool hadoop demos. aignatov had this idea ..... | 19:21 |
| tmckay | (03:10:59 PM) aignatov: tmckay: before the final decision on the example of my last demo at HK summit I got an idea from folks about calculating KLOC metric on openstack sources | 19:21 |
| tmckay | (03:11:06 PM) aignatov: with mapreduce | 19:21 |
| tmckay | (03:12:00 PM) aignatov: between for example havana and icehouse releases :) | 19:21 |
| *** elmiko has joined #openstack-sahara | 19:21 | |
| jayunit100 | If you say the word demo i say "bigpetstore" ! | 19:22 |
| tmckay | jayunit100, I wondered if you had some cool sciency thing we could run | 19:22 |
| tmckay | jayunit100, unfamiliar with bigpetstore | 19:22 |
| aignatov | pigbetstore is something from Java Spring | 19:22 |
| *** jspeidel has left #openstack-sahara | 19:22 | |
| tmckay | okay, wait.... pigbetstore, or bigpetstore? | 19:23 |
| tmckay | They both make sense, sort of | 19:23 |
| aignatov | pig or big? that's the question :) | 19:24 |
| *** themistymay has quit IRC | 19:25 | |
| tmckay | ah, I see, jayunit100 has bigpetstore on github | 19:25 |
| tmckay | aignatov, https://github.com/jayunit100/bigpetstore | 19:26 |
| *** Ch00k has joined #openstack-sahara | 19:26 | |
| aignatov | ok, looking on it | 19:27 |
| tmckay | aignatov, the big advantage is, it's already written :) | 19:27 |
| jayunit100 | heyyy | 19:27 |
| jayunit100 | sorry just got back | 19:27 |
| tmckay | jayunit100, how widespread is that thing? | 19:27 |
| tmckay | Would it still be novel for a Hadoop audience? did you originate it, or just respin it? | 19:28 |
| jayunit100 | tmckay: I'm working on a video for it | 19:28 |
| jayunit100 | well, the "pet store" analogy been around for a while in relational databases | 19:28 |
| jayunit100 | so, time for a bigpetstore ! | 19:29 |
| tmckay | nice. Seems original enough. | 19:29 |
| jayunit100 | tmckay: aignatov im about to make a video for it. Why don't you guys watch the video and then you can decide if its worth your while :) | 19:29 |
| jayunit100 | basically, the idea is this: | 19:29 |
| jayunit100 | 1) BigPetStore is FUN. Everybody likes animals. | 19:29 |
| jayunit100 | 2) BigPetStore generates an arbitrary amount of data (from 1 KB to 1PB). | 19:30 |
| * tmckay thinks they're tasty | 19:30 | |
| tmckay | petabytes baby, now you're talking | 19:30 |
| *** IlyaE has quit IRC | 19:30 | |
| jayunit100 | 3) BigPetStore uses the whole ecosystem, demonstrating the different value of different tools. | 19:30 |
| jayunit100 | 4) BigPetStore has a data model thats easy to understand. No need to read a bunch CSV files to figure out what the examples are doing. They all build off of ONE generated data set: Pet store transactions. | 19:31 |
| tmckay | jayunit100, sounds good. If it can be launched from the hadoop command line, we can launch it from Sahara via Oozie. | 19:32 |
| aignatov | jayunit100: ok, that looks nice | 19:32 |
| jayunit100 | Ive tested it both on EMR hadoop, and on glisterfs hadoop with up to 1 TB of data. In the end you get this: | 19:32 |
| tmckay | aignatov, no reason we can't potentially have multiple demos. Get one in the can, fast, and then if we want we can work on another | 19:32 |
| jayunit100 | https://s3.amazonaws.com/uploads.hipchat.com/30405/198980/RrKpgzy0lXY6X02/bps.jpg | 19:34 |
| tmckay | lol | 19:34 |
| jayunit100 | http://jayunit100.github.io/bigpetstore/ | 19:34 |
| tmckay | jayunit100, where do you get the map rendering? | 19:34 |
| jayunit100 | ^^^ thats the visualization widget. | 19:34 |
| aignatov | tmckay, jayunit100: agree, pet store looks nice, especially it could be visualised well as I see in jayunit100's github | 19:34 |
| jayunit100 | aignatov: you a javascript hacker? | 19:35 |
| tmckay | jayunit100, crobertsrh is | 19:35 |
| aignatov | no :) | 19:35 |
| tmckay | we can kidnap him on this too | 19:35 |
| crobertsrh | more of a hack than a hacker at times :) | 19:35 |
| jayunit100 | crobertsrh: would you be interested in joining a google hangout to help me with the bigpetstore visualization widget ? | 19:35 |
| tmckay | elmiko, ^^ | 19:35 |
| tmckay | you know some javascript, right? | 19:36 |
| crobertsrh | I can't atm, but I would certainly be willing to help | 19:36 |
| elmiko | tmckay: yes | 19:36 |
| *** dmitryme has quit IRC | 19:37 | |
| tmckay | jayunit100, elmiko also has a vested interest in a quality demo :) | 19:37 |
| elmiko | lol, i suppose that's true :) | 19:37 |
| jayunit100 | elmiko: would you be interested in contributing some javascript expertise to bigpetstore? | 19:37 |
| elmiko | sure, i don't have a google account though | 19:37 |
| jayunit100 | if so i can organize a hangout to show you how it all works | 19:37 |
| elmiko | definitely | 19:38 |
| crobertsrh | Is tomorrow morning an option? | 19:38 |
| tmckay | I can come too, maybe I'll pick up some stuff | 19:38 |
| jayunit100 | yes ! | 19:38 |
| elmiko | i could do tomorrow morning | 19:38 |
| jayunit100 | tmckay: great. thanks . | 19:38 |
| tmckay | I have type some javascript before ... | 19:38 |
| jayunit100 | elmiko i guess you'll need a google plus account | 19:38 |
| crobertsrh | that will also give me time to get the google hangout stuff installed again | 19:38 |
| elmiko | i'll set one up tonight | 19:38 |
| jayunit100 | okay | 19:38 |
| jayunit100 | great thanks guys !!!!!!!!!!!! | 19:39 |
| aignatov | guys, can you record your meeting? | 19:39 |
| jayunit100 | yeah we can try to do the hangouts on air thing | 19:39 |
| tmckay | I think I have trevormckay2, but I don't remember the password | 19:39 |
| jayunit100 | okay, we will sync up in here in the morning and stuff | 19:39 |
| tmckay | wow, I remembered :) | 19:40 |
| elmiko | sounds good jayunit100 | 19:40 |
| tmckay | jayunit100, we may want to schedule around a certain internal meeting ... | 19:42 |
| tmckay | 10-11 EST | 19:43 |
| openstackgerrit | Sergey Reshetnyak proposed a change to openstack/sahara: Change tag for vanilla integration test to 'vanilla1' https://review.openstack.org/83056 | 19:46 |
| *** IlyaE has joined #openstack-sahara | 19:47 | |
| jayunit100 | okay tmckay thats cool | 19:48 |
| jayunit100 | brb we will schedule an internal 10-11 meeting | 19:48 |
| tmckay | jayunit100, 10-11 is the slot we *dont* want :) | 19:49 |
| *** themistymay has joined #openstack-sahara | 19:49 | |
| *** witlessb has quit IRC | 19:50 | |
| *** witlessb has joined #openstack-sahara | 19:51 | |
| openstackgerrit | Andrew Lazarev proposed a change to openstack/sahara: [IDH] Integration tests for IDH 3.0.2 https://review.openstack.org/79719 | 19:51 |
| *** alazarev has quit IRC | 19:57 | |
| *** themistymay has quit IRC | 20:05 | |
| *** crobertsrh is now known as _crobertsrh | 20:10 | |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack/sahara: Updated from global requirements https://review.openstack.org/83566 | 20:13 |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack/sahara: Updated from global requirements https://review.openstack.org/83566 | 20:14 |
| jayunit100 | tmckay: oh ok | 20:15 |
| openstackgerrit | Alexander Ignatov proposed a change to openstack/sahara: Add fixed and floating IPs discovery via neutron https://review.openstack.org/82919 | 20:17 |
| *** jspeidel has joined #openstack-sahara | 20:18 | |
| mattf | howdy | 20:18 |
| jspeidel | mattf, hey | 20:18 |
| jspeidel | lets discuss api v2 | 20:18 |
| mattf | hit me | 20:19 |
| jspeidel | looked at the latest CR and made some comments | 20:19 |
| jspeidel | would like to discuss some design aspects of the api | 20:19 |
| mattf | yup, i replied to those comments | 20:19 |
| jspeidel | ok, let me see ... | 20:20 |
| jspeidel | ok, I had responded to your responses | 20:20 |
| mattf | oh, let me look then | 20:20 |
| jspeidel | first concern is the query mechanism | 20:20 |
| jspeidel | instead of adding a new endpoint for each query field, would like to have a generic query framework | 20:21 |
| jspeidel | would also like to see a consistent structure of collection_resource/instance_resource | 20:21 |
| mattf | i don't disagree | 20:21 |
| jspeidel | so in the current proposal we have | 20:22 |
| mattf | i'm mostly just apathetic re the plugins endpoint | 20:22 |
| jspeidel | @rest.get('/plugins/<name>/<version>') | 20:22 |
| jspeidel | version should not be an endpoint | 20:22 |
| jspeidel | instead we should do | 20:22 |
| jspeidel | ugh | 20:23 |
| jspeidel | wont let me send it | 20:23 |
| jspeidel | let me change format | 20:23 |
| jspeidel | "/plugins/:name?version=2" | 20:23 |
| jspeidel | damn | 20:23 |
| mattf | the value of having /name and /name/version escapes me really. why not just have /name provide a list w/ all the details you get from /name/version ? | 20:23 |
| mattf | iirc, when you do /name/version it actually queries a plugin for some config info | 20:24 |
| jspeidel | config info about the plugin with the provided name? | 20:24 |
| SergeyLukjanov | yup, it returns thousands of configs from plugin | 20:25 |
| jspeidel | ugh | 20:25 |
| SergeyLukjanov | for specified version | 20:25 |
| mattf | not really "thousands" | 20:25 |
| SergeyLukjanov | we need it to display in UI | 20:25 |
| mattf | jspeidel, iirc /name/version is a hack to get details of the processes etc the plugin supports to the dashboard for hints | 20:25 |
| jspeidel | what I am proposing is that there is only a cluster collection endpoint such as /plugins | 20:25 |
| jspeidel | and an instance endpoint such as /plugins/HDP | 20:26 |
| jspeidel | everything else is in the query string | 20:26 |
| jspeidel | by default the instance endpoint will return all info about the plugin, unless there are sub-resources | 20:26 |
| * mattf screws his head on straight | 20:26 | |
| jspeidel | maybe configuration ... | 20:26 |
| SergeyLukjanov | on the other hand, we have just several versions per plugin, so, it'll not be bad to return details for all of them I think | 20:26 |
| jspeidel | so /plugins/HDP/configurations | 20:26 |
| jspeidel | anyway, keep the collection/instance model | 20:27 |
| * SergeyLukjanov thinking about dedicating at least one design summit session for API v2 | 20:27 | |
| jspeidel | I would like that very much | 20:27 |
| jspeidel | I designed the ambari rest api so have thought alot about REST api design | 20:27 |
| * mattf nods | 20:28 | |
| mattf | i'm wondering if we aren't just exposing an implementation detail her | 20:28 |
| mattf | e | 20:28 |
| jspeidel | would you like me to write up an abstract for api v2? | 20:28 |
| jspeidel | I am not discussing implementation | 20:28 |
| jspeidel | only api resources | 20:29 |
| mattf | why have plugins = {HDP, IDH, Vanilla} and each plugin have versions = {X, Y, Z} | 20:29 |
| mattf | why not have plugins HDP1 HDP2 IDH2 IDH3 Vanilla1 Vanilla2 | 20:29 |
| jspeidel | yeah, so that was why I had asked in my comments about what would this return | 20:29 |
| mattf | who cares if internally there is one "plugin" that implements HDP1 and HDP2 w/ a few extra params to switch on? | 20:29 |
| *** Ch00k has quit IRC | 20:29 | |
| jspeidel | "/plugins/HDP" | 20:30 |
| jspeidel | what info would this return? | 20:30 |
| jspeidel | is this meaningful | 20:30 |
| jspeidel | or would be need to specify a version for it to be meaningful | 20:30 |
| mattf | it returns the versions | 20:31 |
| jspeidel | ok | 20:31 |
| jspeidel | that makes sense | 20:31 |
| jspeidel | so versions are a sub-resource of the plugin instance endpoint | 20:31 |
| mattf | for better or worse | 20:31 |
| jspeidel | "/plugins/HDP/versions" | 20:31 |
| jspeidel | no, that is good | 20:31 |
| jspeidel | then we would have | 20:32 |
| jspeidel | "/plugins/HDP/versions/2.0" | 20:32 |
| mattf | well, i'm starting to wonder if we're exposing an impl detail, shouldn't have have /plugins return {HDP1, HDP2, IDH2, IDH3, Vanilla1, Vanilla2} | 20:32 |
| mattf | /plguins/:name/versions/:version is more rest-y, at first glance | 20:32 |
| jspeidel | if you query the plugins collections resource you should get HDP, IDH vanilla | 20:32 |
| jspeidel | then you can drill down to the versions for each plugin | 20:33 |
| mattf | why should /plugins return HDP instead of HDP1 and HDP2 ? | 20:33 |
| jspeidel | not an impl detail, an api detail | 20:33 |
| jspeidel | it could, but to me it seems more logical to have a common parent for all related plugin versions | 20:34 |
| jspeidel | HDP1 and HDP2 are not distinct plugins | 20:34 |
| jspeidel | they are the same HDP plugin | 20:34 |
| mattf | it means clients have to make an extra call to get the details they want | 20:34 |
| mattf | <jspeidel> HDP1 and HDP2 are not distinct plugins | 20:34 |
| mattf | <jspeidel> they are the same HDP plugin | 20:34 |
| mattf | ^^ that's the impl detail imho | 20:34 |
| mattf | who cares that they happen to be the same module on the backend (impl)? | 20:35 |
| jspeidel | will they not have any common data? | 20:35 |
| *** themistymay has joined #openstack-sahara | 20:35 | |
| mattf | i'd expect they have data that intersects but doesn't completely overlap | 20:36 |
| mattf | e.g. process names between hdp1 and hdp2 are different | 20:36 |
| mattf | some of the config is the same, but not all | 20:36 |
| mattf | as it stands, /plugins/HDP doesn't give you common data between all versions | 20:36 |
| jspeidel | what does it return? | 20:37 |
| mattf | it's kinda just an extra level (http call) before the client gets to the good stuff | 20:37 |
| mattf | /plugins atm should return HDP, IDH, Vanilla | 20:37 |
| mattf | /plugins/HDP would return the versions 1 & 2 | 20:37 |
| mattf | tmckay, do yuo have a working sahara cli atm, mine is still called savanna and semi busted (gotta fix it still) | 20:38 |
| jspeidel | that makes alot of sense to me | 20:38 |
| mattf | does it make sense primarily because the code is organized into an HDP plugin module that supports two versions of HDP, v1 and v2? | 20:39 |
| mattf | ...because i'd call that an impl detail that doesn't need to be exposed through the api. the client cares mostly for HDP1 or HDP2 | 20:39 |
| tmckay | mattf, yes, I've been testing with master. | 20:39 |
| jspeidel | it makes sense to me because different versions of the same plugin are related | 20:39 |
| mattf | tmckay, will you pastebin the output of sahara plugins-list and plugins-show ? | 20:40 |
| tmckay | sure | 20:40 |
| mattf | tmckay, thanks | 20:40 |
| jspeidel | anyway, perhaps we can get back to this later and continue discussion on other api points | 20:40 |
| jspeidel | such as queries and the basic collection/intance model | 20:41 |
| mattf | we debated queries a few months back, turns out the way to go is to do queries client side | 20:42 |
| jspeidel | if we can agree on these then the above question of what a plugin is should be easy to get resolution on | 20:42 |
| jspeidel | example? | 20:43 |
| mattf | nova, keystone, cinder, neutron, swift clients | 20:43 |
| mattf | openstack common client too | 20:43 |
| jspeidel | so if a user wanted to get a plugin with name=foo and prop2=bar, how would he do it? | 20:44 |
| mattf | the code is less than straight forward. it you look at the Resources you'll find a FindableResource (or something similar) | 20:44 |
| mattf | you can't right now. | 20:44 |
| jspeidel | we would need to have a distinct endpoint for each queryable field? | 20:44 |
| mattf | given the infra we currently have, the code would grab all plugins and do a search client side | 20:45 |
| jspeidel | ugh | 20:45 |
| mattf | indeedn. | 20:45 |
| jspeidel | in v2 are we wanting to do better? | 20:45 |
| tmckay | mattf, http://fpaste.org/89333/39595312/ | 20:45 |
| mattf | (FindResource, not Findable) | 20:46 |
| mattf | tmckay, thanks, can do oyu a --version too for hdp? | 20:46 |
| tmckay | sure | 20:46 |
| mattf | jspeidel, imho, query for plugins at least isn't very valuable | 20:47 |
| jspeidel | what about for any of the resources that are exposed in out api? | 20:47 |
| mattf | jspeidel, for other types query is more valuable, creating the infra service side may be more work than its worth tho | 20:47 |
| jspeidel | mattf, I guess my question is what is the goal of the v2 api | 20:48 |
| jspeidel | mattf, and then how much work do we want to put into it | 20:49 |
| mattf | https://blueprints.launchpad.net/sahara/+spec/v2-api - Goal of the v2 API is a consistent and clean evolution of the v1.0 and v1.1 APIs. | 20:49 |
| mattf | jspeidel, i've already put a lot of work into it | 20:49 |
| jspeidel | sorry | 20:50 |
| jspeidel | *looking* | 20:50 |
| elmiko | is there a documentation of the /v1.1/{tenant id}/clusters endpoint? | 20:51 |
| mattf | elmiko, should be in the api docs | 20:51 |
| jspeidel | mattf, I wasn't implying that there wasn't alot of effort put into the api | 20:51 |
| mattf | http://docs.openstack.org/developer/sahara/restapi/rest_api_v1.1_EDP.html | 20:51 |
| jspeidel | mattf, my apologies if I cam across that way | 20:51 |
| mattf | jspeidel, oh no worries. just saying i've been around the block on it. | 20:52 |
| * mattf didn't take it that way at all | 20:52 | |
| elmiko | mattf: i don't see the /clusters endpoint in there (sorry to interrupt) | 20:52 |
| jspeidel | mattf, in the bp it really doesn't say much about query capabilities and query use cases | 20:52 |
| jspeidel | mattf, I was also questioning the somewhat inconsistent pattern such as ('/plugins/<name>/<version>' | 20:53 |
| jspeidel | which doesnt follow the collection/instance pattern | 20:54 |
| mattf | http://docs.openstack.org/developer/sahara/restapi/rest_api_v1.0.html | 20:54 |
| mattf | elmiko, my bad | 20:54 |
| elmiko | mattf: thanks | 20:54 |
| mattf | tmckay, sorry, i forgot --version was todo because the py client doesn't support it | 20:54 |
| mattf | now i'm wondering why we have it at all | 20:54 |
| jspeidel | mattf, I would think that this would instead be "/plugins/HDP/versions/2.0" | 20:54 |
| mattf | jspeidel, query was pushed off for v3, imho | 20:55 |
| jspeidel | mattf, ok, then we won't discuss query :) | 20:55 |
| mattf | jspeidel, that's perfectly reasonable. i'd like to debate some more if we even expose HDP distinct from HDP2.0 | 20:55 |
| mattf | # TODO(mattf) - saharaclient does not support query w/ version | 20:56 |
| mattf | #@utils.arg('--version', | 20:56 |
| mattf | # metavar='<version>', | 20:56 |
| mattf | # help='Optional version') | 20:56 |
| mattf | def do_plugin_show(cs, args): | 20:56 |
| tmckay | mattf, ack | 20:56 |
| mattf | jspeidel, ^^ the client doesn't even let you get at the version details, so i'm kinda wondering what's going on. i've swapped out all the context... | 20:56 |
| jspeidel | mattf, ^^ ouch | 20:56 |
| mattf | jspeidel, imho /plugins/HDP/versions/2.0 is more rest-y and thus cleaner | 20:57 |
| jspeidel | mattf, so all you can get is the generic plugin info? | 20:57 |
| jspeidel | mattf, +1 | 20:57 |
| *** alazarev has joined #openstack-sahara | 20:58 | |
| *** alazarev has quit IRC | 21:00 | |
| mattf | $ sahara plugin-show --name hdp | 21:02 |
| mattf | Plugin {u'title': u'Hortonworks Data Platform', u'description': u'The Hortonworks OpenStack plugin works with project Savanna to automate the deployment of the Hortonworks data platform on OpenStack based public & private clouds', u'name': u'hdp', u'versions': [u'2.0.6', u'1.3.2']} | 21:02 |
| *** alazarev has joined #openstack-sahara | 21:02 | |
| jspeidel | mattf, seems that we have agreement on the collction/instance pattern and that this is followed for all uri's in the bp except the one we just dicussed | 21:02 |
| jspeidel | mattf, and that we will punt query to v3 | 21:02 |
| mattf | $ grep -A3 get_version saharaclient/api/plugins.py | 21:04 |
| mattf | def get_version_details(self, plugin_name, hadoop_version): | 21:04 |
| mattf | return self._get('/plugins/%s/%s' % (plugin_name, hadoop_version), | 21:04 |
| mattf | 'plugin') | 21:04 |
| mattf | so my TODO is wrong | 21:04 |
| * mattf wipes brow, world is sane | 21:04 | |
| *** themistymay has quit IRC | 21:04 | |
| mattf | jspeidel, yeah, i still want to challenge if we have plugin + version or just plugin w/ version | 21:05 |
| jspeidel | mattf, great | 21:05 |
| jspeidel | mattf, need to run in 5 mins, maybe we can sleep on it and come back tomorrow? | 21:05 |
| mattf | jspeidel, there are still a few things to debate, like access to convert config (from spi), where you're knowledge of good rest apis will come in very handy | 21:06 |
| mattf | jspeidel, we debated a good deal on openstack-dev | 21:06 |
| mattf | i'm pto tomorrow | 21:06 |
| jspeidel | mattf, then monday? | 21:06 |
| mattf | also in some old meeting minutes | 21:06 |
| mattf | monday it is | 21:06 |
| jspeidel | mattf, it's a date | 21:07 |
| *** jspeidel has left #openstack-sahara | 21:07 | |
| *** tmckay has quit IRC | 21:16 | |
| *** alazarev has quit IRC | 21:49 | |
| *** themistymay has joined #openstack-sahara | 21:49 | |
| *** alazarev has joined #openstack-sahara | 21:52 | |
| *** themistymay has quit IRC | 21:53 | |
| openstackgerrit | Andrew Lazarev proposed a change to openstack/sahara: [IDH] Integration tests for IDH 3.0.2 https://review.openstack.org/79719 | 22:07 |
| *** themistymay has joined #openstack-sahara | 22:28 | |
| *** elmiko has quit IRC | 22:29 | |
| *** themistymay has quit IRC | 22:29 | |
| *** alazarev has quit IRC | 22:29 | |
| *** witlessb has quit IRC | 22:39 | |
| *** alazarev has joined #openstack-sahara | 22:51 | |
| *** alazarev has quit IRC | 22:58 | |
| *** alazarev has joined #openstack-sahara | 23:00 | |
| *** ErikB1 has quit IRC | 23:19 | |
| *** tmckay has joined #openstack-sahara | 23:51 | |
| *** IlyaE has quit IRC | 23:55 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!
