Wednesday, 2014-02-05

*** witlessb has quit IRC00:10
*** witlessb has joined #savanna00:11
*** matsuhashi has joined #savanna00:25
openstackgerritAndrew Lazarev proposed a change to openstack/savanna: [DOC] Fixed link to oozie in docs  https://review.openstack.org/7116900:46
*** nosnos has joined #savanna00:57
*** mst89 has quit IRC00:57
*** nosnos has quit IRC01:01
*** nosnos has joined #savanna01:01
openstackgerritAndrew Lazarev proposed a change to openstack/python-savannaclient: Changed base Resource class to prevent changing of passed arguments  https://review.openstack.org/7108602:19
*** crobertsrh has joined #savanna04:00
*** crobertsrh is now known as _crobertsrh04:02
*** ylobankov1 has quit IRC04:22
*** ylobankov has joined #savanna04:23
*** IlyaE has joined #savanna04:28
*** IlyaE has quit IRC04:52
*** akuznetsov has joined #savanna04:56
*** jcooley_ has quit IRC05:19
*** IlyaE has joined #savanna05:23
*** jcooley_ has joined #savanna05:48
openstackgerritJenkins proposed a change to openstack/savanna: Imported Translations from Transifex  https://review.openstack.org/7091806:09
*** jcooley_ has quit IRC07:12
*** jcooley_ has joined #savanna07:12
*** jcooley_ has quit IRC07:17
*** DinaBelova_ is now known as DinaBelova07:19
*** IlyaE has quit IRC07:27
*** dmitryme has joined #savanna08:25
*** akuznetsov has quit IRC08:28
*** akuznetsov has joined #savanna08:31
*** dmitryme has quit IRC08:46
*** dmitryme has joined #savanna08:52
*** matsuhashi has quit IRC08:54
*** matsuhashi has joined #savanna08:55
*** aignatov_ is now known as aignatov08:59
*** aignatov is now known as aignatov_09:02
*** ylobankov has left #savanna09:02
*** ylobankov has joined #savanna09:07
*** aignatov_ is now known as aignatov09:10
*** skolekonov has joined #savanna09:28
openstackgerritA change was merged to openstack/savanna: Fixed HDP plugin to support Heat engine  https://review.openstack.org/7007609:34
*** aignatov is now known as aignatov_10:25
*** nosnos has quit IRC10:53
*** aignatov_ is now known as aignatov11:14
openstackgerritDaniele Venzano proposed a change to openstack/savanna-image-elements: Add a Spark element  https://review.openstack.org/7123711:24
*** dmitryme has quit IRC11:45
openstackgerritSergey Reshetnyak proposed a change to openstack/savanna: Add integration tests to Intel plugin  https://review.openstack.org/6681211:50
openstackgerritSergey Reshetnyak proposed a change to openstack/savanna: Add configuration integration test for IDH plugin  https://review.openstack.org/7081911:53
*** ruhe has joined #savanna11:59
openstackgerritSergey Reshetnyak proposed a change to openstack/savanna: Add configuration integration test for IDH plugin  https://review.openstack.org/7081912:04
*** ruhe is now known as ruhe_12:06
*** ruhe_ is now known as ruhe12:06
*** dmitryme has joined #savanna12:11
*** IvanBerezovskiy has joined #savanna12:26
*** dmitryme has quit IRC12:41
*** ruhe is now known as ruhe_12:44
*** ruhe_ has quit IRC12:48
*** qwerty_nor has joined #savanna13:01
openstackgerritNikita Konovalov proposed a change to openstack/savanna: Moving rest to Pecan/WSME framework  https://review.openstack.org/6390813:02
mattfSergeyLukjanov, why is there still a -2 on the api discussion?13:03
*** ylobankov has left #savanna13:03
*** ylobankov1 has joined #savanna13:03
*** ylobankov1 has quit IRC13:03
*** ylobankov1 has joined #savanna13:04
*** ylobankov1 has left #savanna13:06
*** ylobankov_ has joined #savanna13:06
*** dmitryme has joined #savanna13:10
SergeyLukjanovmattf, please, add note to the commit message that it's for discussion13:11
*** dmitryme has joined #savanna13:13
mattfSergeyLukjanov, it's in the commit message13:14
SergeyLukjanovmattf, are you expecting to merge this change?13:15
SergeyLukjanovmattf, additionally you can take a look on Pecan patch from Nikita13:17
NikitaKonovalovmattf, hi13:19
NikitaKonovalovI've just sent a CR with initial Pecan/WSME framework setup13:20
NikitaKonovalovit still handles v1 requests13:20
*** ruhe has joined #savanna13:22
NikitaKonovalovthink it will take a few days to match all the functionality from v1 using Pecan, because it requires some hacks to handle our config dictionaries13:26
SergeyLukjanovNikitaKonovalov, let's split it to the Pecan/WSME initial work with one example endpoint like Plugins that you already added13:28
SergeyLukjanovand some additional commits like additional WSME models definitions13:28
SergeyLukjanovand etc.13:28
SergeyLukjanovI think mattf will add all other endpoints13:29
NikitaKonovalovthe missing thing right now is auth token check13:29
NikitaKonovalovthere are two possible ways to handle that in Pecan: with hooks and @secure decorator13:30
NikitaKonovalovI'm trying to figure which one is better right now13:30
ruhedon't forget about tests, it's not that easy to build testing infrastructure with Pecan+SQLAlchemy+Alembic infrastructure13:30
NikitaKonovalovv1 api is checked only through integration tests, right?13:32
NikitaKonovalovif so, then nothing should breake, because v1 is still served with Flask13:32
SergeyLukjanovthere is no need to add tests for v1 now13:39
SergeyLukjanovand tests for v2 could be added later13:39
*** ruhe is now known as ruhe_13:46
mattfSergeyLukjanov, depends on if pecan is in a state for prototyping13:54
mattfNikitaKonovalov, hello!13:54
*** ruhe_ is now known as ruhe13:54
*** ruhe has quit IRC13:54
mattfNikitaKonovalov, what's the complexity that ruhe mentioned?13:55
SergeyLukjanovmattf, it was about writing functional tests (not integration) for REST API13:56
mattfis it somehow more difficult w/ pecan than it was w/ flask?13:56
SergeyLukjanovmattf, we're not doing it w/ Flask13:57
mattfNikitaKonovalov, i'll take a look at your first commit for pecan today13:57
mattfSergeyLukjanov, sure, but is it somehow harder to do w/ pecan?13:57
NikitaKonovalovno dfference here, the challenge is to make tests work with different a different database each13:57
SergeyLukjanovmattf, it's difficult because we have rest api framework, sqla and alembic13:58
mattfbecause if the difficult is the same, pointing out the complexity w/ pecan is distracting13:58
mattfs/difficult/difficulty/13:58
SergeyLukjanovmattf, it's the same13:58
SergeyLukjanovI think that ruhe mentioned it to help make the correct extimates13:58
SergeyLukjanovbecause we'd like to functional testing v2 api13:59
SergeyLukjanovso, I'd like to land initial impl of Pecan/WSME support with one simple endpoint like Plugins // when it'll be reviewed and ready13:59
NikitaKonovalovthe solution btw is using the config fixture from oslo13:59
SergeyLukjanovit'll unblock mattf to start working on other enpodints14:00
SergeyLukjanovendpoits*14:00
SergeyLukjanovand we could work in parallel on other improvements like functinonal tests and decoupling from v1 api14:00
SergeyLukjanovfunctional*14:00
*** tmckay has quit IRC14:11
*** dmitryme has quit IRC14:26
*** dmitryme has joined #savanna14:27
openstackgerritNikita Konovalov proposed a change to openstack/savanna: Moving rest to Pecan/WSME framework  https://review.openstack.org/6390814:31
*** ruhe has joined #savanna14:35
*** ruhe is now known as ruhe_14:46
*** ruhe_ is now known as ruhe14:49
*** aignatov is now known as aignatov_14:49
*** skolekonov has quit IRC14:50
*** aignatov_ is now known as aignatov14:53
*** ruhe is now known as ruhe_14:53
*** ruhe_ is now known as ruhe14:53
*** NikitaKonovalov is now known as NikitaKonovalov_15:01
*** ruhe is now known as ruhe_15:01
*** ruhe_ is now known as ruhe15:05
*** ruhe is now known as ruhe_15:07
*** ruhe_ is now known as ruhe15:09
SergeyLukjanovtmckay, aignatov15:10
SergeyLukjanovhttps://blueprints.launchpad.net/savanna/+spec/benchmarking-from-horizon15:10
SergeyLukjanovany thoughts?15:10
mattfSergeyLukjanov, sounds pretty good. i'm not too keen on debugging pecan itself though. the plugin example needs to be functional.15:12
aignatovSergeyLukjanov: interesting idea, but I see huge efforts on that because we need to do some research etc before implementing15:13
aignatovpropose to add as track to next design summit :)15:14
aignatovanyway this will not be landed in icehouse15:14
SergeyLukjanovmattf, sure, it should at least work and probably we'll need one more endpoint to demonstrate all Pecan/WSME usage details15:14
SergeyLukjanovaignatov, yup15:14
SergeyLukjanovaignatov, please add comments to the bp15:14
aignatovalso, OS contains Rally project targeted for benchmarking all components15:15
*** tmckay has joined #savanna15:15
mattfaignatov, it should be possible to run hibench from the savanna api. he can always demonstrate how it works and then push for inclusion in the dashboard, if there's actually anything to include15:16
mattfit might just be a set of steps "upload xyz to swift, make job binary spq, run job abc"15:16
ruhealso, Boris (lead of the Rally) offered his help to include performance tests for Savanna to Rally15:16
*** matsuhashi has quit IRC15:17
*** matsuhashi has joined #savanna15:18
*** IlyaE has joined #savanna15:24
*** ruhe is now known as ruhe_15:27
*** aignatov is now known as aignatov_15:28
*** ruhe_ has quit IRC15:32
*** matsuhashi has quit IRC15:34
openstackgerritTrevor McKay proposed a change to openstack/savanna: Extract configs beginning with "edp." from job_configs['configs']  https://review.openstack.org/6971215:49
openstackgerritTrevor McKay proposed a change to openstack/savanna: Generate streaming tag in mapreduce job  https://review.openstack.org/6972715:49
openstackgerritTrevor McKay proposed a change to openstack/savanna: Add validation check for streaming elements on MapReduce without libs  https://review.openstack.org/6996015:50
*** DinaBelova is now known as DinaBelova_15:53
*** NikitaKonovalov_ is now known as NikitaKonovalov16:01
witlessb_crobertsrh: howdy16:03
witlessbfound the issue with devstack -- one has to list 's-proxy' explicitly in ENABLED_SERVICES16:04
witlessbin localrc16:04
witlessbotherwise glance-swift account doesn't get created in keystone16:05
witlessbwith a bunch of problems after that16:06
openstackgerritTrevor McKay proposed a change to openstack/savanna: Add integration test for streaming mapreduce  https://review.openstack.org/7082916:06
openstackgerritTrevor McKay proposed a change to openstack/savanna: Add integration test for streaming mapreduce  https://review.openstack.org/7082916:08
*** IvanBerezovskiy has left #savanna16:09
*** NikitaKonovalov is now known as NikitaKonovalov_16:18
*** jcooley_ has joined #savanna16:21
*** venza_ has quit IRC16:22
*** venza has joined #savanna16:24
*** DinaBelova_ is now known as DinaBelova16:26
*** jcooley_ has quit IRC16:28
*** jcooley_ has joined #savanna16:33
openstackgerritTrevor McKay proposed a change to openstack/savanna: Move 'main_class' and 'java_opts' into edp.java configs  https://review.openstack.org/6998216:40
*** _crobertsrh is now known as crobertsrh16:41
crobertsrhwitlessb:  glad you got it figured out!16:42
witlessbthat makes two us16:42
witlessbfiled a bug report, we'll see what they'll say16:42
openstackgerritTrevor McKay proposed a change to openstack/savanna: Move 'main_class' and 'java_opts' into edp.java configs  https://review.openstack.org/6998216:43
witlessbcrobertsrh: ahh.16:44
witlessbcrobertsrh: on this page: https://savanna.readthedocs.org/en/latest/devref/devstack.html16:44
witlessbin the section about localrc, it's probably worth to remove swift16:44
witlessbfrom ENABLED_SERVICES alltogether?16:45
crobertsrhI actually use swift, savanna also allows for using swift. Not necessarily required, but supported.16:45
witlessbahh16:45
witlessbsry, I shouldn't have presumed16:45
witlessbthen it's probably worth adding s-proxy to that list16:46
crobertsrhYeah, if that's what saved the day for you, it might be useful.  I don't think I've had to have s-proxy in the past, but it's been awhile since I have grabbed the latest and started up a new stack.16:47
witlessbnods; that's what did the trick16:47
witlessbI'm not sure what would be preferred way to fix the issue, as the devstack team seems to be moving away from account creation using files/keystone_data.sh16:48
crobertsrhHopefully, they will comment on your bug report soon.16:50
witlessbnods16:50
witlessbI'll keep my eye on it16:50
*** bogdando has quit IRC17:11
*** jcooley_ has quit IRC17:22
*** jcooley_ has joined #savanna17:23
*** jcooley_ has quit IRC17:34
*** jcooley_ has joined #savanna17:34
*** jcooley_ has quit IRC17:41
*** jcooley_ has joined #savanna17:42
*** dmitryme has quit IRC17:45
openstackgerritAndrew Lazarev proposed a change to openstack/savanna: Refactored unit tests structure  https://review.openstack.org/7021117:46
*** aignatov_ is now known as aignatov17:57
*** akuznetsov has quit IRC17:58
*** nadya has joined #savanna18:00
*** nadya is now known as Guest8386318:01
*** IlyaE has quit IRC18:19
*** jcooley_ has quit IRC18:19
*** jcooley_ has joined #savanna18:21
*** jcooley_ has quit IRC18:23
*** jcooley_ has joined #savanna18:23
*** aignatov is now known as aignatov_18:24
*** aignatov_ is now known as aignatov18:30
*** IlyaE has joined #savanna18:38
*** mst89 has joined #savanna18:39
*** IlyaE has quit IRC18:40
*** Guest83863 is now known as _nadya_18:41
*** akuznetsov has joined #savanna18:43
*** aignatov is now known as aignatov_18:46
*** IlyaE has joined #savanna18:48
*** _nadya_ has quit IRC18:49
tmckaycrobertsrh, btw, it looks like we are going to go with dotted type names (MapReduce.Streaming)18:52
crobertsrhOk, that works just fine for me.18:52
tmckaycrobertsrh, I'm going to start working on that as soon as I get my rebase damage sorted out on java jobs.  Almost done18:53
*** dmitryme has joined #savanna18:53
crobertsrhOk, I'll start my tentative UI tweaks.18:53
tmckayhot darn, it works again18:58
openstackgerritTrevor McKay proposed a change to openstack/savanna: Move 'main_class' and 'java_opts' into edp.java configs  https://review.openstack.org/6998219:01
*** jcooley_ has quit IRC19:07
*** jcooley_ has joined #savanna19:08
*** dmitryme has quit IRC19:22
*** mst89 has quit IRC19:27
*** mst89 has joined #savanna19:36
*** akuznetsov has quit IRC19:43
openstackgerritSergey Lukjanov proposed a change to openstack/savanna-dashboard: Sync with global-requirements  https://review.openstack.org/7135519:44
openstackgerritSergey Lukjanov proposed a change to openstack/savanna: Sync with global-requirements  https://review.openstack.org/7135619:44
openstackgerritA change was merged to openstack/savanna: [DOC] Fixed link to oozie in docs  https://review.openstack.org/7116919:54
*** mattf is now known as _mattf20:18
*** _mattf is now known as mattf20:19
*** IlyaE has quit IRC20:20
*** DinaBelova is now known as DinaBelova_21:09
openstackgerritChad Roberts proposed a change to openstack/savanna-dashboard: Adding floating ip pool to node groups details for cluster  https://review.openstack.org/7138121:19
mattfSergeyLukjanov, ping21:23
SergeyLukjanovmattf, pong21:23
SergeyLukjanovmattf, I'm agreed with mostly all of your blueprints about not merging them except cli-itests21:24
mattfi just found myself under a bunch of bp email21:24
mattfi need some help understanding what yuo just did21:24
SergeyLukjanovmattf, all cli integration tests bps was reject, all other accepted21:24
mattfdid you just say that it's not important to have integration tests for a bunch of use cases?21:24
SergeyLukjanovprobably I missed several more bps21:24
SergeyLukjanovmattf, it should not be tested for cli21:25
SergeyLukjanovmattf, the common way of testing cli is to test how it transforms cli args to api calls or python client calls21:25
SergeyLukjanovmattf, we should have unit tests for client that will check how we generates jsons21:26
mattffor functional or unit tests, yeah. the bps are about integration tests21:26
mattfyou don't feel there should be integration tests that work through the cli?21:26
mattfmy phrasing may be betraying my opinion21:26
mattfintegration tests don't substitute for the other tests you mention, but they also shouldn't be skipped21:27
SergeyLukjanovmattf, yup, cli just a wrapper and could be tested separately21:27
*** IlyaE has joined #savanna21:27
mattfi'm definitely puzzled by the position that adding new integration tests should not be done21:28
SergeyLukjanovthey should be added  but in other place21:28
SergeyLukjanovlet me explain why21:28
SergeyLukjanovlet's start from which minimal set of tests we already have and must have21:28
SergeyLukjanov1) integration tests for all functionality using python client (we already have a bunch of them in savanna repo)21:29
mattfmap maybe? user -magic-> cli -python-> client -rest-> savanna-api -> *21:30
SergeyLukjanov1) == scenarios tests21:30
SergeyLukjanov2) functional tests for API (in future) that will test mocked world but check how API works fast21:31
SergeyLukjanovwhy we don't need massive integration tests for cli21:31
SergeyLukjanovbecause we're using python bindings in many places21:31
SergeyLukjanovcli, horizon, heat21:32
SergeyLukjanovin integration test too21:32
SergeyLukjanovand there is no need to duplicate all of the integration tests from python bindings to cli21:32
SergeyLukjanovwe can just unit test that cli correctly transforms cli arguments to python bindings calls21:32
mattfwhat does it harm?21:33
SergeyLukjanovyou can take a look on nova client tests21:33
SergeyLukjanovmattf, IMO integration tests for cli are useless because they will duplicate other integration tests21:33
SergeyLukjanovat least we'll never run them to not duplicate resources usage21:34
mattfi'll argue that we should have integration tests for the dashboard too21:34
mattfthe concern is time that the test suite will take to run?21:34
SergeyLukjanovmattf, yup21:35
SergeyLukjanovmattf, because it could be replaced with a fast unit tests21:35
mattfdashboard testing w/ selenium or something similar has value too21:35
SergeyLukjanovmattf, exactly, the reason of writing real tests w/o mocks for horizon is that we don't understand how to correctly mock all the world :(21:35
mattfoh, there's much more you get by using the real world vs mocks21:36
mattfmocks are a narrow snapshot of things understood by a few people and not likely to have real-world issues21:36
mattfthey serve a great purpose, but leave quality gaps21:37
SergeyLukjanovmattf, it makes sense for UI that is the end-user part21:37
mattfi can look into getting more hardware to speed things up so we can have more complete test coverage. that's a no brainer $$$ ask.21:37
SergeyLukjanovmattf, there are no strong objections for duplicate integration tests for CLI21:38
SergeyLukjanovnot now21:38
mattfcli+ui imho21:38
mattfi was just lazy and had the cli on my mind, so filed re cli21:38
SergeyLukjanovin future we could write main tests in such manner that http calls could be duplicated by cli analogues21:38
mattfi don't understand "duplicated by cli analogues"21:39
* mattf may need lunch21:40
SergeyLukjanovif you'll look at savanna integration tests21:40
SergeyLukjanovyou'll find that in fact by replacing http calls with cli calls you will have cli tests21:40
SergeyLukjanovbtw I'm still on position that cli integration tests currently absolutely useless, only cli args to python bindings calls transformations should be checked21:41
SergeyLukjanovwe could return back to the integrate-test-the-world-through-cli after all other test will be finished21:41
mattfi'm of the opinion that if someone is going to expand our integration test coverage we should be happy. if it may slow things down we prioritize runs or get more hardware.21:42
SergeyLukjanovheh, missed one more big fact - all OpenStack projects following the same approach21:42
openstackgerritTrevor McKay proposed a change to openstack/savanna: Add utilities for supporting dotted job types  https://review.openstack.org/7138721:42
SergeyLukjanovmattf, it should be expanded in other places21:42
mattfyou know what i'm gonna say!21:43
mattffile some bps w/ your ideas on what's missing21:43
* mattf smiles21:43
openstackgerritTrevor McKay proposed a change to openstack/savanna: Add utilities for supporting dotted job types  https://review.openstack.org/7138721:44
tmckayforgot the unit test, heh21:44
mattfthanks for going over all those bps. i need to run to a meeting (why a 5pm meeting!?). i'll ask about hardware $$$ too.21:44
SergeyLukjanovmattf, heh, it's 1:44 am here ;)21:44
mattfyeah, why are you still up!21:45
SergeyLukjanovmattf, to talk with you about your bps ;)21:45
* mattf is dreadfully sorry then21:45
SergeyLukjanovmattf, np, it should be discussed and I was like to explain my ops21:46
mattfbtw, new name suggestion: super awesome sauce21:46
tmckaysas21:46
mattfSergeyLukjanov, i'll be up early tomorrow too21:47
tmckaysass21:47
SergeyLukjanovbtw I'll ask our qa guys about add missed blueprints for integration tests21:47
SergeyLukjanovheh, sass is already taken :(21:47
mattfsassy21:47
SergeyLukjanovSass: Syntactically Awesome Style Sheets21:47
tmckaySassvanna21:47
SergeyLukjanovSleepyCachalot21:48
tmckayhow about StackaDoop?21:49
SergeyLukjanovtmckay, nice :)22:06
SergeyLukjanovtmckay, about https://review.openstack.org/#/c/69712/22:06
SergeyLukjanovtmckay, will it change REST API calls/responses?22:06
openstackgerritA change was merged to openstack/python-savannaclient: Fixed issue when savanna url is not provided  https://review.openstack.org/7034022:07
tmckaySergey, it will affect streaming mapreduce and java jobs.  Good point.  Not the API so much, but if we have an example JSON in the REST doc it may need to change, yes.22:07
tmckayI'll take a look22:07
tmckaySergeyLukjanov, even22:07
SergeyLukjanovtmckay, ok, thank you, I'll -1 it to prevent merge, please, comment about your research there22:09
tmckaywill do.22:09
SergeyLukjanovtmckay, so, it'll only rename specific configs in jobs to edp.XXX, am I right?22:09
SergeyLukjanovonly examples or existing integration code could be broken so22:10
tmckaySergeyLukjanov, yes.  The only exception is for java jobs, we had "main_class" and "java_opts" at the top level of the JSON (with cluster_id, etc).  Those fields go away and move into configs as "edp.java.main_class" and "edp.java.java_opts"22:10
tmckayI fixed the integration test22:10
SergeyLukjanovtmckay, yup, got it finally ;)22:11
SergeyLukjanovcould you, please, add a note to docs about this job type - specific behaviour?22:11
SergeyLukjanovif it's the only missed part, it could be done in separated CR to not block this one22:11
SergeyLukjanovtmckay, ^^22:12
tmckayYes.  looking at execute now.  I'll have an answer for you by tomorrow morning, I have a few hours to kill.22:12
tmckayack on the separate CR22:12
SergeyLukjanovtmckay, thx22:14
tmckaynp.  Good night!22:14
openstackgerritTrevor McKay proposed a change to openstack/savanna: Add utilities for supporting dotted job types  https://review.openstack.org/7138722:17
*** crobertsrh is now known as _crobertsrh22:21
openstackgerritTrevor McKay proposed a change to openstack/savanna: Update the edp user doc to discuss "edp." configs for Java jobs  https://review.openstack.org/7140322:50
openstackgerritTrevor McKay proposed a change to openstack/savanna-extra: Small tweak to the wordcount example README  https://review.openstack.org/7141323:10

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!