Monday, 2015-02-02

« Sunday, 2015-02-01 Index Tuesday, 2015-02-03 »
*** Networkn3rd has joined #openstack-sahara00:47
*** oikawa has joined #openstack-sahara01:36
*** Networkn_ has joined #openstack-sahara01:44
*** Networkn3rd has quit IRC01:47
*** macjacktw has quit IRC01:59
*** macjack has joined #openstack-sahara02:27
openstackgerritJamie Lennox proposed openstack/python-saharaclient: Rework authentication  https://review.openstack.org/14777302:43
*** Longgeek has joined #openstack-sahara03:04
*** Longgeek has joined #openstack-sahara03:04
openstackgerritKen Chen proposed openstack/sahara-specs: Implement a CM API libary in CDH plugin We plan to implement a subset of CM APIs and add it into Sahara project, so that we do not need to depend on the external third- party library. implements bp: add-lib-subset-cm-api Change-Id: Id5c9712b2034a2e537650d  https://review.openstack.org/15197803:09
openstackgerritweiting-chen proposed openstack/sahara: Add Impala service test in cdh plugin integration test  https://review.openstack.org/15193403:19
*** Networkn_ has quit IRC03:39
*** hdd has joined #openstack-sahara03:44
*** hdd has quit IRC03:58
*** oikawa has quit IRC04:04
*** tellesnobrega has joined #openstack-sahara04:05
*** Longgeek has quit IRC04:30
*** Longgeek has joined #openstack-sahara04:34
*** akuznetsov has joined #openstack-sahara04:48
*** akuznetsov has quit IRC04:50
*** Longgeek has quit IRC05:27
*** Nikolay_St has quit IRC05:30
*** Longgeek has joined #openstack-sahara05:32
openstackgerritlu huichun proposed openstack/sahara: Add Flume service test in cdh plugin integration test  https://review.openstack.org/15193505:36
openstackgerritlu huichun proposed openstack/sahara: Add Flume service test in cdh plugin integration test  https://review.openstack.org/15193505:59
*** hdd has joined #openstack-sahara06:17
*** tellesnobrega_ has joined #openstack-sahara06:30
*** tellesnobrega has quit IRC06:32
*** hdd has quit IRC06:37
*** sgotliv_ has joined #openstack-sahara06:46
openstackgerritlu huichun proposed openstack/sahara: Specify the package name when executing Java type edp jobs  https://review.openstack.org/15200907:20
*** Longgeek has quit IRC07:21
*** ViswaV has joined #openstack-sahara07:29
*** Longgeek has joined #openstack-sahara07:30
*** Longgeek has quit IRC07:34
*** Longgeek has joined #openstack-sahara07:49
*** sgotliv_ has quit IRC07:50
*** Longgeek has quit IRC07:52
*** chlong has quit IRC08:00
*** sgotliv has joined #openstack-sahara08:04
*** sgotliv has quit IRC08:27
*** sgotliv has joined #openstack-sahara08:28
*** skolekonov has joined #openstack-sahara08:43
*** Nikolay_St has joined #openstack-sahara09:12
*** vgridnev_ has joined #openstack-sahara09:15
*** Nikolay_1t has joined #openstack-sahara09:18
*** Nikolay_1t has quit IRC09:19
openstackgerritVitaly Gridnev proposed openstack/sahara: Remove log module from common modules  https://review.openstack.org/15205709:22
openstackgerritVitaly Gridnev proposed openstack/sahara: Remove log module from common modules  https://review.openstack.org/15205709:30
*** IvanBerezovskiy has joined #openstack-sahara09:33
*** tosky has joined #openstack-sahara09:54
openstackgerritSergey Reshetnyak proposed openstack/sahara: Add indirect VMs access implementation  https://review.openstack.org/13359009:59
openstackgerritVitaly Gridnev proposed openstack/sahara: Apply event-log feature for Vanilla plugins  https://review.openstack.org/14997210:12
*** Nikolay_St has quit IRC10:13
openstackgerritVitaly Gridnev proposed openstack/sahara: Waiting should depends on cluster state  https://review.openstack.org/15039910:14
*** AndreyPavlov has quit IRC10:41
*** tellesnobrega_ has quit IRC10:42
stannie42Hi10:47
stannie42is Andrey around ?10:47
stannie42(Andrey Pavlov)10:47
*** tosky has quit IRC10:47
*** Nikolay_St has joined #openstack-sahara10:47
*** vgridnev_ has quit IRC10:49
*** tosky has joined #openstack-sahara11:04
*** AndreyPavlov has joined #openstack-sahara11:04
openstackgerritSergey Reshetnyak proposed openstack/python-saharaclient: Use pretty-tox for better test output  https://review.openstack.org/15208611:15
openstackgerritSergey Reshetnyak proposed openstack/python-saharaclient: Use pretty-tox for better test output  https://review.openstack.org/15208611:16
AndreyPavlovhi, Pierre11:27
AndreyPavlovyep, I'm here11:27
*** tellesnobrega has joined #openstack-sahara11:34
*** tosky has quit IRC12:16
*** tosky_ has joined #openstack-sahara12:16
stannie42Hi AndreyPavlov12:21
*** miqui_away has quit IRC12:21
stannie42I wanted to contact about the spec for CLI refactoring12:26
stannie42but since I couldn't read you, I added my comments in the review12:27
*** vgridnev_ has joined #openstack-sahara12:27
AndreyPavlovok, I'll read them12:29
openstackgerritSergey Reshetnyak proposed openstack/python-saharaclient: Use pretty-tox for better test output  https://review.openstack.org/15208612:38
openstackgerritEvgeny Sikachev proposed openstack/sahara: Refactoring methods for terminating  https://review.openstack.org/15077813:05
openstackgerritVitaly Gridnev proposed openstack/sahara: Apply event-log feature for Vanilla plugins  https://review.openstack.org/14997213:19
*** Nikolay_St has quit IRC13:29
openstackgerritPierre Padrixe (stannie) proposed openstack/sahara-specs: Add spec for Delete by multiple names or ids in CLI  https://review.openstack.org/15211513:40
*** tosky_ has quit IRC13:51
*** egafford has joined #openstack-sahara13:51
*** tosky_ has joined #openstack-sahara13:54
openstackgerritAndrey Pavlov proposed openstack/sahara-specs: Spec for sahara client CLI refactoring  https://review.openstack.org/15212214:02
openstackgerritVitaly Gridnev proposed openstack/sahara: Apply event-log feature for Vanilla plugins  https://review.openstack.org/14997214:05
*** _crobertsrh is now known as crobertsrh14:08
*** Nikolay_St has joined #openstack-sahara14:23
openstackgerritMerged openstack/sahara: fix Direct engine moves cluster to "Scaling" twice  https://review.openstack.org/14644714:25
*** zigo has quit IRC14:31
openstackgerritVitaly Gridnev proposed openstack/sahara: Add provision steps to Spark Plugin  https://review.openstack.org/15213214:33
*** Nikolay_St has quit IRC14:34
*** zigo has joined #openstack-sahara14:35
* mattf nudges SergeyLukjanov - https://review.openstack.org/#/c/150409/14:35
*** jodah has quit IRC14:37
openstackgerritAndrey Pavlov proposed openstack/sahara-specs: Spec for sahara client CLI refactoring  https://review.openstack.org/15212214:37
*** Nikolay_St has joined #openstack-sahara14:40
*** jodah has joined #openstack-sahara14:41
*** openstackgerrit has quit IRC14:52
*** openstackgerrit has joined #openstack-sahara14:52
*** tmckay has joined #openstack-sahara14:55
openstackgerritVitaly Gridnev proposed openstack/sahara: Add provisioning steps to Storm plugin  https://review.openstack.org/15214315:04
*** vgridnev_ has quit IRC15:12
openstackgerritSergey Reshetnyak proposed openstack/sahara-specs: Fix tests for specs  https://review.openstack.org/15215315:25
*** Nikolay_St has quit IRC15:30
openstackgerritKen Chen proposed openstack/sahara: Separate the codes of CDH5 and CDH5.3.0  https://review.openstack.org/14793315:30
*** earnThis has joined #openstack-sahara15:33
earnThisIs devstack robust enough for a complete sahara install, in a lab type/POC situation?15:34
elmikoearnThis: how big a cluster are you wanting to build?15:35
elmikoearnThis: for example, i've run some of our example jobs on 5 node cluster with devstack15:35
earnThisSomething small, just wanted to get my feet with sahara15:38
elmikothen yes, definitely15:38
earnThiselmiko: great, thanks15:38
elmikoit's even integrated into devstack, so you should be able to turn it on in the local.conf and see it under "data processing" in horizon15:38
earnThiselmiko: yea I just noticed that, which is nice. I thought it was going to be a more manual endeavor15:39
elmiko=)15:39
*** skolekonov has quit IRC15:40
openstackgerritVitaly Gridnev proposed openstack/sahara: Waiting should depends on cluster state  https://review.openstack.org/15039915:43
egaffordHi all. At https://review.openstack.org/#/c/150117/, I'm seeing a post-merge CI failure with 80 rechecks. However, this change only hit the Spark plugin, and the Spark CI job succeeds. How should I interpret this? I'm happy to just post ci recheck to the review, but figured I'd check to see if I'm missing something.15:47
* elmiko looks15:49
egaffordThanks, elmiko.15:49
elmikoegafford: is it the same tests that keep failing?15:50
egaffordelmiko: Honestly, I haven't looked all that closely yet; given the nature of the change and the failed jobs, I imagined there might be a quick "oh yeah, CI just does that sometimes" sort of answer. I'll dig deeper and report back.15:53
elmikoegafford: well yea, usually when i see the cluster in error state i just recheck15:54
elmikobut if you're seeing 80 rechecks, then i might start investigating which errors are occuring. especially if it's the same ones.15:55
egaffordelmiko: Question on that: does the CI automatically recheck? No rechecks were manually requested. Is there really a max automatic recheck count of 80 before notification (which is what it looks like from here)? That seems... generous.15:57
elmikoegafford: i think you need to manually recheck15:57
elmikoegafford: and fwiw, it looks like the current spate of failures are all occuring because those clusters can't resolve the external urls needed to download parts of cdh and hdp plugins15:58
egaffordOkay. So what does "Sahara Hadoop Cluster CI check (80 rechecks)" mean, in the context that CI was never rechecked? Yes, I agree.15:58
elmikoif they persist we might need to inquire the cluster's access to the public internet15:59
egaffordYup.15:59
elmikoegafford: where does it say 80 rechecks?15:59
elmikooh nvm15:59
egaffordCool.15:59
elmikothat's a good question, i'm not sure what that means.15:59
egaffordOkay, thanks.16:00
elmikomaybe it means it did 80 rechecks on the last cycle, and if you kick another recheck it will do 80 cycles again(if necessayr)16:00
egaffordThat was my thought, but that seems like a lot of cluster time (and weak approval in case of success.)16:01
openstackgerritSergey Reshetnyak proposed openstack/sahara-specs: Fix tests for specs  https://review.openstack.org/15215316:01
elmikomight have to ask some of the mirantis folks about that16:02
egaffordelmiko: Reasonable; thanks.16:03
openstackgerritTrevor McKay proposed openstack/sahara: Add Swift integration with Spark  https://review.openstack.org/14665916:06
*** witlessb has joined #openstack-sahara16:12
openstackgerritMerged openstack/sahara-image-elements: Remove ability to building vanilla 2.3 and 2.4 images  https://review.openstack.org/14997116:14
egaffordelmiko: Well, whatever that was, it went away on the 81st (manual) recheck.16:14
elmikoegafford: lol, great success!16:14
elmikoegafford: it just needed your human touch ;)16:14
egaffordelmiko: Heh.16:15
openstackgerritlu huichun proposed openstack/sahara: Make test_transient_gating support different plugin  https://review.openstack.org/15187016:18
*** Networkn3rd has joined #openstack-sahara16:33
*** hogepodge has quit IRC16:40
*** hdd has joined #openstack-sahara16:41
*** IvanBerezovskiy has left #openstack-sahara17:01
tmckayelmiko, crobertsrh, if you go to http://apache-spark-user-list.1001560.n3.nabble.com/ can you see a post from me about cdh5 at 11:55?  I got a failure notice saying I'm not subscribed, but I am, and I got a copy of it.17:01
* elmiko looks17:04
elmiko"Plans for distribution tarball based on cdh5-1.2.0_5.3.0? (jackson issue)", i see it17:04
tmckayyeah, cool, thanks.  Weird17:04
tmckayspam failure17:04
elmikoyea, weird...17:05
*** hogepodge has joined #openstack-sahara17:05
*** hogepodge has quit IRC17:07
*** hogepodge has joined #openstack-sahara17:09
*** coolsvap is now known as coolsvap|afk17:12
openstackgerritKen Chen proposed openstack/sahara: Separate the codes of CDH5 and CDH5.3.0  https://review.openstack.org/14793317:27
*** sgotliv has quit IRC17:28
crobertsrha bit late tmckay, but yes, I see it :)17:40
tmckaycrobertsrh, thanks17:40
*** tosky_ has quit IRC17:45
*** amcrn has joined #openstack-sahara17:47
*** tosky_ has joined #openstack-sahara17:48
*** coolsvap|afk is now known as coolsvap17:52
*** Networkn_ has joined #openstack-sahara17:59
*** Networkn3rd has quit IRC18:00
*** ViswaV has quit IRC18:08
*** ViswaV has joined #openstack-sahara18:08
*** hdd has quit IRC18:13
*** Networkn_ has quit IRC18:16
*** Networkn3rd has joined #openstack-sahara18:16
*** Nikolay_St has joined #openstack-sahara18:17
*** Nikolay_St has quit IRC18:26
*** tosky_ has quit IRC18:28
openstackgerritYaroslav Lobankov proposed openstack/sahara: Open port 8088 for HDP 2.0.6  https://review.openstack.org/15222518:44
*** Nikolay_St has joined #openstack-sahara18:45
*** witlessb has quit IRC18:48
*** witlessb has joined #openstack-sahara18:50
*** Nikolay_St has quit IRC18:51
*** earnThis has quit IRC18:56
*** hdd has joined #openstack-sahara18:57
*** Nikolay_St has joined #openstack-sahara19:06
*** Nikolay_St has quit IRC19:11
openstackgerritMerged openstack/sahara: Fixed minor errors in Sahara DB comments  https://review.openstack.org/15182019:23
*** Nikolay_St has joined #openstack-sahara19:25
*** coolsvap is now known as coolsvap|afk19:36
*** hdd has quit IRC19:57
*** sgotliv has joined #openstack-sahara20:18
*** Nikolay_St has quit IRC20:18
*** sgotliv has quit IRC20:19
tmckaycrobertsrh, does this mean anything to you? /opt/stack/horizon/openstack_dashboard/templates/project/data_processing.job_binaries/job_binaries_form_script.html (File does not exist)20:27
tmckaytrying to run from a devstack master with sahara endpoint added to the catalog20:27
crobertsrhYes.20:27
tmckayeasy fix?20:27
crobertsrhI think that file was removed in a recent patch20:27
tmckayah, should I roll back the horizon version a bit?20:28
crobertsrhit might be the easiest path to victory20:28
crobertsrhwhere is it being referenced from?20:28
* tmckay goes for the big red button20:28
crobertsrhah...on the jobs page20:29
tmckayI went to the "jobs" tab on a horizon running as part of devstack, sahara is independent but added to the catalog20:29
crobertsrhdoh20:29
crobertsrhProbably a bug20:29
crobertsrhIf you just remove line 87 in data_processing/jobs/templates/data_processing.jobs/jobs.html, you should be fine20:30
tmckayoh, cool20:30
tmckaycroberts, heh, magic20:31
tmckaythanks20:31
crobertsrhnp :)20:32
*** ViswaV has quit IRC20:35
*** ViswaV has joined #openstack-sahara20:54
venzatmckay: ping20:54
tmckayvenza, hi20:55
venzahi!20:55
venzaI read your email20:55
tmckaywhat do you think?20:55
tmckayI just started thinking about this late Friday, when I found that the newest cdh branch can resolve the jackson version issue20:56
venzaif I understand correctly the easiest way would be to package spark while we wait for a fixed cdh 5.3 package?20:56
venzabut it is not clear to me the relation with swift20:57
tmckayYes, if we build our own spark assembly (and make sure we use sane flags to compile it), we could make that tgz available (probably from mirantis.com)20:57
tmckayvenza, oh, swift.20:57
tmckayWell, the hadoop-openstack.jar (or hadoop-swift.jar for the version carried by Sahara) calls into jackson-mapper-asl at runtime20:58
venzaah ok20:58
tmckayif jackson-core and jackson-mapper do not match versions, you need to play games with the classpath to avoid the error20:58
tmckayI don't like patching the classpath :)20:58
tmckayI would rather have a consistent spark assembly20:58
venzayes, I agree20:59
tmckaythe other question is should we add complexity to DIB and Sahara to support multiple simultaneous versions for the spark plugin, like vanilla, hdp, and cdh20:59
tmckayI think "yes"20:59
tmckayBut, maybe not for kilo, unless we have volunteers :)21:00
tmckayI can bring this up on Thursday at the meeting21:00
venzasorry for my questions, I am not able to follow everything, but why do we need multiple versions of spark?21:01
tmckayvenza, just convention.  We are trying to support the previous version of a plugin for at least a cycle after a new one is introduced and the old one is deprecated21:01
venzaah ok21:01
tmckaySo, for instanche, we had vanilla 2.4 for a while, now it's going away21:02
tmckayWe should keep keep spark on cdh4 for a cyle, along with cdh5, and then phase it out21:02
tmckay(typos)21:02
venzaok so the problem is the cdh version, not the spark version, I understand better now21:03
tmckaybut that also means in the Sahara pulldowns, you have a choice.  And the Sahara internals need to handle the choice21:03
tmckayvenza, well, it *is* actually the spark version.  The classes in the assembly21:03
venzayes in the end the user sees the spark plugin21:03
tmckayI was looking at the spark git branches21:03
tmckayIf the same spark assembly can be built for cdh4, that would be fine21:04
tmckaybut I'm not sure what dependencies on cdh5 there might be in the spark git21:04
tmckayIt is labeled like this:21:04
tmckaycdh5-1.2.0_5.3.021:05
tmckaythe change I need is in the pox.xml for spark21:05
tmckaymaybe it doesn't have to be tied to cdh5, I don't know21:05
venzaon spark website you can download spark 1.2 for cdh421:06
venzathe binary compiled21:06
tmckayI tried that, assembly still has mixed classes21:06
tmckayI checked the pom files in the cdh branches, none of them have the change until the cdh5-1.2.0-5.3.0 branch.  I suppose we could make our own.21:07
venzapietro, who is our spark expert will be back next week, I'll ask him if he has any idea21:08
venzaI understand the convention and the reasons behind it21:08
tmckayvenza, okay.  So in summary, it is really just a technical issue with which jackson deps are pulled in to the spark assembly build21:09
venzabut to me it seems a waste of time in this case21:09
tmckayvenza, maybe so21:09
venzaas for volunteers, for now we are willing, but really no time to spare :(21:10
tmckayI understand21:11
venzado you have any idea of how many sahara-spark users are there?21:11
tmckayno21:12
venzaI think I will go to bed now, thank you very much for the explanations21:12
tmckaygood night, thanks!21:12
venzatmckay, one last thing that just came to me21:18
tmckaysure21:18
venzathe spark plugin may require very little to support two versions21:18
venzato go from cdh4 to cdh5 I didn't have to change anything21:19
tmckayyes, I agree.  I think maybe only some changes in config-helper to deal with the EDP stuff21:19
tmckayso maybe it's not that bad21:19
venzait just expects an image filled with the right bits21:19
venzait is mostly the dib elements21:20
venzathat would need to be changed21:20
tmckayI think you're right21:20
tmckaybut we still need specs, and integration tests, etc etc21:20
venzayep21:21
tmckayIf someone is free to work on it, there is plenty of time I think.  But, if it's being added on top of other stuff, maybe not21:21
crobertsrhIf anyone has a chance for a quick horizon review...https://review.openstack.org/#/c/152272/21:22
tmckaydone21:24
crobertsrhThanks guys21:28
*** egafford has quit IRC21:40
*** ViswaV has quit IRC21:50
*** crobertsrh is now known as _crobertsrh21:55
openstackgerritAndrew Lazarev proposed openstack/python-saharaclient: Fixed work with 'data_processing' service spelling  https://review.openstack.org/14940122:01
*** ViswaV has joined #openstack-sahara22:02
*** ViswaV_ has joined #openstack-sahara22:04
*** ViswaV has quit IRC22:07
alazarevtmckay, hi, have a minute to talk about https://review.openstack.org/#/c/150640/ ?22:13
tmckayalazarev, hi22:17
alazarevtmckay, I'm thinking what can we do to link execution and resulting data...22:19
tmckaymaybe a field in the job_execution?22:19
alazarevtmckay, with value for all variables?22:20
alazarevtmckay, and don't allow several %RND% (all of them will be replaced with the same value then)22:21
tmckayyes, that might work.  A dictionary with the placeholders and the values22:21
tmckaynot sure what else could be done, the job_execution is the logical place to store that22:22
alazarevthe original idea was to allow placeholder parameters... e.g. length of random string22:23
tmckayright, that part makes sense to me.  I just wonder how, after you run it, you can be sure which output came from which job22:23
tmckayif you do it multiple times22:24
alazarevthe easiest way is to store urls in job executions22:24
tmckaythe spark edp stuff works like this, except that the job dir has the execution id as a name, so it's easy to find22:24
tmckayI think so22:24
alazarevwe can use existing "info" field for that22:25
tmckayagreed22:25
tmckaythat will make the updates easy22:25
alazarevlooks like a kind of info22:25
tmckayand it can be recorded when the status goes to "running"22:25
tmckaywe already have an update there, to store the job id, etc22:26
tmckayalazarev, +1, sounds good to me.  time to go, I'll review again tomorrow if you udpate22:27
alazarevtmckay, ok, will update spec22:28
*** tmckay has left #openstack-sahara22:29
alazarevtmckay, I took a look into code, job_execution creation is the most logical place to make that22:32
openstackgerritAndrew Lazarev proposed openstack/sahara-specs: Allow placeholders in datasource URLs  https://review.openstack.org/15064022:41
openstackgerritMerged openstack/sahara: Specify the package name when executing Java type edp jobs  https://review.openstack.org/15200922:51
*** ViswaV_ has quit IRC22:59
*** chlong has joined #openstack-sahara23:11
*** ViswaV has joined #openstack-sahara23:22
« Sunday, 2015-02-01 Index Tuesday, 2015-02-03 »

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!