Wednesday, 2015-02-25

« Tuesday, 2015-02-24 Index Thursday, 2015-02-26 »
openstackgerritweiting-chen proposed openstack/sahara: Add Key Value Store service test in cdh plugin integration test  https://review.openstack.org/15894001:05
*** alazarev has quit IRC01:08
*** aignatov has quit IRC01:08
*** SergeyLukjanov has quit IRC01:08
*** aignatov has joined #openstack-sahara01:09
*** openstack has joined #openstack-sahara01:11
*** alazarev has joined #openstack-sahara01:11
*** SergeyLukjanov has joined #openstack-sahara01:14
*** sgotliv has quit IRC01:16
*** elmiko has quit IRC01:16
*** Longgeek has joined #openstack-sahara01:18
*** Longgeek has quit IRC01:18
*** elmiko has joined #openstack-sahara01:20
*** amcrn has quit IRC01:20
*** dmitryme2 has joined #openstack-sahara01:23
*** dmitryme has quit IRC01:24
*** dmitryme2 is now known as dmitryme01:24
*** sgotliv has joined #openstack-sahara01:25
*** Longgeek has joined #openstack-sahara01:25
*** pashkin has quit IRC01:35
*** ekarlso has quit IRC01:35
*** juice has quit IRC01:35
*** anteaya has quit IRC01:35
*** ruhe has quit IRC01:35
*** NikitaKonovalov has quit IRC01:35
*** coolsvap_ has quit IRC01:35
*** ekarlso has joined #openstack-sahara01:39
*** pashkin has joined #openstack-sahara01:39
*** ruhe has joined #openstack-sahara01:39
*** NikitaKonovalov has joined #openstack-sahara01:39
*** coolsvap_ has joined #openstack-sahara01:39
*** juice has joined #openstack-sahara01:39
*** anteaya has joined #openstack-sahara01:39
*** pashkin has quit IRC01:40
*** ekarlso has quit IRC01:40
*** juice has quit IRC01:40
*** anteaya has quit IRC01:40
*** ruhe has quit IRC01:40
*** NikitaKonovalov has quit IRC01:40
*** coolsvap_ has quit IRC01:40
*** sreshetnyak has quit IRC01:40
*** ekarlso has joined #openstack-sahara01:41
*** pashkin has joined #openstack-sahara01:41
*** ruhe has joined #openstack-sahara01:41
*** NikitaKonovalov has joined #openstack-sahara01:41
*** coolsvap_ has joined #openstack-sahara01:41
*** juice has joined #openstack-sahara01:41
*** anteaya has joined #openstack-sahara01:41
*** Longgeek has quit IRC01:49
*** jamielennox is now known as jamielennox|away01:58
*** sreshetnyak has joined #openstack-sahara01:59
*** jamielennox|away is now known as jamielennox02:06
*** sreshetnyak has quit IRC02:10
*** sreshetnyak has joined #openstack-sahara02:16
openstackgerritweiting-chen proposed openstack/sahara: Add Key Value Store service test in cdh plugin integration test  https://review.openstack.org/15894002:16
*** jamielennox is now known as jamielennox|away02:17
*** Longgeek has joined #openstack-sahara02:18
*** jamielennox|away is now known as jamielennox02:27
openstackgerritweiting-chen proposed openstack/sahara: Add Impala service test in cdh plugin integration test  https://review.openstack.org/15193402:36
*** jamielennox is now known as jamielennox|away02:38
*** jamielennox|away is now known as jamielennox02:47
openstackgerritMichael McCune proposed openstack/sahara-specs: Adding improved secret storage spec  https://review.openstack.org/15743202:49
openstackgerritTIngting Bao proposed openstack/sahara: Remove unused field in job_execution table  https://review.openstack.org/15896403:04
*** devlaps has joined #openstack-sahara03:14
*** jamielennox is now known as jamielennox|away03:22
*** jamielennox|away is now known as jamielennox03:33
*** hogepodge has quit IRC03:34
*** hogepodge has joined #openstack-sahara03:39
*** devlaps has quit IRC03:40
*** Longgeek has quit IRC03:52
*** ViswaV has quit IRC04:01
*** mahito has joined #openstack-sahara04:03
*** ViswaV has joined #openstack-sahara04:05
*** coolsvap_ is now known as coolsvap04:16
*** hdd has quit IRC04:36
*** hdd has joined #openstack-sahara04:37
*** hdd has quit IRC04:38
*** Longgeek has joined #openstack-sahara04:46
*** akuznetsov has joined #openstack-sahara05:03
*** Longgeek has quit IRC05:07
*** Longgeek has joined #openstack-sahara05:07
*** chen123 has joined #openstack-sahara05:09
*** chandankumar has joined #openstack-sahara05:26
*** Longgeek has quit IRC05:50
*** chandankumar has quit IRC05:50
*** Longgeek has joined #openstack-sahara05:52
*** mahito has quit IRC06:03
*** mahito has joined #openstack-sahara06:13
*** Longgeek has quit IRC06:19
*** chandankumar has joined #openstack-sahara06:20
*** hdd has joined #openstack-sahara06:27
*** mahito has quit IRC07:02
*** mahito has joined #openstack-sahara07:06
*** tnovacik_ has joined #openstack-sahara07:08
*** mahito has quit IRC07:11
*** mahito has joined #openstack-sahara07:12
*** mahito has quit IRC07:36
*** mahito has joined #openstack-sahara07:37
*** hdd has quit IRC07:42
*** ekarlso has quit IRC07:48
openstackgerritEvgeny Sikachev proposed openstack/sahara: Add support of several scenario files in integration tests  https://review.openstack.org/15871008:08
*** skolekonov has joined #openstack-sahara08:09
*** witlessb has joined #openstack-sahara08:16
*** ViswaV has quit IRC08:27
*** mahito has quit IRC08:31
openstackgerritEvgeny Sikachev proposed openstack/sahara: Add unit-tests for new integration tests  https://review.openstack.org/15529808:43
*** IvanBerezovskiy_ has joined #openstack-sahara08:44
*** ekarlso has joined #openstack-sahara08:52
openstackgerritDenis Egorenko proposed stackforge/sahara-ci-config: Divide jobs log to different files  https://review.openstack.org/15873708:55
openstackgerritDenis Egorenko proposed stackforge/sahara-ci-config: Divide jobs log to different files  https://review.openstack.org/15873708:58
openstackgerritEvgeny Sikachev proposed openstack/sahara: Add support of several scenario files in integration tests  https://review.openstack.org/15871009:04
*** akuznetsov has quit IRC09:07
openstackgerritEvgeny Sikachev proposed openstack/sahara: Add unit-tests for new integration tests  https://review.openstack.org/15529809:11
openstackgerritDenis Egorenko proposed stackforge/sahara-ci-config: Divide jobs log to different files  https://review.openstack.org/15873709:12
openstackgerritDenis Egorenko proposed stackforge/sahara-ci-config: Divide jobs log to different files  https://review.openstack.org/15873709:21
*** akuznetsov has joined #openstack-sahara09:21
openstackgerritMerged stackforge/sahara-ci-config: Divide jobs log to different files  https://review.openstack.org/15873709:27
openstackgerritEvgeny Sikachev proposed openstack/sahara: Add unit-tests for new integration tests  https://review.openstack.org/15529810:02
openstackgerritEvgeny Sikachev proposed openstack/sahara: Add support of several scenario files in integration tests  https://review.openstack.org/15871010:16
*** tnovacik_ has quit IRC10:22
*** macjacktw has joined #openstack-sahara10:23
*** macjacktw1 has joined #openstack-sahara10:24
*** macjacktw1 has quit IRC10:25
*** macjack has quit IRC10:25
*** macjack has joined #openstack-sahara10:26
*** macjack has quit IRC10:26
*** macjacktw has quit IRC10:27
*** tosky has joined #openstack-sahara10:33
*** macjack has joined #openstack-sahara10:36
*** tnovacik_ has joined #openstack-sahara11:17
openstackgerritEvgeny Sikachev proposed openstack/sahara: Add support of several scenario files in integration tests  https://review.openstack.org/15871011:25
openstackgerritMerged openstack/sahara: Minor - Added missing check for 'Deleting' state  https://review.openstack.org/15586112:42
*** tnovacik_ has quit IRC12:53
*** tnovacik_ has joined #openstack-sahara13:02
openstackgerritEvgeny Sikachev proposed openstack/sahara: Add support of several scenario files in integration tests  https://review.openstack.org/15871013:08
openstackgerritMerged openstack/sahara: Remove unused code (timed decorator)  https://review.openstack.org/15864913:11
openstackgerritOpenStack Proposal Bot proposed openstack/sahara: Updated from global requirements  https://review.openstack.org/15877513:22
openstackgerritOpenStack Proposal Bot proposed openstack/sahara-dashboard: Updated from global requirements  https://review.openstack.org/15911813:22
*** ylobankov has quit IRC13:34
*** _crobertsrh is now known as crobertsrh13:50
openstackgerritVitaly Gridnev proposed openstack/sahara: Provide ability to get events directly from cluster  https://review.openstack.org/15913113:59
openstackgerritVitaly Gridnev proposed openstack/sahara: Provide ability to get events directly from cluster  https://review.openstack.org/15913114:02
openstackgerritVitaly Gridnev proposed openstack/sahara: Provide ability to get events directly from cluster  https://review.openstack.org/15913114:04
openstackgerritChad Roberts proposed openstack/sahara: Adding ability to edit cluster templates  https://review.openstack.org/15746014:06
openstackgerritChad Roberts proposed openstack/sahara: Adding ability to edit cluster templates  https://review.openstack.org/15746014:23
*** witlessb has quit IRC14:27
*** witlessb has joined #openstack-sahara14:29
*** openstackgerrit has quit IRC15:08
*** openstackgerrit has joined #openstack-sahara15:08
openstackgerritChad Roberts proposed openstack/sahara: Adding ability to edit cluster templates  https://review.openstack.org/15746015:36
*** chandankumar has quit IRC15:47
*** chandankumar has joined #openstack-sahara15:53
openstackgerritArtem Osadchiy proposed openstack/sahara: Add Drill support for MapR plugin  https://review.openstack.org/14835016:00
*** tnovacik_ has quit IRC16:09
*** tnovacik_ has joined #openstack-sahara16:12
*** hdd has joined #openstack-sahara16:15
*** ViswaV has joined #openstack-sahara16:16
openstackgerritDenis Egorenko proposed stackforge/sahara-ci-config: Migrate jobs to new integrations tests config  https://review.openstack.org/15530116:17
openstackgerritEvgeny Sikachev proposed openstack/sahara: Add support of several scenario files in integration tests  https://review.openstack.org/15871016:18
*** coolsvap is now known as coolsvap_16:28
*** tnovacik_ has quit IRC16:28
*** openstackstatus has joined #openstack-sahara16:43
*** ChanServ sets mode: +v openstackstatus16:43
*** IvanBerezovskiy_ has quit IRC16:53
*** vgridnev_ has joined #openstack-sahara17:05
*** tnovacik_ has joined #openstack-sahara17:06
vgridnev_ping tmckay17:06
tmckayvgridnev_, hi17:06
vgridnev_hi17:06
vgridnev_when i'm use data.get('progress', False) it would be string, right?17:08
vgridnev_in my patch which i proposed today17:08
tmckayyes17:09
vgridnev_ok, thanks17:09
tmckayvgridnev_, I was just doing the same thing in another patch.  I expected it to be a boolean :)17:10
vgridnev_Today Nikita and Sergey Lukjanov we decided, that is bad approach to have separate endpoint for events17:10
vgridnev_Today with*17:10
*** chandankumar has quit IRC17:11
*** skolekonov has quit IRC17:13
tmckayah, I se17:16
tmckaysee17:16
*** tmckay is now known as tmckay_lunch17:16
*** tnovacik_ has quit IRC17:21
*** jamielennox is now known as jamielennox|away17:22
*** jamielennox|away is now known as jamielennox17:30
elmikovgridnev_: why the move away from an endpoint for clusterevents?17:38
vgridnev_it's was noticed during implementation for horizon17:38
elmikobut why remove it?17:39
vgridnev_we have problem: if need full info about provision progress, it's required to make 2 api calls17:39
elmikoone for the cluster and one for the events?17:39
vgridnev_yes17:40
elmikohmm17:40
egaffordSergeyLukjanov: An oddity: on https://review.openstack.org/#/c/159118/, I do not have +2 (though I'm stable maint core now,) while elmiko (who is sahara-core but not stable maint core) does have +2 on this. It seems that sahara-dashboard stable/icehouse may not be playing by group rules.17:40
elmikovgridnev_: we probably need to patch the spec, imo17:40
vgridnev_good proposition, elmiko17:41
openstackgerritSergey Reshetnyak proposed openstack/sahara: Collect errors in new integration tests  https://review.openstack.org/15784217:53
*** tnovacik_ has joined #openstack-sahara17:56
openstackgerritMerged openstack/sahara: Updated from global requirements  https://review.openstack.org/15877517:59
*** jamielennox is now known as jamielennox|away18:02
*** hdd has quit IRC18:06
vgridnev_folks, how to ping guys from Cloudera/Intel?18:10
*** egafford has quit IRC18:11
*** jamielennox|away is now known as jamielennox18:15
*** hdd has joined #openstack-sahara18:20
crobertsrhAre rechecks just slow today?  Or is it me?18:24
toskycrobertsrh: I had to launch 2 rechecks for two different issues (one fixed today), so I suspect the queues are full18:26
crobertsrhOk.  I feel less bad if it is slow for everyone :)18:26
*** tmckay_lunch is now known as tmckay19:10
tmckayvgridnev_, sometimes they check the channel (maybe through eavesdrop) but you could email them directly19:12
tmckayor leave a note on a review19:12
tmckayvgridnev_, who are you trying to find?19:12
toskytmckay, vgridnev_: aren't they hanging around during the meeting? Especially tomorrow, during the UTC-afternoon time?19:14
*** macjack has quit IRC19:15
elmikotosky: +119:16
tmckaytosky, yes, but you can find them more quickly if you need to19:17
toskyoh, there are always emails19:17
openstackgerritMerged openstack/sahara: Add support for oslo_debug_helper to tox.ini  https://review.openstack.org/15881219:18
elmikoor ring the sahara tower bells ;)19:19
*** tosky has quit IRC19:30
vgridnev_i'm trying to find ken chen for reviewing this patch: https://review.openstack.org/#/c/157728/19:31
*** egafford has joined #openstack-sahara19:33
tmckayvgridnev_, I would send an email.  They have contacted me by email about Oozie questions19:35
elmikosend to the ml too19:35
vgridnev_ok, thanks tmkay19:36
*** chandankumar has joined #openstack-sahara19:50
*** akuznetsov has quit IRC20:00
openstackgerritAndrew Lazarev proposed openstack/sahara: Implemented support of placeholders in datasource URLs  https://review.openstack.org/15890920:25
*** chandankumar has quit IRC21:01
openstackgerritAndrew Lazarev proposed openstack/sahara-specs: Use trusts for cluster creation and scaling  https://review.openstack.org/15925121:15
*** amcrn has joined #openstack-sahara21:15
tmckayalazarev, ping21:18
alazarevtmckay, pong21:18
tmckayalazarev, lots of thinking out loud on your CR :)21:19
tmckaymy last comment (just posted) I think might make the most sense21:19
tmckaymaybe we just need placeholder replacement calls in the data source resolution routine, instead of just copying the URL21:19
tmckayalazarev, line 228 in service/edp/job_utils.py21:20
tmckaywe would have to pass job_execution id in to that routine, too21:21
tmckaythat would handle the data source substitution cases.  But not manually typed URLs with placeholders.21:24
tmckayBut, those could be checked for too during processing21:25
alazarevtmckay, do you think we need support manually typed URLs?21:26
alazarevtmckay, I thought the feature is for datasources only21:26
tmckayalazarev, well, I don't know.  It seems inconsistent to me not to21:27
tmckaythe case would be someone running a Java job mulitple times, with an arg giving an output dir21:27
alazarevtmckay, for manually types URLs we could type whatever we want, for datasource we can't21:27
tmckayalazarev, I guess we could say that if you want placeholders, you must use the data source substitution feature.  That's the rule.21:28
tmckayalazarev, but it would be the relaunch case.  I guess you could change the value and relaunch.  Okay, I can accept that, sure.21:28
alazarevtmckay, e.g. we could add %DATASOURCE_ID% var that will not have sense without datasource, also we could add %DATASOURCE_TYPE%, etc.21:29
tmckayso, manual is not supported.  You can do that even with a Java job by using data_source substitution, so kay21:29
tmckayokay21:29
*** tnovacik_ has quit IRC21:29
tmckayalazarev, maybe we should change the syntax for data source references to match your placeholders21:30
alazarevtmckay, why do you think that updating job_execution in engine is bad?21:30
tmckayIt seemed like good policy to have a single point of update, so that when working on code in the job_manager the developer could be sure that it had not changed.  I think it makes setting status, etc, less error-prone21:31
tmckayalazarev, it doesn't have to be set in stone, if there is a good case not to, but I think it helps in sanely managing job status, etc21:33
alazarevtmckay, we don't have such policy for other objects21:34
alazarevtmckay, cluster status is changed in many places21:34
alazarevtmckay, and having long list in return statement looks even worse for me21:35
alazarevtmckay, datasources are not supported in spark, right? Only manual URLs21:37
tmckayalazarev, using the data source reference substitution mechanism, they are21:39
tmckaythis is new21:39
tmckayso you reference one by uuid in the arg list and turn substitution on21:40
tmckay(or by name, with datasource://name I think)21:40
alazarevtmckay, and URL will be added to configs... and now I dont21:40
alazarevtmckay, and URL will be added to configs... and now I don't update placeholder... right?21:40
tmckayright.21:41
tmckaybut I think it can be fixed near line 228 of job_utils, easily21:41
alazarevI update it later, but value in config will still contain placeholder21:41
tmckayright21:41
alazarevand to fix that we need 1. all info to make replacemnt 2. way to return generated URL21:43
tmckayalazarev, maybe your're right about job_execution.  For status updates, it seemed better to do it in the job_manger, but for other fields, maybe it doesn't matter.  It seemed unnecessary to have each implementation of cancel_job() for instance do the status update.21:43
tmckayalazarev, I think you have most of #1.  I think all you need is the job_execution id which can be passed in, and the url generation routine is already in the same file, isn't it?21:44
alazarevtmckay, yeap, looks so21:45
tmckaydatasource is already retrieved from database, so you have that too21:45
tmckayoh, it url constructor takes the whole job_exeuction but that's okay. It can take an id, or the data source reference routine can just take job_execution21:46
alazarevwith new ability to reference from config... is it possible that the same datasource referenced twice?21:47
tmckayyes21:47
alazarevtmckay, not good :)21:47
alazarevtmckay, is it Ok to have the same URL for all duplicated?21:48
tmckayheh.  It didn't matter for my case.  But, we could add a cache to that routine.21:48
tmckayhmmm .... interesting case.21:49
alazarevtmckay, because I return id->URL dict as result21:50
*** ViswaV has quit IRC21:50
tmckaywe don't know what an app is going to do with them ... do we make it illegal?  not sure21:50
tmckayI could see inputs being passed multiple times, maybe.  But outputs ... probably not21:51
tmckaythe trouble is we have no idea how an app is written21:51
*** ViswaV has joined #openstack-sahara21:52
*** vgridnev_ has quit IRC21:53
tmckayalazarev, what if  resolve_data_source_references() built a dictionary as it went, and looked up references in the dict? And then returned it?21:53
alazarevtmckay, this is exactly what I was thinking about21:54
tmckayThat way, all duplicate references in job_configs would be the same, and if input_source or output_source referenced them too, they would also be the same21:54
alazarevtmckay, exactly21:54
tmckay^^ this last one is a weird case, which hopefully will get better with egafford's job arg mapping spec21:54
tmckayI don't like fixed data sources for mapreduce jobs, for example21:55
elmikowouldn't that dict need to be saved somehow in case of process restart?21:55
tmckayalazarev, as long as we are consistent, it is up to the user to write a good job21:55
alazarevelmiko, it will be saved in job_execution21:55
elmikoalazarev: ack, thanks21:55
tmckayelmiko, ack.  alazarev, more argument for update as soon as possible.  I remove my objection :)21:56
tmckaySahara is becoming powerful, and making my head hurt21:56
openstackgerritTIngting Bao proposed openstack/sahara: Remove unused field in job_execution table  https://review.openstack.org/15896421:56
elmikotmckay: lol21:57
tmckayalazarev, so by adding these changes, Spark would automatically be fixed up.21:58
alazarevtmckay, yeap21:59
tmckaynothing else to do, I believe21:59
tmckayhmm, except for the job_execution update call21:59
egaffordtmckay: Soon; soon (re: job arg mapping spec.)22:01
*** crobertsrh is now known as _crobertsrh22:01
alazarevtmckay, shouldn't spark be able to work with external hdfs? why this code oozie specific?22:02
tmckayalazarev, you mean the configure cluster for hdfs code?  Hmm.  Maybe you're right22:04
tmckayNever tried it22:04
tmckayalazarev, yes, I think you're right.  Oversight.22:07
tmckayI think it was missed because of the lack of data sources22:08
tmckayAnd it brings up a good point, too.  If you run a java job with url args that reference an external hdfs, this could would not run either22:09
tmckay"this code"22:10
tmckayit would only work if you referenced data sources22:10
tmckayalazarev, separate bug?22:15
alazarevtmckay, definitely22:16
alazarevtmckay, it looks that resolve_data_source_references should handle all that stuff22:17
tmckayalazarev, I think so.  If we add the h.configure_cluster_for_hdfs() call to spark in another CR, data_sources for external hdfs referenced in job_configs will be caught. Only one thing left:22:19
tmckaymanual URLs again.  For manually typed URLs referencing external hdfs, we won't catch them.  Mostly a Java and Spark case.22:20
tmckayIt might be possible to check for those in resolve, or do another pass.22:21
alazarevtmckay, we can add configure_cluster_for_hdfs to resolve_data_source_references22:21
alazarevthe same lines as in my patch :)22:22
tmckayalazarev, seems like a separate function to me.  One is dealing with fixing up values in job_exeuction, one is modifying cluster config22:24
alazarevtmckay, but you need to return list of URLs from configs22:25
tmckayyeah, that's the other option, another return value22:26
tmckayor another pass, but that's inefficient22:26
tmckayTradeoffs, I'm okay with it either way22:26
tmckayalazarev, I'll add a bug for external hdfs support for Spark (if you did not already)22:28
alazarevtmckay, I didn't22:29
alazarevtmckay, please also add bug for hdfs in configs for oozie22:29
tmckayokay, will do22:29
alazarevtmckay, hdfs for spark looks more like bp22:30
tmckayyeah, unintentionally missed feature22:30
openstackgerritAndrew Lazarev proposed openstack/sahara-specs: Two step scaling with Heat engine  https://review.openstack.org/15927822:41
tmckayhttps://blueprints.launchpad.net/sahara/+spec/edp-spark-external-hdfs22:53
tmckayhttps://bugs.launchpad.net/sahara/+bug/142573122:53
openstackLaunchpad bug 1425731 in Sahara "[EDP][Oozie] Configuration of cluster for external hdfs missed for URLs in job_configs" [Undecided,New]22:53
tmckayalazarev, fyi, thanks for noticing ^^22:53
*** tmckay is now known as tmckay_bbl22:54
*** ViswaV has quit IRC22:54
openstackgerritAndrew Lazarev proposed openstack/sahara-specs: Use trusts for cluster creation and scaling  https://review.openstack.org/15925122:55
*** ViswaV has joined #openstack-sahara22:59
*** egafford has quit IRC23:00
*** ViswaV has quit IRC23:04
*** ViswaV has joined #openstack-sahara23:04
*** hdd has quit IRC23:22
*** hdd has joined #openstack-sahara23:28
*** witlessb has quit IRC23:37
*** macjack has joined #openstack-sahara23:42
*** chlong has quit IRC23:43
*** chlong_ has quit IRC23:44
*** chlong has joined #openstack-sahara23:48
*** macjack has quit IRC23:48
openstackgerritTIngting Bao proposed openstack/sahara: Remove unused field in job_execution table  https://review.openstack.org/15896423:49
*** macjack has joined #openstack-sahara23:49
*** hdd has quit IRC23:55
openstackgerritAndrew Lazarev proposed openstack/sahara: Implemented support of placeholders in datasource URLs  https://review.openstack.org/15890923:56
« Tuesday, 2015-02-24 Index Thursday, 2015-02-26 »

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!