Thursday, 2015-03-19

« Wednesday, 2015-03-18 Index Friday, 2015-03-20 »
*** mahito has joined #openstack-sahara00:00
openstackgerritTrevor McKay proposed openstack/sahara: Add a CLI tool for managing default templates  https://review.openstack.org/16364900:06
elmikotmckay_bbl: i take it you're adding the template name stuff?00:07
*** tmckay_bbl is now known as tmckay00:07
tmckayelmiko, yeah, it was as piece of cake and it looks nice00:08
elmikotmckay: cool!00:08
tmckayI tried out the sample conf with the cdh clusters, worked beautifully00:08
tmckaycheck out the sample conf attached to the CR00:08
elmikok, might not get to it tonight but it's on the list00:09
elmikoapi-wg meeting right now00:09
tmckayelmiko, there are a few rough edges around rollback, but I think it's pretty solid00:09
tmckayah00:09
tmckayalright, have fun! :)00:09
elmikonice, glad it worked out00:09
*** jamielennox is now known as jamielennox|away00:09
openstackgerritTrevor McKay proposed openstack/sahara: Default templates for CDH  https://review.openstack.org/16499100:10
openstackgerritTrevor McKay proposed openstack/sahara: Default templates for Spark  https://review.openstack.org/16479700:11
openstackgerritTrevor McKay proposed openstack/sahara: Default templates for Vanilla  https://review.openstack.org/16478900:11
openstackgerritTrevor McKay proposed openstack/sahara: Default templates for HDP  https://review.openstack.org/16477200:12
openstackgerritTrevor McKay proposed openstack/sahara: Default templates for HDP  https://review.openstack.org/16477200:17
*** jamielennox|away is now known as jamielennox00:23
*** tmckay has quit IRC00:53
openstackgerritMerged openstack/sahara: Apply event-log feature for HDP plugin  https://review.openstack.org/15247200:58
openstackgerritJamie Lennox proposed openstack/python-saharaclient: Rework authentication  https://review.openstack.org/14777301:06
*** Longgeek has joined #openstack-sahara01:09
*** macjack1 has joined #openstack-sahara02:22
*** macjack1 has quit IRC02:23
*** Poornima has joined #openstack-sahara02:35
*** macjack1 has joined #openstack-sahara02:42
*** macjack1 has quit IRC02:48
*** coolsvap|afk is now known as coolsvap03:04
*** coolsvap is now known as coolsvap|afk03:32
*** coolsvap|afk is now known as coolsvap03:39
*** macjack has joined #openstack-sahara03:46
*** macjack has quit IRC03:56
*** mahito has quit IRC03:59
*** mahito has joined #openstack-sahara04:01
*** jamielennox is now known as jamielennox|away04:06
openstackgerritJacob Bin Wang proposed openstack/sahara: Fix log import error in tempest tests for Sahara  https://review.openstack.org/16568404:07
*** Poornima has quit IRC04:08
*** Poornima has joined #openstack-sahara04:09
openstackgerritJacob Bin Wang proposed openstack/sahara: Fix log import error in tempest tests for Sahara  https://review.openstack.org/16568404:10
*** Nikolay_St has quit IRC04:46
*** Nikolay_St has joined #openstack-sahara04:47
*** chen12 has quit IRC05:17
*** mahito has quit IRC05:25
*** mahito has joined #openstack-sahara05:29
*** sgotliv has joined #openstack-sahara05:40
*** coolsvap is now known as coolsvap|afk05:41
*** chen12 has joined #openstack-sahara05:43
*** coolsvap|afk is now known as coolsvap05:48
openstackgerritYuhara Motoki proposed openstack/python-saharaclient: add --name for CLI command(job-binary-data-create) to assign name to job-binary-internal  https://review.openstack.org/16569805:54
*** Nikolay_St has quit IRC05:58
*** sgotliv has quit IRC06:01
*** pcaruana has quit IRC06:33
openstackgerritJacob Bin Wang proposed openstack/sahara: Fix log import error in tempest tests for Sahara  https://review.openstack.org/16568406:54
*** oikawa has quit IRC07:07
*** tnovacik has joined #openstack-sahara07:13
*** sreshetn1 has joined #openstack-sahara07:24
openstackgerritAndrey Pavlov proposed openstack/sahara: Docs updated with instance locality feature  https://review.openstack.org/16468307:40
*** Poornima is now known as Poornima_Lunch07:45
*** chlong has quit IRC07:47
*** Nikolay_St has joined #openstack-sahara08:19
*** chandankumar has joined #openstack-sahara08:38
*** Nikolay_St has quit IRC08:42
*** Poornima has joined #openstack-sahara08:53
*** mahito has quit IRC09:04
*** skolekonov has joined #openstack-sahara09:14
*** logan2 has quit IRC09:22
*** logan2 has joined #openstack-sahara09:22
*** Longgeek has quit IRC09:27
*** witlessb has joined #openstack-sahara09:28
*** Longgeek has joined #openstack-sahara09:29
*** Nikolay_St has joined #openstack-sahara09:29
*** Longgeek has quit IRC09:31
openstackgerritSebastien Badia proposed stackforge/puppet-sahara: policy.json: Allow one to manage them from the puppet module  https://review.openstack.org/15247009:32
openstackgerritMerged stackforge/puppet-sahara: Migrate postgresql backend to use openstacklib::db::postgresql  https://review.openstack.org/14551109:36
*** Longgeek has joined #openstack-sahara09:37
*** sgotliv has joined #openstack-sahara10:03
*** chandankumar has quit IRC10:04
*** witlessb_ has joined #openstack-sahara10:12
*** Longgeek has quit IRC10:13
*** Longgeek has joined #openstack-sahara10:13
*** witlessb has quit IRC10:13
*** witlessb_ is now known as witlessb10:13
*** chandankumar has joined #openstack-sahara10:18
*** tosky has joined #openstack-sahara10:31
*** coolsvap is now known as coolsvap|afk10:41
openstackgerritMerged stackforge/sahara-ci-config: Global refactoring of CI scripts  https://review.openstack.org/16210210:41
*** chandankumar_ has joined #openstack-sahara10:44
*** chandankumar has quit IRC10:47
*** oikawa has joined #openstack-sahara10:58
openstackgerritMerged openstack/sahara: Add usages for step_type field  https://review.openstack.org/16299711:02
*** pcaruana has joined #openstack-sahara11:05
*** macjack has joined #openstack-sahara11:13
openstackgerritMerged openstack/sahara: Move updating provision progress to conductor  https://review.openstack.org/15942111:15
*** sgotliv has quit IRC11:30
*** coolsvap|afk is now known as coolsvap11:36
*** sgotliv has joined #openstack-sahara11:38
*** macjack has quit IRC11:47
*** sreshetn1 has quit IRC11:56
*** Nikolay_St has quit IRC12:03
*** witlessb has quit IRC12:07
*** chlong has joined #openstack-sahara12:14
*** witlessb has joined #openstack-sahara12:16
*** Longgeek has quit IRC12:20
*** Longgeek has joined #openstack-sahara12:20
*** chlong has quit IRC12:26
*** witlessb_ has joined #openstack-sahara12:34
*** witlessb has quit IRC12:35
*** witlessb_ is now known as witlessb12:35
*** tmckay has joined #openstack-sahara12:36
*** tmckay has quit IRC12:37
*** tmckay has joined #openstack-sahara12:37
*** Longgeek_ has joined #openstack-sahara12:37
*** Longgeek has quit IRC12:41
*** sgotliv has quit IRC12:41
*** chlong has joined #openstack-sahara12:42
*** Nikolay_St has joined #openstack-sahara12:43
*** sgotliv has joined #openstack-sahara12:45
*** sgotliv has quit IRC12:46
*** sgotliv has joined #openstack-sahara12:46
openstackgerritJacob Bin Wang proposed openstack/sahara: Fix order of arguments in assertEqual - Part1  https://review.openstack.org/16527112:55
*** crobertsrh has joined #openstack-sahara12:57
*** pnayak has joined #openstack-sahara13:03
tmckayvgridnev, ping13:04
*** chandankumar_ has quit IRC13:04
*** pnayak has quit IRC13:04
tmckayelmiko, ping13:04
elmikotmckay: pong13:04
tmckayelmiko, hi, I just posted a comment on https://review.openstack.org/#/c/163463/.  Looks to me like _is_accessible is not called?  Am I missing something?13:05
*** Poornima has quit IRC13:05
tmckayhate to -2 something that is already approved, but I didn't see it13:05
elmikotmckay: taking another look at that patch13:09
*** Networkn3rd has quit IRC13:10
elmikotmckay: look at the decorator13:12
elmikoi'm still not sure it's added somewhere, but it looks like the magic might be in there13:14
tmckayah, maybe13:14
elmikoodd though, still don't see how it gets actively added to the poll list13:16
tmckayelmiko, in the _ips_assign case though the method is called13:18
tmckayfrom _await_networks13:18
tmckaygotta relocate, looking forward to the answer13:19
*** vgridnev_ has joined #openstack-sahara13:20
*** chlong has quit IRC13:21
elmikovgridnev_: ping13:22
vgridnev_elmiko, pong13:23
elmikovgridnev_: hey, tmckay has some questions about https://review.openstack.org/#/c/16346313:23
vgridnev_i see13:23
*** tmckay has quit IRC13:23
elmikook13:23
vgridnev_seems that missed it during rebase13:23
elmikogot it, good spot on tmckay's part hehe13:24
*** Longgeek_ has quit IRC13:35
openstackgerritMichael McCune proposed openstack/sahara: Moving cdh cloudera manager password to config file  https://review.openstack.org/16507713:36
*** pnayak has joined #openstack-sahara13:36
openstackgerritMichael McCune proposed openstack/sahara: Moving cdh cloudera manager password to config file  https://review.openstack.org/16507713:38
openstackgerritMichael McCune proposed openstack/sahara: Moving cdh cloudera manager password to config file  https://review.openstack.org/16507713:45
*** tnovacik has quit IRC13:46
openstackgerritMichael McCune proposed openstack/sahara: Moving cdh cloudera manager password to config file  https://review.openstack.org/16507713:47
openstackgerritVitaly Gridnev proposed openstack/sahara: Add usages of poll util for service modules  https://review.openstack.org/16346313:49
vgridnev_tmkay, issue in this patch was resolved13:50
vgridnev_https://review.openstack.org/#/c/163463/813:50
elmikovgridnev_: thanks13:50
*** tmckay has joined #openstack-sahara13:53
*** tmckay has quit IRC13:54
*** tmckay has joined #openstack-sahara13:55
openstackgerritMerged openstack/sahara-specs: Provide default templates for each plugin  https://review.openstack.org/16220913:58
vgridnev_where can i find FFE list?13:59
openstackgerritDenis Egorenko proposed stackforge/sahara-ci-config: Fix minor issues  https://review.openstack.org/16582813:59
tmckayvgridnev_, I was looking for that too the other day14:00
elmikovgridnev_: https://wiki.openstack.org/wiki/Sahara/Kilo/FPF_Exceptions that?14:00
tmckaythat's FPF, FFE should be a different list14:02
tmckaynow that we're entering FF14:02
elmikook, didn't see an ffe in my cache14:02
tmckaynot sure it's been created yet14:02
*** Nikolay_St has quit IRC14:02
elmikomaybe not14:02
*** Nikolay_St has joined #openstack-sahara14:03
vgridnev_kilo-3 was released14:04
elmikocool14:04
tmckayelmiko, so did you get an answer on how those cdh resource scripts were being used?14:05
tmckayre the password stuff? (looking now)14:05
elmikoi did14:05
openstackgerritVitaly Gridnev proposed openstack/sahara: Add usages of poll util for service modules  https://review.openstack.org/16346314:05
elmikoken added a nice response to the review14:05
tmckaycool, I'll look for it14:06
elmikothey are used by an admin, and he expects that they should be hand tailored before running14:06
tmckayexcellent14:06
elmikotoo bad https://review.openstack.org/#/c/164684/ didn't make it into k-3. i fear we've added an unnecessary requirement14:08
*** egafford has joined #openstack-sahara14:08
SergeyLukjanovtmckay, vgridnev_, we've just got the kilo-3 tag, so, FF is active14:09
SergeyLukjanovtmckay, vgridnev_, I've got 3 FFEs already and it's already assigned to kilo-rc1 https://launchpad.net/sahara/+milestone/kilo-rc114:10
SergeyLukjanovtmckay, vgridnev_, so it's polling timeouts, default templates and job types endpoint14:11
elmikoSergeyLukjanov: i'm going to write a patch for the improved-secret-storage spec, since that is Liberty am i ok to put a patch to the spec?14:11
*** egafford has quit IRC14:11
tmckaySergeyLukjanov, thank you.  I am proud to be on the exception list ;-)14:11
SergeyLukjanovelmiko, yeah, sure14:11
SergeyLukjanovtmckay,  :)14:11
elmikoSergeyLukjanov: ack, thanks14:11
elmikolol14:11
*** Nikolay_St has quit IRC14:12
*** egafford has joined #openstack-sahara14:15
vgridnev_hm, 50 emails after release in two minutes : )14:17
elmikoouch!14:17
*** dhellmann has quit IRC14:19
*** dhellmann has joined #openstack-sahara14:19
tmckayI have a few things left for templates.  Fix up MANIFEST.in file, maybe tweak log messages to include name and id both (just id is not so helpful, especially on delete), and maybe see if I can fix rollback of update14:22
tmckaybut, it's working well, I have been using it a lot14:22
openstackgerritSergey Lukjanov proposed openstack/sahara-specs: Defere "Allow placeholders in datasource URLs" to Liberty We're already meet Feature Freeze  https://review.openstack.org/16584114:22
tmckayjob type endpoint is good, but config hints are still not good.  That might have to wait to Liberty to fix14:23
tmckayvgridnev_, okay, only those 3 poll CRs for you for Kilo, right?14:26
tmckayI'll review now and then finish my own stuff before I review more :)14:26
vgridnev_yeah14:27
*** Nikolay_St has joined #openstack-sahara14:27
openstackgerritDenis Egorenko proposed stackforge/sahara-ci-config: Fix minor issues  https://review.openstack.org/16582814:28
openstackgerritSergey Lukjanov proposed openstack/sahara-specs: Defere "Migrate to HEAT HOT language" to Liberty We're already meet Feature Freeze  https://review.openstack.org/16585314:35
openstackgerritSergey Lukjanov proposed openstack/sahara-specs: Defere "Improvement of Sahara logs" to Liberty We're already meet Feature Freeze  https://review.openstack.org/16585414:35
openstackgerritVitaly Gridnev proposed openstack/sahara: Add usages of plugin poll - part 1  https://review.openstack.org/16076914:36
*** evgeny has joined #openstack-sahara14:42
vgridnev_elmiko, can you explain what secret flag means in your patch?14:44
elmikovgridnev_: the secret option prevents that config value from appearing in logs14:44
*** evgeny has left #openstack-sahara14:44
vgridnev_ok, thanks14:45
*** evgeny has joined #openstack-sahara14:46
*** evgeny has left #openstack-sahara14:46
elmikovgridnev_: http://docs.openstack.org/developer/oslo.config/cfg.html?highlight=secret if you want a little more detail, there's not much in the docs though14:46
*** esikachev has joined #openstack-sahara14:47
*** Nikolay_St has quit IRC14:47
*** esikachev has quit IRC14:47
*** Nikolay_St has joined #openstack-sahara14:48
*** evgeny has joined #openstack-sahara14:50
*** evgeny has left #openstack-sahara14:50
*** chandankumar_ has joined #openstack-sahara14:58
*** Nikolay_St has quit IRC15:02
*** Nikolay_St has joined #openstack-sahara15:03
*** Networkn3rd has joined #openstack-sahara15:06
*** Nikolay_St has quit IRC15:14
*** tmckay has quit IRC15:14
*** Nikolay_St has joined #openstack-sahara15:19
SergeyLukjanovfolks, here is a list of FFEs - https://wiki.openstack.org/wiki/Sahara/Kilo/FF_Exceptions15:25
*** chandankumar_ has quit IRC15:29
openstackgerritMerged openstack/sahara-specs: Defere "Allow placeholders in datasource URLs" to Liberty We're already meet Feature Freeze  https://review.openstack.org/16584115:38
*** Nikolay_St has quit IRC15:39
openstackgerritJacob Liberman proposed openstack/sahara: Replace direct http requests by sahara client in Quick start guide  https://review.openstack.org/16519215:40
*** tmckay has joined #openstack-sahara15:43
openstackgerritMerged stackforge/sahara-ci-config: Fix minor issues  https://review.openstack.org/16582815:44
*** Nikolay_St has joined #openstack-sahara15:45
*** chandankumar_ has joined #openstack-sahara15:45
*** tmckay has quit IRC15:50
*** chandankumar_ has quit IRC15:52
*** tmckay has joined #openstack-sahara15:52
vgridnev_elmiko, ping15:54
elmikovgridnev_: pong15:54
vgridnev_i know reasons, why removing option from plugins/base cause problems15:55
vgridnev_maybe15:55
vgridnev_it's required to add new entry for that in sahara/config.py15:55
elmikomy thought is that it's no longer included in the namespace15:55
elmikoyea15:55
elmikohere's my issue,15:56
elmikoif i add it to a cdh/config_helper.py then i need to add that namespace to the sahara/config.py15:56
elmikoso it's still being added specifically even though it's in a plugin15:56
vgridnev_yes15:56
elmikoso i feel like either way we have this plugin dependent option being added. it seemed more benign to add in the base with the plugin option15:57
elmikoeven though having it in the base makes things a little less modular15:57
elmikobut i figure, if you had to adjust the cdh plugin, you might need to change the plugin option in the base anyways15:58
elmikodoes that make sense?15:59
vgridnev_maybe, i think16:00
elmikoif you think it would just be cleaner to create a config_helper in the cdh plugin i can do that, i'll just need to add that namespace specifically (like you mentioned)16:01
elmikowhat are your thoughts?16:02
openstackgerritMerged openstack/sahara-specs: Defere "Migrate to HEAT HOT language" to Liberty We're already meet Feature Freeze  https://review.openstack.org/16585316:04
*** Nikolay_St has quit IRC16:06
*** tnovacik has joined #openstack-sahara16:07
*** Nikolay_St has joined #openstack-sahara16:13
vgridnev_exactly, i think it would be better16:16
elmikovgridnev_: better to move it?16:18
openstackgerritMerged openstack/sahara-specs: Defere "Improvement of Sahara logs" to Liberty We're already meet Feature Freeze  https://review.openstack.org/16585416:18
vgridnev_yes, better to move it to separate file, i think16:19
elmikook, will do16:19
*** crobertsrh has quit IRC16:23
tmckayso what command do we have that actually uses the MANIFEST.in file so that I can know whether or not it's right?16:32
elmikoi think if you run a clean install using setup.py it will pull files from the manifest16:33
elmikomake sure there is no sahara.egg.* dir in your directory before you run it though16:33
tmckayelmiko, python setup.py install?16:33
tmckaybecause build, sdist, bdist don't seem to use it16:33
tmckayhmm, maybe it's the egg16:33
*** Nikolay_St has quit IRC16:33
elmikoright16:34
elmikomake a new virtualenv, run `python setup.py install`, then check the install in the venv16:34
elmikovgridnev_: only downside is that i need to import the plugin at the file scope instead of in list_opts, but i don't think it will negatively affect things16:36
*** crobertsrh has joined #openstack-sahara16:36
*** chandankumar_ has joined #openstack-sahara16:37
openstackgerritMichael McCune proposed openstack/sahara: Moving cdh cloudera manager password to config file  https://review.openstack.org/16507716:38
*** Nikolay_St has joined #openstack-sahara16:39
openstackgerritGael Chamoulaud proposed stackforge/puppet-sahara: add back availability to manage logging and syslog confs  https://review.openstack.org/15627816:41
openstackgerritDenis Egorenko proposed stackforge/sahara-ci-config: Fix exit code for scripts  https://review.openstack.org/16591216:42
*** crobertsrh has quit IRC16:43
openstackgerritGael Chamoulaud proposed stackforge/puppet-sahara: spec: updates for rspec-puppet 2.x and rspec 3.x  https://review.openstack.org/16506416:43
openstackgerritGael Chamoulaud proposed stackforge/puppet-sahara: Add Puppet 4.x lint checks  https://review.openstack.org/16500916:43
*** crobertsrh has joined #openstack-sahara16:44
*** hdd has joined #openstack-sahara16:49
*** skolekonov has quit IRC16:56
*** Nikolay_St has quit IRC16:59
elmikotmckay: when your cdh clusters failed because of too small flavors, did they end up in error state?17:02
*** Nikolay_St has joined #openstack-sahara17:05
*** Networkn3rd has quit IRC17:05
*** pcaruana has quit IRC17:05
*** Poornima has joined #openstack-sahara17:09
openstackgerritPino Toscano proposed openstack/sahara-image-elements: Bump diskimage-builder version to 0.1.41  https://review.openstack.org/16593017:15
*** Networkn3rd has joined #openstack-sahara17:15
tmckayelmiko, yeah17:19
elmikotmckay: was it a timeout waiting for cloudera manager?17:20
tmckayelmiko, you know why this MANIFEST thing is confusing?  Because if you don't set the environment variable SKIP_GIT_SDIST=1  pbr uses the git to find the file set17:20
elmikointeresting, i hadn't run it that17:21
tmckaydefault is git generation.  so my files are there by default, MANIFEST or no17:21
elmikobut i had lots of issues with manifest when packaging sahara17:21
tmckaywhich is why we commonly miss files in the MANIFEST17:21
elmikoright17:21
elmikoit happened all the time when running the rpmbuild17:21
tmckayso the only way to really check is set that env and run sdist17:21
elmikogood find17:22
elmikothat must have been added more recently, because i was testing as last year and didn't run into that17:22
tmckayelmiko, yeah, weird.  Makes me wonder how the different distros are building, and how they get the file set to begin with17:24
tmckayif an original tarball is generated using pbr from a git, then shouldn't all the files be there anyway?17:24
elmikotmckay: how did you run the setup.py?17:24
*** Nikolay_St has quit IRC17:24
elmikowe excise pbr on fedora17:24
tmckayelmiko, remember when the launch_command.py went missing from the spark stuff?  I don't know how that happened17:25
elmikoand rhel/rhos17:25
tmckaymaybe that's it, then17:25
elmikoat least for rh stuff17:25
elmikobut we basically run the setup.py to do the install for the package17:25
tmckayokay, well in the absence of pbr it will do the right thing17:26
elmikotmckay: so, on the cdh cluster, you did flavor 4 on the namenode, then 2 on all the others?17:26
elmikohaha, yea17:26
elmikoor flavor 4 on the secondary namenode as well?17:26
tmckayand pbr is supposed to be python build reasonableness ... hmmm17:26
tmckayelmiko, just the namenode17:26
elmikohmm17:26
elmikoi think my compute node is just too slow for this17:27
tmckaycould be17:27
elmikoit takes about 30min just to get the instances up17:27
tmckaywhoa17:27
elmikoyea17:27
elmikoi need more ram!!!!17:27
tmckaymy test bed was much faster, do you want to use it?17:27
elmikowhen i get to serious testing i will, for now i'm just playing around to see if i can get this working17:27
tmckayok17:28
openstackgerritMerged stackforge/puppet-sahara: policy.json: Allow one to manage them from the puppet module  https://review.openstack.org/15247017:28
*** Nikolay_1t has joined #openstack-sahara17:30
*** pnayak has quit IRC17:31
*** coolsvap is now known as coolsvap|afk17:33
openstackgerritMichael McCune proposed openstack/sahara: Moving cdh cloudera manager password to config file  https://review.openstack.org/16507717:38
*** weiting has joined #openstack-sahara17:45
*** chandankumar_ has quit IRC17:48
openstackgerritMerged stackforge/puppet-sahara: add back availability to manage logging and syslog confs  https://review.openstack.org/15627817:53
*** Nikolay_1t has quit IRC17:53
*** Nikolay_St has joined #openstack-sahara17:54
*** crobertsrh has left #openstack-sahara17:55
openstackgerritVitaly Gridnev proposed openstack/sahara: Sync with latest oslo-incubator  https://review.openstack.org/16502117:55
elmikomeeting in a few minutes17:57
vgridnev_at openstack-meeting-alt?17:57
elmikoyes17:57
*** sreshetn1 has joined #openstack-sahara17:58
SergeyLukjanovmeeting time :)17:58
openstackgerritMerged stackforge/sahara-ci-config: Fix exit code for scripts  https://review.openstack.org/16591218:00
openstackgerritMerged stackforge/puppet-sahara: Add Puppet 4.x lint checks  https://review.openstack.org/16500918:10
openstackgerritMerged stackforge/puppet-sahara: spec: updates for rspec-puppet 2.x and rspec 3.x  https://review.openstack.org/16506418:11
*** coolsvap|afk has quit IRC18:11
openstackgerritDenis Egorenko proposed stackforge/sahara-ci-config: Remove source of nonexistent file in pep8 check  https://review.openstack.org/16595218:16
*** Nikolay_St has quit IRC18:18
*** Nikolay_St has joined #openstack-sahara18:23
openstackgerritDenis Egorenko proposed stackforge/sahara-ci-config: Fix return code for dib-scenario jobs  https://review.openstack.org/16595718:30
*** Nikolay_St has quit IRC18:39
*** hdd has quit IRC18:42
openstackgerritPino Toscano proposed openstack/sahara-image-elements: Bump diskimage-builder version to 0.1.41  https://review.openstack.org/16593018:43
openstackgerritSergey Kolekonov proposed stackforge/sahara-ci-config: Fix return code for dib-scenario jobs  https://review.openstack.org/16595718:45
openstackgerritSergey Lukjanov proposed openstack/sahara-specs: Added dir for image elements specs  https://review.openstack.org/16596518:47
*** Nikolay_St has joined #openstack-sahara18:50
*** sreshetn1 has quit IRC18:51
*** sreshetn1 has joined #openstack-sahara18:59
*** IBerezovskiy has quit IRC19:01
*** tosky has quit IRC19:06
*** Nikolay_St has quit IRC19:09
*** Nikolay_St has joined #openstack-sahara19:14
openstackgerritMerged stackforge/sahara-ci-config: Remove source of nonexistent file in pep8 check  https://review.openstack.org/16595219:23
*** hdd has joined #openstack-sahara19:26
openstackgerritTrevor McKay proposed openstack/sahara: Add a CLI tool for managing default templates  https://review.openstack.org/16364919:27
*** Poornima has quit IRC19:29
*** Nikolay_St has quit IRC19:36
*** Nikolay_St has joined #openstack-sahara19:42
*** weiting has quit IRC19:43
openstackgerritMerged stackforge/sahara-ci-config: Fix return code for dib-scenario jobs  https://review.openstack.org/16595719:44
*** Poornima has joined #openstack-sahara19:44
elmikotmckay: ping20:01
*** Nikolay_St has quit IRC20:01
*** Poornima has quit IRC20:01
*** sreshetn1 has quit IRC20:05
tmckayelmiko, hey20:07
elmikotmckay: hey, i want to help with the config hints stuff if you think it's doable before final freeze20:08
tmckayelmiko, btw, someone else asked you about "secret" for an oslo config, what does that do? (so I don't have to read the sources :) )20:08
elmikotmckay: secret=True means that the config option will not appear in any logs20:08
elmikonice option, i wasn't aware of it either20:09
tmckayelmiko, that's what I thought, wasn't sure.  config-hints, I think it's doable.  We can at least improve it20:09
tmckayelmiko, here is the skinny on config-hints, in no particular order:20:09
elmikotmckay: cool, i don't remember, is there a spec up or something i can review?20:09
elmikoah, cool20:09
tmckaynot really20:09
elmikogo ahead, sorry to interrupt20:10
tmckayactually, the Sahara EDP user pages talk about them20:10
* elmiko looks20:10
tmckayyou can look there for some general info20:10
elmikook, i'll do some reading. might have more questions tomorrow ;)20:11
tmckayfirst thing I noticed is that for some of the job types we send back empty lists, and the type of "configs" is not right there.  It should be a dict.  job_configs is a dict which contains 3 things -- configs (key value pairs), params (key value pairs), and args (a list)20:11
tmckayso at the very list, all the blank config values should go from [] to {} :)20:12
*** Nikolay_St has joined #openstack-sahara20:12
tmckayelmiko, the other easy fruit is adding the various "edp.xxx" configs that Sahara intercepts for spark/java jobs20:12
tmckayand, the edp.substitute_xxxx configs for data source references20:13
tmckay(applies to all jobs I think)20:13
elmikoand is there a bug or something to mark CRs against?20:14
tmckayI would put them against the job-types-endpoint bp20:14
tmckay(I think that's the name)20:14
elmikook, cool20:14
tmckayelmiko, the harder part is that for cases where we do have actual configs currently, they all come from a single routine in the Oozie EDP stuff.  Config hints ideally are specific to a particular job type for a particular version of a particular plugin20:15
tmckaythat doesn't mean that they will vary across all of those, but they could20:16
tmckayFor instance, it's likely that something is different between hadoop1 and hadoop2 versions of the the various plugins20:16
*** sreshetn1 has joined #openstack-sahara20:16
elmikook20:17
tmckayelmiko, so I would probably take something well-defined, like vanilla1 and vanilla2, and see how the configs vary there, and break those out of the current Oozie engine method20:17
elmikoand just to be clear, you are cleaning up the job-types endpoints20:17
tmckaymake sense? Kind of a voyage of discovery20:17
elmikoit makes sense, i'll need to dig a little because i haven't used these features before20:17
tmckayelmiko, already done.  It's reporting hints now, driven from the plugin EDP engine, but they are all calling back into the current stuff20:18
elmikobut, i think the key manager stuff will be on hold for at least a few weeks. so i think it's probably better if i help get this stuff in order.20:18
elmikotmckay: ack20:18
tmckayelmiko, cool.  If you look at my CR list, you'll see a bunch of "Implement xxxx" changes related to the endpoints for the plugins20:18
tmckayIn those, you can see the new get_edp_config_hints(job_type) routines.20:19
tmckaythat's where they potentially break away20:19
elmikook, cool20:19
tmckayelmiko, I don't know if we need a separate spec for this or not20:20
elmikoi guess we'll see20:20
elmikokinda depends on how big the change becomes20:20
tmckaycurrent spec for job type endpoint mentions config-hints, but not in detail20:20
elmikoif it's just cleaning up and making them clearer then probably not20:20
tmckayelimiko, cool thing is it is really easy to test with REST using the new endpoint20:21
elmikoyea20:21
tmckaynothing to pass but flags20:21
tmckayelmiko, currently it's "one size fits all" which probably means "one size doesn't fit most"20:21
elmikohehe20:21
elmikotrue that20:21
elmikotmckay: thanks for the info, i've got some reading and experimenting to do20:22
tmckayelmiko, launching a cluster and looking in the various .xml files on a cluster node may be a helpful place to look, also all of the various .xml in Sahara resource directories20:22
tmckayelmiko, ack, thanks20:23
elmikoack20:23
elmikoi can test some of the clusters here, but i'll probably need grid1 for cdh at a minimum20:23
*** witlessb has quit IRC20:26
*** sreshetn1 has quit IRC20:27
*** Nikolay_St has quit IRC20:28
*** witlessb has joined #openstack-sahara20:36
*** Nikolay_St has joined #openstack-sahara20:39
*** Nikolay_St has quit IRC20:46
*** Nikolay_St has joined #openstack-sahara20:52
*** crobertsrh has joined #openstack-sahara20:53
*** sreshetn1 has joined #openstack-sahara20:54
*** crobertsrh has left #openstack-sahara21:02
*** Networkn3rd has quit IRC21:13
*** sreshetn1 has quit IRC21:19
elmikotmckay: ping21:30
*** vgridnev_ has quit IRC21:34
*** crobertsrh has joined #openstack-sahara21:35
tmckayelmiko, hi21:38
elmikotmckay: hey, about to grab dinner. was just curious if about the job-types endpoint21:40
elmikowe can talk tomorrow though21:40
tmckayok21:40
*** crobertsrh has quit IRC21:55
*** hdd has quit IRC21:59
elmikotmckay: still here?22:03
tmckayyeah, refactoring :)22:04
elmikocool22:04
tmckaymaybe :)22:04
elmikoso i just ran from master and hit the job-types endpoint22:04
elmikolol22:04
tmckaydid it work?22:04
elmikoi just got back an empty list22:05
elmikobasically { "job-types": [] }22:05
tmckayhmmm.  do you have plugins loaded?22:05
elmikoyea, vanilla, spark, and cd22:05
elmikocdh22:05
tmckayoh, wait22:05
elmikois there a patch i need?22:05
tmckayhdp was merged as part of the original, but I didn't add the others til everything was approved22:06
elmikoah, ok22:06
tmckayso the others are all on review22:06
elmikogot it, i'll start with reviews then ;)22:06
tmckayyou can patch them all in, shouldn't be any conflicts22:06
elmikok22:06
elmikothanks22:06
tmckayelmiko, one thing to take a look at --22:06
tmckayI included a basic plugin object under each job type, that might be a little repetitive22:07
tmckayI wonder if there is a better way to represent that.22:07
tmckayThe idea was to have a list of plugins and versions for each job type for the current Sahara22:07
elmikoyea22:08
elmikoi guess the other way would be to list all the plugins with job-types under them?22:08
tmckayif you think of a good way to shorten it that is still easily handled by the client, let me know22:08
elmikoack22:08
elmikoi need to understand a little better, then start hacking22:08
elmikoit looks pretty straight forward so far though22:09
elmikoand the spec is nice and thorough ;)22:09
*** jamielennox|away is now known as jamielennox22:09
*** sreshetn1 has joined #openstack-sahara22:15
elmikotmckay: one question about https://review.openstack.org/#/c/165231/122:17
elmikois that validation stuff part of the job-types endpoint?22:17
tmckaylooking22:19
elmikoin edp_engine.py22:19
*** sreshetn1 has quit IRC22:19
tmckayelmiko, special case for spark22:20
elmikook, cool22:20
tmckayelmiko, for all the other plugins, we introduced EDP for all versions22:20
elmikogotcha22:20
tmckayelmiko, for spark we only introduced EDP in 1.0.0 and skipped 0.9.122:20
elmikoi thought there was a reason, just wanted to make sure22:21
tmckayI fixed up the static checks that were there a little bit22:21
tmckayelmiko, oh, yeah, I added that edp_supported() method, and then refactored the other stuff22:22
elmikoright22:22
tmckayelmiko, so that validate_job_execution has a different purpose, not part of the endpoint stuff22:22
elmikoi thought it might be nice to mention in commit msg, but not -1 worthy22:22
tmckayI figured it was self-evident :)  Apparently not22:22
elmikoit just seemed out of place with the other job-type stuff22:23
elmikoor i'm just bad at reading ;)22:23
tmckayyes, definitely an outlier22:23
tmckayI would have done EDP for spark 0.9.1, but spark-submit didn't exist22:23
elmikowas it just the spark command at that point? or did you need an app?22:23
tmckayyou had to use some other spark class submission thing22:23
elmikoah, ok22:24
tmckayso it would have been a separate EDP implementation22:24
tmckaytime for dinner22:24
elmikoenjoy!22:24
*** egafford has quit IRC22:40
*** witlessb has quit IRC22:43
*** crobertsrh has joined #openstack-sahara22:49
*** crobertsrh has left #openstack-sahara22:50
*** sreshetn1 has joined #openstack-sahara23:15
*** sreshetn1 has quit IRC23:20
*** chlong has joined #openstack-sahara23:27
« Wednesday, 2015-03-18 Index Friday, 2015-03-20 »

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!