*** IlyaE has quit IRC | 00:01 | |
openstackgerrit | Trevor McKay proposed a change to openstack/sahara: Refactor the job manager to allow multiple execution engines https://review.openstack.org/100622 | 00:06 |
---|---|---|
*** IlyaE has joined #openstack-sahara | 00:17 | |
*** _RobLevas has joined #openstack-sahara | 00:23 | |
*** _RobLevas has quit IRC | 00:32 | |
*** Networkn3rd has joined #openstack-sahara | 01:17 | |
*** Networkn3rd has quit IRC | 01:22 | |
*** _RobLevas has joined #openstack-sahara | 01:30 | |
*** _RobLevas has quit IRC | 01:34 | |
*** zhiyan_ is now known as zhiyan | 01:47 | |
*** IlyaE has quit IRC | 01:50 | |
*** zhiyan is now known as zhiyan_ | 01:56 | |
*** zhiyan_ is now known as zhiyan | 01:56 | |
*** bradd1 has quit IRC | 01:58 | |
*** bradd1 has joined #openstack-sahara | 02:30 | |
*** IlyaE has joined #openstack-sahara | 02:37 | |
*** IlyaE has quit IRC | 02:51 | |
*** IlyaE has joined #openstack-sahara | 03:06 | |
*** ViswaV has joined #openstack-sahara | 03:07 | |
*** ViswaV__ has joined #openstack-sahara | 03:07 | |
*** ViswaV_ has quit IRC | 03:09 | |
*** ViswaV has quit IRC | 03:11 | |
*** IlyaE has quit IRC | 03:12 | |
*** _RobLevas has joined #openstack-sahara | 03:32 | |
*** _RobLevas has quit IRC | 03:36 | |
*** IlyaE has joined #openstack-sahara | 03:40 | |
*** ViswaV__ has quit IRC | 04:13 | |
*** ViswaV has joined #openstack-sahara | 04:14 | |
*** IlyaE has quit IRC | 04:21 | |
*** _RobLevas has joined #openstack-sahara | 04:34 | |
*** _RobLevas has quit IRC | 04:39 | |
*** akuznetsov has quit IRC | 04:46 | |
*** akuznetsov has joined #openstack-sahara | 04:53 | |
*** bradd1 has quit IRC | 05:01 | |
*** IlyaE has joined #openstack-sahara | 05:03 | |
*** akuznetsov has quit IRC | 05:12 | |
*** IlyaE has quit IRC | 05:25 | |
*** akuznetsov has joined #openstack-sahara | 05:26 | |
*** IlyaE has joined #openstack-sahara | 05:59 | |
openstackgerrit | OpenStack Proposal Bot proposed a change to openstack/sahara: Imported Translations from Transifex https://review.openstack.org/105358 | 06:01 |
*** xianghui has joined #openstack-sahara | 06:07 | |
*** Longgeek has joined #openstack-sahara | 06:36 | |
*** IlyaE has quit IRC | 06:56 | |
*** xianghui has quit IRC | 07:04 | |
*** IlyaE has joined #openstack-sahara | 07:07 | |
*** skolekonov has joined #openstack-sahara | 07:08 | |
*** IlyaE has quit IRC | 07:15 | |
openstackgerrit | A change was merged to stackforge/sahara-ci-config: Resolve problem with java downloading for spark images https://review.openstack.org/105388 | 07:50 |
*** jpich has joined #openstack-sahara | 07:58 | |
*** xianghui has joined #openstack-sahara | 08:06 | |
*** julienvey has joined #openstack-sahara | 08:15 | |
openstackgerrit | A change was merged to openstack/python-saharaclient: Remove docutils pin https://review.openstack.org/105627 | 08:18 |
*** yiconglu has joined #openstack-sahara | 08:26 | |
*** witlessb has joined #openstack-sahara | 08:53 | |
*** tosky has joined #openstack-sahara | 08:59 | |
openstackgerrit | A change was merged to openstack/sahara: Imported Translations from Transifex https://review.openstack.org/105358 | 09:03 |
*** Timotey has joined #openstack-sahara | 09:14 | |
*** yiconglu has quit IRC | 09:19 | |
*** _RobLevas has joined #openstack-sahara | 09:19 | |
*** Longgeek has quit IRC | 09:27 | |
openstackgerrit | A change was merged to openstack/sahara: Update docs to reflect the changes in security group section in horizon https://review.openstack.org/104687 | 09:38 |
openstackgerrit | A change was merged to openstack/sahara: Updated from global requirements https://review.openstack.org/105174 | 09:42 |
*** vrovachev has joined #openstack-sahara | 09:46 | |
*** Longgeek has joined #openstack-sahara | 09:58 | |
*** Longgeek has quit IRC | 10:05 | |
*** xianghui has quit IRC | 10:49 | |
*** xianghui has joined #openstack-sahara | 11:03 | |
*** _RobLevas has quit IRC | 11:32 | |
*** Longgeek has joined #openstack-sahara | 11:38 | |
*** _RobLevas has joined #openstack-sahara | 11:42 | |
openstackgerrit | Matthew Farrellee proposed a change to openstack/sahara: Update image registry docs to use cli https://review.openstack.org/105715 | 12:02 |
*** _crobertsrh is now known as crobertsrh | 12:12 | |
openstackgerrit | Sergey Reshetnyak proposed a change to openstack/sahara: Add vanilla plugin with Hadoop 2.4.0 https://review.openstack.org/105099 | 12:16 |
openstackgerrit | Sergey Reshetnyak proposed a change to openstack/sahara: Add vanilla plugin with Hadoop 2.4.0 https://review.openstack.org/105099 | 12:20 |
openstackgerrit | Sergey Lukjanov proposed a change to openstack/sahara: Add oslo.i18n lib to requirements https://review.openstack.org/105723 | 12:31 |
openstackgerrit | Sergey Lukjanov proposed a change to openstack/sahara: Use oslo.i18n https://review.openstack.org/105724 | 12:31 |
openstackgerrit | Sergey Lukjanov proposed a change to openstack/sahara: Remove oslo-incubator's gettextutils https://review.openstack.org/105725 | 12:31 |
*** akuznetsov has quit IRC | 12:34 | |
openstackgerrit | Sergey Lukjanov proposed a change to openstack/sahara: Use oslo.i18n https://review.openstack.org/105724 | 12:36 |
*** sballe has joined #openstack-sahara | 13:01 | |
tellesnobrega | hey, storm uses only one config file and it is a YAML file. I need to parse it so it can be loaded, is it ok to use pyyaml lib to parse it? this means including in the dependencies for sahara | 13:04 |
tellesnobrega | actually i'm a little confused how this is going to work. This files contains storm running configurations, like master ip, zookeeper ip and so on. This can be written without the user, by using the information of node groups we can figure out this ips and write the configuration file. Other configurations could be managed by the user. For a first prototype, should i have it all or just the basics to have it running, and leave t | 13:09 |
elmiko | tellesnobrega: do you have a spec or blueprint talking about the storm change? | 13:20 |
tellesnobrega | not yet | 13:20 |
elmiko | i would imagine you could probably get pyyaml brought in, but first we'll probably have to discuss the spec and understand the impact | 13:21 |
tellesnobrega | sure | 13:21 |
tellesnobrega | i will work on this spec now | 13:21 |
elmiko | as for the config stuff, it does sound like you will need to create the yaml config dynamically | 13:22 |
tellesnobrega | it needs to be because of the zookeeper and master node information | 13:22 |
elmiko | you might follow the design pattern used for the workflow.xml files and create a yaml template that can be filled in at runtime. then you wouldn't necessarily need pyyaml. | 13:22 |
elmiko | just throwin ideas out, i don't know much about storm | 13:23 |
tellesnobrega | elmiko: sounds ok. it may be easier to use pyyaml this way so i could have a dictionary and it translates to yaml (i'm not sure that it does, but i think so) | 13:24 |
*** Longgeek has quit IRC | 13:25 | |
tellesnobrega | this file needs to be sent to every node in the topology. I'm not sure what configurations sahara needs | 13:25 |
tellesnobrega | pyyaml writes the file alright | 13:26 |
tmckay | SergeyLukjanov, aignatov, hi, I've got gate-sahara-pep8 failures on jenkins for https://review.openstack.org/#/c/100622/, but in my local branch tox -e pep8 passes (even after wiping out tox and rebuilding). Is jenkins using a new pep8 definition that we don't have in the sahara sources? It would be much simpler if they matched. | 13:26 |
elmiko | tellesnobrega: ack | 13:26 |
*** Longgeek has joined #openstack-sahara | 13:35 | |
*** venza_ has joined #openstack-sahara | 13:43 | |
tellesnobrega | elmiko: created the blueprint, im gonna start writing the spec | 13:46 |
elmiko | sounds good, then we can at least get some ideas on the table about the changes needed | 13:46 |
*** venza has quit IRC | 13:48 | |
*** anteaya has quit IRC | 13:48 | |
*** anteaya has joined #openstack-sahara | 13:55 | |
tmckay | SergeyLukjanov, elmiko, croberts, okay, apparently this is related to changes from aignatov in https://review.openstack.org/#/c/100090/2. the sahara/tox.ini has "ignores" for some errors, but the gate tests must not have those ignores -- looks like maybe gate tests start with icehouse as a base? Not too familiar with those. However, if I edit tox.ini and do not ignore H305 and H307 I can get the same errors for https://review. | 14:00 |
elmiko | tmckay: i thought those were being pulled out with the pep8 fix change? | 14:01 |
aignatov | tmckay: probably you’ll need to rebase on the latest commit in sahara master | 14:02 |
tmckay | well, apparently, the gate tests are lagging. So if you want a clean bill of health from jenkins, you might have to unignore something | 14:02 |
aignatov | You’ll see all these errors | 14:02 |
tmckay | aignatov, I just rebased yesterday :) I'll try again | 14:02 |
dmitryme | tmckay: I think gate rebases your change on master before testing it | 14:02 |
SergeyLukjanov | tmckay, ack - gate always rebase your changes to master before running jobs | 14:03 |
tmckay | okay, I'll get an up to date master and try again | 14:03 |
SergeyLukjanov | tellesnobrega, ack - let's discuss on spec | 14:03 |
SergeyLukjanov | but IMO pyyaml will be ok | 14:04 |
openstackgerrit | Trevor McKay proposed a change to openstack/sahara: Refactor the job manager to allow multiple execution engines https://review.openstack.org/100622 | 14:04 |
tellesnobrega | SergeyLukjanov: sure, i'm writing it now. | 14:04 |
aignatov | tmckay: Also https://review.openstack.org/#/c/101845/ could affects you :) | 14:04 |
SergeyLukjanov | tellesnobrega, cool, thx! | 14:04 |
dmitryme | tmckay: I agree with aignatov: most probably the CR he pointed out affected you | 14:05 |
tmckay | aignatov, dmitryme, SergeyLukjanov, thanks! I guessed I missed the rebase by a few hours, lol. | 14:06 |
tmckay | I have the same errors now | 14:07 |
tmckay | I was confused :) | 14:07 |
* tmckay writes down "When in doubt, always rebase first" | 14:07 | |
openstackgerrit | Vitaly Gridnev proposed a change to openstack/sahara: Fixed configuring instances for Vanilla 2.0 https://review.openstack.org/105525 | 14:07 |
*** ViswaV_ has joined #openstack-sahara | 14:13 | |
*** ViswaV__ has joined #openstack-sahara | 14:14 | |
*** ViswaV has quit IRC | 14:16 | |
*** ViswaV_ has quit IRC | 14:18 | |
openstackgerrit | Trevor McKay proposed a change to openstack/sahara: Refactor the job manager to allow multiple execution engines https://review.openstack.org/100622 | 14:18 |
*** Networkn3rd has joined #openstack-sahara | 14:27 | |
openstackgerrit | A change was merged to openstack/python-saharaclient: Jar files for pig and mapreduce EDP tests https://review.openstack.org/105603 | 14:42 |
tellesnobrega | SergeyLukjanov: i'm a little lost here what is the impact in the api. Haven't thought about it yet. Do you know of api impact when adding new plugins? | 14:54 |
openstackgerrit | Vitaly Gridnev proposed a change to openstack/sahara: Fix scaling cluster Vanilla for Hadoop 2.3 https://review.openstack.org/105756 | 14:54 |
*** _RobLevas is now known as RobLevas | 14:57 | |
*** IlyaE has joined #openstack-sahara | 15:02 | |
SergeyLukjanov | tellesnobrega, no impact | 15:04 |
tellesnobrega | great :) | 15:04 |
tellesnobrega | thats what i thougt, but wasnt sure | 15:04 |
SergeyLukjanov | tellesnobrega, by API impact we mean new endpoints addition or updating existing ones | 15:04 |
*** Networkn3rd has quit IRC | 15:09 | |
*** zhiyan is now known as zhiyan_ | 15:10 | |
*** Hao has joined #openstack-sahara | 15:11 | |
*** sballe has quit IRC | 15:21 | |
*** Longgeek has quit IRC | 15:36 | |
*** Longgeek has joined #openstack-sahara | 15:37 | |
*** Longgeek has quit IRC | 15:40 | |
*** Longgeek has joined #openstack-sahara | 15:40 | |
*** RobLevas has quit IRC | 16:10 | |
openstackgerrit | Sofiia Kostiuchenko proposed a change to openstack/sahara: Fixed EDP job execution failure https://review.openstack.org/105778 | 16:11 |
*** vrovachev has quit IRC | 16:11 | |
*** ViswaV has joined #openstack-sahara | 16:12 | |
*** ViswaV__ has quit IRC | 16:13 | |
*** Hao has quit IRC | 16:14 | |
tmckay | SergeyLukjanov, aignatov, maybe dmitryme :), I am wondering about using paramiko Channel objects to run ssh commands on the remote host asynchronously. The trouble is, that depending on how a park job is launched with spark-submit from the command line, the driver program will run synchronously in the shell, or the driver program will be deployed into the cluster. Driver deployment does not work for all cases (python spark prog | 16:15 |
tmckay | I am thinking about launching spark synchronously in ssh but asynchronously to Sahara run_job, because of course it could take a really, really long time | 16:16 |
tmckay | Any problem with paramiko Channels that you know of? Everything else we do is synchronous | 16:16 |
tmckay | My other thought is to use synchronous ssh inside a python subprocess in Sahara, and then track them for job status | 16:17 |
tmckay | launch a subprocess per spark job run | 16:17 |
* tmckay has to look more closely at subprocesses/threads in Sahara | 16:18 | |
tmckay | crobertsrh, elmiko, too, ^^ in case you have experience here | 16:18 |
* tmckay needs some food | 16:19 | |
openstackgerrit | Telles Mota Vidal Nóbrega proposed a change to openstack/sahara-specs: Storm Integration https://review.openstack.org/105784 | 16:20 |
dmitryme | tmckay: keeping ssh connection for a long time does not sound good | 16:22 |
dmitryme | can we use ssh to run command in the background | 16:22 |
dmitryme | we’ll have to run another ssh commands later to track job status | 16:24 |
*** jpich has quit IRC | 16:25 | |
*** miqui has quit IRC | 16:27 | |
*** miqui has joined #openstack-sahara | 16:28 | |
crobertsrh | sorry, just got back from lunch....looking back | 16:32 |
*** shakamunyi has joined #openstack-sahara | 16:33 | |
*** shakamunyi has quit IRC | 16:33 | |
*** RobLevas has joined #openstack-sahara | 16:33 | |
tmckay | dmitryme, yes, ssh background is one way, I have seen examples using "screen" and "nohup" but from the comment it seems difficult to always get it right :) That's why I wondered about Paramiko channels. | 16:33 |
tmckay | They let you do things asynchronously already. | 16:33 |
*** shakamunyi has joined #openstack-sahara | 16:34 | |
*** bandarji has joined #openstack-sahara | 16:34 | |
dmitryme | tmckay: they do. Our calls are synchronous because we explicitly block - https://github.com/openstack/sahara/blob/master/sahara/utils/ssh_remote.py#L129 | 16:36 |
dmitryme | but again, I don’t like idea of long waiting ssh | 16:37 |
tmckay | okay, I see. So, it would be a possibility with some new flag to execute_command, but I'll see if I can get ssh to return and leave the command running | 16:37 |
tmckay | dmitryme, thanks | 16:38 |
dmitryme | on the other side, I think people are dramatizing when they are speaking about running in background | 16:38 |
dmitryme | at least I did it once with nohup and it worked | 16:38 |
openstackgerrit | Telles Mota Vidal Nóbrega proposed a change to openstack/sahara-specs: Storm Integration https://review.openstack.org/105784 | 16:39 |
tmckay | okay thanks :) many people have asked this question. That is encouraging. | 16:39 |
*** Networkn3rd has joined #openstack-sahara | 16:39 | |
*** Hao has joined #openstack-sahara | 16:42 | |
*** miqui_ has joined #openstack-sahara | 16:54 | |
*** miqui has quit IRC | 16:55 | |
elmiko | +1 about not wanting long running ssh | 17:05 |
*** Networkn3rd has quit IRC | 17:05 | |
*** julienvey has quit IRC | 17:15 | |
*** IlyaE has quit IRC | 17:39 | |
*** julienvey has joined #openstack-sahara | 17:46 | |
*** julienvey has quit IRC | 17:48 | |
*** julienve_ has joined #openstack-sahara | 17:48 | |
*** tosky has quit IRC | 17:49 | |
*** tosky__ has joined #openstack-sahara | 17:49 | |
*** ViswaV has quit IRC | 17:52 | |
*** julienve_ has quit IRC | 17:52 | |
*** ViswaV has joined #openstack-sahara | 17:52 | |
crobertsrh | I'm trying to organize a list of dashboard changes that need to be done for juno (with some thought as to whether or not there will be time). Please feel free to add any thoughts you might have. https://etherpad.openstack.org/p/sahara-juno-post-merge-changes I'm hoping to get started on these very soon (ideally the merge will be done very soon--even if it isn't, I'll be adding these changes as deps on top of it). | 17:53 |
*** tosky__ is now known as tosky | 17:54 | |
*** IlyaE has joined #openstack-sahara | 17:57 | |
elmiko | crobertsrh: well, you got my concern :) | 17:59 |
*** Hao has quit IRC | 17:59 | |
crobertsrh | I'm assuming that I have missed many things with head down in merge land. | 17:59 |
*** Longgeek has quit IRC | 18:01 | |
elmiko | it's nice that you're finally starting to see daylight there | 18:01 |
crobertsrh | For sufficiently low values of daylight | 18:06 |
elmiko | lol | 18:06 |
*** xianghui has quit IRC | 18:11 | |
tmckay | ssh -i ~/.ssh/id_rsa.pub ubuntu@10.3.8.240 '/opt/spark/bin/spark-submit /opt/spark/lib/spark-examples-1.0.0-hadoop2.0.0-mr1-cdh4.2.0.jar --class org.apache.spark.examples.SparkPi --master spark://spark-test-one-spark-master-1-001:7077 > /tmp/app 2>&1 & echo $!' | 18:27 |
tmckay | 2503 | 18:27 |
tmckay | dmitryme, ^^ :-D | 18:27 |
tmckay | that last number is the pid | 18:27 |
tmckay | and the spark log trace in this case is in /tmp/app. We could maybe save that under /mnt/spark/appid.log or something | 18:28 |
tmckay | and it returns immediately, of course | 18:28 |
tmckay | So, we can know when the pid is done, and we can save the log | 18:29 |
tmckay | the trick to the return is apparently just to redirect or close stderr and stdout | 18:29 |
*** IlyaE has quit IRC | 18:30 | |
*** IlyaE has joined #openstack-sahara | 18:31 | |
*** julienvey has joined #openstack-sahara | 18:49 | |
openstackgerrit | Trevor McKay proposed a change to openstack/sahara: Allow plugins to choose the EDP implementation https://review.openstack.org/101016 | 18:51 |
*** julienvey has quit IRC | 18:53 | |
dmitryme | tmckay: yep, looks good to me | 18:54 |
*** Hao has joined #openstack-sahara | 19:02 | |
*** Hao has quit IRC | 19:11 | |
*** IlyaE has quit IRC | 19:12 | |
*** ViswaV has quit IRC | 19:20 | |
*** shakamunyi has quit IRC | 19:20 | |
*** ViswaV has joined #openstack-sahara | 19:23 | |
*** Hao has joined #openstack-sahara | 19:23 | |
*** Hao has quit IRC | 19:30 | |
*** IlyaE has joined #openstack-sahara | 19:34 | |
*** miqui_ is now known as miqui | 19:37 | |
*** shakamunyi has joined #openstack-sahara | 19:49 | |
*** shakamunyi has quit IRC | 19:49 | |
*** julienvey has joined #openstack-sahara | 19:49 | |
*** julienvey has quit IRC | 19:54 | |
*** IlyaE has quit IRC | 20:03 | |
*** bandarji has quit IRC | 20:04 | |
*** bandarji has joined #openstack-sahara | 20:04 | |
*** bandarji has quit IRC | 20:05 | |
*** IlyaE has joined #openstack-sahara | 20:07 | |
*** shakamunyi has joined #openstack-sahara | 20:15 | |
*** shakamunyi has quit IRC | 20:15 | |
*** bandarji has joined #openstack-sahara | 20:16 | |
*** IlyaE has quit IRC | 20:20 | |
*** shakamunyi has joined #openstack-sahara | 20:24 | |
*** Hao has joined #openstack-sahara | 20:27 | |
*** bandarji has quit IRC | 20:28 | |
*** IlyaE has joined #openstack-sahara | 20:30 | |
*** Hao has quit IRC | 20:32 | |
*** julienvey has joined #openstack-sahara | 20:50 | |
*** julienvey has quit IRC | 20:55 | |
*** IlyaE has quit IRC | 20:58 | |
*** bandarji has joined #openstack-sahara | 20:59 | |
*** IlyaE has joined #openstack-sahara | 21:03 | |
*** shakamunyi has quit IRC | 21:08 | |
*** shakamunyi has joined #openstack-sahara | 21:09 | |
*** shakamunyi has quit IRC | 21:09 | |
*** shakamunyi has joined #openstack-sahara | 21:10 | |
*** IlyaE has quit IRC | 21:20 | |
*** Hao has joined #openstack-sahara | 21:29 | |
*** crobertsrh is now known as _crobertsrh | 21:29 | |
*** Hao has quit IRC | 21:34 | |
*** sballe has joined #openstack-sahara | 21:42 | |
*** julienvey has joined #openstack-sahara | 21:51 | |
*** IlyaE has joined #openstack-sahara | 21:52 | |
*** shakamunyi has quit IRC | 21:53 | |
*** shakamunyi has joined #openstack-sahara | 21:54 | |
*** shakamunyi has quit IRC | 21:54 | |
*** julienvey has quit IRC | 21:56 | |
*** ViswaV has quit IRC | 21:59 | |
*** IlyaE has quit IRC | 22:05 | |
*** IlyaE has joined #openstack-sahara | 22:12 | |
*** miqui_ has joined #openstack-sahara | 22:13 | |
*** miqui has quit IRC | 22:13 | |
*** IlyaE has quit IRC | 22:23 | |
*** bandarji has quit IRC | 22:23 | |
*** RobLevas has quit IRC | 22:25 | |
*** Hao has joined #openstack-sahara | 22:30 | |
*** RobLevas has joined #openstack-sahara | 22:31 | |
*** RobLevas is now known as _RobLevas | 22:31 | |
*** Hao has quit IRC | 22:36 | |
*** IlyaE has joined #openstack-sahara | 22:37 | |
*** boris-42 has quit IRC | 22:37 | |
*** boris-42 has joined #openstack-sahara | 22:39 | |
*** ViswaV has joined #openstack-sahara | 22:43 | |
*** elmiko is now known as _elmiko | 22:45 | |
*** julienvey has joined #openstack-sahara | 22:52 | |
*** julienvey has quit IRC | 22:56 | |
*** ViswaV has quit IRC | 23:04 | |
*** witlessb has quit IRC | 23:25 | |
*** _RobLevas has quit IRC | 23:32 | |
*** Hao has joined #openstack-sahara | 23:32 | |
*** Hao has quit IRC | 23:37 | |
*** julienvey has joined #openstack-sahara | 23:52 | |
*** IlyaE has quit IRC | 23:53 | |
*** julienvey has quit IRC | 23:57 | |
sballe | SergeyLukjanov, hi. I am trying to get sahara working using the vanilla plug-in. Have you seen an error like this before: export HEAT_INCLUDE_PASSWORD=1? | 23:59 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!