| *** shuyingya has joined #openstack-sahara | 00:07 | |
| *** shuyingya has quit IRC | 00:11 | |
| *** tosky has quit IRC | 00:15 | |
| *** shuyingya has joined #openstack-sahara | 00:43 | |
| *** shuyingya has quit IRC | 00:50 | |
| *** shuyingya has joined #openstack-sahara | 01:13 | |
| *** tuanluong has joined #openstack-sahara | 01:38 | |
| *** https_GK1wmSU has joined #openstack-sahara | 02:10 | |
| *** https_GK1wmSU has left #openstack-sahara | 02:13 | |
| *** dave-mcc_ has joined #openstack-sahara | 02:41 | |
| *** dave-mccowan has quit IRC | 02:42 | |
| *** dave-mccowan has joined #openstack-sahara | 03:12 | |
| *** dave-mcc_ has quit IRC | 03:14 | |
| *** esikachev has joined #openstack-sahara | 03:23 | |
| *** esikachev has quit IRC | 03:27 | |
| *** dave-mccowan has quit IRC | 03:33 | |
| *** links has joined #openstack-sahara | 03:48 | |
| *** esikachev has joined #openstack-sahara | 04:44 | |
| *** esikachev has quit IRC | 04:59 | |
| *** Poornima has joined #openstack-sahara | 05:12 | |
| *** ssmith has joined #openstack-sahara | 05:16 | |
| *** esikachev has joined #openstack-sahara | 05:42 | |
| *** anshulbehl has joined #openstack-sahara | 05:42 | |
| *** pgadiya has joined #openstack-sahara | 05:48 | |
| *** hoonetorg has quit IRC | 06:02 | |
| *** hoonetorg has joined #openstack-sahara | 06:15 | |
| *** pcaruana has joined #openstack-sahara | 06:34 | |
| *** rcernin has joined #openstack-sahara | 06:46 | |
| *** shuyingya has quit IRC | 07:12 | |
| *** shuyingya has joined #openstack-sahara | 07:12 | |
| *** esikachev has quit IRC | 07:31 | |
| *** tesseract has joined #openstack-sahara | 07:48 | |
| *** ssmith has quit IRC | 08:00 | |
| *** esikachev has joined #openstack-sahara | 08:09 | |
| *** zemuvier has joined #openstack-sahara | 08:10 | |
| openstackgerrit | Alina Nesterova proposed openstack/sahara-ci-config master: Remove README.md https://review.openstack.org/488420 | 08:20 |
|---|---|---|
| *** pgadiya has quit IRC | 08:25 | |
| openstackgerrit | Alina Nesterova proposed openstack/sahara-ci-config master: Remove README.md https://review.openstack.org/488420 | 08:28 |
| *** openstackgerrit has quit IRC | 08:33 | |
| *** pgadiya has joined #openstack-sahara | 08:41 | |
| *** abalutoiu has joined #openstack-sahara | 09:11 | |
| *** esikachev has quit IRC | 09:42 | |
| *** tosky has joined #openstack-sahara | 09:48 | |
| *** shuyingya has quit IRC | 09:53 | |
| *** anshulbehl has quit IRC | 10:06 | |
| *** tuanluong has quit IRC | 10:07 | |
| *** anshulbehl has joined #openstack-sahara | 10:18 | |
| *** esikachev has joined #openstack-sahara | 10:26 | |
| *** shuyingya has joined #openstack-sahara | 10:38 | |
| *** Poornima has quit IRC | 10:48 | |
| *** esikachev has quit IRC | 10:50 | |
| *** anshulbehl has quit IRC | 10:54 | |
| *** esikachev has joined #openstack-sahara | 10:54 | |
| *** Poornima has joined #openstack-sahara | 10:54 | |
| *** pgadiya has quit IRC | 10:56 | |
| *** anshulbehl has joined #openstack-sahara | 11:07 | |
| *** pgadiya has joined #openstack-sahara | 11:08 | |
| *** esikachev has quit IRC | 11:28 | |
| *** abalutoiu has quit IRC | 11:38 | |
| *** Poornima_K has joined #openstack-sahara | 11:50 | |
| *** Poornima has quit IRC | 11:51 | |
| *** Poornima_K has quit IRC | 12:00 | |
| *** Poornima has joined #openstack-sahara | 12:00 | |
| *** openstackgerrit has joined #openstack-sahara | 12:15 | |
| openstackgerrit | Alina Nesterova proposed openstack/sahara-ci-config master: [wip] Add playbook to run Apache2 https://review.openstack.org/490002 | 12:15 |
| *** Poornima has quit IRC | 12:23 | |
| *** shuyingya has quit IRC | 12:35 | |
| *** abalutoiu has joined #openstack-sahara | 12:49 | |
| *** jeremyfreudberg has joined #openstack-sahara | 12:58 | |
| *** shuyingya has joined #openstack-sahara | 12:58 | |
| *** shuyingya has quit IRC | 13:03 | |
| *** shuyingya has joined #openstack-sahara | 13:03 | |
| *** lucasxu has joined #openstack-sahara | 13:12 | |
| *** aolwas has quit IRC | 13:21 | |
| tellesnobrega | jeremyfreudberg, remember the error with spark.py and swift? | 13:21 |
| jeremyfreudberg | tellesnobrega, i remember discussing it, yes | 13:22 |
| tellesnobrega | goes down to java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.swift.snative.SwiftNativeFileSystem not found | 13:22 |
| *** aolwas has joined #openstack-sahara | 13:22 | |
| jeremyfreudberg | so, hadoop-swift jar is not in classpath | 13:23 |
| tellesnobrega | apparently yes, but looking into /opt/spark/conf there is a spark-defaults.conf file | 13:24 |
| tellesnobrega | which has spark.executor.extraClassPath /usr/lib/hadoop-mapreduce/hadoop-openstack.jar | 13:24 |
| jeremyfreudberg | that is true | 13:24 |
| jeremyfreudberg | i'm in a spark cluster now seeing that | 13:25 |
| tellesnobrega | which has the class | 13:25 |
| tellesnobrega | for some reason it is not on the classpath | 13:25 |
| jeremyfreudberg | can you go into the job execution's folder in /tmp/spark-edp/....... and look at launch_command.log, too? | 13:26 |
| tellesnobrega | yes | 13:26 |
| tellesnobrega | what would you like to see? | 13:27 |
| tellesnobrega | /opt/spark/bin/spark-submit --class main --master spark://spark21-spark21-master-0:7077 --deploy-mode client /tmp/wcount.py swift://bigsea.sahara/text.py | 13:27 |
| tellesnobrega | that is command | 13:27 |
| jeremyfreudberg | that's what i wanted to see | 13:28 |
| *** pgadiya has quit IRC | 13:29 | |
| jeremyfreudberg | i think you have edp.spark.adapt_for_swift (this is a checkbox in dashboard ui, otherwise it's a config value) as false | 13:29 |
| tellesnobrega | it is false | 13:29 |
| jeremyfreudberg | with that true, it adds --driver-class-path /usr/lib/hadoop-mapreduce/hadoop-openstack.jar as an argument to spark submit | 13:29 |
| jeremyfreudberg | BUT | 13:29 |
| jeremyfreudberg | it also uses the SparkWrapper jar | 13:29 |
| jeremyfreudberg | and some other stuff | 13:30 |
| jeremyfreudberg | which i thought was glitchy with pyspark | 13:30 |
| jeremyfreudberg | which is why we should try it out | 13:30 |
| tellesnobrega | sure | 13:31 |
| tellesnobrega | i got a different error now | 13:35 |
| tellesnobrega | Usage: wordcount <file> | 13:35 |
| jeremyfreudberg | the input args is not getting there | 13:35 |
| tellesnobrega | it is | 13:36 |
| *** abalutoiu has quit IRC | 13:36 | |
| tellesnobrega | but there is some stuff in between | 13:36 |
| tellesnobrega | spark.xml and main | 13:36 |
| *** abalutoiu has joined #openstack-sahara | 13:36 | |
| jeremyfreudberg | right, in this edp.spark.adapt_for_swift=true case, the last arguments are input and output, and before that should be mainclass | 13:38 |
| jeremyfreudberg | the main class of the wordcount jar, i mean | 13:38 |
| jeremyfreudberg | --deploy-mode client <file name> spark.xml <main class> <input> <output> | 13:39 |
| jeremyfreudberg | and i don't mean wordcount jar | 13:39 |
| jeremyfreudberg | it's py | 13:40 |
| jeremyfreudberg | main class we determined was "main", but possibly dummy | 13:40 |
| tellesnobrega | still failing with wordcount <file> | 13:42 |
| tellesnobrega | --deploy-mode client /tmp/wcount.py spark.xml main swift://bigsea.sahara/text.py swift://bigsea.sahara/output | 13:43 |
| jeremyfreudberg | hmm | 13:44 |
| jeremyfreudberg | trying something | 13:47 |
| tellesnobrega | ok | 13:49 |
| jeremyfreudberg | well, my finding is that when running with edp.spark.adapt_for_swift=true, a lot of stuff makes in into sys.argv of the python script that wouldn't, if you were running with adapt=false | 13:55 |
| *** abalutoiu_ has joined #openstack-sahara | 13:56 | |
| jeremyfreudberg | confirmed with meteos codebase as well, see https://github.com/openstack/meteos/blob/master/meteos/engine/drivers/generic.py#L191 they have two relevant args, but https://github.com/openstack/meteos/blob/master/meteos/cluster/binary/meteos-script-1.6.0.py#L648 shows that they start way down in sys.argv[3], with some other stuff before it | 13:57 |
| -openstackstatus- NOTICE: We have disable infracloud-vanilla due to the compute host running mirror.regionone.infracloud-vanilla.o.o being offline. Please recheck your failed jobs to schedule them to another cloud. | 13:58 | |
| *** abalutoiu has quit IRC | 13:58 | |
| *** abalutoiu__ has joined #openstack-sahara | 13:59 | |
| *** zemuvier has quit IRC | 14:02 | |
| tellesnobrega | jeremyfreudberg, I'm seeing something similar here | 14:02 |
| *** abalutoiu_ has quit IRC | 14:03 | |
| tellesnobrega | removed a lot of stuff and it kinda worked | 14:03 |
| tellesnobrega | failed on swift credentials | 14:03 |
| jeremyfreudberg | removed stuff from the spark-submit command? or what? | 14:03 |
| tellesnobrega | yes | 14:09 |
| tellesnobrega | cleaned up the command | 14:09 |
| tellesnobrega | a bit | 14:09 |
| tellesnobrega | for some reason it is not reading the spark.xml | 14:10 |
| tellesnobrega | it is passed --files | 14:10 |
| jeremyfreudberg | ah yes, finally, that's the issue i was trying to remember! | 14:11 |
| tellesnobrega | but it fails saying that the config is missing fs.swift.service.sahara.username | 14:12 |
| *** esikachev has joined #openstack-sahara | 14:15 | |
| tellesnobrega | jeremyfreudberg, https://spark.apache.org/docs/2.1.0/storage-openstack-swift.html | 14:16 |
| jeremyfreudberg | tellesnobrega, so the configs are the right thing, but they can't be accessed by pyspark job. you remember every time we discuss this and I say, "hmm... I think there's some weird issue between SparkWrapper jar and pyspark jobs, but I can't remember what it is", this is the issue... it can't read spark.xml because of the way that SparkWrapper is injecting those fs.swift.* configs at runtime is only accessible to JAR-file jobs | 14:16 |
| jeremyfreudberg | in the next release, we can invent a edp.spark.adapt_for_swift_pyspark that injects the configs in a compatible way | 14:17 |
| tellesnobrega | by pyspark you mean spark-submit? | 14:18 |
| jeremyfreudberg | i mean spark job written in python | 14:18 |
| tellesnobrega | ok | 14:18 |
| tellesnobrega | because pyspark used to be something no spark that isn't supported anymore | 14:18 |
| jeremyfreudberg | oh, i see | 14:19 |
| tellesnobrega | can you try running without the sparkwrapper? | 14:19 |
| jeremyfreudberg | without the spark wrapper (And the extra args that get put there because of it) it should mostly work, since that's the same as edp.spark.adapt_for_swift=false, but you'll still need --driver-classpath and someway for the swift username and password to get in | 14:21 |
| tellesnobrega | yes | 14:21 |
| jeremyfreudberg | at some point i had written something like this in a job: | 14:22 |
| jeremyfreudberg | sc = SparkContext(appName="PythonWordCount") | 14:22 |
| jeremyfreudberg | sc._jsc.hadoopConfiguration().set('fs.swift.service.sahara.username', sys.argv[3]) | 14:22 |
| jeremyfreudberg | and i recall that working | 14:22 |
| tellesnobrega | yes, from that link I sent you the right way to run it is by putting the configuration into the job itself | 14:23 |
| tellesnobrega | considering the risk of having sensitive information in a file somewhere | 14:24 |
| tellesnobrega | I'm not sure what would be the best set up for us here. I get that having the config done on our side makes things easier for the user | 14:26 |
| tellesnobrega | but we need to do this right, if it doesn't work then we have nothing | 14:27 |
| jeremyfreudberg | well, i'm thinking about a few things | 14:27 |
| jeremyfreudberg | right now, it basically works if you tell the user "put the credentials in the script, or read them from args" | 14:27 |
| jeremyfreudberg | but that's annoying/bad because it means the python spark jobs are different for no reason apparent to user | 14:28 |
| jeremyfreudberg | and you can't use proxy user temp credentials | 14:28 |
| tellesnobrega | are you sure that works? do you still have the job you wrote? | 14:28 |
| jeremyfreudberg | i can test it later, but it works on meteos | 14:28 |
| tellesnobrega | cool, I will try to add that to my wcount.py test as well | 14:28 |
| jeremyfreudberg | but what i was saying regarding proxy users, is that sahara assumes the credentials can be read from edp configs in fs.swift.service.sahara.* | 14:29 |
| jeremyfreudberg | but a python spark can't access these configs, which eventually end up in spark.xml, as we saw, at least in the current state | 14:29 |
| tellesnobrega | yes | 14:30 |
| tellesnobrega | I assume that this is what the wrapper does, reads this config before sending the job up | 14:30 |
| tellesnobrega | or am I wrong? my java is a little rusty | 14:30 |
| jeremyfreudberg | that's more or less right, it reads the config and puts it "somewhere", then invokes the main method of the user's actual job after config has been properly loaded | 14:31 |
| jeremyfreudberg | so, let's remember that the point of EDP is to provide a generic interface for jobs that should behave more or less the same regardless of underlying services | 14:31 |
| jeremyfreudberg | so we are kind of failing that goal now regarding the python spark situation | 14:32 |
| tellesnobrega | true | 14:32 |
| jeremyfreudberg | ideally we could write some new kind of wrapper specifically for python spark jobs, whicih would preserve the user experience and make edp feel generic as it should be | 14:32 |
| tellesnobrega | that is exactly my point | 14:33 |
| tellesnobrega | we are completely failing it right now | 14:33 |
| jeremyfreudberg | so, we should definitely investigate if that's possible to resolve | 14:33 |
| tellesnobrega | I'm trying a basic word count and it doesn't work | 14:33 |
| tellesnobrega | yes | 14:33 |
| *** Poornima has joined #openstack-sahara | 14:35 | |
| *** Poornima has quit IRC | 14:35 | |
| *** Poornima has joined #openstack-sahara | 14:37 | |
| jeremyfreudberg | in any case, i'll add it to etherpad, now that we actually have a clear view of the limitations | 14:42 |
| tellesnobrega | yes | 14:42 |
| jeremyfreudberg | exact solution will take some time to flesh out | 14:42 |
| tellesnobrega | thanks | 14:42 |
| *** links has quit IRC | 15:01 | |
| jeremyfreudberg | hey tellesnobrega, do you know how i can get added to the launchpad group for sahara, so i can set bug importance (hi/medium/etc)? | 15:08 |
| * tellesnobrega check | 15:08 | |
| *** esikachev has quit IRC | 15:14 | |
| tellesnobrega | jeremyfreudberg, can you check if you can add yourself to the team? | 15:14 |
| jeremyfreudberg | tellesnobrega, not sure if I'm even looking at the right team, sahara drivers, sahara bug team | 15:15 |
| jeremyfreudberg | ? | 15:15 |
| tellesnobrega | sahara drivers should be the one | 15:16 |
| tellesnobrega | I'm not an administrator of the group | 15:17 |
| *** esikachev has joined #openstack-sahara | 15:17 | |
| jeremyfreudberg | tellesnobrega, try adding me to sahara bug team, if you are member of drivers team then you are admin there | 15:19 |
| jeremyfreudberg | https://launchpad.net/~sahara-bugs/+members#active | 15:19 |
| tellesnobrega | ok | 15:20 |
| jeremyfreudberg | if it can't be done, it's not the end of the world, but it would be nice | 15:20 |
| tellesnobrega | that I could do | 15:21 |
| *** esikachev has quit IRC | 15:22 | |
| jeremyfreudberg | tellesnobrega, thanks, that works, i have full control over bugs now | 15:22 |
| tellesnobrega | awesome | 15:22 |
| jeremyfreudberg | you should add tosky and esikachev too | 15:22 |
| tellesnobrega | will do | 15:28 |
| *** tomtomtom has quit IRC | 15:32 | |
| *** tomtomtom has joined #openstack-sahara | 15:33 | |
| *** esikachev has joined #openstack-sahara | 15:33 | |
| tosky | uhm, talking about the usage of the main gates for more complex jobs... https://review.openstack.org/#/c/489773/ | 15:36 |
| tosky | (see '[trove] Can we move some non-voting broken jobs to the experimental queue?' on openstack-dev) | 15:36 |
| *** abalutoiu__ has quit IRC | 15:38 | |
| elmiko | jeremyfreudberg esikachev tosky congrats to you all on the cores =) | 15:41 |
| esikachev | thanks) | 15:42 |
| * elmiko just saw the "paperwork" go through | 15:42 | |
| *** anshulbehl has quit IRC | 15:42 | |
| *** jeremyfreudberg has quit IRC | 15:44 | |
| openstackgerrit | Shu Yingya proposed openstack/sahara-dashboard master: Fix the compatibility after django 1.11 support https://review.openstack.org/490064 | 15:49 |
| *** shuyingya has quit IRC | 15:51 | |
| *** shuyingya has joined #openstack-sahara | 15:51 | |
| *** jeremyfreudberg has joined #openstack-sahara | 15:54 | |
| jeremyfreudberg | elmiko, thanks :) | 15:55 |
| *** shuyingya has quit IRC | 15:56 | |
| jeremyfreudberg | tosky, looking | 15:56 |
| tellesnobrega | tosky, I don't see how that affect us, maybe I wasn't during a previous discussion, is this regarding adding our jobs to the openstack ci? | 15:57 |
| *** shuyingya has joined #openstack-sahara | 15:57 | |
| tosky | tellesnobrega: it does not affect us; I was just pointing out the number of jobs triggered | 15:57 |
| tellesnobrega | oh, I see | 15:57 |
| tellesnobrega | I guess that if we push we can get some of ours there | 15:57 |
| tosky | exactly; I was worrying too much maybe | 15:58 |
| tellesnobrega | maybe | 15:58 |
| *** links has joined #openstack-sahara | 15:58 | |
| *** esikachev has quit IRC | 16:02 | |
| *** shuyingya has quit IRC | 16:04 | |
| *** shuyingya has joined #openstack-sahara | 16:04 | |
| *** Poornima has quit IRC | 16:05 | |
| *** rcernin has quit IRC | 16:08 | |
| *** shuyingya has quit IRC | 16:09 | |
| *** pcaruana has quit IRC | 16:14 | |
| *** hoonetorg has quit IRC | 16:30 | |
| tosky | openstack-dev: [all] Rollout of Zuul v3 at the PTG | 16:35 |
| * tosky rejoices AND screams at the same time | 16:35 | |
| jeremyfreudberg | lol | 16:36 |
| *** hoonetorg has joined #openstack-sahara | 16:42 | |
| *** jeremyfreudberg has quit IRC | 16:43 | |
| *** shuyingya has joined #openstack-sahara | 16:48 | |
| *** shuyingya has quit IRC | 16:53 | |
| *** esikachev has joined #openstack-sahara | 16:59 | |
| *** esikachev has quit IRC | 17:03 | |
| *** shuyingya has joined #openstack-sahara | 17:06 | |
| *** tesseract has quit IRC | 17:07 | |
| openstackgerrit | Shu Yingya proposed openstack/sahara-dashboard master: Fix the compatibility after django 1.11 support https://review.openstack.org/490064 | 17:10 |
| *** shuyingya has quit IRC | 17:11 | |
| *** esikachev has joined #openstack-sahara | 17:34 | |
| *** tosky has quit IRC | 17:36 | |
| *** esikachev has quit IRC | 17:38 | |
| *** jeremyfreudberg has joined #openstack-sahara | 17:41 | |
| *** esikachev has joined #openstack-sahara | 17:47 | |
| openstackgerrit | Merged openstack/python-saharaclient stable/pike: Update .gitreview for stable/pike https://review.openstack.org/488806 | 17:53 |
| *** links has quit IRC | 18:31 | |
| *** esikachev has quit IRC | 18:32 | |
| *** esikachev has joined #openstack-sahara | 18:33 | |
| *** esikachev has quit IRC | 18:39 | |
| *** shuyingya has joined #openstack-sahara | 19:21 | |
| *** shuyingya has quit IRC | 19:25 | |
| tellesnobrega | jeremyfreudberg, about tosky question if we need to backport the reno patch | 19:56 |
| tellesnobrega | the answer is no | 19:56 |
| jeremyfreudberg | tellesnobrega, cool, although it still seems strange to me, what if we wanted a release note to be included for pike-eol release, as an example. still, i'll trust your answer | 19:58 |
| *** hoonetorg has quit IRC | 20:03 | |
| *** hoonetorg has joined #openstack-sahara | 20:20 | |
| tellesnobrega | jeremyfreudberg, when you have time can you check that python job for me? | 20:20 |
| jeremyfreudberg | tellesnobrega, sure | 20:21 |
| *** lucasxu has quit IRC | 21:00 | |
| dhellmann | jeremyfreudberg : the job that builds release notes for our CI always runs against the master branch, and reno scans the other branches based on the instructions in the rst files. So the rst file doesn't have to exist on the stable branch for our publishing to work. Now if someone wants to package that file, you would have to backport it. | 21:12 |
| jeremyfreudberg | ok, dhellmann, thanks for stepping in to clarify. that actually makes a lot of sense | 21:13 |
| dhellmann | always happy to help :-) | 21:13 |
| *** shuyingya has joined #openstack-sahara | 21:20 | |
| *** shuyingya has quit IRC | 21:25 | |
| *** jeremyfreudberg has quit IRC | 21:38 | |
| *** shuyingya has joined #openstack-sahara | 22:29 | |
| *** shuyingya has quit IRC | 22:33 | |
| *** https_GK1wmSU has joined #openstack-sahara | 22:57 | |
| *** https_GK1wmSU has left #openstack-sahara | 22:59 | |
| *** shuyingya has joined #openstack-sahara | 23:18 | |
| *** shuyingya has quit IRC | 23:23 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!