Thursday, 2016-02-18

« Wednesday, 2016-02-17 Index Friday, 2016-02-19 »
*** sgotliv has quit IRC00:11
*** sgotliv has joined #openstack-sahara00:11
*** witlessb has quit IRC00:43
*** rcernin has quit IRC00:46
*** crobertsrh is now known as _crobertsrh01:41
rickflareI did a crap ton of reading and I now under so much more!02:36
rickflareI have a crap ton of questions for you guys tomorrow02:37
openstackgerritzhongshengping proposed openstack/puppet-sahara: Add api_paste type/provider for Sahara  https://review.openstack.org/28161302:50
*** Poornima has joined #openstack-sahara04:02
rickflareguys I am now seeing the following when running my spark job04:14
rickflarehttp://pastebin.com/pqiRRSWT04:14
openstackgerritzhongshengping proposed openstack/puppet-sahara: Add api_paste type/provider for Sahara  https://review.openstack.org/28161305:32
*** dave-mccowan has quit IRC05:38
openstackgerritVitaly Gridnev proposed openstack/sahara: honor api_insecure parameters  https://review.openstack.org/27999606:07
*** apavlov has joined #openstack-sahara06:08
*** apavlov has quit IRC06:39
openstackgerritJaxon Wang proposed openstack/sahara-tests: Add more infomation when create cluster failed for scenario test  https://review.openstack.org/28109506:53
*** nkrinner has joined #openstack-sahara06:58
openstackgerritJaxon Wang proposed openstack/sahara: Update CDH user doc for CDH 5.5.0  https://review.openstack.org/28167007:16
openstackgerritGrigoriy Rozhkov proposed openstack/sahara: Remove unsupported MapR plugin versions  https://review.openstack.org/26644407:16
openstackgerritVitaly Gridnev proposed openstack/sahara: CDH plugin config helper refactoring  https://review.openstack.org/25582507:18
openstackgerritVitaly Gridnev proposed openstack/sahara: CDH plugin edp engine code refactoring  https://review.openstack.org/25730907:18
*** esikachev has joined #openstack-sahara07:18
openstackgerritVitaly Gridnev proposed openstack/sahara: Add CDH 5.5 support  https://review.openstack.org/27996407:28
openstackgerritEvgeny Sikachev proposed openstack/sahara-tests: Fix READMEs location for sahara_tests  https://review.openstack.org/28073007:31
openstackgerritMerged openstack/sahara-tests: Add CDH 5.5.0 scenario test  https://review.openstack.org/28109207:35
openstackgerritJinxing Fang proposed openstack/sahara: Update the roadmap  https://review.openstack.org/28167807:40
*** rcernin has joined #openstack-sahara07:51
openstackgerritJaxon Wang proposed openstack/sahara-tests: Add more infomation when create cluster failed for scenario test  https://review.openstack.org/28109508:03
*** pcaruana has joined #openstack-sahara08:05
openstackgerritEvgeny Sikachev proposed openstack/sahara-tests: Put input datasources to hdfs in Pig job  https://review.openstack.org/28070108:08
openstackgerritEvgeny Sikachev proposed openstack/sahara-tests: Put input datasources to hdfs in Pig job  https://review.openstack.org/28070108:11
openstackgerritEvgeny Sikachev proposed openstack/sahara-tests: Disable ssl_verify as default  https://review.openstack.org/28076208:14
*** esikachev has quit IRC08:23
openstackgerritJaxon Wang proposed openstack/sahara: Update CDH user doc for CDH 5.5.0  https://review.openstack.org/28167008:29
openstackgerritGrigoriy Rozhkov proposed openstack/sahara: Remove unsupported MapR plugin versions  https://review.openstack.org/26644408:38
openstackgerritMerged openstack/sahara-specs: Remove unsupported versions of MapR plugin  https://review.openstack.org/25862008:44
*** vgridnev has joined #openstack-sahara08:48
*** esikachev has joined #openstack-sahara09:11
*** witlessb has joined #openstack-sahara09:18
*** apavlov has joined #openstack-sahara09:23
openstackgerritJaxon Wang proposed openstack/sahara: CDH plugin edp engine code refactoring  https://review.openstack.org/25730909:57
*** openstackgerrit has quit IRC10:02
*** openstackgerrit has joined #openstack-sahara10:03
openstackgerritEvgeny Sikachev proposed openstack/sahara-tests: Adding ability use default templates  https://review.openstack.org/28022510:22
openstackgerritEvgeny Sikachev proposed openstack/sahara-tests: Put input datasources to hdfs in Pig job  https://review.openstack.org/28070110:27
*** apavlov has quit IRC10:40
openstackgerritEvgeny Sikachev proposed openstack/sahara-tests: Add autoregistering of image  https://review.openstack.org/28131510:43
*** Poornima has quit IRC10:43
openstackgerritzhongshengping proposed openstack/puppet-sahara: Add the capability to configure api-paste.ini with config.pp  https://review.openstack.org/28175610:47
*** apavlov has joined #openstack-sahara10:55
*** _degorenko|afk is now known as degorenko11:01
*** tellesnobrega is now known as tellesnobrega_af11:02
*** vgridnev has quit IRC11:11
*** vgridnev has joined #openstack-sahara11:21
*** tellesnobrega_af is now known as tellesnobrega11:24
*** vgridnev has quit IRC11:24
*** vgridnev has joined #openstack-sahara11:26
*** vgridnev has quit IRC11:28
*** vgridnev has joined #openstack-sahara11:28
openstackgerritzhongshengping proposed openstack/puppet-sahara: Add the capability to configure api-paste.ini with config.pp  https://review.openstack.org/28175611:34
openstackgerritzhongshengping proposed openstack/puppet-sahara: Add the capability to configure api-paste.ini with config.pp  https://review.openstack.org/28175611:37
openstackgerritzhongshengping proposed openstack/puppet-sahara: Add the capability to configure api-paste.ini with config.pp  https://review.openstack.org/28175611:38
openstackgerritEvgeny Sikachev proposed openstack/sahara-tests: Add check of scaling for CDH and Ambari  https://review.openstack.org/27467511:43
*** apavlov has quit IRC11:44
*** apavlov has joined #openstack-sahara11:47
*** raildo-afk is now known as raildo12:09
*** apavlov has quit IRC12:09
*** dave-mccowan has joined #openstack-sahara12:26
openstackgerritEvgeny Sikachev proposed openstack/sahara-tests: Fix using proxy node for checks  https://review.openstack.org/27944712:41
*** esikachev_afk has joined #openstack-sahara12:49
*** esikachev has quit IRC12:53
*** esikachev has joined #openstack-sahara12:53
*** esikachev has left #openstack-sahara12:53
*** esikachev_afk is now known as esikachev12:58
* esikachev is now away: Away from keyboard13:02
*** esikachev is now known as esikachev_afk13:02
*** esikachev_afk is now known as esikachev13:14
*** thumpba has joined #openstack-sahara13:15
*** apavlov has joined #openstack-sahara13:25
*** nkrinner has quit IRC13:28
*** nkrinner has joined #openstack-sahara13:28
*** _crobertsrh is now known as crobertsrh13:31
vgridnevcrobertsrh, is there understanding what's up with integration tests in your dashboard changes?13:58
crobertsrhI'm guessing that there will be some rewriting of integration tests still to come.13:58
crobertsrhHoping that those changes will be small, but I still need to look at them.13:59
*** thumpba has quit IRC14:09
*** dave-mccowan has quit IRC14:09
*** dave-mccowan has joined #openstack-sahara14:10
*** tmckay has joined #openstack-sahara14:10
*** vgridnev has quit IRC14:11
*** vgridnev has joined #openstack-sahara14:12
*** crobertsrh1 has joined #openstack-sahara14:27
*** egafford has joined #openstack-sahara14:27
rickflaremorning14:29
* rickflare finally read 90% of the sahara docs and is ashamed he didnt do it sooner14:29
openstackgerritMerged openstack/sahara-dashboard: Change color of status field  https://review.openstack.org/27824114:31
*** tmckay has quit IRC14:34
elmikorickflare: good bedtime reading ;)14:35
rickflareso14:37
rickflareelmiko14:37
rickflaretell me what you think about this14:37
rickflarefew things in my reading that I have questions about14:37
elmikok14:37
*** vgridnev has quit IRC14:38
rickflareone is once sahara starts a cluster lets say you have a massive map reduce job failure or a cascading failure of name nodes14:38
rickflareall of which I have seen14:39
rickflareusing horizon can you restart the hadoop services14:39
rickflareor does one have to manually go into each node and restart the services?14:39
elmikohmm14:40
elmikowell if the job fails, then i would expect the job execution in horizon to report as failed. at which point you could restart the job.14:40
rickflarethis is a huge huge issue for production clusters14:40
rickflarenot related to the job14:40
rickflareive seen ingestion or random things cause name nodes to die14:41
elmikodo those issue end up cascading back to the master and causing a job failure?14:41
elmikoor just node drop outs14:41
rickflareI am not aware that sahara is using high availility for name nodes14:41
rickflarenodes can just drop out14:42
elmikowe do have some HA options14:42
elmikook, so yo might want to look at the health check patches that vgridnev is working on14:42
elmikowe are implementing a system that will determine the health of the cluster14:42
rickflareso HA on the name nodes is something that should be top priority14:42
rickflareif its not already implemented14:42
elmikoso, to your original question. currently, i don't think you can restart individual nodes in the cluster14:42
elmikoyou would need to login to the node and fix whatever is wrong, or restart manually14:43
rickflarewhat happens if you have to reboot the entire cluster14:43
rickflareie kernel patches etc14:43
rickflareyou have to manually restart all the cluster services14:43
elmikoin that case, you would need to rebuild the image you are using for the cluster and rebuild the cluster14:43
*** vgridnev has joined #openstack-sahara14:43
elmikowe don't have any option currently to update software packages per node14:43
rickflarewell not the cluster software but os related14:44
rickflareand that approach wont work in most cases14:44
rickflareesp if the data in hdfs can not be reproduced14:44
*** vgridnev has quit IRC14:44
rickflareie pcap etc14:44
elmikoi would imagine you would want to create an external hdfs store, and treat the cluster as something that can be dropped and respawned if needed14:45
rickflareso in that case you would not be using hdfs at all14:46
rickflarejust the task trackers14:46
rickflarewhich is not a compelling use14:46
rickflarefor hardware consolidation14:47
rickflareso are you familar with cloudera manager?14:47
elmikoa little14:47
elmikowait, why wouldn't you be using hdfs?14:47
rickflareso ensuring sahara has the basic functionality of cloudera manager should high on the list14:47
elmikosahara includes the cloudera manager in the cdh plugin images14:48
rickflarewell if you are going to have a external hdfs store14:48
rickflareyou might as well use bare metal14:48
rickflareat that point14:48
*** dave-mccowan has quit IRC14:48
elmikowhy is that?14:48
rickflarebecause you can trust the data persistance of the openstack cluster14:48
rickflarelike if rebooting14:49
rickflarerequires a cluster rebuild14:49
rickflarethat is not effective14:49
elmikosure14:49
*** tellesnobrega is now known as tellesnobrega_af14:49
elmikowe are currently working on improving the cluster health options, but we don't have some of these potions yet14:49
elmikowe do have some HA availability in the cluster, but this is nothing like rolling o/s upgrades or anything14:50
elmikoi'm still curious about the external hdfs stuff, and why that is a poor choice14:50
*** vgridnev has joined #openstack-sahara14:50
rickflareok14:50
rickflareill try to articulate that more14:51
elmikocool14:51
rickflareill since with my co workers14:51
rickflareand provide you with something in detail14:51
rickflareas for the cluster health14:51
rickflarethat is helpful14:52
rickflarebut service control14:52
rickflareis HUGE14:52
rickflareI have written ruby scripts to manage hadoop clusters14:52
rickflareie formating name nodes14:52
rickflareand ensuring HA to starting and stoping name nodes14:53
elmikonice, sounds like a good improvement for sahara14:53
rickflarethese scripts would perform checks on  the status of the name nodes prior to doing anything related to the datanodes14:53
elmikoimo, if you are free this afternoon (EST), you should bring these topics up at our meeting14:53
rickflareok great!14:53
elmikodefinitely look at the patches that vgridnev is working on for cluster health too14:54
elmikohe is creating a system that will be very expandable for performing health checks14:54
rickflarethat another thing I struggle with14:54
rickflarei am not yet very knowledgable about who is doing what within the project14:54
elmikototally understandable14:54
elmikorickflare: http://eavesdrop.openstack.org/#OpenStack_Data_Processing_%28Sahara%29_Team_Meeting14:55
elmikocoming to our meetings is one the best way to stay in touch with what we are working on14:55
elmikoand if you make it to summit, i'm sure you will find no shortage of folks who would like to talk about improvements to sahara14:56
rickflarei am for sure coming14:56
elmikowe could even make plans for future cycles about improvements to cluster health and rolling reboots, etc.14:56
rickflareI am working tirelessly to getting my guys14:56
rickflareto start seriously doing work14:57
rickflarefor this14:57
elmiko\o/14:57
rickflareand providing feedback14:57
elmikowe love that14:57
elmikogetting good user/operator feedback is something we really would like to get more of14:57
rickflareyou guys are going to get it14:57
rickflarei am putting all my chips on openstack and sahara14:58
elmikogoing "all in" eh?14:58
elmiko;)14:58
rickflarei see more power and control in this that aws14:58
rickflarealso I just dont see running these services in containers making sense at this point14:58
elmikoyea, that's still up in the air, i suppose14:59
elmikoi've done a little work with spark in containers14:59
rickflareyea14:59
rickflareso the containers issue is interesting15:00
rickflareits hard to know what will happen15:00
rickflarei mean docker is powerful and kubernetes is sick15:00
rickflarethe complexity is sky high though15:00
elmikoyea15:01
elmikoespecially the networking issues15:01
rickflarewhat are your thoughts on it?15:01
rickflareomg yes15:01
elmikowell, i think it's interesting, and i do like how quickly the containers can spin up15:01
elmikogetting a spark cluster running on containers, i've seen the clusters spin up way faster than vm15:02
rickflareoh yea15:02
rickflarebecause you essentially only need to deal with that process15:02
*** dave-mccowan has joined #openstack-sahara15:02
elmikoso, i imagine it would make more difference in situations where you care about elastic scaling speed15:02
elmikoright15:02
rickflarei just worry that the pace of development with containers is moving so fast15:03
rickflareit may leave vm's in the dust before even leaving the starting line15:03
elmikohehe15:03
elmikoa valid concern15:03
elmikoi think the whole containers/vm debate needs to be taken with a grain of salt15:04
elmikothere are some situations where having a vm is preferable, and sometimes the opposite15:04
rickflareit worries me though15:04
rickflareas a CEO15:04
elmikowhat worries you about it?15:04
rickflareof a small company my choices of tech direction have huge implications15:04
elmikoah, right15:04
rickflareesp since we can not be masters of every domain15:05
rickflareits just not possible15:05
elmikoright15:05
rickflareso thats the biggest concern I have15:05
elmikototally valid15:06
rickflarewhat time is the meeting15:07
rickflarei have a lot I think I can provide15:07
elmikoi think 1pm eastern today15:07
rickflareok cool15:07
rickflarei am also going to work on this blueprint15:07
rickflaretoday15:07
rickflareand get that in15:07
elmikoyea, sounds like you have some great ideas15:07
rickflareon another note15:09
rickflareI am still getting failures on my spark jobs15:09
elmikowell that stinks =(15:10
rickflareyea15:10
rickflarehttp://pastebin.com/pqiRRSWT15:10
rickflareis what im getting15:11
elmikoweird, so, some connection issue to the keystone controller15:11
rickflareyea15:12
rickflareand I dont know why15:12
elmikomight be worth running some curl commands to the keystone server to see if you can issue a token create from that node15:12
rickflareenlighten me15:13
elmikohttp://docs.openstack.org/developer/keystone/api_curl_examples.html?highlight=curl#service-api-examples-using-curl15:13
elmikothat shows some examples of running cli curl commands to a keystone api controller15:13
elmikosomething i might try to debug, is logging in to the node that is failing to make the token, and manually run the curl command to that endpoint to see if you can generate a token15:14
elmikoyou *may* find that there is some networking issue affecting communication between the tenant network and the control plane network15:14
elmikoi've seen issues like not allowing inbound traffic, etc...15:15
elmiko(although your sec rules looked fine the other day)15:15
rickflareworked just fine15:15
elmikook, so it's something specific to when spark attempts to make the connection15:15
elmikothis is where things get fuzzy, because you are dealing with spark using the hadoop-openstack.jar to do comms with keystone15:16
rickflarehumm15:17
elmikoi'm guessing this is to access some swift object?15:17
rickflareyes sir15:17
elmikoit's weird that the curl would work, but spark fails15:18
elmikoand socket timeout definitely indicates that it's a networking issue, not a bad request to the keystone controller15:18
elmikowe are now approaching the edge of my knowledge on debugging spark15:19
elmikojust fyi...15:19
rickflareok15:20
elmikohmm, could it be that one of the nodes in the cluster is attempting this request and perhaps that node doesn't have good connectivity to the keystone server?15:21
rickflarethey all do15:24
elmikohuh15:24
*** Akanksha08 has joined #openstack-sahara15:30
rickflareyou know what15:32
rickflareelmiko15:32
rickflareyou might be right15:32
elmikorickflare: sorry, i don't have any other specific advice. maybe write a small spark app to try and access the keystone controller, or if pyspark is on those images you could try running something from the pyspark repl15:32
rickflarei might have had a ip overlap15:32
rickflareim checking now15:32
elmikoah, interesting ;)15:32
rickflaregreat catch15:33
rickflarei actually think that may have been it15:33
rickflarerebuilding the cluster in a different15:33
elmiko\o/15:33
rickflareip space15:35
rickflarerunning the job now15:39
crobertsrhvgridnev:  I'm seeing what I think is strange output when I run the integration tests locally (different failures than what I think I'm seeing in the gate logs).  Any tips on running the integration tests?15:40
*** tmckay has joined #openstack-sahara15:40
* rickflare reading pep8 and realizing he knows nothing15:40
rickflareelmiko15:42
rickflarenope same15:42
rickflaretime out error15:42
elmikohuh...15:43
elmikorickflare: so, yea, at this point you might need to debug this from inside the spark app. maybe by writing a custom app to ping the keystone server. i might use pyspark if it's available on those nodes15:46
elmikoit should be fairly easy to write a small pyspark app to test connectivity with the keystone controller15:46
rickflareso I am seeing this in keystone15:48
rickflarehttp://pastebin.com/XLEy3vUC15:48
elmikointeresting....15:48
* esikachev is now away: Away from keyboard15:48
*** esikachev is now known as esikachev_afk15:49
elmikoso, it's not a connection issue but a data issue15:49
*** thumpba has joined #openstack-sahara15:49
elmikoi'm surprised that keystone doesn't return a 50015:49
elmikoor maybe it does and the hadoop-openstack connector mis-reports15:49
elmikorickflare: does it show the request that was made?15:50
elmiko(also, you may want to se debug=true in your keystone conf to see more output)15:50
elmikoyou'll probably want to debug the actual body that is sent to the token POST15:51
*** vgridnev has quit IRC15:52
rickflarek15:56
*** krotscheck_dcm is now known as krotscheck16:00
*** raildo is now known as raildo-afk16:02
*** zigo has quit IRC16:03
openstackgerritGrigoriy Rozhkov proposed openstack/sahara: Remove unsupported MapR plugin versions  https://review.openstack.org/26644416:04
*** zigo has joined #openstack-sahara16:05
*** coolsvap|away has quit IRC16:06
*** pcaruana has quit IRC16:15
*** nkrinner has quit IRC16:16
*** raildo-afk is now known as raildo16:17
*** rcernin has quit IRC16:17
*** tellesnobrega_af is now known as tellesnobrega16:19
*** coolsvap|away has joined #openstack-sahara16:20
openstackgerritTim Kelsey proposed openstack/sahara: Fixes to make bandit integration tests work with sahara  https://review.openstack.org/28194016:35
openstackgerritEvgeny Sikachev proposed openstack/sahara-tests: Fix using proxy node for checks  https://review.openstack.org/27944716:39
*** tellesnobrega is now known as tellesnobrega_af16:44
*** tellesnobrega_af is now known as tellesnobrega16:57
openstackgerritGrigoriy Rozhkov proposed openstack/sahara-tests: Add MapR-FS support to sahara scenario framework  https://review.openstack.org/28196317:05
*** vgridnev has joined #openstack-sahara17:11
*** vgridnev has quit IRC17:15
*** vgridnev has joined #openstack-sahara17:18
*** vgridnev has quit IRC17:19
*** vgridnev has joined #openstack-sahara17:27
*** krotscheck is now known as krotscheck_dr17:27
*** vgridnev has quit IRC17:28
*** vgridnev has joined #openstack-sahara17:34
*** vgridnev has quit IRC17:35
*** rcernin has joined #openstack-sahara17:36
*** vgridnev has joined #openstack-sahara17:42
*** vgridnev has quit IRC17:45
rickflarehey17:50
rickflarewhat channel is the meeting in?17:50
elmikorickflare: openstack-meeting-alt (today)17:52
*** vgridnev has joined #openstack-sahara17:52
*** esikachev_afk is now known as esikachev17:53
*** degorenko is now known as _degorenko|afk18:02
*** tellesnobrega is now known as tellesnobrega_af18:05
*** tellesnobrega_af is now known as tellesnobrega18:05
*** tellesnobrega is now known as tellesnobrega_af18:07
*** raildo is now known as raildo-afk18:10
*** tellesnobrega_af is now known as tellesnobrega18:11
*** raildo-afk is now known as raildo18:22
openstackgerritGrigoriy Rozhkov proposed openstack/sahara: Add Hue 3.9.0 to MapR plugin  https://review.openstack.org/27521718:28
*** apavlov has quit IRC18:51
rickflaretmckay you around?18:51
tmckayyeah18:52
rickflaresooo18:52
rickflaregot some new errors18:52
rickflarei got the job to run18:52
rickflareas long as I dont write to swift18:52
tmckayhad to fight with my cluster this morning18:52
*** vgridnev has quit IRC18:52
openstackgerritlu huichun proposed openstack/sahara: [EDP] Add suspend_job() for sahara edp engine(oozie implementation)  https://review.openstack.org/20144818:53
tmckayrickflare, ack. The swift issue is very strange -- I've got a spark 131 out of the box, with no special config, and it works18:53
rickflarereally18:53
rickflarewth18:53
tmckayrickflare, you could explore integrating with manila, for fun18:53
rickflaremanilla?18:53
*** witlessb has quit IRC18:53
tmckayrickflare, yeah, i've never seen the permission issue we ran into. For me, if you can read from swift, it's been fine18:54
tmckaymanila is an NFS share service in openstack18:54
tmckayWe did some work to integrate sahara with it18:54
tmckayYou can set up manila, and host your data on manila shares instead of swift, and then have sahara mount the shares on your cluster18:54
* esikachev is now away: Away from keyboard18:55
*** esikachev is now known as esikachev_afk18:55
tmckayegafford and crobertsrh did most of the work on it, I did some18:55
rickflareits failing on the create file18:55
rickflareit as if18:55
rickflareI dont have permission to write to the container18:55
elmikoooh, you installed with packstack?18:55
tmckayI wonder if you created the container using different credentials?18:55
rickflarenaw18:55
elmikoyou may want to check the swift acls to make sure that the user has permission to write in that project on swift18:56
tmckayelmiko, some default swift acl junk?18:56
elmikoright18:56
elmikocould be18:56
elmikoor18:56
*** witlessb has joined #openstack-sahara18:56
elmikohave you tried using those creds to just make a file in swift from the cli or something?18:56
rickflareyea18:56
tmckayelmiko, yeah, the initial upload18:56
rickflareand it works18:56
rickflareso check this out18:56
tmckaywonder if it's an acl tied to an ip?18:57
rickflarei cant seem to delete any of them object in the containers18:57
tmckayrickflare, anything in the swift logs?18:57
rickflaresome are listed as pseudo-folders18:57
elmikotmckay: doubtful18:57
rickflareall puts in swift18:58
rickflareno failures18:58
tmckayrickflare, on a side note, I verified that the bug I reported is valid for swift and java jobs both, so any time you're ready I can point you to where it needs to be fixed and you can work on getting that ATC status18:58
rickflareyea18:59
rickflarepm me18:59
rickflarelets do that18:59
tmckayhmmm. java thinks it has a permission error somewhere, but where?? never seen it18:59
openstackgerritChad Roberts proposed openstack/sahara-dashboard: Fixing up integration tests after UI reorganization  https://review.openstack.org/28200919:01
elmikorickflare: but it was a fail to POST in keystone though right?19:03
elmikoi still think you need to track down the request that was made to keystone and figure out why it denied permission19:03
elmikolook at user, creds, project, etc..19:04
rickflareits fine in keystone now19:09
rickflarei have the wrong password in my spark.xml19:09
rickflarenow its a permission issue again19:09
tmckayweirdest thing. read from swift, write to hdfs works on that job19:14
*** witlessb has quit IRC19:14
rickflaretmckay I just pm'd ya19:15
*** witlessb has joined #openstack-sahara19:48
*** rcernin has quit IRC19:55
*** crobertsrh is now known as _crobertsrh20:00
*** pino|work_ has joined #openstack-sahara20:09
*** pino|work has quit IRC20:12
*** kgalanov has quit IRC20:15
*** kgalanov has joined #openstack-sahara20:17
elmikotmckay, rickflare is it a public container?20:17
rickflare?20:17
rickflareyes20:17
rickflareit is20:17
elmikoso, anyone can read without perms20:17
tmckayelmiko, yeah, read is working, write is failing20:18
elmikobecause it's public20:18
tmckayactually, we might check the core-site.xml on the node to see what tenant it's trying to use20:18
elmikoyou aren't by chance writing to an object that exists already?20:18
tmckaypossible it's in 2 different tenants ...20:18
tmckaynope, we tried that20:18
elmikok20:18
elmikohad to ask20:19
tmckaycrazy thing, it was writing to output2, right?  so we watched it actually create swift://sparkstuff.sahara/output220:19
elmikooh weird...20:19
tmckayand swift://sparkstuff.sahara/output2/_temporary_file20:19
rickflareyup20:20
tmckaythen it dies with permission error with a java trace20:20
tmckayalmost like it was failing on staging an intermediate file20:20
elmikoyea20:20
tmckaycould even be in hdfs, for all I know20:20
tmckaya config issue between master and workers?20:20
elmikodidn't you mention hacking the core-site.xml file?20:22
elmikodoes it need to be distributed to all nodes?20:22
openstackgerritBrandon James proposed openstack/sahara: Check that main-class value is not null in job execution validator  https://review.openstack.org/28205020:32
*** thumpba has quit IRC20:53
*** Akanksha08 has quit IRC20:54
*** Akanksha08 has joined #openstack-sahara20:58
tmckaytellesnobrega, ping21:02
tellesnobregatmckay, pong21:02
tmckaytellesnobrega, hey. so this change right above, I tested for spark and java but not for storm. It ensures that main-class is present and not null21:03
tmckaytellesnobrega, that should be the case for storm as well, correct?  main class is required and can't be empty?21:03
tellesnobregayes21:04
tmckaySomeday we may want to support Manifest entries in certain cases that name the main class (but at least Oozie does not seem to be able to do that)21:04
tmckaytellesnobrega, okay, thanks, wanted to be sure21:04
tmckayI think from what I've read though that manifests could work with spark, without a main class config option21:05
tellesnobregatmckay, np, the way the command line for storm is implemented we require a main class, if it is missing the command line won't be complete and fail the job launch21:06
*** raildo is now known as raildo-afk21:06
tmckaytellesnobrega, gotcha, yeah if it's modeled after spark that totally makes sense21:06
tellesnobregait is21:06
tmckayI forgot that :)21:07
openstackgerritJulian proposed openstack/sahara: Add package installation methods to ssh_remote util  https://review.openstack.org/28205821:10
tellesnobregatmckay, :)21:12
*** chlong has quit IRC21:57
*** chlong_ has joined #openstack-sahara21:58
*** tmckay has left #openstack-sahara22:06
*** Akanksha08 has quit IRC22:18
*** dave-mccowan has quit IRC22:26
*** egafford has quit IRC22:27
*** witlessb has quit IRC22:39
*** egafford has joined #openstack-sahara23:15
*** jamielennox is now known as jamielennox|away23:20
*** dave-mccowan has joined #openstack-sahara23:41
*** openstackgerrit has quit IRC23:47
*** openstackgerrit_ is now known as openstackgerrit23:47
*** openstackgerrit_ has joined #openstack-sahara23:48
*** openstackgerrit_ is now known as openstackgerrit23:48
*** openstackgerrit_ has joined #openstack-sahara23:49
*** chlong_ has quit IRC23:52
*** openstackgerrit_ has quit IRC23:55
*** egafford has quit IRC23:56
*** openstackgerrit_ has joined #openstack-sahara23:57
*** sgotliv has quit IRC23:57
« Wednesday, 2016-02-17 Index Friday, 2016-02-19 »

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!