Friday, 2013-08-09

*** NikitaKonovalov has joined #savanna00:31
*** NikitaKonovalov has quit IRC00:35
*** aignatov_ has quit IRC00:46
*** dina_belova has joined #savanna00:54
*** dina_belova has quit IRC00:58
*** benl has quit IRC01:08
*** aignatov has quit IRC01:10
*** aignatov has joined #savanna01:11
*** NikitaKonovalov has joined #savanna01:31
*** NikitaKonovalov has quit IRC01:36
*** benl has joined #savanna01:39
*** benl has quit IRC01:48
*** dina_belova has joined #savanna01:54
*** dina_belova has quit IRC01:59
*** NikitaKonovalov has joined #savanna02:32
*** NikitaKonovalov has quit IRC02:36
*** IlyaE has joined #savanna02:50
*** dina_belova has joined #savanna02:55
*** shredder12 has quit IRC02:59
*** mattf is now known as _mattf03:00
*** dina_belova has quit IRC03:00
*** NikitaKonovalov has joined #savanna03:32
*** NikitaKonovalov has quit IRC03:37
*** SergeyLukjanov has joined #savanna03:46
*** dina_belova has joined #savanna03:55
*** dina_belova has quit IRC04:00
*** SergeyLukjanov has quit IRC04:11
*** dina_belova has joined #savanna04:14
*** dina_belova has quit IRC04:29
*** NikitaKonovalov has joined #savanna04:33
*** NikitaKonovalov has quit IRC04:38
*** aignatov_ has joined #savanna04:44
*** aignatov_ has quit IRC05:03
*** Nadya has joined #savanna05:22
*** bmahe has joined #savanna05:33
*** Nadya has quit IRC05:39
*** IlyaE has quit IRC06:11
*** bmahe has quit IRC06:20
*** IlyaE has joined #savanna06:21
*** bmahe has joined #savanna06:33
*** NikitaKonovalov has joined #savanna06:34
*** NikitaKonovalov has quit IRC06:34
*** NikitaKonovalov has joined #savanna06:36
*** NikitaKonovalov has quit IRC06:36
*** NikitaKonovalov has joined #savanna06:37
*** IlyaE has quit IRC07:04
*** Nadya has joined #savanna07:12
*** dina_belova has joined #savanna07:15
*** dina_belova has quit IRC07:16
*** Nadya has quit IRC07:26
*** NikitaKonovalov has quit IRC07:27
*** NikitaKonovalov has joined #savanna07:45
*** SergeyLukjanov has joined #savanna07:49
*** nmakhotkin has joined #savanna07:52
*** dmitryme has joined #savanna08:01
*** nmakhotkin has quit IRC08:02
*** dina_belova has joined #savanna08:08
openstackgerritA change was merged to stackforge/savanna: Implement to_dict() method for Resource  https://review.openstack.org/4090408:27
openstackgerritYaroslav Lobankov proposed a change to stackforge/savanna: Integration test updating for "HDP" plugin  https://review.openstack.org/4036908:30
*** nprivalova has joined #savanna08:37
*** dina_belova has quit IRC08:55
*** dmitryme_ has joined #savanna09:00
*** dmitryme has quit IRC09:01
*** dmitryme_ is now known as dmitryme09:01
*** dina_belova has joined #savanna09:14
*** dina_belova has quit IRC09:14
*** dina_belova has joined #savanna09:20
*** dina_belova has quit IRC09:36
*** NikitaKonovalov has quit IRC09:37
*** NikitaKonovalov has joined #savanna09:42
openstackgerritAlexander Kuznetsov proposed a change to stackforge/savanna: Added first version of model for EDP  https://review.openstack.org/4022709:48
openstackgerritAlexander Kuznetsov proposed a change to stackforge/savanna: Added conductor API for DataSource object  https://review.openstack.org/4059109:48
*** bmahe has quit IRC09:52
*** NikitaKonovalov has quit IRC10:03
*** dina_belova has joined #savanna10:13
*** dina_belova has quit IRC10:15
*** dina_belova has joined #savanna10:15
*** nprivalova has quit IRC10:24
*** dina_belova has quit IRC10:27
*** nprivalova has joined #savanna10:28
openstackgerritSergey Reshetnyak proposed a change to stackforge/savanna: Refactoring hdp plugin  https://review.openstack.org/3973310:34
*** nprivalova has quit IRC10:40
*** dina_belova has joined #savanna10:41
*** dina_belova has quit IRC10:41
*** NikitaKonovalov has joined #savanna10:51
*** NikitaKonovalov has quit IRC10:55
*** NikitaKonovalov has joined #savanna11:01
*** dina_belova has joined #savanna11:06
*** nprivalova has joined #savanna11:06
*** NikitaKonovalov has quit IRC11:12
*** NikitaKonovalov has joined #savanna11:14
*** benl_ has joined #savanna11:22
*** akuznetsov has joined #savanna11:31
*** dina_belova has quit IRC11:33
*** dmitryme has quit IRC11:34
*** dina_belova has joined #savanna11:42
*** ruhe has joined #savanna11:45
*** tmckayrh has quit IRC11:49
openstackgerritNikita Konovalov proposed a change to stackforge/savanna: Migration to Conductor  https://review.openstack.org/4090211:52
*** akuznetsov has quit IRC11:57
*** dina_belova has quit IRC11:57
*** NikitaKonovalov has quit IRC11:57
*** ruhe has quit IRC12:01
*** SergeyLukjanov has quit IRC12:08
*** dmitryme has joined #savanna12:14
openstackgerritA change was merged to stackforge/savanna: Added first version of model for EDP  https://review.openstack.org/4022712:21
*** nprivalova has quit IRC12:26
*** dina_belova has joined #savanna12:28
*** nprivalova has joined #savanna12:31
*** ruhe has joined #savanna12:31
*** _mattf is now known as mattf12:31
*** dina_belova has quit IRC12:36
*** dmitryme_ has joined #savanna12:49
*** dmitryme has quit IRC12:52
*** dmitryme_ is now known as dmitryme12:53
*** NikitaKonovalov has joined #savanna12:54
*** ruhe has quit IRC12:58
*** NikitaKonovalov has quit IRC12:59
openstackgerritJonathan Maron proposed a change to stackforge/savanna: Made Ambari RPM location configurable  https://review.openstack.org/4047913:05
*** nprivalova has quit IRC13:27
*** _tstclair is now known as tstclair13:30
*** tmckayrh has joined #savanna13:38
tmckayrhhello savanna folks.  I have a question on debugging savanna-ci results.  I have a failure on https://review.openstack.org/40936 and I've grepped the console log for "fail", "debug", "error" and I can't find the reason for the failure.  Any suggestions on how to understand what savanna-ci is telling me?13:40
* tmckayrh grabs snacks13:40
dmitrymetmckayrh: As you see, CI consists of 5 different jobs, of which two fail13:44
dmitrymedon't mind coverage job for now13:44
tmckayrhah, maybe all of this wasn't my fault.    File "/var/lib/jenkins/jobs/gate-savanna-coverage/workspace/savanna/db/model_base.py", line 19, in <module>     from sqlalchemy.ext import declarative ImportError: No module named ext13:45
dmitrymeit is experimental and actually should never fail13:45
dmitrymehmm, where do you see output you've just posted?13:46
tmckayrhthat's from coverage, actually.... hold on13:46
dmitrymeforget about coverage13:46
dmitrymeit is broken13:46
tmckayrhokay.13:46
dmitrymethat leaves only integration tests13:46
dmitrymeto understand, what is failing go to jenkins via the link13:47
tmckayrhokay  http://jenkins.savanna.mirantis.com/job/gate-savanna-integration/532/console13:47
dmitrymein the menu on left click on 'Console Output' [raw]13:48
dmitrymespecifically, click on [raw]13:48
tmckayrhit ends with what looks like a failure on "test_swift".13:48
dmitrymeyep, that is it13:48
tmckayrhI did change an EDP enum value of "Swift" to lowercase but that seems not likely...  still13:49
dmitrymeactually that is a known problem, timeouts for that job are not finely tuned yet, so it fails from time to time13:49
*** nprivalova has joined #savanna13:49
dmitrymeEDP should not affect the test13:49
tmckayrhdmitryme, can I run the same test from my dev environment?  I'd like to see the test log itself13:50
dmitrymeat all13:50
dmitrymeI don't think it is your fault swift test failed13:50
dmitrymeas I said, it fails from time to time13:50
tmckayrhokay.  this is what I find confusing... there is an error here, but no pointer to a detailed log13:51
dmitrymeas for running tests manually, I don't know how to do that13:51
tmckayrhI have to repatch anyway, I'll upload a new change set and see what happens.  Thanks.13:51
dmitrymeand I don't see any folks who know around13:51
dmitrymeyep, I think that is the best approach13:51
tmckayrhI appreciate the help.13:52
dmitrymeno problem :-)13:52
tmckayrhnprivalova, ping13:52
*** nprivalova has quit IRC14:01
*** dmitryme has quit IRC14:08
*** benl_ has quit IRC14:08
*** dmitryme has joined #savanna14:09
*** alazarev has joined #savanna14:12
*** benl_ has joined #savanna14:12
*** akuznetsov has joined #savanna14:15
*** nprivalova has joined #savanna14:16
nprivalovahi tmckayrh14:16
*** ruhe has joined #savanna14:16
tmckayrhakuznetsov, I saw your note on credentials for edp -- nprivalova also mentioned alternate auth methods yesterday.  I am looking at how to use "oneof" in a json schema so that we can have different credential forms but still validate14:17
tmckayrhthat would be ideal14:17
tmckayrhIf I can't make it work, I'll just loosen the restrictions14:17
nprivalovatmckayrh, I think we should consider only user+password case14:19
*** alazarev has left #savanna14:19
akuznetsovtmckayrh I think for now it will enough to object type for this field, as for data source component14:20
nprivalovatmckayrh, I told you about trust-mechanism but we need to investigate it first14:21
tmckayrhnprivalova, well, proof of concept then?  I can always leave "oneof" with a single option, for later expansion.  akuznetsov mentioned "what about tokens in the future?" as you did.14:21
*** alazarev has joined #savanna14:21
*** alazarev has quit IRC14:21
*** benl_ has quit IRC14:24
nprivalovatmckayrh, proof, yes14:26
*** ruhe has quit IRC14:30
*** IlyaE has joined #savanna14:32
*** benl_ has joined #savanna14:33
*** ruhe has joined #savanna14:37
openstackgerritNikolay Mahotkin proposed a change to stackforge/savanna: Added Hive configuration, new nodeprocess - hiveserver  https://review.openstack.org/4084714:48
*** akuznetsov has quit IRC14:49
aignatovtmckayrh, ping14:57
aignatovcould you take a look at my patch https://review.openstack.org/#/c/40590/14:58
tmckayrhaignatov, yes, sure14:59
tmckayrhotp at the moment15:00
*** benl_ has quit IRC15:02
* tmckayrh has triumped with the "oneOf" keyword in json15:02
tmckayrhtriumphed even15:02
openstackgerritNikolay Mahotkin proposed a change to stackforge/savanna: Added Hive configuration, new nodeprocess - hiveserver  https://review.openstack.org/4084715:02
tmckayrhaignatov, oops, I missed the email where I was assigned to be a reviewer :)  I should change my mail filter...15:03
aignatovnot a problem, anyway I'll ping you if needed ^_^15:09
*** bogdando has quit IRC15:23
*** sacharya has joined #savanna15:26
openstackgerritNadya Privalova proposed a change to stackforge/savanna: Added conductor API for Job Object  https://review.openstack.org/4111915:27
openstackgerritNadya Privalova proposed a change to stackforge/savanna: Added conductor API for Job Object  https://review.openstack.org/4111915:31
openstackgerritNadya Privalova proposed a change to stackforge/savanna: Added conductor API for Job Object  https://review.openstack.org/4111915:37
*** rnirmal has joined #savanna15:38
openstackgerritDmitry Mescheryakov proposed a change to stackforge/savanna: Conductor objects are re-populated on update  https://review.openstack.org/4112215:38
sacharyaHi all15:42
tmckayrhhi15:42
sacharyaIs there a more flushed out version of this https://etherpad.openstack.org/savanna_API_draft_EDP_extensions15:42
tmckayrhsacharya, work in progress.  In fact, I am just committing the JobOrigin api now.  The best doc at the moment is code15:43
tmckayrhsacharya, do you have a specific question?  Maybe I know the answer :)15:44
openstackgerritTrevor McKay proposed a change to stackforge/savanna:     Add initial version of the REST api for the job origin component  https://review.openstack.org/4093615:44
sacharyaIts hard to say without looking at the request response for each of those ;)15:44
sacharyaThere are so many different ways of executing a job in Hadoop15:45
tmckayrhsacharya, yes, I see your point :)  Okay, so the response will be dependent on the database object that is defined.  I am about to start working on defining the JobOrigin object, I think the other edp objects do not have database models yet either.15:46
tmckayrhas you can see. the EDP work in savanna is very new15:46
sacharyaYeah, thats fine. I just wanted to make sure I was not missing some secret doc somewhere15:47
aignatovbtw, sacharya, here is another doc https://etherpad.openstack.org/edp_v3_components15:47
tmckayrhnope :)  We are zeroing in on what reality will be15:47
sacharyaThanks aignatov15:48
aignatovregarding different ways of job executing, we'll try to support most popular ones - pig, hive, jar files)15:48
sacharyahow about streaming ?15:49
sacharyapython scripts15:49
*** nprivalova has quit IRC15:49
aignatovnot sure about python scrits, but streaming and pips should be there :)15:50
aignatov*pipes15:50
sacharyathat will cover it15:50
aignatovah, ok15:50
sacharyastupid question tmckayrh15:52
sacharyawhat is a data-source? say a swift endpoint or a particular container in swift that will act as a hadoop input ?15:52
tmckayrhsacharya, not a stupid question :)  A data-source is a particular file, for lack of a better term.  Maybe a dir, for data (I could imagine processing a whole dir)15:53
*** akuznetsov has joined #savanna15:53
aignatovi think is's a particular container15:53
sacharyaIn that case, do we need another api call to list supported data-source types eg swift, hdfs, db etc?15:55
tmckayrhsacharya, could be something other than swift (hdfs).  But in swift terms, I would think it was a container or an object.  Same for output.15:55
tmckayrhsacharya, we have talked about that -- an ability to browse "endpoints" that the system currently knows about15:56
tmckayrhto answer the question "hey, I'm new here.  Where can I find stuff to process, and where can I put my results?"15:56
tmckayrhsimilar question for "where can I store my job binaries anyway?"15:57
tmckayrhnot strictly necessary at first during development -- you just "know" what you can use, or ask the site admin :)15:57
sacharyayeah, i agree15:58
tmckayrhsacharya, feel free to post comments on the etherpad -- that's not a formal doc15:58
tmckayrhjust throw up an irc nick or an email so we can find you again :)15:59
sacharyaok sure15:59
aignatovright, but don't forget to add tag with your nickname)15:59
* tmckayrh wonders if he has done that everywhere....15:59
aignatovhaha :)16:00
*** akuznetsov has quit IRC16:00
aignatovtmckayrh, I have to go now, but I'll look at your patchset more deeply on the weekend)16:01
tmckayrhaignatov, okay, thanks.  I'm going to start working on the db object definition....16:01
*** ruhe has quit IRC16:02
*** benl_ has joined #savanna16:02
*** benl_ has quit IRC16:09
openstackgerritNikolay Mahotkin proposed a change to stackforge/savanna: Added Hive configuration, new nodeprocess - hiveserver  https://review.openstack.org/4084716:11
*** dmitryme has quit IRC16:11
sacharyatmckayrth, i am assuming these job apis can be configured before user has a cluster? only the execute action needs a cluster.16:21
sacharyatmckayrh16:21
tmckayrhsacharya, yes, that sounds right16:26
sacharyai was wondering why you need to store the job binaries in database, that looks like the reason.16:26
tmckayrhsacharya, we are postulating that jobs (binaries at first) can be stored in swift or in the savanna sqlite database, or an external hdfs and then will be moved to the hadoop cluster for execution16:27
tmckayrhyes, the db stores the location information so that savanna can be used to move things around later16:28
sacharyamakes sense16:29
tmckayrh2 things going on there -- 1) the job record, which contains a uri of some sort and 2) in some instances, the actual job binary itself will also be in the db (but the preferred method is to have it somewhere else)16:29
*** ruhe has joined #savanna16:52
*** ruhe has quit IRC16:52
*** ruhe has joined #savanna17:00
*** ruhe has quit IRC17:05
*** IlyaE has quit IRC17:12
openstackgerritJohn Speidel proposed a change to stackforge/savanna: Allow Ambari users to be specified in configuration  https://review.openstack.org/4114617:14
openstackgerritJohn Speidel proposed a change to stackforge/savanna: Allow Ambari users to be specified in configuration  https://review.openstack.org/4114617:29
*** akuznetsov has joined #savanna17:38
*** akuznetsov has quit IRC17:49
*** IlyaE has joined #savanna17:55
*** benl_ has joined #savanna17:58
*** IlyaE has quit IRC18:02
*** dmitryme has joined #savanna18:03
*** Nadya has joined #savanna18:04
*** akuznetsov has joined #savanna18:04
mattfso much for back to talk in the AM ET. sorry folks, i've been otp all day.18:06
*** dmitryme has quit IRC18:25
*** dmitryme has joined #savanna18:34
*** akuznetsov has quit IRC18:40
*** hub_cap has left #savanna18:51
*** tmckayrh has quit IRC18:51
*** dmitryme has quit IRC19:45
*** tmckayrh has joined #savanna19:52
*** benl_ has quit IRC20:21
*** dmitryme has joined #savanna20:32
*** IlyaE has joined #savanna20:38
*** benl_ has joined #savanna20:46
*** rnirmal has quit IRC21:24
*** tstclair is now known as _tstclair21:36
*** sacharya has quit IRC21:40
*** dmitryme has quit IRC21:43
tmckayrhanyone around handy with sqlalchemy and know why I get this?  I added a table, but apparently not completely :)21:52
tmckayrh  File "/home/tmckay/src/savanna/.tox/venv/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 1397, in add21:52
tmckayrh    raise exc.UnmappedInstanceError(instance)21:52
tmckayrhUnmappedInstanceError: Class 'savanna.db.models_edp.JobOrigin' is not mapped21:52
tmckayrh:) got it.  Forgot to derive from SavannaBase21:57
tmckayrh__tablename__ attribute was missing21:57
Nadyatmckayrh, I believe you forget to add JobOrigin in one of 4(or 6) layers...21:57
Nadyaoh. ok :)21:57
tmckayrh4 or 6 layers?  uh oh :)21:57
tmckayrhSo, I apparently created a record.21:57
tmckayrhI wonder if it's persisting21:58
Nadyalook at patches from me or from Alex K21:59
Nadyathere is tests on models21:59
tmckayrhah, okay, thanks21:59
NadyaI think it may help21:59
tmckayrhIt's late there....22:00
tmckayrhor early :)22:01
Nadyalate...:)22:09
tmckayrhis there a controlled way to stop savanna-api running in tox, or is it ctrl-c safe?22:13
tmckayrhmy firefox SQLiteManager plugin is showing the JobOrigin, yay22:14
tmckayrhbbl maybe22:21
tmckayrh_tmckayrh22:21
Nadyactrl-c is perfect!22:43
*** _tstclair has quit IRC22:47
*** Nadya has quit IRC22:47
*** IlyaE has quit IRC22:49
*** _tstclair has joined #savanna22:52
*** _tstclair has quit IRC22:57
*** IlyaE has joined #savanna23:08
*** benl_ has quit IRC23:09
*** tstclair has joined #savanna23:18
*** IlyaE has quit IRC23:35
*** sacharya has joined #savanna23:47

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!