Tuesday, 2013-10-15

*** matsuhashi has joined #savanna00:48
*** sacharya has joined #savanna00:56
*** IlyaE has joined #savanna00:59
*** IlyaE has quit IRC01:01
*** nosnos has joined #savanna01:05
*** sacharya has quit IRC01:16
*** sacharya has joined #savanna01:19
*** SergeyLukjanov has joined #savanna03:33
*** sacharya has quit IRC03:53
*** SergeyLukjanov is now known as _SergeyLukjanov03:58
*** sacharya has joined #savanna03:58
*** _SergeyLukjanov has quit IRC03:58
*** sacharya has quit IRC04:45
*** matsuhashi has quit IRC05:11
*** SergeyLukjanov has joined #savanna05:11
*** matsuhashi has joined #savanna05:11
*** matsuhashi has quit IRC05:13
*** matsuhashi has joined #savanna05:14
*** matsuhashi has quit IRC05:22
*** matsuhashi has joined #savanna05:23
*** matsuhashi has quit IRC05:27
*** matsuhashi has joined #savanna05:34
*** akuznetsov has quit IRC07:28
*** dafter has joined #savanna07:38
*** dafter has joined #savanna07:38
*** SergeyLukjanov has quit IRC07:46
*** DinaBelova has joined #savanna08:03
*** akuznetsov has joined #savanna08:14
*** matsuhas_ has joined #savanna08:14
*** matsuhashi has quit IRC08:15
*** tvb|afk has joined #savanna08:15
*** matsuhashi has joined #savanna08:15
*** dafter has quit IRC08:19
*** matsuhas_ has quit IRC08:19
*** tvb|afk has quit IRC08:49
*** DinaBelova has quit IRC09:08
*** tmckay has joined #savanna09:30
*** openstackgerrit has quit IRC09:38
*** dafter has joined #savanna09:42
*** dafter has quit IRC09:43
*** DinaBelova has joined #savanna09:45
*** tvb|afk has joined #savanna09:49
*** akuznetsov has quit IRC10:09
*** akuznetsov has joined #savanna10:10
*** nprivalova has quit IRC10:22
*** DinaBelova has quit IRC10:27
*** nosnos has quit IRC10:28
*** nosnos has joined #savanna10:28
*** nosnos has quit IRC10:33
*** tvb|afk has quit IRC10:37
*** dafter has joined #savanna10:37
*** tmckay has quit IRC10:42
*** tvb|afk has joined #savanna10:42
*** dafter has quit IRC10:43
*** nprivalova has joined #savanna10:52
*** akuznetsov has quit IRC10:53
*** DinaBelova has joined #savanna10:59
*** tvb|afk has quit IRC11:01
*** dafter has joined #savanna11:02
*** nprivalova has quit IRC11:20
*** dmitryme has joined #savanna11:27
*** tvb|afk has joined #savanna11:28
*** dafter_ has joined #savanna11:29
*** dafter_ has joined #savanna11:30
*** dafter has quit IRC11:32
*** dafter_ has quit IRC11:32
*** tvb|afk has quit IRC11:33
*** nprivalova has joined #savanna11:36
*** dmitryme has quit IRC11:38
*** akuznetsov has joined #savanna11:51
*** dafter has joined #savanna11:52
*** tvb|afk has joined #savanna11:57
*** tvb|afk has joined #savanna11:57
*** akuznetsov has quit IRC12:00
*** dafter has quit IRC12:00
*** dmitryme has joined #savanna12:09
*** nprivalova has quit IRC12:13
*** nprivalova has joined #savanna12:21
*** akuznetsov has joined #savanna12:34
*** openstackgerrit has joined #savanna13:02
*** tvb|afk has quit IRC13:04
*** dafter has joined #savanna13:07
*** ruhe has joined #savanna13:11
*** _crobertsrh is now known as crobertsrh13:13
*** dafter has quit IRC13:15
*** dafter has joined #savanna13:15
*** tvb|afk has joined #savanna13:18
*** dafter has quit IRC13:20
*** dmitryme has quit IRC13:26
*** tmckay has joined #savanna13:28
*** tmckay has left #savanna13:28
*** tmckay has joined #savanna13:29
*** ruhe has quit IRC13:29
*** ruhe has joined #savanna13:30
tmckaySo I was starting up a cluster from savanna.  The  instances were created (1 master, 1 worker) and the cluster was in the Waiting status for a long time.  Eventually it went to error.  When I use the REST api to retrieve the cluster info, it contains this:13:32
tmckay           "status": "Error",13:33
tmckay            "status_description": "Unauthorized (HTTP 401)",13:33
tmckayDoes anybody know what this means?13:33
tmckaycrobertsrh, this is where I ended up last evening ^^13:33
* tmckay goes to look through devstack logs13:33
crobertsrhI haven't seen that one13:38
tmckaysomething is unauthorized, but I don't know what...13:40
tmckaymaybe I should try again, it's been looping all night.13:40
crobertsrhsecurity rules set up to allow ssh?13:41
*** ruhe has quit IRC13:43
*** DinaBelova has quit IRC13:46
*** ruhe has joined #savanna13:49
*** tmckay has quit IRC13:51
aignatovtmckay, may be token is expired?13:54
*** nprivalova is now known as nadya13:56
*** dmitryme has joined #savanna14:03
*** ruhe has quit IRC14:06
*** ruhe has joined #savanna14:11
*** tmckay has joined #savanna14:15
*** SergeyLukjanov has joined #savanna14:16
*** akuznetsov has quit IRC14:24
*** sacharya has joined #savanna14:27
*** SergeyLukjanov has quit IRC14:32
*** ruhe has quit IRC14:33
*** matsuhashi has quit IRC14:34
*** ruhe has joined #savanna14:35
*** matsuhashi has joined #savanna14:35
*** akuznetsov has joined #savanna14:36
*** SergeyLukjanov has joined #savanna14:38
*** SergeyLukjanov is now known as _SergeyLukjanov14:38
*** matsuhashi has quit IRC14:39
*** _SergeyLukjanov has quit IRC14:39
*** sacharya has quit IRC14:41
*** SergeyLukjanov has joined #savanna14:52
*** rnirmal has joined #savanna15:02
*** sacharya has joined #savanna15:13
*** mattf is now known as _mattf15:18
*** ruhe has quit IRC15:26
*** sreshetnyak has quit IRC15:31
*** ruhe has joined #savanna15:33
openstackgerritAlexander Ignatov proposed a change to openstack/savanna: Added REST API v1.1 section  https://review.openstack.org/5094615:36
*** DennyZhang has joined #savanna15:36
openstackgerritA change was merged to openstack/savanna: Update end time in job execution after job complete  https://review.openstack.org/5180915:40
openstackgerritA change was merged to openstack/savanna: Unconditionally monkey patch nova.keypairs.get  https://review.openstack.org/5169315:42
*** akuznetsov has quit IRC15:52
crobertsrhDid the validation rules for job binary file names change?  hadoop-examples-1.2.1.jar is not a valid file name?  Are "-" no longer allowed?15:52
*** DennyZhang has quit IRC15:55
*** crobertsrh is now known as crobertsrh-lunch15:56
*** tmckay has quit IRC15:57
aignatovmmm, let me check16:01
aignatovyes, "error_message": "u'hadoop-examples-1.2.1.jar' is not a 'valid_name'",16:02
aignatovalso the name "hadoopexamples1.1jar" is not allowed16:04
aignatovbecause of digit after dot16:04
*** nadya has quit IRC16:04
aignatovI think we shoud change it, but it seems not in 0.3 release16:05
*** ruhe has quit IRC16:05
SergeyLukjanovoops, yeah, looks like too strong validation16:06
*** dmitryme has quit IRC16:08
*** tvb|afk has quit IRC16:16
*** SergeyLukjanov has quit IRC16:20
*** _mattf is now known as mattf16:21
mattf(OperationalError) no such column: clusters.trust_id <-- anyone remember off the top of their head how to resolve this? it's a savanna update. the db is old.16:22
mattfsavanna-db-manage db-sync is no help16:24
*** SergeyLukjanov has joined #savanna16:41
*** crobertsrh-lunch is now known as crobertsrh16:41
*** SergeyLukjanov is now known as _SergeyLukjanov16:42
*** _SergeyLukjanov is now known as SergeyLukjanov16:42
*** SergeyLukjanov is now known as _SergeyLukjanov16:42
*** _SergeyLukjanov is now known as SergeyLukjanov16:42
*** ruhe has joined #savanna16:43
*** akuznetsov has joined #savanna17:07
*** dmitryme has joined #savanna17:18
*** DennyZhang has joined #savanna17:25
*** tmckay has joined #savanna17:26
*** akuznetsov has quit IRC17:26
*** dafter has joined #savanna17:27
*** dafter has quit IRC17:32
openstackgerritDmitry Mescheryakov proposed a change to openstack/savanna: Remove KeypairManager.get monkey patch  https://review.openstack.org/5173417:37
*** openstackgerrit has quit IRC17:37
*** openstackgerrit has joined #savanna17:38
openstackgerritSergey Lukjanov proposed a change to openstack/savanna-extra: Add example edp jobs folder and pig example  https://review.openstack.org/5189217:47
*** sreshetnyak has joined #savanna17:47
*** DennyZhang has quit IRC17:49
openstackgerritSergey Lukjanov proposed a change to openstack/savanna-extra: Add simple tox.ini to make gate working  https://review.openstack.org/5189517:50
openstackgerritSergey Lukjanov proposed a change to openstack/savanna-image-elements: Add simple tox.ini to make gate working  https://review.openstack.org/5189617:50
openstackgerritA change was merged to openstack/savanna-image-elements: Add simple tox.ini to make gate working  https://review.openstack.org/5189617:51
openstackgerritA change was merged to openstack/savanna-extra: Add simple tox.ini to make gate working  https://review.openstack.org/5189517:52
openstackgerritSergey Lukjanov proposed a change to openstack/savanna-extra: Add example edp jobs folder and pig example  https://review.openstack.org/5189217:53
tmckayhey folks, I have 2 instances from a cluster launced from savanna, which I can ssh to, but the cluster status is still "Waiting" and has been for a long time.  What is savanna trying to do to connect to them?18:23
tmckaythis is where I'm getting to lately, and ultimately it errors out18:23
SergeyLukjanovtmckay, hi18:24
SergeyLukjanovtmckay, possible it could be related to the floating ips config18:24
SergeyLukjanovwhich version of OS are you using?18:24
SergeyLukjanovand nova-network or neutroin18:24
SergeyLukjanovneutron*18:24
tmckaymy host is F19, my images are fedora18:24
dmitrymecan you post your logs via paste18:25
tmckayI think I'm using the default network18:25
tmckayunfortunately I have savanna logging to the screen18:25
tmckayI should change that18:25
dmitrymeyep18:25
tmckayopenstack I have logging to disk18:25
dmitrymealso can you rebase to the latest master18:25
dmitrymeit has additional logging18:25
tmckayokay.  Is there a place in the savanna code you can point me for the command it is trying to use now to connect?  If it's a shell command maybe I can debug it.18:26
dmitrymeSince it is hanging in "Waiting" state, I think it should be in the following method: https://github.com/openstack/savanna/blob/master/savanna/service/instances.py#L3718:27
tmckaycool, thanks.  Hmm, not sure if I have any floating ips assigned.  how can I check?  Each instance only shows an address from the "fixed" range in horizon18:28
tmckayah, I see, maybe stuck in _await_active (hence no floating ip)18:29
SergeyLukjanovtmckay, are instance avaiable from the host where savanna installed through the internal ips?18:30
dmitrymeanother question, how value 'use_floating_ips' in config is set to?18:31
tmckayyes.  I have a fixed range of 10.0.3.x, and I can ssh -l fedora from the savanna host to 10.0.3.2 and 10.0.3.318:31
tmckaydmitryme, commented out18:32
tmckay#use_floating_ips=True18:32
*** ruhe has quit IRC18:34
dmitrymeI think it should start working if you uncomment it and set to false18:34
tmckayokay, I'll try18:34
SergeyLukjanovyeah, the default value is true18:34
tmckaywhy would that break it?18:35
tmckay_check_if_active looks pretty simple18:35
SergeyLukjanovit waits for floating ips to access vms18:35
*** SergeyLukjanov has quit IRC18:35
tmckayso maybe I'm stuck in _assign_floating_ips? or _await_networks?18:36
dmitrymeit would work your current config if you had nova-network and auto_assign_floating_ip=True in your nova.conf18:36
dmitrymein that case floating IPs would be assigned automatically by Nova18:37
tmckayokay, thanks.  I'll check that.  crobertsrh was just mentioning something about that too18:37
dmitrymeif there is no auto assignment, then as far as I recall you need to pass pool_id, or something like that18:38
dmitrymewith each node group18:38
dmitrymeSavanna will itself assign a floating IP taken from that pool18:39
*** SergeyLukjanov has joined #savanna18:39
tmckayokay.  So next question, is it safe to "delete cluster" to get this out of the waiting state?18:39
*** ruhe has joined #savanna18:41
tmckayalright, retrying with use_floating_ips=False18:44
tmckaydmitryme, oh my!  It's configuring!18:46
tmckayAnd Active!  Are you kidding me?18:46
dmitrymegreat to hear that :-)18:47
SergeyLukjanovtmckay, it's safe enough to delete cluster in waiting state18:47
tmckaySergeyLukjanov, thanks.  It was my floating_ip config.  Thank you very much.  I think I had other issues earlier with nova sec groups, etc.  Maybe I can run a job now.18:49
tmckaySergeyLukjanov, also, thanks for the pointer to the "create_cluster" routine.  Seems like a useful place for debug.18:51
crobertsrhWhile trying to run the pig demo, I get the following (in both fedora and ubuntu image, so it seems like something I'm doing incorrectly):  http://fpaste.org/46992/81863631/ (oozie log).  My job get to RUNNING state briefly, then KILLED.  Thanks for any tips!19:01
*** ruhe has quit IRC19:04
*** dmitryme has quit IRC19:07
mattfSergeyLukjanov, https://review.openstack.org/#/c/51892 -- should the output be checked in?19:34
crobertsrhaignatov:  is there anything special required from your cluster config to successfully run your pig demo?  Both tmckay and I are trying to run it, but our job winds-up as "KILLED".19:39
*** SergeyLukjanov is now known as _SergeyLukjanov19:46
*** _SergeyLukjanov has quit IRC19:47
openstackgerritA change was merged to openstack/savanna: Remove KeypairManager.get monkey patch  https://review.openstack.org/5173419:49
*** ruhe has joined #savanna19:53
crobertsrhHmm, the pig job fails with error message:  Workflow failed, error message[Main class [org.apache.oozie.action.hadoop.PigMain], exit code [2]]19:57
*** SergeyLukjanov has joined #savanna19:58
*** ruhe has quit IRC20:00
tmckaycrobertsrh, yes, I was just looking at main class messages before an emgency dog walk20:10
*** SergeyLukjanov has quit IRC20:11
crobertsrhtmckay:  A useful tool...the oozie web UI  http://<your master ip>:11000/oozie/20:14
tmckaycrobertsrh, where did you find that PigMain error?20:15
crobertsrhclick down a few levels in the web UI20:15
crobertsrhI actually originally found it in oozie.log, but the web UI is nicer20:15
tmckayah, there we go.  JA01820:23
tmckaythat's got to mean something20:24
crobertsrhYep, same code I see20:24
crobertsrhI wonder if our images are just out of date?20:24
*** rnirmal has quit IRC20:26
tmckayfound my way to jobdetails.  oozie.launcher.error counter is 1 on Map.  I would guess this means that Oozie doesn't know how to find the main class for Map.20:30
*** dmitryme has joined #savanna20:45
crobertsrhAh, finally got my job to run.....question.  Is it a bug or a feature that the name that I give my "job binary" is actually used in the execution of the job?  My problem was that I called gave udf.jar a name of "udf-jar" when I created my binary.  When I called it "udf.jar", it started working properly.20:51
*** mattf is now known as _mattf20:54
*** SergeyLukjanov has joined #savanna21:12
*** crobertsrh is now known as _crobertsrh21:13
*** dmitryme has quit IRC21:18
*** sreshetnyak has quit IRC21:20
*** sacharya has quit IRC22:28
*** SergeyLukjanov is now known as _SergeyLukjanov22:34
*** _SergeyLukjanov has quit IRC22:34
*** SergeyLukjanov has joined #savanna22:36
*** SergeyLukjanov has quit IRC22:44
*** akuznetsov has joined #savanna23:20
*** tmckay has quit IRC23:59

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!