Friday, 2014-05-09

*** IlyaE has joined #openstack-sahara00:26
*** witlessb has quit IRC00:27
*** ViswaV has quit IRC00:33
*** tmckay has joined #openstack-sahara01:36
*** witlessb has joined #openstack-sahara01:56
*** witlessb has quit IRC02:00
*** tmckay has quit IRC02:20
*** IlyaE has quit IRC02:25
*** bradd1 has joined #openstack-sahara02:25
openstackgerritOpenStack Proposal Bot proposed a change to openstack/sahara: Updated from global requirements  https://review.openstack.org/9295502:29
*** IlyaE has joined #openstack-sahara02:34
*** mattf is now known as _mattf02:41
*** _mattf is now known as mattf02:43
*** mattf is now known as _mattf02:44
openstackgerritOpenStack Proposal Bot proposed a change to openstack/python-saharaclient: Updated from global requirements  https://review.openstack.org/9295902:49
openstackgerritOpenStack Proposal Bot proposed a change to openstack/sahara: Updated from global requirements  https://review.openstack.org/9295502:49
openstackgerritOpenStack Proposal Bot proposed a change to openstack/sahara-dashboard: Updated from global requirements  https://review.openstack.org/9296002:49
*** IlyaE has quit IRC02:54
*** IlyaE has joined #openstack-sahara02:55
*** IlyaE has quit IRC03:08
*** ViswaV has joined #openstack-sahara03:25
*** ViswaV_ has joined #openstack-sahara03:26
*** ViswaV has quit IRC03:26
*** ViswaV has joined #openstack-sahara03:28
*** ViswaV_ has quit IRC03:31
*** bradd1 has quit IRC03:33
*** IlyaE has joined #openstack-sahara04:17
*** Ch00k has joined #openstack-sahara04:34
*** ViswaV has quit IRC04:42
*** IlyaE has quit IRC04:50
*** sballe__ has joined #openstack-sahara04:56
*** sballe_ has quit IRC04:59
*** IlyaE has joined #openstack-sahara05:12
*** bradd1 has joined #openstack-sahara05:33
*** IlyaE has quit IRC05:38
*** IlyaE has joined #openstack-sahara06:05
*** bradd1 has joined #openstack-sahara06:35
*** zhiyan_ is now known as zhiyan06:42
*** bradd1 has joined #openstack-sahara06:53
*** IlyaE has quit IRC06:59
*** IlyaE has joined #openstack-sahara06:59
*** IlyaE has quit IRC07:03
*** zhiyan is now known as zhiyan_07:05
*** witlessb has joined #openstack-sahara07:33
*** tnovacik has joined #openstack-sahara07:34
*** tnovacik is now known as tnovacik|gone07:51
*** tnovacik|gone has quit IRC07:55
*** bradd1 has joined #openstack-sahara07:59
*** bradd1 has quit IRC07:59
*** Ch00k has quit IRC08:16
*** Ch00k has joined #openstack-sahara10:17
*** bradd1 has joined #openstack-sahara10:35
*** _crobertsrh is now known as crobertsrh12:34
*** elmiko_ is now known as elmiko12:45
*** bradd1 has joined #openstack-sahara13:00
*** bradd1 has quit IRC13:05
*** bradd1 has joined #openstack-sahara13:12
*** bradd1 has quit IRC13:17
*** tmckay has joined #openstack-sahara13:24
openstackgerritChad Roberts proposed a change to stackforge/puppet-sahara: Adding RHEL support for puppet-sahara  https://review.openstack.org/9236813:27
*** bradd1 has joined #openstack-sahara14:22
*** _mattf is now known as mattf14:23
tmckayHi all.  Trying to launch an hdp 2.0.6 cluster, it's in "Waiting" with active instances so I may be all right.  But, in case it fails, does anyone have a list handy of the node processes for master/worker I need for HDP 2.0.6 for a cluster that can run Java and Pig actions?  Not too familiar with HDP layout.14:25
tmckaythanks to elmiko, in another window http://elmiko.github.io/openstack/sahara/2014/04/24/centos-hdp2-cluster-creation-debug-notes.html14:27
tmckayaignatov, taking another run through slides for spelling, etc.  Tell me, if I want to add a slide or two to introduce the demo, can I do it right from the share?14:45
*** ViswaV has joined #openstack-sahara15:14
*** ViswaV_ has joined #openstack-sahara15:16
*** ViswaV has quit IRC15:18
*** tmckay1 has joined #openstack-sahara15:25
*** tmckay has quit IRC15:27
*** tmckay1 has left #openstack-sahara15:28
*** tmckay has joined #openstack-sahara15:28
*** mattf is now known as _mattf15:31
*** ViswaV_ has quit IRC15:35
*** ViswaV has joined #openstack-sahara15:47
*** IlyaE has joined #openstack-sahara15:54
tmckayso my first attempt at a Java action failed, but I left out mapreduce2client on the workers. Hoping that was the issue.  2nd call for node process lists for HDP 2.0.6 cluster that can run Java/Pig.15:56
tmckayaignatov, ping, you there?15:59
*** crobertsrh has left #openstack-sahara16:04
*** crobertsrh has joined #openstack-sahara16:04
*** elmiko is now known as elmiko_afk16:15
aignatovtmckay: hey16:19
aignatovfeel free to update slides16:19
aignatov:)16:19
*** bradd1 has quit IRC16:19
*** bradd1 has joined #openstack-sahara16:19
tmckayaignatov, yeah, adding some demo intro slides16:19
tmckayaignatov, I have 1 feedback item after review (copied from email I was going to send you)16:20
aignatovif you thihnk that those slides will help for your spech adding new slides is strongly recommended :)16:20
tmckay- Supported job types, slide 12:16:20
tmckayCurrently says "Jar", "Hive", "Pig".16:20
tmckayShould this say Java, MapReduce, Streaming MapReduce, Hive, and Pig?  Also, do we want to claim Hive?  Maybe with an asterisk, since it only works on vanilla 1.2.1 as far as I know.16:20
tmckayYeah, just an intro, then we can switch to the web browser16:21
aignatovif it works even in 1.2.1 that we can say that Sahara suports it :)16:21
aignatovas for MapReduce different types I will fix that, thx16:22
tmckayaignatov, okay.  And "Jar" should be "Java" if we want to align with the pulldowns :)16:23
tmckayaignatov, do you like my 2 demo slides?  Anything else you think I should add?16:24
tmckayaignatov, and any expertise you have about EDP on hdp 2.0.6 is welcome.  So far, my jobs fail.16:25
tmckayI may have to use vanilla16:25
aignatovtmckay: use vanilla!16:25
aignatov:)16:25
tmckayaignatov, what would be really cool, if I can make it work, is to run on vanilla, then use relaunch to run the same job on hdp :)16:26
tmckayThat would be great.16:26
aignatovyes, but do yuo think you’ll have a time to do all of that?16:27
aignatovyou planned16:27
aignatovah, you will use already provisioned clusters for the demo :)16:28
tmckayyes16:28
tmckayGoing to skip over the cluster launch, assuming that is old news from last summit :)16:28
tmckayI can briefly show the templates16:29
aignatovactually it’s not needed :)16:29
aignatovtemplates and clusters is the old things right16:30
tmckayaignatov, I plan to build 1 of each type of object (Job Binary, Data Source, Job) and have all of the rest of the objects ready to go16:30
tmckayThen I can just launch them, and if there's time I can show a relaunch16:31
tmckayvoila16:31
tmckayI think I'll add canned data to a slide, so I don't have to log in to hdfs to show it16:31
tmckayelmiko_afk, if I've launched an hdp cluster, what's the login for the ambari console?16:33
tmckayaignatov, ha!  oozie said bigpetstore worked on hdp.  But I have to find the data, there is no hadoop user ...16:40
* tmckay goes on a treasure hunt16:41
tmckayelmiko_afk, ^^16:41
tmckay/user/hdfs/bigpetstore16:42
tmckaycrobertsrh, I'm so excited I can hardly stand it16:43
crobertsrhHeh, a very nerdy christmas :)16:43
*** ViswaV has quit IRC16:43
tmckaylol16:43
*** ViswaV has joined #openstack-sahara16:44
elmiko_afktmckay: just happened to walk by, admin/admin16:44
tmckaycool, thanks16:44
*** ViswaV has quit IRC16:45
*** ViswaV has joined #openstack-sahara16:45
openstackgerritChad Roberts proposed a change to stackforge/puppet-sahara: Adding RHEL support for puppet-sahara  https://review.openstack.org/9236816:54
tmckaycrobertsrh, will relaunch work for a job that is pending/running?  do you check the status?  I'm think I can launch on vanilla, then immediately turn around and relaunch on hdp.17:02
tmckaycool17:02
crobertsrhYeah, it should work just fine17:02
crobertsrhIt will not kill the original job17:02
tmckayrun in parallel.  that would be boss17:02
crobertsrhjust start another job17:02
tmckaythanks17:02
*** elmiko_afk is now known as elmiko17:14
elmikotmckay: did that work for you?17:14
tmckayyeah, I got the ambari console.17:15
tmckayI wish I knew why the routing for the other consoles fails17:16
tmckaysome network guru thing17:16
*** IlyaE has quit IRC17:16
elmikolike, you can't route between the nodes?17:19
*** ViswaV_ has joined #openstack-sahara17:22
*** ViswaV has quit IRC17:22
*** ViswaV has joined #openstack-sahara17:23
*** ViswaV_ has quit IRC17:23
*** ViswaV_ has joined #openstack-sahara17:24
*** ViswaV has quit IRC17:24
*** ViswaV_ has quit IRC17:30
*** IlyaE has joined #openstack-sahara17:43
*** ViswaV has joined #openstack-sahara17:57
aignatovtmckay: in hdp there is ‘hdfs’ user18:03
tmckayaignatov, yeah, found that :)18:05
tmckayaignatov, weird, my hdp cluster can run the jobs, but something is wrong.  2 min for generation on vanilla, 4 min for clean, 3 minutes for analyze.  On hdp, 12 minutes for generation, 6 for clean, 6 for analyze18:07
tmckayno idea what the difference is18:08
tmckaywell, one is hadoop 2.0.6, one is 2.3.018:08
tmckaymaybe 2.3.0 is faster18:08
aignatov2.0.6 is a version of plugin not hadoop :)18:08
tmckayhmm, okay18:09
aignatovwhich flavor are using with HDP nodes?18:09
tmckaylarge, on everything18:10
tmckayhadoop version says Hadoop 2.2.0.2.0.6.0-10118:10
tmckayso, slightly diferent than 2.3.0.  I don't know where to being with hadoop performance eval, so we'll just call it slow18:15
tmckayhmm, I can still do my relaunch trick with a second vanilla cluster18:15
*** elmiko has quit IRC18:16
*** elmiko has joined #openstack-sahara18:18
* tmckay finds some food18:19
aignatovtmckay: relaunch on vanilla would be fine as well :)18:24
tmckayaignatov, yes.  I'll do both.  Lots of pet supply processing.  Something will finish if I run 3 clusters :)  The internal stack can handle it.18:57
openstackgerritChad Roberts proposed a change to stackforge/puppet-sahara: Adding RHEL support for puppet-sahara  https://review.openstack.org/9236818:59
tmckayaignatov, okay, I think I'm done with slides.  Do you want to give them your ok?  I don't want to invent the roadmap (last slide), just citing possibilities :)19:11
aignatovdo you plan to show slides before the actual demo?19:12
tmckayFirst few slides before, then the last slide after19:12
tmckayI thought I would show them quickly what the data is supposed to look like19:13
openstackgerritA change was merged to openstack/sahara-dashboard: Updated from global requirements  https://review.openstack.org/9296019:13
aignatovProbably all text in your slides shold be with the same Font19:14
aignatovPT Sans I think19:14
*** ViswaV has quit IRC19:16
*** IlyaE has quit IRC19:16
aignatovthe last slide: all bullet point should be w/o ‘?’ sign at the end19:16
aignatov:)19:16
tmckayaignatov, ok, I'll adjust.  I used PT Sans Narrow in a few places because it didn't fit19:16
tmckayok, no ? mark19:17
*** ViswaV has joined #openstack-sahara19:17
aignatovalso ‘Other ideas?’ should be deleted, we don’t ask people in the talk what to so next :) we give them an our plans :)19:18
tmckayaignatov, but it's open source ;-)  Don't we want their ideas?19:18
aignatovbut you can say that “any others ideas are welcome”19:18
tmckayokay, I'll delete it.  Thanks19:18
tmckayok, good idea19:19
aignatovand wwlcome people to our design session :)19:19
*** openstackgerrit has quit IRC19:19
*** openstackgerrit has joined #openstack-sahara19:20
aignatovinformation about where to find us could be at the one nwe last slide19:20
aignatov*new19:20
openstackgerritChad Roberts proposed a change to stackforge/puppet-sahara: Adding RHEL support for puppet-sahara  https://review.openstack.org/9236819:20
*** IlyaE has joined #openstack-sahara19:22
aignatovI’ve added two last slides :)19:23
openstackgerritA change was merged to openstack/sahara-image-elements: Turns off ambari and hadoop2 services for initial VM boot in HDP images  https://review.openstack.org/9271519:23
tmckayaignatov, okay, am I seeing your updates in real time?19:24
tmckayah, there they are19:24
tmckaygreat19:24
tmckaystill fixing fonts19:24
aignatovtime to launch19:24
aignatovbrb19:25
tmckayhmm, design sessions may have to be another slide.19:25
openstackgerritChad Roberts proposed a change to stackforge/puppet-sahara: Adding RHEL support for puppet-sahara  https://review.openstack.org/9236819:25
aignatovhmm, I think there is  :)no neeed to add new slide about design sessions only19:26
aignatov:)19:26
*** IlyaE has quit IRC19:32
tmckayaignatov, they wouldn't fit19:33
tmckayShould I take it out?  whatever you think19:33
tmckayI think I'm done editing19:34
tmckaygoodbye, vanilla 1.2.1 cluster.  You have been replaced with vanilla 2.19:38
tmckayIt was nice working with you.19:38
tmckaycrobertsrh, any experience getting rid of instances that won't delete?19:42
elmikotmckay: do they list as error state?19:43
tmckayI'm afraid they're sitting around eating up resources19:43
tmckayno, running.  2 centos images have been that way for several days, and now only 1 of my vanilla 121 nodes went away (out of 5)19:44
elmikotmckay: i had instances that wouldn't delete before and i needed to use `nova reset-state` on them to get them back to a place i could delete19:44
*** Ch00k has quit IRC19:44
tmckayooo, that sounds good19:44
crobertsrhI have removed them from disk19:46
crobertsrhnova reset-state is probably more civilized19:46
* elmiko lifts pinky19:46
tmckayelmiko, ooo, nova delete id19:48
tmckayinstant nuke19:48
tmckaythanks for the idea19:48
tmckaysomething in horizon seems to wrong with deletes19:49
tmckaysometimes it works, sometimes not19:49
tmckayretry from the UI "You are not allowed to delete instance blah"19:51
elmikoyea, i had to reset-state before i could delete in the past19:51
elmikousually when i saw that message about not being allowed, then i needed to reset-state19:51
tmckaybut the UI crushed it like yesterday's coke can19:51
elmikoyea, once i reset-state i could operate the ui again too19:52
tmckayCLI19:52
tmckayrather19:52
elmikofor me, i would see the status of the instance in the ui be something like running, then i would reset-state and it would show up as error19:52
elmikothen i could delete through either cli or ui19:52
tmckayweird.  I just tried delete on the Sahara cluster that was already in delete, and it cleaned it all up the second time19:54
elmikoat least it worked19:58
tmckaycrobertsrh, ever seen this?  Trying to reuse my vanilla 230 cluster template, "Flavor 8 not found"19:58
tmckayshould be large.  I wonder if someone edited the flavor, I think that changes the id19:58
* tmckay does copy and modify19:58
tmckaycrobertsrh, course I'm not an admin so it won't let me see the flavor table, I think20:01
elmikotmckay: i think you can use the rest interface to get a list of all the flavors20:01
tmckayrighto20:02
tmckayfor some reason "8" doesn't cut it anymore.  Looks the same.20:03
elmikodon't those need to be int's instead of strings?20:04
tmckayelmiko, weird, I just copied the node group templates, rechose flavor from the pulldown, made a new cluster templ and launched.20:09
tmckaysome enumeration value changed?  Hmm, we did update sahara, I think....20:09
tmckaymaybe there is a mapping issue20:09
elmikomaybe20:10
tmckayI don't know.  Mystery for anther day.20:10
elmikowhenever i've used flavor from the json, i've always checked first by querying sahara20:11
elmikobut i have had issues with it20:11
tmckaysomewhere there is a translation from "m1.large" to "8"20:11
elmikoahh ok20:11
tmckayhmm, maybe a launchpad bug to track this20:14
tmckayok, and now it works20:15
tmckayI just tried to launch the same templ to get the exact wording of the error, and it started building it20:16
tmckayor maybe not.  Fails again.  I might have picked the wrong one20:17
elmikois this the centos cluster?20:19
*** ViswaV has quit IRC20:37
*** ViswaV has joined #openstack-sahara20:43
*** IlyaE has joined #openstack-sahara21:03
*** crobertsrh is now known as _crobertsrh21:06
tmckayelmiko, no, this was vanilla, hadoop 2, ubuntu21:20
tmckayaignatov, SergeyLukjanov, _mattf, hmmm, I must not have hdp configured correctly, something is wrong.  Very slow compared to vanilla, running on the same stack.  If we have time maybe on Monday you guys can help hotrod it, otherwise I'll just use 2 vanilla clusters to show relaunch and parallel execution.21:22
aignatovtmckay: yes, you always can show it using vanilla21:24
tmckayaignatov, the engineer in me wants to dive into hdp performance, lol.  But that probably has to wait.  I wonder what I set up incorrectly....21:25
tmckayI think I'm all set for the demo, though.21:26
aignatovshame on me, I21:26
aignatovnever ran hdp :)21:26
tmckayaignatov, I ran hadoop 1, but bigpetstore is hadoop 2 only.  It could be ported to hadoop 1, I'm sure, but I didn't have time to try21:27
tmckaythis is my first hadoop 2 stack21:27
tmckayfor hdp21:27
aignatovyou are right21:27
tmckaygood news is that generate, clean, and analyze are all only a few minutes on vanilla21:29
tmckayso I can easily run all 321:29
aignatovthat’s good, but do you plan to prepare words in these gaps? :)21:29
aignatovwhat do you think to do during 2 minutes?21:30
tmckayyes, sure.21:30
aignatovbtw, how long will the all demo?21:30
aignatovhow long?21:30
aignatovin time :)21:30
tmckayI might open it up for questions while the jobs run, or give a quick overview of the cluster launching stuff (node groups, templates) for folks who haven't seen it before21:31
tmckayor show some of the cluster links (the oozie console)21:31
tmckay10 - 15 minutes, easy.21:32
tmckayI can make it shorter or longer21:32
aignatovI haven’t mesured my spech time yet but I exect only 10 mins maximum21:34
aignatovso my 7-10 mis + 15 mins of your time + 5-7 of SergeyLukjanov speech should be fine, what do you think SergeyLukjanov and tmckay ?21:36
tmckayokay.  I'm sure I can fill the time.  I can even get crazy, pull up the oozie console, and show them details of the oozie workflow we generate :)21:36
tmckayI can talk about the CLI too, if there are no questions21:36
tmckayit doesn't work with the internal cluster, because that's still using cert tokens and our sahara instance isn't set up for it.21:37
tmckaybut, I can run a devstack somewhere else and run the CLI against it if we want to21:37
tmckayonly if nobody has questions21:38
openstackgerritA change was merged to openstack/python-saharaclient: Updated from global requirements  https://review.openstack.org/9295921:38
* elmiko starts working on crazy questions21:38
tmckaylol, a plant!21:38
tmckayawesome21:38
tmckaymaybe elimiko can make a guest appearance and talk about image building21:39
elmikono!21:39
aignatovthree men woul be enough I think, tmckay :)21:40
elmikowhat day is that presentation?21:41
tmckayTuesday21:41
elmikocool, should be fun21:41
aignatovelmiko: and on Monday, SergeyLukjanov, mattf and jspiedel will show their talk :)21:42
aignatovthat’s funny too21:43
elmikowhat time are the talks, i'm not seeing them on the schedule21:43
elmikothe monday one is the "Ops Meetup" ?21:44
aignatovypu can find it here http://openstacksummitmay2014atlanta.sched.org/21:44
elmikoaignatov: thanks21:45
aignatovMonday, May 12 • 2:00pm - 2:40pm21:46
elmikoman, there is so much going on21:47
aignatovyep, you can click on all sessions you plan to attend21:48
aignatovand you ll get ypur schedule21:48
aignatov*your21:48
elmikooh nice21:48
elmikowhat time is the talk on tuesday?21:48
aignatovtmckay: slides looks good21:48
tmckayelmikio, hmm, I'll have to check the schedule21:49
aignatovnot sure if lower font in 20 and 21 will look fine on the scrren21:49
tmckayaignatov, I just made a cheat sheet of stuff to talk about while the jobs run.  I could go on for hours.21:50
tmckayokay.  Maybe we can get there early and test the fong?21:50
tmckayfont21:50
aignatovnot sure, last time some organizer ask people to leave present room as fast as they can21:53
aignatovso there are not time for it21:53
tmckaywhich slides are you concerned about?  Maybe we can make them bigger.  On one, I stretched the margins to fit on one line22:00
tmckayhmm, my hdp cluster is almost acting like it is resource challenged.  I used to get the same kind of behavior with vanilla on my laptop.  Maybe respin with xlarge flavor22:00
*** openstackgerrit has quit IRC22:05
*** openstackgerrit has joined #openstack-sahara22:05
tmckayelmiko, Tuesday at 2.  took me a while to find it22:09
elmikoahh, thanks dude22:09
elmikoi'm having trouble putting my schedule together. too many good talks22:09
tmckayalright, time to go.  Maybe see you guys tomorrow, or Sunday night :)22:11
*** tmckay has quit IRC22:11
openstackgerritA change was merged to openstack/sahara: Extended plugin SPI with methods to communicate with EDP  https://review.openstack.org/7372922:12
*** IlyaE has quit IRC22:17
*** IlyaE has joined #openstack-sahara22:19
*** IlyaE has quit IRC22:38
elmikonight all22:48
*** elmiko is now known as elmiko_22:48
*** elmiko_ has quit IRC22:56
*** ViswaV has quit IRC23:39
openstackgerritOpenStack Proposal Bot proposed a change to openstack/sahara: Updated from global requirements  https://review.openstack.org/9295523:56

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!