Thursday, 2014-05-29

*** IlyaE has joined #openstack-sahara00:06
*** alazarev has quit IRC00:10
*** alazarev has joined #openstack-sahara00:25
*** sunrenjie6 has joined #openstack-sahara00:30
*** ViswaV has joined #openstack-sahara00:59
*** ViswaV has quit IRC01:00
*** ViswaV has joined #openstack-sahara01:02
*** ViswaV_ has joined #openstack-sahara01:05
*** bradd1 has joined #openstack-sahara01:06
*** ViswaV has quit IRC01:06
*** shakamunyi has quit IRC01:16
openstackgerritA change was merged to openstack/sahara: Fix intermittent transient cluster tests failure  https://review.openstack.org/9586101:45
*** alazarev has quit IRC01:46
*** alazarev has joined #openstack-sahara02:01
*** alazarev has quit IRC02:12
*** zhiyan_ is now known as zhiyan02:34
*** rhodgin has joined #openstack-sahara02:41
*** IlyaE has quit IRC02:48
*** IlyaE has joined #openstack-sahara03:05
*** ViswaV_ has quit IRC03:10
*** ViswaV has joined #openstack-sahara03:37
*** bradd1 has quit IRC03:38
*** zhiyan is now known as zhiyan_03:47
*** shakamunyi has joined #openstack-sahara03:48
*** ViswaV has quit IRC03:49
*** ViswaV_ has joined #openstack-sahara03:49
*** shakamunyi has quit IRC04:05
*** zhiyan_ is now known as zhiyan04:10
*** akuznetsov has joined #openstack-sahara04:11
*** alazarev has joined #openstack-sahara04:17
*** ViswaV has joined #openstack-sahara04:22
*** ViswaV_ has quit IRC04:22
*** zhiyan is now known as zhiyan_04:52
*** alazarev has quit IRC04:53
*** shakamunyi has joined #openstack-sahara05:02
*** alazarev has joined #openstack-sahara05:05
*** ViswaV has quit IRC05:08
*** shakamunyi has quit IRC05:14
*** IlyaE has quit IRC05:20
*** IlyaE has joined #openstack-sahara05:27
*** zhiyan_ is now known as zhiyan05:29
*** shakamunyi has joined #openstack-sahara05:40
*** shakamunyi has quit IRC05:50
*** ghenriks has joined #openstack-sahara06:05
*** ghenriks_ has quit IRC06:07
*** alazarev has quit IRC06:13
*** alazarev has joined #openstack-sahara06:13
*** tnovacik|gone has joined #openstack-sahara06:15
*** alazarev has quit IRC06:27
*** shakamunyi has joined #openstack-sahara06:47
*** shakamunyi has quit IRC06:59
openstackgerritAlexander Ignatov proposed a change to openstack/sahara: Added ability to run HDFS service only with Hadoop 2  https://review.openstack.org/9637107:08
*** IlyaE has quit IRC07:10
*** shakamunyi has joined #openstack-sahara07:25
*** bogdando has quit IRC07:26
*** bogdando has joined #openstack-sahara07:26
*** witlessb has joined #openstack-sahara07:27
openstackgerritA change was merged to openstack/sahara: Updated from global requirements  https://review.openstack.org/9627207:34
*** shakamunyi has quit IRC07:36
*** shakamunyi has joined #openstack-sahara08:03
*** shakamunyi has quit IRC08:14
*** shakamunyi has joined #openstack-sahara08:40
*** sunrenjie6 has quit IRC08:52
*** sunrenjie6 has joined #openstack-sahara08:52
*** shakamunyi has quit IRC08:52
*** sunrenjie6 has quit IRC08:57
*** tnovacik|gone is now known as tnovacik09:05
*** jpich has joined #openstack-sahara09:11
*** shakamunyi has joined #openstack-sahara09:19
*** skolekonov has joined #openstack-sahara09:23
*** zhiyan is now known as zhiyan_09:30
*** shakamunyi has quit IRC09:31
*** tosky has joined #openstack-sahara09:52
*** shakamunyi has joined #openstack-sahara09:58
*** shakamunyi has quit IRC10:05
*** shakamunyi has joined #openstack-sahara10:05
openstackgerritSergey Lukjanov proposed a change to openstack/sahara: Rework keystone auth_token middleware configs  https://review.openstack.org/7468810:06
*** shakamunyi has quit IRC10:10
*** shakamunyi has joined #openstack-sahara10:10
*** shakamunyi has quit IRC10:20
*** shakamunyi has joined #openstack-sahara10:47
*** shakamunyi has quit IRC11:00
*** IvanBerezovskiy has joined #openstack-sahara11:11
*** shakamunyi has joined #openstack-sahara11:27
*** tellesnobrega_ has joined #openstack-sahara11:32
*** _crobertsrh is now known as crobertsrh11:37
*** shakamunyi has quit IRC11:41
*** tmckay has left #openstack-sahara12:05
*** shakamunyi has joined #openstack-sahara12:07
*** shakamunyi has quit IRC12:20
openstackgerritA change was merged to openstack/sahara: Replaced RuntimeErrors with specific errors  https://review.openstack.org/9475012:20
*** themistymay has joined #openstack-sahara12:56
*** _elmiko is now known as elmiko12:57
*** miqui has joined #openstack-sahara12:58
openstackgerritSergey Lukjanov proposed a change to openstack/sahara: Add upgrade notes for sahara-api to sahara-all  https://review.openstack.org/9572613:00
openstackgerritSergey Lukjanov proposed a change to openstack/sahara: Add warn re alpha readiness of distrib mode  https://review.openstack.org/9645313:00
openstackgerritMichael McCune proposed a change to openstack/sahara-image-elements: Adding options to control DIB repo branch  https://review.openstack.org/9624813:00
SergeyLukjanovcrobertsrh, tmckay https://review.openstack.org/#/c/96453/13:01
elmikoSergeyLukjanov: that's a good change, i got bit by using sahara-api instead of sahara-all13:11
*** zhiyan_ is now known as zhiyan13:11
SergeyLukjanovelmiko, yup, I think it'll help all of us a lot in debugging "Why nothing working??"13:13
elmikoagreed13:14
*** shakamunyi has joined #openstack-sahara13:16
*** shakayumi has joined #openstack-sahara13:18
*** shakamunyi has quit IRC13:21
openstackgerritSergey Lukjanov proposed a change to openstack/sahara: Add warn re alpha readiness of distrib mode  https://review.openstack.org/9645313:27
*** tmckay has joined #openstack-sahara13:30
*** shakayumi has quit IRC13:34
openstackgerritMichael McCune proposed a change to openstack/sahara: Adding disconnected mode fixes to hdp plugin  https://review.openstack.org/9446013:34
openstackgerritA change was merged to openstack/python-saharaclient: Updated from global requirements  https://review.openstack.org/9627013:38
openstackgerritA change was merged to openstack/sahara: Synced jsonutils from oslo-incubator  https://review.openstack.org/9569213:38
*** witlessb has quit IRC13:44
*** witlessb has joined #openstack-sahara13:46
*** shakayumi has joined #openstack-sahara13:48
*** IlyaE has joined #openstack-sahara13:55
openstackgerritA change was merged to openstack/sahara: Removed unused global var and unnessary param  https://review.openstack.org/9472413:58
openstackgerritMichael McCune proposed a change to openstack/sahara-image-elements: Adding options to control DIB repo branch  https://review.openstack.org/9624813:59
openstackgerritA change was merged to openstack/sahara: remove default=None for config options  https://review.openstack.org/9608314:08
openstackgerritA change was merged to openstack/sahara: Added validation check that network provided for neutron  https://review.openstack.org/9504714:16
openstackgerritA change was merged to openstack/sahara: Fix working sahara with heat and nova-network  https://review.openstack.org/9343714:16
*** alazarev has joined #openstack-sahara14:19
*** skolekonov has quit IRC14:26
*** _mattf is now known as mattf14:27
*** zhiyan is now known as zhiyan_14:52
*** bradd1 has joined #openstack-sahara14:57
*** rhodgin has quit IRC14:58
*** alazarev has quit IRC15:06
tmckaycrobertsrh, add https://bugs.launchpad.net/sahara/+bug/1324582.  Please comment if I missed something15:20
openstackgerritA change was merged to openstack/sahara: Add upgrade notes for sahara-api to sahara-all  https://review.openstack.org/9572615:20
crobertsrhI'll take a look, thanks15:20
crobertsrhI added a blurb about my generate_setup_script addition to produce the log output there.  Looks good though.15:24
tmckaycool, thanks.  Insidious.15:29
*** akuznetsov has quit IRC15:33
*** alazarev has joined #openstack-sahara15:37
*** rhodgin has joined #openstack-sahara15:38
mattfelmiko, re detached head, i'm a fan of git-archive because why do we really care what the state of the checkout is from a working-git-repo perspective15:38
mattfit could just as easily be a tarball download, a yum or pip install15:38
mattfelmiko, SergeyLukjanov, if you're going to change the clone to clone+checkout, just go ahead and change it to git-archive15:40
*** ViswaV has joined #openstack-sahara15:41
SergeyLukjanovour elements are broken again because of www.apache.org downloading :(15:42
SergeyLukjanovhttps://bugs.launchpad.net/sahara/+bug/132459015:42
SergeyLukjanovanybody want to fix?15:42
elmikomattf: ack, i'll work up another patch15:42
openstackgerritIvan Berezovskiy proposed a change to openstack/sahara-image-elements: Unset CentOS image parameter that affects on another images  https://review.openstack.org/9650215:42
*** ViswaV_ has joined #openstack-sahara15:43
SergeyLukjanovmattf, elmiko, I'm absolutely neutral re using git-archive vs. clone+checkout in this case15:43
openstackgerritIvan Berezovskiy proposed a change to openstack/sahara-image-elements: Unset CentOS image parameters that affect on another images  https://review.openstack.org/9650215:43
mattfSergeyLukjanov, the joys of depending on external artifacts!15:44
mattfoof, incubator, it graduated awhile ago15:44
mattfi'll update the url15:44
SergeyLukjanovmattf, thx!15:44
tmckayIvanBerezovskiy, did you see https://bugs.launchpad.net/sahara/+bug/1324582? crobertsrh and I just finished tracking this down.  My thought is maybe that the /mnt changes should be done by the plugins, not rc.local.15:44
elmikoi agree with mattf about not needing the git repo structure, archive might be clearer. i'm gonna do some investigation and come up with a patch.15:45
mattfyou probably just change clone to archive, remove the -b and pipe to tar15:45
mattftho might not have to pipe to tar in modern git version15:45
elmikomattf: i changed the clone to a clone+checkout15:45
* mattf did all his git-archive stuff back in '0815:46
*** ViswaV has quit IRC15:46
*** zhiyan_ is now known as zhiyan15:47
IvanBerezovskiytmckay, if we chown /mnt in plugin, we shouldn't chown it in rc.local script i think15:49
tmckayIvanBerezovskiy, agreed.  Not sure why that chown is there in rc.local.  The plugins are all different, though -- we'll have to check each one.15:50
tmckayit may be as simple as taking that out of rc.local15:50
crobertsrhI've only experienced this issue for fedora vanilla 1 images.  Not to say it isn't possible elsewhere.15:50
*** tellesnobrega_ has quit IRC15:50
IvanBerezovskiytmckay, but we must be sure that we can make: 'sudo chown' from plugin, because owner of /mnt isn't root. That's why i did chown15:52
elmikomattf: i'm not seeing how to pull a remote repo through git-archive, or were you suggesting clone then archive?15:54
tmckaycrobertsrh, IvanBerezovskiy, we'll have to play and look at/test each plugin to see what the expectations are for /mnt.  Not sure how each deals with /mnt, or what hadoop expectations are in 2.x vs 115:54
IvanBerezovskiytmckay, agreed, need to test15:55
mattfelmiko, try --remote15:55
elmikomattf: that's what i'm messing with, but it's not doing what i expect lol15:56
tmckayIvanBerezovskiy, I just wanted to make sure someone familiar with DIB saw the bug :)15:56
openstackgerritMatthew Farrellee proposed a change to openstack/sahara-image-elements: Use stable apache urls in spark elements  https://review.openstack.org/9650615:57
*** zhiyan is now known as zhiyan_15:57
openstackgerritA change was merged to openstack/python-saharaclient: use strutils.to_slug() instead of utils.slugify()  https://review.openstack.org/9592715:57
mattfSergeyLukjanov, ^^15:58
*** shakayumi is now known as shakamunyi15:59
elmikomattf: not sure git-archive will work in this case, i think the server doesn't allow it for https protocols16:02
SergeyLukjanovmattf, does pre-graduated versions of spark available by this url?16:03
mattfSergeyLukjanov, yeah, but we shouldn't care16:05
mattfelmiko, i do remember it being kinda non-intuitive16:05
SergeyLukjanovmattf, just curios16:05
mattfSergeyLukjanov, they're suffix'd w/ -incubation16:05
elmikomattf: i googled around a little, but i think this error is from the server end. http://paste.openstack.org/show/82057/16:06
*** tellesnobrega_ has joined #openstack-sahara16:06
tmckayIvanBerezovskiy, crobertsrh, hmm, with guestfish I can see that at least in sahara-icehouse-vanilla-1.2.1-fedora-20.qcow2 the /mnt dir is already owned by root when the image is created16:06
tmckay><fs> ll /16:07
tmckaytotal 8416:07
tmckay...16:07
openstackgerritOpenStack Proposal Bot proposed a change to openstack/sahara: Updated from global requirements  https://review.openstack.org/9651016:07
tmckaydrwxr-xr-x  2 root root  4096 Aug  7  2013 mnt16:07
tmckaythat would suggest removing the chown will not change anything, just get rid of the race16:07
tmckayassuming it's okay that /mnt/log/hadoop could be created later16:08
tmckayjust thought I'd check that16:08
* tmckay updates bug16:08
mattfelmiko, bummer, ok16:09
elmikoyea.. :(16:09
mattftmckay, https://review.openstack.org/#/c/96506/ pls16:12
*** zhiyan_ is now known as zhiyan16:12
IvanBerezovskiytmckay, good if it's ok16:12
tmckaymattf, ack.  Maybe I can give my first +2 :)16:13
*** elmiko is now known as _elmiko16:14
tmckaywoohoo, hope I didn't break it16:15
openstackgerritA change was merged to openstack/sahara-image-elements: Use stable apache urls in spark elements  https://review.openstack.org/9650616:16
tmckayhmm, maybe I should have waited for savanna-ci. mattf, SergeyLukjanov, do we delay +2 until after ci?16:16
tmckayoops, too late16:16
IvanBerezovskiytmckay, sahara-ci doesn't test spark images, only vanilla images now16:17
mattftmckay, savanna-ci will -1, the cr fixes a dl error16:17
mattfoh, that too16:17
tmckaymattf, but in general.... we should wait for ci (with a sanity check on ci expectations), correct?16:17
mattfyeah16:17
mattfbefore +A16:17
* tmckay puts away +2 Sword of Overzealousness16:18
*** vrovachev has quit IRC16:19
*** IvanBerezovskiy has left #openstack-sahara16:20
*** zhiyan is now known as zhiyan_16:21
SergeyLukjanovwe're planning to add testing of all plugin for elements in sahara-ci16:23
mattfthe magical sahara-ci, when does it slay the savanna-ci?16:23
SergeyLukjanovbut it requires adding more compute power to our lab, so, not so fast as we want16:23
SergeyLukjanovmattf, it'll be renamed some day ;016:24
SergeyLukjanov:)16:24
mattfi'm bummed that we don't have more external facing hardware to help out w/ this16:24
SergeyLukjanovmattf, sure, I'm afraid noone has such hardware :)16:26
mattfit's amazing what it takes to put hardware on the world wide web16:27
tmckaymattf, btw, another victory for cli integration testing -- I added a spark plugin test with a few lines of code that will launch a spark cluster16:29
* mattf <316:29
tmckaynow all we need is spark edp16:29
* tmckay reminds self to add cluster scale cli blueprint16:30
* mattf ponders the implications16:32
mattfso you have a working spark image16:32
mattfand the plugin works16:32
mattflooks like it's close to getting merged too16:32
tmckaymattf, trying the new patch, haven't tried since before bug day.  ' RuntimeError: Plugins couldn't be loaded: spark', investigating.  Probably something in my env16:33
*** themistymay has quit IRC16:34
openstackgerritSofiia Kostiuchenko proposed a change to openstack/sahara: [HDP] Integration tests for HDP 2.0.6  https://review.openstack.org/9573916:35
*** tnovacik has quit IRC16:35
*** jpich has quit IRC16:36
*** themistymay has joined #openstack-sahara16:37
* tmckay watches spark cluster build16:49
SergeyLukjanovmattf, could you please sync oslo excluding oslo/db?16:50
*** themistymay has quit IRC16:50
SergeyLukjanovmattf, oslo/db will require some magic code updates16:50
mattfSergeyLukjanov, yeah, i was planning to do that post hadoop summit next week16:51
SergeyLukjanovmattf, it'll be great to have it in j116:51
mattfdo you know of anyone who wants to do the oslo/db magic?16:51
SergeyLukjanovmattf, alazarev will do it16:51
mattfgreat16:51
alazarevyes, will do16:51
SergeyLukjanovmattf, any chance you can sync in the next few days? :)16:52
mattfmaybe monday on my flight, but i'm not optimistic16:53
mattfif there's an urgent need, i can sit this sync out and pick up for the next16:54
*** bradd1 has quit IRC16:55
*** themistymay has joined #openstack-sahara16:56
SergeyLukjanovmattf, it's not a blocker, but it'll be really great, I see many changes16:56
*** IlyaE has quit IRC16:56
mattfyeah, it's overdue for sure16:56
*** alazarev has quit IRC16:57
*** bradd1 has joined #openstack-sahara16:57
*** ylobankov has quit IRC17:00
openstackgerritSergey Lukjanov proposed a change to openstack/sahara: Rework keystone auth_token middleware configs  https://review.openstack.org/7468817:01
*** _elmiko is now known as elmiko17:04
elmikowhat version of git is on savanna-ci?17:05
*** alazarev has joined #openstack-sahara17:11
*** tellesnobrega_ has quit IRC17:11
*** zhiyan_ is now known as zhiyan17:12
*** zhiyan is now known as zhiyan_17:22
*** IlyaE has joined #openstack-sahara17:23
*** themistymay has quit IRC17:28
openstackgerritMichael McCune proposed a change to openstack/sahara-image-elements: Adding options to control DIB repo branch  https://review.openstack.org/9624817:37
*** akuznetsov has joined #openstack-sahara17:47
SergeyLukjanovelmiko, I presume some 1.7.X17:56
SergeyLukjanovmeeting in 5 mins17:56
*** openstackgerrit has quit IRC17:57
mattffyi, i'm going to have to leave early for a phone call17:58
SergeyLukjanovmeeting time18:01
elmikoSergeyLukjanov: thanks, think i got the git command fixed for older version18:02
*** zhiyan_ is now known as zhiyan18:13
*** zhiyan is now known as zhiyan_18:23
*** IlyaE has quit IRC18:25
*** IlyaE has joined #openstack-sahara18:37
*** akuznetsov has quit IRC18:38
*** akuznetsov has joined #openstack-sahara18:42
*** IlyaE has quit IRC18:44
*** IlyaE has joined #openstack-sahara18:45
*** IlyaE has quit IRC18:50
*** IlyaE has joined #openstack-sahara18:51
*** akuznetsov has quit IRC19:00
*** witlessb has quit IRC19:06
*** openstackgerrit has joined #openstack-sahara19:07
*** akuznetsov has joined #openstack-sahara19:13
*** openstackgerrit has quit IRC19:13
*** zhiyan_ is now known as zhiyan19:14
*** akuznetsov has quit IRC19:21
*** zhiyan is now known as zhiyan_19:24
*** openstackgerrit has joined #openstack-sahara19:35
*** ViswaV_ has quit IRC19:35
tmckayaignatov, just getting into this now but it looks like in spark you can do "sc.textFile("hdfs://blahblah") for an external hdfs19:46
tmckayaignatov, so, I'm guessing that we could extend spark to handle swift urls as well, just like hadoop19:46
tmckaywe may be able to include our own JAR (or python module for python jobs) with a derived class from sc.textfile that would handle swift without it even being upstream.  Just an idea.19:47
*** notmyname has left #openstack-sahara19:48
tmckaybtw, I have found a guy running Spark from Oozie and contacted him via email.  He sent me some links to his blog, etc.19:50
aignatovtmckay: very nice to hear that :)19:54
tmckaymaybe we can get him to join Sahara ;-)19:55
tmckayfree trip to Paris maybe?19:55
aignatovmaybe :)19:55
tmckayaignatov, even better20:07
aignatovhow?20:07
tmckayfrom a colleague....20:07
tmckaytmckay, currently it just delegates to hadoop's FileInputFormat (which I guess handles local, hdfs, and some other types)20:08
tmckayI asked him how hard it would be to make spark handle swfit:// urls20:09
tmckayHe said it already passes it to Hadoop, which we've already patched20:09
tmckayIt might just work20:09
aignatovI think so, FileInputFormat should be unique for each fs implementation20:11
*** ViswaV has joined #openstack-sahara20:14
*** zhiyan_ is now known as zhiyan20:15
*** IlyaE has quit IRC20:17
*** IlyaE has joined #openstack-sahara20:18
openstackgerritSergey Lukjanov proposed a change to openstack/sahara: Add warn re alpha readiness of distrib mode  https://review.openstack.org/9645320:19
*** openstackgerrit has quit IRC20:25
*** zhiyan is now known as zhiyan_20:26
*** ViswaV has quit IRC20:37
*** tmckay has left #openstack-sahara20:40
*** ViswaV has joined #openstack-sahara20:42
*** shakamunyi has quit IRC20:51
*** shakamunyi has joined #openstack-sahara20:52
*** ViswaV has quit IRC21:01
*** bradd1 has quit IRC21:05
*** ViswaV has joined #openstack-sahara21:05
*** shakamunyi has quit IRC21:14
*** zhiyan_ is now known as zhiyan21:18
*** zhiyan is now known as zhiyan_21:27
*** openstackgerrit has joined #openstack-sahara21:34
*** openstackgerrit has quit IRC21:47
*** IlyaE has quit IRC21:57
*** IlyaE has joined #openstack-sahara21:58
*** witlessb has joined #openstack-sahara22:00
*** shakamunyi has joined #openstack-sahara22:11
*** elmiko is now known as _elmiko22:13
*** IlyaE has quit IRC22:18
*** zhiyan_ is now known as zhiyan22:18
*** IlyaE has joined #openstack-sahara22:18
*** shakayumi has joined #openstack-sahara22:22
*** shakamunyi has quit IRC22:22
*** witlessb has quit IRC22:26
*** zhiyan is now known as zhiyan_22:27
*** IlyaE has quit IRC22:48
*** ViswaV has quit IRC22:50
*** ViswaV has joined #openstack-sahara22:53
*** ViswaV has quit IRC22:53
*** IlyaE has joined #openstack-sahara22:54
*** ViswaV has joined #openstack-sahara22:55
*** zhiyan_ is now known as zhiyan23:19
*** zhiyan is now known as zhiyan_23:28
*** rhodgin has quit IRC23:59

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!