Tuesday, 2014-04-08

*** openstackgerrit has quit IRC00:24
*** _mattf is now known as mattf00:36
*** crobertsrh has joined #openstack-sahara00:47
*** ErikB1 has joined #openstack-sahara00:58
*** elmiko is now known as elmiko_afk01:21
*** openstack has joined #openstack-sahara01:48
*** bradd1 has joined #openstack-sahara01:54
*** bradd1 has quit IRC01:56
*** ErikB1 has quit IRC02:16
*** akuznetsov has joined #openstack-sahara03:31
*** akuznetsov has quit IRC03:42
*** akuznetsov has joined #openstack-sahara04:38
*** akuznetsov has quit IRC04:48
*** akuznetsov has joined #openstack-sahara04:50
*** akuznetsov has quit IRC05:01
*** akuznetsov has joined #openstack-sahara05:06
*** akuznetsov has quit IRC05:17
*** RockKuo has joined #openstack-sahara05:20
*** akuznetsov has joined #openstack-sahara05:29
openstackgerritJenkins proposed a change to openstack/sahara: Imported Translations from Transifex  https://review.openstack.org/8271906:06
*** akuznetsov has quit IRC06:16
*** IlyaE has quit IRC06:18
*** akuznetsov has joined #openstack-sahara06:19
*** akuznetsov has quit IRC06:42
*** dmitryme has joined #openstack-sahara06:45
*** akuznetsov has joined #openstack-sahara06:46
*** akuznetsov has quit IRC07:06
*** akuznetsov has joined #openstack-sahara07:07
*** akuznetsov has quit IRC07:12
*** skolekonov has joined #openstack-sahara07:14
*** dmitryme has quit IRC07:15
openstackgerritA change was merged to openstack/sahara: Update EDP requirements for hadoop v2  https://review.openstack.org/8501707:15
*** witlessb has joined #openstack-sahara08:05
openstackgerritA change was merged to openstack/sahara-image-elements: Removes the service stop call for gmetad  https://review.openstack.org/8574108:14
openstackgerritA change was merged to openstack/sahara-image-elements: Add hadoop v2 support for IDH plugin  https://review.openstack.org/8306208:20
*** akuznetsov has joined #openstack-sahara08:24
openstackgerritA change was merged to openstack/sahara-image-elements: Removes default usage of root-passwd element from HDP image generation  https://review.openstack.org/8514308:25
openstackgerritA change was merged to openstack/sahara-image-elements: Updated from global requirements  https://review.openstack.org/8584108:48
openstackgerritA change was merged to openstack/sahara-extra: Updated from global requirements  https://review.openstack.org/8584008:49
openstackgerritA change was merged to openstack/sahara: Fix big job binary objects in mysql  https://review.openstack.org/8546609:10
openstackgerritA change was merged to openstack/sahara: Add a custom filter method to scan wrapped dict results  https://review.openstack.org/8488109:10
*** dmitryme has joined #openstack-sahara09:32
*** RockKuo has quit IRC09:39
*** bogdando has quit IRC09:39
*** bogdando has joined #openstack-sahara09:39
*** RockKuo has joined #openstack-sahara09:39
*** bogdando has quit IRC09:39
*** akuznetsov has quit IRC09:39
*** openstackgerrit has quit IRC09:39
*** sreshetnyak has quit IRC09:39
*** skostiuchenko has quit IRC09:39
*** openstack has joined #openstack-sahara09:49
*** mattf has quit IRC09:52
*** mattf has joined #openstack-sahara09:52
*** tosky has joined #openstack-sahara09:56
*** tosky has quit IRC09:59
*** tosky has joined #openstack-sahara09:59
*** ruhe has quit IRC10:03
*** ruhe has joined #openstack-sahara10:04
*** NikitaKonovalov has quit IRC10:26
*** bogdando has quit IRC10:26
*** sreshetnyak has quit IRC10:26
*** dmitryme has quit IRC10:26
*** mattf has quit IRC10:26
*** bogdando has joined #openstack-sahara10:27
*** mattf has joined #openstack-sahara10:27
*** NikitaKonovalov has joined #openstack-sahara10:27
*** dmitryme has joined #openstack-sahara10:27
*** sreshetnyak has joined #openstack-sahara10:32
*** tosky has quit IRC10:39
*** crobertsrh has quit IRC10:39
*** tosky has joined #openstack-sahara10:40
*** crobertsrh has joined #openstack-sahara10:40
*** tosky has quit IRC10:40
*** tosky has joined #openstack-sahara10:40
*** dmitryme has quit IRC11:17
tellesnobregai have one question about oozie, when i install sahara, do i have one oozie server? or they are created for each cluster that is created?11:25
*** RockKuo has quit IRC11:28
*** dmitryme has joined #openstack-sahara11:44
dmitrymecrobertsrh: the Hadoop 2 images are listed in the docs and are available for download: http://docs.openstack.org/developer/sahara/userdoc/vanilla_plugin.html11:47
*** tosky has quit IRC11:55
tellesnobregadmitryme: i have one question about oozie, when i install sahara, do i have one oozie server? or are there one oozie server for each cluster that is created?12:10
SergeyLukjanovtellesnobrega, one oozie server per cluster12:10
*** tosky has joined #openstack-sahara12:14
toskyuhm, what are the tests running on Sahara CI? For example in this review https://review.openstack.org/#/c/85017/ I see12:15
toskyhttp://www.fpaste.org/92526/96959118/12:15
toskyare they all tempest tests, or are there (it doesn't seem so) also the tests in sahara repository?12:16
tellesnobregaSergeyLukjanov: does it depends on the plugin that is being used? for example, could i create a new plugin that doesnt use oozie?12:17
SergeyLukjanovtellesnobrega, we're using only oozie atm and it's not pluggable12:29
SergeyLukjanovtellesnobrega, but we have some plans to discuss it on summit12:29
SergeyLukjanovtellesnobrega, and to extract current code to the plugin to support different workload management tools12:29
SergeyLukjanovfor ex. to run spark jobs12:29
SergeyLukjanovand etc.12:30
SergeyLukjanovtosky, we're running sahara/tests/integration in savanna-ci12:30
crobertsrhthanks dmitryme :)12:30
toskySergeyLukjanov: thanks!12:30
SergeyLukjanovtosky, all our tempest stuff is executed by jenkins12:30
SergeyLukjanovtosky, 3 *dsvm* jobs12:31
toskySergeyLukjanov: isn't savanna-ci a jenkins instance too?12:31
* tosky a bit confused12:31
tellesnobregaSergeyLukjanov: i think you already know, i want to have real time processing in sahara, i have seen that oozie can schedule stuff other than mapreduce, like pig jobs and hive, but i'm not sure if this will suffice for real time, or will i need some other scheduler12:32
SergeyLukjanovtellesnobrega, yup, anyway, it's not only job management tool, so, we'll implement pluggability for EDP12:33
*** dmitryme has quit IRC12:39
tellesnobregaSergeyLukjanov: i see, so the user can choose between using hadoop or other precessing technique, like the one i'm proposing12:39
SergeyLukjanovtellesnobrega, yup12:40
tellesnobregaSergeyLukjanov: sounds great.12:41
SergeyLukjanovtellesnobrega, we don't know how it'll looks like atm, but it's on the summit agenda12:41
tellesnobregaSergeyLukjanov: I'm not very familiar with sahara, but for what i've seen, i think that i will need to create a new plugin (i.e. realtime-plugin) and configure it, just like vanilla and hdp plugin that exists today right?12:43
tellesnobregaSergeyLukjanov: i will try to attend the conversation12:43
SergeyLukjanovtellesnobrega, probably, there will two types of plugins - for provisioning data processing cluster and for running jobs on top of it12:44
*** tosky has quit IRC12:44
SergeyLukjanovtellesnobrega, btw I'm talking about http://openstack.org/summit/openstack-summit-atlanta-201412:45
tellesnobregaSergeyLukjanov: yeah, i will be there :)12:46
SergeyLukjanovtellesnobrega, oh, great ;)12:46
SergeyLukjanovtellesnobrega, our design summit sessions will be12:46
SergeyLukjanovlet me remember...12:46
SergeyLukjanovThu afternoon + Friday morning12:47
tellesnobregaSergeyLukjanov: i was doing some work with keystone and need to talk the guys into accepting our proposals ;) but if the time doesn't conflict i will try to attend it12:47
SergeyLukjanovtellesnobrega, our slots have very small intersection with keystone atm12:48
*** elmiko_afk is now known as elmiko12:49
tellesnobregaSergeyLukjanov: i think that i will be able to attend it, it wont be necessary for me to attend all keystone sessions, and since this proposal here is going to be important for my masters degree i will make time for it.12:50
SergeyLukjanovtellesnobrega, cool12:56
*** tmckay has joined #openstack-sahara13:00
*** dmitryme has joined #openstack-sahara13:01
tmckayaignatov, good morning, ping13:01
aignatovtmckay: hi, pong13:02
aignatovI'm in a meeting13:02
tmckayaignatov, I have a Pig that won't oink :)13:02
aignatovwill ping you soon, ok?13:02
tmckayaignatov, okay, after the meeting.  thanks.13:02
*** tosky has joined #openstack-sahara13:03
tmckaytosky, just in case, have you managed to run a Pig job on a hadoop 2 cluster?13:03
tmckaytosky, my mr and mr streaming jobs work, but Pig hangs.  ubuntu machine image.13:03
toskytmckay: no, I had that Authentication error and then I did other stuff13:04
toskytmckay: did you try the streaming job with keystone v2?13:04
tmckaytosky, hmm, I used whatever the default setup is.  No attempt to change the auth api.13:05
toskytmckay: I remember that you tried with keystone v2 and that was a workaround for that authentication issue13:06
tmckaytosky, lol, you remember more than I do, then :)13:06
tmckaymaybe too many things happening at once13:07
toskyo.O13:07
toskytmckay: you can easily check the configuration so that we are sure about this13:08
tmckaytosky, okay, where should I check?13:08
tmckaylocalrc, sahara.conf?13:08
toskytmckay: sahara.conf, wherever it is13:08
tmckaytosky, okay, I have use_identity_api_v3 commented out in my sahara.conf, and the default from keystone.py appears to be True, so I assume I used v313:12
tmckaytosky, would you like me to try a mr-streaming job with v2?13:13
toskytmckay: maybe the pig job, but well, if you don't have that error, I'm not sure it would help13:13
toskyI could try again with ubuntu, I was using a centos vm13:14
toskytmckay: is the devstack checkout a fresh one?13:14
tmckaytosky, the pig job starts running, but then never finishes the map, never creates the output dir, and gives no errors that I can find except log4j errors13:14
tmckaytosky, well, the devstack source itself is from March 18.  Maybe worth a pull.13:15
tmckaywhat have I got to loose? :)13:15
toskytmckay: not sure if devstack created a branch for icehouse; I manually had to checkout some milestone-proposed branches13:15
toskytmckay: otherwise you will go to juno and not to icehouse (even if the delta is small at this point I guess)13:15
tmckayright, gotcha.  There's got to be an icehouse tag somewhere...13:16
tmckayhmm, maybe not13:16
*** openstackgerrit has joined #openstack-sahara13:23
elmikotmckay: are you getting that JA018 error in your log?13:23
*** openstackgerrit has quit IRC13:23
tmckayelmiko, I don't think so, not for Pig13:24
*** openstackgerrit has joined #openstack-sahara13:24
elmikotmckay: just saw your email, that error ended up resolving to something with the output directory. had to ask13:25
*** akuznetsov has quit IRC13:25
tmckayelmiko, thanks.  If I can get a cluster running here, I'll try command line Pig.  I wouldn't care so much except that bigpetstore needs Pig, doh13:30
toskytmckay: "command line Pig" == use the hadoop cluster directly, without sahara-based job submission?13:31
tmckaytosky, yes13:31
tmckaytosky, I tried yesterday with command line Oozie, still failed.  "hadoop" is one level lower13:32
tmckaymaybe it will give me a decent error13:32
tmckayExcept, at the moment, my devstack startup gives me:13:32
tmckayTraceback (most recent call last):13:32
tmckay  File "/opt/stack/cinder/bin/cinder-manage", line 62, in <module>13:32
tmckay    from oslo import messaging13:32
tmckayImportError: cannot import name messaging13:32
tmckaygreat, thank you, devstack.  you worked yesterday13:32
*** akuznetsov has joined #openstack-sahara13:33
toskylet me restack my stack13:34
*** Hao has joined #openstack-sahara13:41
*** dmitryme has quit IRC13:54
elmikoi want to create a blueprint for sahara-dashboard, do i create it at https://launchpad.net/sahara ?14:00
*** sreshetnyak has quit IRC14:13
crobertsrhelmiko:  yes14:14
elmikocool, thanks14:16
aignatovtmckay: I'm back here :)14:16
*** sreshetnyak has joined #openstack-sahara14:16
aignatovis your Pig oinking already?14:17
aignatovoink - > nice word  :) never heard before :)14:17
tmckayaignatov, no :)  Right now my devstack is crashing (oslo.messaging, trying to install by hand).  but that's a different problem14:17
tmckayaignatov, lol, yeah, every American toddler learns that a pig says oink14:17
tmckayaignatov, okay, so I'm wondering if the images for hadoop 2 on savanna-ci can be published14:18
aignatovtmckay: I think they are already published14:18
tmckayaignatov, I know Pig is running on vanilla2 in ci, but my Pig doesn't.  so, it has to be the image, or a config.  I thought maybe I would start with the image.14:18
tmckayaignatov, anyway to get a listing for the contents of sahara-files.mirantis.com?  Doesn't seem to have a public index that I can find14:19
aignatovtmckay: actually it has no listing :)14:19
aignatovrecently today sreshetnyak updated links to the docs14:19
tmckayaignatov, dimitryme thought ysterday maybe they (hadoop 2 images) were missing14:19
aignatovsorry, to the images14:19
tmckayokay, I'll take a look.  the 1.2.1 links were there, didn't see the 2.3.014:20
aignatovtmckay: it seems links are updated14:22
aignatovhttp://docs.openstack.org/developer/sahara/userdoc/vanilla_plugin.html14:22
*** akuznetsov has quit IRC14:22
tmckayaignatov, so anyway, I got a vanilla hadoop 2 cluster up (driven from the cli test) and I ran mr and mr-streaming.  Just need to see Pig work, and a Java action, then I can try to run bigpetstore14:23
aignatovalso another issue why Oozie is not working properly when your cluster doesn't contain jobhistory process14:23
tmckayaignatov, oink, my pig thanks you!  That wasn't there yesterday.  How do you say thank you in Russian?14:23
aignatovхрюкать :)14:23
tmckayyeah, I added a history server14:24
aignatovtmckay, lol, I just posted translation of oinking in Russian ^ - хрюкать14:26
aignatovand "thank you" Russian says "спасибо"14:26
tmckayaignatov, heh, if I travel there someday I'll have to oink at people and see what they say.14:27
*** vrovachev has joined #openstack-sahara14:27
SergeyLukjanovtmckay, I'm afraid that it'll be very difficult to translate their answers ;)14:27
tmckayalright, this is too much.  In order to get devstack to work, I had to go to /opt/stack/oslo.messaging and do "python setup.py build; python setup.py install" manually.14:28
tmckaythat is way, way too "inside baseball".  No dev should ever have to do that.14:28
openstackgerritSergey Lukjanov proposed a change to openstack/sahara: [WIP] Start using oslotest as base for all tests  https://review.openstack.org/8604514:29
tmckayaignatov, so do you know which image savanna-ci uses for vanilla 2? centos, ubuntu, fedora?  And, do you know where to find the sahara.conf used by ci? (or the thing that generates it?)14:41
aignatovsahara.conf is here https://github.com/savanna-project/savanna-ci/blob/master/scripts/gate-savanna-integration-new.sh#L121-L13114:43
aignatovtmckay: as I remember savanna-ci uses ubuntu-based image during tests running14:44
toskySergeyLukjanov: is that [WIP] change part of the "let's port to tempest-style"?14:44
aignatovfor vanilla14:44
tmckayaignatov, okay, great, thanks.  Hopefully this will work for me this morning14:44
*** skostiuchenko has quit IRC14:45
*** juice has quit IRC14:45
*** skolekonov has quit IRC14:45
*** tellesnobrega has quit IRC14:45
*** ylobankov has quit IRC14:45
*** SergeyLukjanov has quit IRC14:45
tmckayaignatov, how is your sloc thing going?14:46
*** ylobankov has joined #openstack-sahara14:48
* tmckay doesn't think savanna-ci conf is very special ....14:49
*** SergeyLukjanov has joined #openstack-sahara14:49
*** skostiuchenko has joined #openstack-sahara14:49
SergeyLukjanovtosky, nope, it's just a migration from obsolete unittest2.TestCase to super duper oslotest.base.BaseTestCase (based on testtools)14:49
SergeyLukjanovit's an OpenStack way14:49
*** tellesnobrega has joined #openstack-sahara14:49
aignatovtmckay: actually I didn't do anything in this area yet14:49
*** juice has joined #openstack-sahara14:50
*** skolekonov has joined #openstack-sahara14:50
tmckayaignatov, well, summit is a ways away14:53
*** bradd1 has joined #openstack-sahara14:55
*** skolekonov has quit IRC14:58
aignatovtmckay: I just have another activity in Mirantis, I'll take this action on top of my priorities :)14:59
*** IlyaE has joined #openstack-sahara15:00
*** Hao has quit IRC15:05
*** IlyaE has quit IRC15:13
*** Hao has joined #openstack-sahara15:14
*** IlyaE has joined #openstack-sahara15:16
openstackgerritRobert Nettleton proposed a change to openstack/sahara-image-elements: Updates hadoop-hdp element to use sahara-based download URL for hadoop-swift  https://review.openstack.org/8606515:37
*** DenisEgorenko has left #openstack-sahara15:39
openstackgerritRobert Nettleton proposed a change to openstack/sahara-image-elements: Updates hadoop-hdp to use new project name in URL  https://review.openstack.org/8606515:41
*** ErikB has joined #openstack-sahara15:41
*** bob_nettleton has joined #openstack-sahara15:47
*** mattf has quit IRC15:50
*** mattf has joined #openstack-sahara15:51
*** bob_nettleton has quit IRC15:57
aignatovErikB: I've updated your CR today about info about hdp plugin in the docs16:11
aignatovhttps://review.openstack.org/#/c/84797/16:12
aignatovand noticed that in previous CR you misconfigured link to hdp 2.0 image16:13
aignatovplease my comment in this CR16:13
*** dmitryme has joined #openstack-sahara16:30
*** mattf is now known as _mattf17:09
*** Hao has quit IRC17:11
*** Hao has joined #openstack-sahara17:11
openstackgerritSergey Lukjanov proposed a change to openstack/sahara: [WIP] Start using oslotest as base for all tests  https://review.openstack.org/8604517:12
*** Hao has quit IRC17:15
*** Hao has joined #openstack-sahara17:19
*** Hao has quit IRC17:23
*** Hao has joined #openstack-sahara17:42
*** Hao has quit IRC17:46
*** sballe has joined #openstack-sahara17:52
*** IlyaE has quit IRC18:15
*** Hao has joined #openstack-sahara18:22
*** Hao has quit IRC18:26
*** dmitryme has quit IRC18:28
*** IlyaE has joined #openstack-sahara18:31
*** akuznetsov has joined #openstack-sahara18:34
*** akuznetsov has quit IRC18:51
*** tosky has quit IRC19:02
*** ErikB has quit IRC19:10
*** ErikB has joined #openstack-sahara19:11
*** Hao has joined #openstack-sahara19:22
crobertsrhtmckay:  I forget, did you ever come up with a way to run the "pi" example as a Java Action?19:26
*** Hao has quit IRC19:26
tmckaycrobertsrh, I don't recall.  I didn't add it to examples.19:28
crobertsrhheh :)19:29
crobertsrhOk.  I suspect the trick might have been getting the stdout sent to a file somewhere19:30
*** _mattf is now known as mattf19:49
mattfdid you know, you have to go the TENTH page of google results to see why we renamed? https://www.google.com/search?q=savanna+hadoop&start=9019:49
* mattf didn't even know there was a 10th page before19:49
*** dmitryme has joined #openstack-sahara19:51
*** sballe_ has joined #openstack-sahara19:56
*** sballe has quit IRC19:59
*** Hao has joined #openstack-sahara20:23
tmckayso let me ask you all this.  Trying to run devstack with neutron.  When it comes up, I have a public network and private network and horizon lists the project value for them as "demo" and "admin".  Why are they in different projects, and does it matter?20:26
*** Hao has quit IRC20:27
dmitrymetmckay: saw the same in setups I used20:30
dmitrymeseems like it does not really matter20:31
dmitrymeit actually causes no trouble if they are marked as 'shared'20:31
dmitryme(I don't remember if that is configured automatically)20:32
tmckayok, thanks20:36
*** mattf is now known as _mattf20:40
*** dmitryme has left #openstack-sahara20:46
openstackgerritAndrew Lazarev proposed a change to openstack/sahara-image-elements: [DIB] Added qemu-img dependency for CentOS host system  https://review.openstack.org/8615220:48
*** crobertsrh is now known as _crobertsrh20:54
*** dmitryme has joined #openstack-sahara20:55
*** ruhe2 has joined #openstack-sahara21:03
*** ruhe2 has quit IRC21:06
*** dmitryme2 has joined #openstack-sahara21:06
*** dmitryme has left #openstack-sahara21:09
*** dmitryme2 has left #openstack-sahara21:10
*** dmitryme has joined #openstack-sahara21:12
*** Hao has joined #openstack-sahara21:24
*** dmitryme has left #openstack-sahara21:24
*** dmitryme has joined #openstack-sahara21:24
*** Hao has quit IRC21:28
*** ErikB has quit IRC21:34
*** ErikB1 has joined #openstack-sahara21:40
*** ErikB1 has quit IRC21:40
*** elmiko is now known as elmiko_afk21:48
*** dmitryme has left #openstack-sahara21:56
*** dmitryme has joined #openstack-sahara21:57
*** Hao has joined #openstack-sahara22:21
*** Hao has quit IRC22:23
*** Hao has joined #openstack-sahara22:23
*** ErikB1 has joined #openstack-sahara22:26
*** Hao has quit IRC22:28
*** witlessb has quit IRC22:30
*** Hao has joined #openstack-sahara23:24
*** Hao has quit IRC23:28
*** IlyaE has quit IRC23:30
*** IlyaE has joined #openstack-sahara23:49
*** ErikB1 has quit IRC23:52

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!