Wednesday, 2014-08-20

*** julienvey has joined #openstack-sahara00:25
*** julienvey has quit IRC00:29
*** Networkn3rd has joined #openstack-sahara00:43
*** xianghui has quit IRC00:54
*** xianghui has joined #openstack-sahara00:58
*** xianghui has quit IRC01:07
*** xianghui has joined #openstack-sahara01:07
*** alazarev has quit IRC01:08
*** julienvey has joined #openstack-sahara01:25
*** julienvey has quit IRC01:27
*** julienvey has joined #openstack-sahara01:27
*** julienvey has quit IRC01:32
*** witlessb has joined #openstack-sahara01:56
*** witlessb has quit IRC02:01
*** shakamunyi has quit IRC02:01
*** xianghui has quit IRC02:07
*** xianghui has joined #openstack-sahara02:07
*** julienvey has joined #openstack-sahara02:28
*** julienvey has quit IRC02:33
*** shakamunyi has joined #openstack-sahara02:34
*** shakamunyi has quit IRC02:41
*** shakamunyi has joined #openstack-sahara02:42
*** xianghui has quit IRC03:09
*** xianghui has joined #openstack-sahara03:09
*** xianghui has quit IRC03:16
*** xianghui has joined #openstack-sahara03:17
*** julienvey has joined #openstack-sahara03:29
*** julienvey has quit IRC03:33
*** Networkn3rd has quit IRC03:50
*** xianghui has quit IRC04:17
*** xianghui has joined #openstack-sahara04:17
*** akuznetsov has joined #openstack-sahara04:28
*** julienvey has joined #openstack-sahara04:30
*** julienvey has quit IRC04:34
*** ViswaV has joined #openstack-sahara04:36
*** ViswaV_ has joined #openstack-sahara04:37
*** ViswaV has quit IRC04:41
*** ViswaV has joined #openstack-sahara04:57
*** ViswaV_ has quit IRC05:01
*** ViswaV has quit IRC05:04
*** alazarev has joined #openstack-sahara05:13
*** julienvey has joined #openstack-sahara05:31
*** julienvey has quit IRC05:35
*** alazarev has quit IRC05:58
*** julienvey has joined #openstack-sahara06:31
*** julienvey has quit IRC06:36
*** shakamunyi has quit IRC06:43
*** shakamunyi has joined #openstack-sahara06:44
*** shakamunyi has quit IRC06:47
*** shakamunyi has joined #openstack-sahara06:47
*** akuznetsov has quit IRC07:03
*** akuznetsov has joined #openstack-sahara07:23
*** julienvey has joined #openstack-sahara07:32
*** julienvey has quit IRC07:37
openstackgerritA change was merged to stackforge/sahara-ci-config: Revert "Temporary set jobs with Vanilla 2.4 non-voting"  https://review.openstack.org/11537108:07
openstackgerritDenis Egorenko proposed a change to stackforge/sahara-ci-config: Optimize set of integration tests  https://review.openstack.org/11553908:19
*** witlessb has joined #openstack-sahara08:24
*** shakamunyi has quit IRC08:25
*** IvanBerezovskiy has joined #openstack-sahara08:32
*** julienvey has joined #openstack-sahara08:33
*** julienvey has quit IRC08:37
*** Longgeek has joined #openstack-sahara08:46
*** Longgeek has quit IRC08:48
*** Longgeek has joined #openstack-sahara08:48
*** shakamunyi has joined #openstack-sahara08:52
*** shakamunyi has quit IRC08:56
*** julienve_ has joined #openstack-sahara09:34
*** julienve_ has quit IRC09:38
*** tosky has joined #openstack-sahara09:43
*** shakamunyi has joined #openstack-sahara09:53
*** shakamunyi has quit IRC09:57
*** openstackgerrit has quit IRC10:10
*** julienve_ has joined #openstack-sahara10:34
*** julienve_ has quit IRC10:39
*** shakamunyi has joined #openstack-sahara10:54
*** tnovacik has quit IRC10:54
*** shakamunyi has quit IRC10:58
*** tnovacik has joined #openstack-sahara11:11
*** julienve_ has joined #openstack-sahara11:35
*** julienve_ has quit IRC11:39
*** ylobankov has joined #openstack-sahara11:50
*** shakamunyi has joined #openstack-sahara11:54
*** shakamunyi has quit IRC11:59
*** tmckay has joined #openstack-sahara12:32
*** julienve_ has joined #openstack-sahara12:36
*** julienve_ has quit IRC12:40
*** miqui has joined #openstack-sahara12:43
*** shakamunyi has joined #openstack-sahara12:55
*** shakamunyi has quit IRC13:00
*** _elmiko is now known as elmiko13:05
*** julienve_ has joined #openstack-sahara13:37
*** julienve_ has quit IRC13:41
*** Networkn3rd has joined #openstack-sahara13:53
*** Longgeek has quit IRC13:55
*** shakamunyi has joined #openstack-sahara13:56
*** Longgeek has joined #openstack-sahara13:56
*** shakamunyi has quit IRC14:00
*** tnovacik has quit IRC14:20
*** tnovacik has joined #openstack-sahara14:35
*** julienve_ has joined #openstack-sahara14:37
*** julienve_ has quit IRC14:42
*** tnovacik is now known as tnovacik|gone14:48
*** tnovacik|gone has quit IRC14:55
*** shakamunyi has joined #openstack-sahara15:00
*** openstackgerrit has joined #openstack-sahara15:12
tmckayhmm, so I launched a cluster, and I deleted the security group on the master node, and replace it with an empty security group, and I can still get to the Oozie server on port 11000 from my Sahara host.  Why?15:13
tmckayI thought an empty security group meant that every port was blocked.15:13
tmckayOr, are ports always open to the private network, regardless of sec group?15:14
tmckay(no floating ips in this case)15:14
*** Longgeek_ has joined #openstack-sahara15:14
elmikoi think within the private network the ports are open15:14
tmckayokay, so I have to play with this with floating ips set up15:15
*** Networkn3rd has quit IRC15:16
*** alazarev has joined #openstack-sahara15:16
*** Longgeek has quit IRC15:17
tmckayelmiko, my test was the standard "all ports open" from tcp 1-10999 and 11001-6553515:19
tmckayI just want to see it work :)15:20
elmikocool15:20
*** julienve_ has joined #openstack-sahara15:38
elmikotmckay: if you run `tox -e py27` on master, are you getting errors?15:41
tmckaychecking15:42
elmikothanks15:42
*** julienve_ has quit IRC15:43
*** alazarev has quit IRC15:47
tmckayelmiko, still installing deps ...15:48
elmikotmckay: k, looks like a bunch of the current reviews are getting hit by this too15:49
*** tosky has quit IRC15:56
*** alazarev has joined #openstack-sahara15:59
elmikotmckay: i wonder if this is a version issue with something16:04
tmckayelmiko, looks like I have some errors, too16:05
tmckayelmiko, but I've also got local files which are contributing16:06
tmckaylet me try again16:06
*** IvanBerezovskiy has left #openstack-sahara16:07
elmikok16:07
tmckayI've got a couple of test_ng_template errors16:08
elmikosame for me, time for a litte git-bisect16:09
tmckayelmiko, git blame for the failing line shows me this commit 29abc401faba80966f18e5e61ebb44ebf0200bd7, although it could be somewhere else16:12
elmikotmckay: i'm running the tests again, i'll check it out after16:13
*** tellesnobrega has quit IRC16:14
elmikotmckay: that commit certainly hits the offending code16:14
tmckayah, in my case it is this16:15
tmckayone string has -1, one has -1.016:15
elmikoyup, same for me16:15
tmckayI wonder if internationalization changed it16:15
tmckayor some ancillary commit16:15
elmikodunno, it looks like whatever is returning that string is not giving a unicode string, and doesn't have the float-ish value16:16
elmikothe commit before that one works fine though16:16
*** DenisEgorenko has left #openstack-sahara16:17
elmikothat commit is ancient16:19
tmckayelmiko, ah, interesting.  This is "minimum" being applied by the standard json validation stuff16:20
elmikotmckay: huh, makes me wonder if some external dep. changed16:20
tmckayyeah, could be16:20
tmckayelmiko, I'm following the chain down to sahara/utils/api_validator.py16:22
tmckayhmm, nothing obvious to me16:25
*** k4n0 has quit IRC16:25
*** tmckay has quit IRC16:25
elmikook, now it has disappeared for me... wtf16:26
elmikoi think i need to regen .tox after each test16:26
elmikoweird16:29
*** Networkn3rd has joined #openstack-sahara16:31
*** julienve_ has joined #openstack-sahara16:39
*** julienve_ has quit IRC16:43
*** akuznetsov has quit IRC16:44
*** tosky has joined #openstack-sahara16:44
*** tmckay has joined #openstack-sahara16:49
*** tmckay has quit IRC16:50
*** tmckay has joined #openstack-sahara16:50
*** tellesnobrega has joined #openstack-sahara16:53
*** _mattf is now known as mattf17:11
*** akuznetsov has joined #openstack-sahara17:35
openstackgerritAndrew Lazarev proposed a change to openstack/sahara: Implemented get_open_ports method for vanilla hadoop2  https://review.openstack.org/11051817:37
openstackgerritAndrew Lazarev proposed a change to openstack/sahara: Added ability to create security group automatically  https://review.openstack.org/10939417:37
*** julienvey has joined #openstack-sahara17:40
openstackgerritAndrew Lazarev proposed a change to openstack/sahara: Moved URI getters from plugin SPI to edp_engine  https://review.openstack.org/11540017:41
openstackgerritAndrew Lazarev proposed a change to openstack/sahara: Moved get_oozie_server from plugin SPI to edp_engine  https://review.openstack.org/11540317:41
openstackgerritAndrew Lazarev proposed a change to openstack/sahara: Added create_hdfs_dir method to oozie edp engine  https://review.openstack.org/11511517:41
*** akuznetsov has quit IRC17:42
*** julienvey has quit IRC17:44
openstackgerritAndrew Lazarev proposed a change to openstack/sahara: Do not rely on hash ordering in tests  https://review.openstack.org/11273617:48
*** tellesnobrega has quit IRC17:48
*** tnovacik has joined #openstack-sahara17:53
openstackgerritAndrew Lazarev proposed a change to openstack/sahara: Updated docs with security group management feature  https://review.openstack.org/11051718:00
*** tellesnobrega has joined #openstack-sahara18:01
elmikoalazarev: have you seen these errors before? http://logs.openstack.org/54/115654/1/check/gate-sahara-python27/479ef7a/console.html18:02
alazarevelmiko: https://bugs.launchpad.net/sahara/+bug/135928418:02
elmikoalazarev: makes sense, thanks!18:03
alazarevelmiko: and fix at https://review.openstack.org/#/c/112736/418:04
alazarevelmiko: but it looks we need it faster… simpliest solution is to update requirements to either “jsonschema>=2.4.0” or “jsonschema<2.0.0”18:05
*** ViswaV has joined #openstack-sahara18:05
alazarevdo you know our policy about requirements?18:06
elmikoalazarev: i do not know the policy about requirements, i agree though we need the fix soon.18:06
*** alazarev has quit IRC18:08
*** tosky has quit IRC18:24
*** alazarev has joined #openstack-sahara18:26
*** miqui has quit IRC18:31
tmckayelmiko, ping18:40
*** julienvey has joined #openstack-sahara18:40
elmikotmckay: hey18:44
*** julienvey has quit IRC18:45
elmikotmckay: re: that bug earlier, alazarev showed me https://bugs.launchpad.net/sahara/+bug/1359284 and https://review.openstack.org/#/c/112736/418:45
tmckayelmiko, hey.  This horse is not quite dead ... so I set up floating ips and got a cluster (with nova).  I left out port 11000, and I can't hit it from the public ip.  I can hit it from private.  But, that means that anyone who can launch an instance on this openstack instance can hit the port from a new machine.18:45
tmckay(cool on the bug)18:45
tmckaySo, I just fired off a new instance, not part of the cluster, and it's on the same private network18:46
elmikotmckay: i would expect that. if you can put a machine on the private net, you will have access.18:46
elmikotmckay: that's kinda why i liked the iptables approach as a first measure18:46
alazarevSahara doesn’t have own requirements (we even don’t have check_requirements job)18:47
alazarevso, I commited to global: https://review.openstack.org/11573518:47
elmikoalazarev: very nice18:47
tmckayelmiko, yeah, iptables.  Because, consider something big like os1 with a lot of people in the same tenant18:47
tmckayYou still may not want me perusing your swift objects18:48
elmikotmckay: exactly18:48
elmikotmckay: actually though, if we all exist in the same tenant you can't stop me from looking at your swift containers18:48
tmckayelmiko, yeah, I just had that thought18:50
tmckayelmiko, so same tenant gets you everybody's swift, and network access to everybody's box, unless they have layered iptables on that host specifically18:51
tmckayelmiko, so if that's the case, ^^, then secgroup blocks access from public if you're using floating ips and an oozie server.  If you're in the tenant, there is no restriction except via iptable18:57
tmckayand even then, you're not really blocking because swift is open to you anyway18:57
tmckaymultiple people in the same tenant really seems to mean "I inherently trust you"18:58
*** miqui has joined #openstack-sahara18:58
elmikotmckay: i'm not sure about the tenant-network mapping.19:00
elmikotmckay: about same tenant trust though, yea if you are all in the same tenant(called project in v3), you share access to many things. this can be limited with roles though, i believe.19:01
*** alazarev has quit IRC19:07
*** mattf is now known as _mattf19:17
*** alazarev has joined #openstack-sahara19:25
*** tmckay has quit IRC19:26
*** tnovacik has quit IRC19:35
*** julienvey has joined #openstack-sahara19:41
*** julienvey has quit IRC19:46
elmikoalazarev: why take out PYTHONHASHSEED in that patch? (https://review.openstack.org/#/c/112736/4)19:46
alazarevelmiko: to make it random as recommended by tox19:47
alazarevelmiko: tests must not rely on results of random() calls19:47
elmikoalazarev: k, just saw that bug #134881819:47
elmikoalazarev: i thought we had just put that in for a different bug19:47
alazarevelmiko: why different? this is exactly about that19:48
alazarevelmiko: I’ll remove #1359284 from that CR19:49
elmikoalazarev: no, no reason to. i was mistaken19:49
elmikoalazarev: i thought we had just put PYTHONHASHSEED=0 in to resolve a different bug.19:50
elmikoi just didn't understand why it was being taken out19:50
openstackgerritAndrew Lazarev proposed a change to openstack/sahara: Do not rely on hash ordering in tests  https://review.openstack.org/11273619:52
elmikoalazarev: that CR does fix #1359284 for me19:55
elmikoalazarev: either way, you've got my +119:56
alazarevelmiko: yes, it fixed #1359284, but we need to fix it faster because of broken Jenkins, so I commited https://review.openstack.org/11573519:57
elmikoalazarev: yea, i saw 115735, wasn't sure if i should +1 though19:57
elmikoi don't have much experience with the global requirements19:57
alazarevelmiko: I too :)19:58
*** tmckay has joined #openstack-sahara19:58
elmikoalazarev: lol, more than me! =)19:58
alazarevelmiko: Sahara had own requirements before integration… so I though I just need to update it19:58
alazarevelmiko: but now reqs automatically sync with global19:59
elmikoalazarev: ahh, ok20:01
*** Networkn3rd has quit IRC20:12
*** Longgeek_ has quit IRC20:31
*** ViswaV has quit IRC20:34
*** julienvey has joined #openstack-sahara20:42
*** julienvey has quit IRC20:47
*** julienve_ has joined #openstack-sahara20:47
*** ViswaV has joined #openstack-sahara21:09
openstackgerritMichael McCune proposed a change to openstack/sahara: Adding configuration and check for proxy domain  https://review.openstack.org/11565421:48
ViswaVHi all, quick question,  does sahara require/mandate access to a PyPi/debian or other such package servers from the guest agent , in order to install applications on the guest agent or is it left to the choice of users of sahara service to provide fully baked images with everything needed on guest agent side vs providing PyPi and debian pkg servers etc?22:01
elmikoViswaV: currently, we use pregenerated images that can be found online or produced using the sahara-image-elements project22:04
elmikoViswaV: with the exception of the centos hdp images, i don't think they require access to pypi or pkg servers22:04
ViswaVThanks @elmiko . I believe Trove does something similar as well (pre-generated images  via disk image builder) . I am trying to understand if there is a strong bias within the Openstack satellite service projects ecosystem towards pre-generated images for their guest agents (if the services have guest agent side components),  and if so , are there strong reasons for that.22:08
elmikoViswaV: not sure if there is a greater bias across OpenStack. in the case of Sahara, i believe the pre-generated images are used to avoid issues with configuring hadoop and the other services.22:09
ViswaVIt seems assuming the availability of a standard debian pkg server or PyPi server within the Openstack infra (especially to the tenant/vm space), is a bit brittle.22:09
elmikoViswaV: that being said, in the case of at least 1 of our images(centos hdp) it does try to download packages from Hortonworks22:10
ViswaVok. got it. Thx for pointing to that case.22:10
elmikoViswaV: i think the reasoning for using so many pregenerated images is that you might not always be able to count on having network access to the public internet22:11
elmikonp =)22:11
ViswaVYeah..that is the reason I have heard from many folks. And also security implications of providing access to public internet to the VMs..22:11
elmikoexactly22:11
elmikowe had a bug about the centos hdp images not working without access to the internet.22:12
ViswaVWhat did it turn out to be due to?22:13
elmikoin the Sahara plugin for those images, there were remote ssh commands attempting to update hdp packages from Hortonworks.22:13
ViswaVAh. ok.22:14
elmikothe images actually contained older versions of those components, so we were able to refine the plugin to use the installed versions if no access was available.22:14
ViswaVok. How do you guys manage to keep up pace with newer versions of hadoop ecosystem of applications if you rely on pre-generated images? Generating images to catchup with a new version of some hadoop tool/project is cumbersome and time consuming right?22:15
ViswaVI guess the sahara-image-elements, probably alleviates that burden since it is automated the image creation a lot.22:16
elmikoViswaV: right, so we update the image-elements when needed and we have developers working on the plugins for various technologies22:18
ViswaVMakes sense.22:19
elmikoViswaV: for example, we just released a 2.4.0 plugin for Hadoop and we have Storm and Spark 1.0 plugins in the pipeline.22:19
ViswaVwhat does the 'plugin' mean in this context? Specifically storm/spark plugins.. Are they part of sahara-image-elements project or sahara project?22:20
elmikoViswaV: sorry, the plugins are part of Sahara. they interface the different platforms we wish to execute jobs on.22:22
ViswaVGot it. thx.22:22
elmikoi believe Hadoop was the initial target, but the project has grown since.22:22
ViswaVStorm is more real time data processing (Stream processing) and hadoop is batch map reduce… how are these coming together under same Sahara umbrella… is the scope of sahara increasing to encompass any big data related stuff?22:23
elmikoas i understand it, we are accepting patches from folks who wish to see various platforms integrated. the growth to Spark and Storm is just starting to take off, so we are starting to abstract the interfaces away from being Hadoop specific.22:25
elmikobut i've heard talk about attempting to bring Mesos as an option over Oozie, for some cases.22:25
elmikoand the Hortonworks crew maintains an image that uses Ambari and their setup on top of Hadoop22:26
ViswaVInteresting.22:27
elmikoyea, lots of activity going on22:27
ViswaVAny others beyond storm and spark… like kafka, samza etc.. ?22:29
elmikonot that i know of, i've only heard about Spark and Storm, both of which are coming along nicely.22:30
elmikowe had a Spark 0.9 version, but recently there has been work to bring it up to 1.022:30
elmikoand i think the Storm patches are under review, but don't quote me on that lol22:31
ViswaVIt's been a while since I looked at Sahara. Looks like lot of new stuff added.22:32
elmikowe're making nice progress :)22:32
ViswaVWhere could I find more information on what other real time stream processing plugins are on the roadmap? I looked at the blueprints page and only found Storm and Spark related blueprints so far.22:38
elmikoi think the roadmap is up to date, not sure if any other groups are working on other plugins22:39
ViswaVok thx.22:39
elmikonp, glad to help22:39
*** elmiko is now known as _elmiko22:44
*** tmckay has quit IRC22:48
*** tmckay has joined #openstack-sahara22:49
*** julienve_ has quit IRC22:59
openstackgerritAndrew Lazarev proposed a change to openstack/sahara: Fixed wrong use of testtools.ExpectedException  https://review.openstack.org/11581423:04
*** shakamunyi has quit IRC23:13
*** witlessb has quit IRC23:40
*** openstackgerrit has quit IRC23:54
alazarevtmckay: last patch in edp_engine series: https://review.openstack.org/#/c/115823/23:57
*** openstackgerrit has joined #openstack-sahara23:57
alazarevtmckay: do you have something in mind that can be moved too?23:58

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!