*** julienvey has joined #openstack-sahara | 00:25 | |
*** julienvey has quit IRC | 00:29 | |
*** Networkn3rd has joined #openstack-sahara | 00:43 | |
*** xianghui has quit IRC | 00:54 | |
*** xianghui has joined #openstack-sahara | 00:58 | |
*** xianghui has quit IRC | 01:07 | |
*** xianghui has joined #openstack-sahara | 01:07 | |
*** alazarev has quit IRC | 01:08 | |
*** julienvey has joined #openstack-sahara | 01:25 | |
*** julienvey has quit IRC | 01:27 | |
*** julienvey has joined #openstack-sahara | 01:27 | |
*** julienvey has quit IRC | 01:32 | |
*** witlessb has joined #openstack-sahara | 01:56 | |
*** witlessb has quit IRC | 02:01 | |
*** shakamunyi has quit IRC | 02:01 | |
*** xianghui has quit IRC | 02:07 | |
*** xianghui has joined #openstack-sahara | 02:07 | |
*** julienvey has joined #openstack-sahara | 02:28 | |
*** julienvey has quit IRC | 02:33 | |
*** shakamunyi has joined #openstack-sahara | 02:34 | |
*** shakamunyi has quit IRC | 02:41 | |
*** shakamunyi has joined #openstack-sahara | 02:42 | |
*** xianghui has quit IRC | 03:09 | |
*** xianghui has joined #openstack-sahara | 03:09 | |
*** xianghui has quit IRC | 03:16 | |
*** xianghui has joined #openstack-sahara | 03:17 | |
*** julienvey has joined #openstack-sahara | 03:29 | |
*** julienvey has quit IRC | 03:33 | |
*** Networkn3rd has quit IRC | 03:50 | |
*** xianghui has quit IRC | 04:17 | |
*** xianghui has joined #openstack-sahara | 04:17 | |
*** akuznetsov has joined #openstack-sahara | 04:28 | |
*** julienvey has joined #openstack-sahara | 04:30 | |
*** julienvey has quit IRC | 04:34 | |
*** ViswaV has joined #openstack-sahara | 04:36 | |
*** ViswaV_ has joined #openstack-sahara | 04:37 | |
*** ViswaV has quit IRC | 04:41 | |
*** ViswaV has joined #openstack-sahara | 04:57 | |
*** ViswaV_ has quit IRC | 05:01 | |
*** ViswaV has quit IRC | 05:04 | |
*** alazarev has joined #openstack-sahara | 05:13 | |
*** julienvey has joined #openstack-sahara | 05:31 | |
*** julienvey has quit IRC | 05:35 | |
*** alazarev has quit IRC | 05:58 | |
*** julienvey has joined #openstack-sahara | 06:31 | |
*** julienvey has quit IRC | 06:36 | |
*** shakamunyi has quit IRC | 06:43 | |
*** shakamunyi has joined #openstack-sahara | 06:44 | |
*** shakamunyi has quit IRC | 06:47 | |
*** shakamunyi has joined #openstack-sahara | 06:47 | |
*** akuznetsov has quit IRC | 07:03 | |
*** akuznetsov has joined #openstack-sahara | 07:23 | |
*** julienvey has joined #openstack-sahara | 07:32 | |
*** julienvey has quit IRC | 07:37 | |
openstackgerrit | A change was merged to stackforge/sahara-ci-config: Revert "Temporary set jobs with Vanilla 2.4 non-voting" https://review.openstack.org/115371 | 08:07 |
---|---|---|
openstackgerrit | Denis Egorenko proposed a change to stackforge/sahara-ci-config: Optimize set of integration tests https://review.openstack.org/115539 | 08:19 |
*** witlessb has joined #openstack-sahara | 08:24 | |
*** shakamunyi has quit IRC | 08:25 | |
*** IvanBerezovskiy has joined #openstack-sahara | 08:32 | |
*** julienvey has joined #openstack-sahara | 08:33 | |
*** julienvey has quit IRC | 08:37 | |
*** Longgeek has joined #openstack-sahara | 08:46 | |
*** Longgeek has quit IRC | 08:48 | |
*** Longgeek has joined #openstack-sahara | 08:48 | |
*** shakamunyi has joined #openstack-sahara | 08:52 | |
*** shakamunyi has quit IRC | 08:56 | |
*** julienve_ has joined #openstack-sahara | 09:34 | |
*** julienve_ has quit IRC | 09:38 | |
*** tosky has joined #openstack-sahara | 09:43 | |
*** shakamunyi has joined #openstack-sahara | 09:53 | |
*** shakamunyi has quit IRC | 09:57 | |
*** openstackgerrit has quit IRC | 10:10 | |
*** julienve_ has joined #openstack-sahara | 10:34 | |
*** julienve_ has quit IRC | 10:39 | |
*** shakamunyi has joined #openstack-sahara | 10:54 | |
*** tnovacik has quit IRC | 10:54 | |
*** shakamunyi has quit IRC | 10:58 | |
*** tnovacik has joined #openstack-sahara | 11:11 | |
*** julienve_ has joined #openstack-sahara | 11:35 | |
*** julienve_ has quit IRC | 11:39 | |
*** ylobankov has joined #openstack-sahara | 11:50 | |
*** shakamunyi has joined #openstack-sahara | 11:54 | |
*** shakamunyi has quit IRC | 11:59 | |
*** tmckay has joined #openstack-sahara | 12:32 | |
*** julienve_ has joined #openstack-sahara | 12:36 | |
*** julienve_ has quit IRC | 12:40 | |
*** miqui has joined #openstack-sahara | 12:43 | |
*** shakamunyi has joined #openstack-sahara | 12:55 | |
*** shakamunyi has quit IRC | 13:00 | |
*** _elmiko is now known as elmiko | 13:05 | |
*** julienve_ has joined #openstack-sahara | 13:37 | |
*** julienve_ has quit IRC | 13:41 | |
*** Networkn3rd has joined #openstack-sahara | 13:53 | |
*** Longgeek has quit IRC | 13:55 | |
*** shakamunyi has joined #openstack-sahara | 13:56 | |
*** Longgeek has joined #openstack-sahara | 13:56 | |
*** shakamunyi has quit IRC | 14:00 | |
*** tnovacik has quit IRC | 14:20 | |
*** tnovacik has joined #openstack-sahara | 14:35 | |
*** julienve_ has joined #openstack-sahara | 14:37 | |
*** julienve_ has quit IRC | 14:42 | |
*** tnovacik is now known as tnovacik|gone | 14:48 | |
*** tnovacik|gone has quit IRC | 14:55 | |
*** shakamunyi has joined #openstack-sahara | 15:00 | |
*** openstackgerrit has joined #openstack-sahara | 15:12 | |
tmckay | hmm, so I launched a cluster, and I deleted the security group on the master node, and replace it with an empty security group, and I can still get to the Oozie server on port 11000 from my Sahara host. Why? | 15:13 |
tmckay | I thought an empty security group meant that every port was blocked. | 15:13 |
tmckay | Or, are ports always open to the private network, regardless of sec group? | 15:14 |
tmckay | (no floating ips in this case) | 15:14 |
*** Longgeek_ has joined #openstack-sahara | 15:14 | |
elmiko | i think within the private network the ports are open | 15:14 |
tmckay | okay, so I have to play with this with floating ips set up | 15:15 |
*** Networkn3rd has quit IRC | 15:16 | |
*** alazarev has joined #openstack-sahara | 15:16 | |
*** Longgeek has quit IRC | 15:17 | |
tmckay | elmiko, my test was the standard "all ports open" from tcp 1-10999 and 11001-65535 | 15:19 |
tmckay | I just want to see it work :) | 15:20 |
elmiko | cool | 15:20 |
*** julienve_ has joined #openstack-sahara | 15:38 | |
elmiko | tmckay: if you run `tox -e py27` on master, are you getting errors? | 15:41 |
tmckay | checking | 15:42 |
elmiko | thanks | 15:42 |
*** julienve_ has quit IRC | 15:43 | |
*** alazarev has quit IRC | 15:47 | |
tmckay | elmiko, still installing deps ... | 15:48 |
elmiko | tmckay: k, looks like a bunch of the current reviews are getting hit by this too | 15:49 |
*** tosky has quit IRC | 15:56 | |
*** alazarev has joined #openstack-sahara | 15:59 | |
elmiko | tmckay: i wonder if this is a version issue with something | 16:04 |
tmckay | elmiko, looks like I have some errors, too | 16:05 |
tmckay | elmiko, but I've also got local files which are contributing | 16:06 |
tmckay | let me try again | 16:06 |
*** IvanBerezovskiy has left #openstack-sahara | 16:07 | |
elmiko | k | 16:07 |
tmckay | I've got a couple of test_ng_template errors | 16:08 |
elmiko | same for me, time for a litte git-bisect | 16:09 |
tmckay | elmiko, git blame for the failing line shows me this commit 29abc401faba80966f18e5e61ebb44ebf0200bd7, although it could be somewhere else | 16:12 |
elmiko | tmckay: i'm running the tests again, i'll check it out after | 16:13 |
*** tellesnobrega has quit IRC | 16:14 | |
elmiko | tmckay: that commit certainly hits the offending code | 16:14 |
tmckay | ah, in my case it is this | 16:15 |
tmckay | one string has -1, one has -1.0 | 16:15 |
elmiko | yup, same for me | 16:15 |
tmckay | I wonder if internationalization changed it | 16:15 |
tmckay | or some ancillary commit | 16:15 |
elmiko | dunno, it looks like whatever is returning that string is not giving a unicode string, and doesn't have the float-ish value | 16:16 |
elmiko | the commit before that one works fine though | 16:16 |
*** DenisEgorenko has left #openstack-sahara | 16:17 | |
elmiko | that commit is ancient | 16:19 |
tmckay | elmiko, ah, interesting. This is "minimum" being applied by the standard json validation stuff | 16:20 |
elmiko | tmckay: huh, makes me wonder if some external dep. changed | 16:20 |
tmckay | yeah, could be | 16:20 |
tmckay | elmiko, I'm following the chain down to sahara/utils/api_validator.py | 16:22 |
tmckay | hmm, nothing obvious to me | 16:25 |
*** k4n0 has quit IRC | 16:25 | |
*** tmckay has quit IRC | 16:25 | |
elmiko | ok, now it has disappeared for me... wtf | 16:26 |
elmiko | i think i need to regen .tox after each test | 16:26 |
elmiko | weird | 16:29 |
*** Networkn3rd has joined #openstack-sahara | 16:31 | |
*** julienve_ has joined #openstack-sahara | 16:39 | |
*** julienve_ has quit IRC | 16:43 | |
*** akuznetsov has quit IRC | 16:44 | |
*** tosky has joined #openstack-sahara | 16:44 | |
*** tmckay has joined #openstack-sahara | 16:49 | |
*** tmckay has quit IRC | 16:50 | |
*** tmckay has joined #openstack-sahara | 16:50 | |
*** tellesnobrega has joined #openstack-sahara | 16:53 | |
*** _mattf is now known as mattf | 17:11 | |
*** akuznetsov has joined #openstack-sahara | 17:35 | |
openstackgerrit | Andrew Lazarev proposed a change to openstack/sahara: Implemented get_open_ports method for vanilla hadoop2 https://review.openstack.org/110518 | 17:37 |
openstackgerrit | Andrew Lazarev proposed a change to openstack/sahara: Added ability to create security group automatically https://review.openstack.org/109394 | 17:37 |
*** julienvey has joined #openstack-sahara | 17:40 | |
openstackgerrit | Andrew Lazarev proposed a change to openstack/sahara: Moved URI getters from plugin SPI to edp_engine https://review.openstack.org/115400 | 17:41 |
openstackgerrit | Andrew Lazarev proposed a change to openstack/sahara: Moved get_oozie_server from plugin SPI to edp_engine https://review.openstack.org/115403 | 17:41 |
openstackgerrit | Andrew Lazarev proposed a change to openstack/sahara: Added create_hdfs_dir method to oozie edp engine https://review.openstack.org/115115 | 17:41 |
*** akuznetsov has quit IRC | 17:42 | |
*** julienvey has quit IRC | 17:44 | |
openstackgerrit | Andrew Lazarev proposed a change to openstack/sahara: Do not rely on hash ordering in tests https://review.openstack.org/112736 | 17:48 |
*** tellesnobrega has quit IRC | 17:48 | |
*** tnovacik has joined #openstack-sahara | 17:53 | |
openstackgerrit | Andrew Lazarev proposed a change to openstack/sahara: Updated docs with security group management feature https://review.openstack.org/110517 | 18:00 |
*** tellesnobrega has joined #openstack-sahara | 18:01 | |
elmiko | alazarev: have you seen these errors before? http://logs.openstack.org/54/115654/1/check/gate-sahara-python27/479ef7a/console.html | 18:02 |
alazarev | elmiko: https://bugs.launchpad.net/sahara/+bug/1359284 | 18:02 |
elmiko | alazarev: makes sense, thanks! | 18:03 |
alazarev | elmiko: and fix at https://review.openstack.org/#/c/112736/4 | 18:04 |
alazarev | elmiko: but it looks we need it faster… simpliest solution is to update requirements to either “jsonschema>=2.4.0” or “jsonschema<2.0.0” | 18:05 |
*** ViswaV has joined #openstack-sahara | 18:05 | |
alazarev | do you know our policy about requirements? | 18:06 |
elmiko | alazarev: i do not know the policy about requirements, i agree though we need the fix soon. | 18:06 |
*** alazarev has quit IRC | 18:08 | |
*** tosky has quit IRC | 18:24 | |
*** alazarev has joined #openstack-sahara | 18:26 | |
*** miqui has quit IRC | 18:31 | |
tmckay | elmiko, ping | 18:40 |
*** julienvey has joined #openstack-sahara | 18:40 | |
elmiko | tmckay: hey | 18:44 |
*** julienvey has quit IRC | 18:45 | |
elmiko | tmckay: re: that bug earlier, alazarev showed me https://bugs.launchpad.net/sahara/+bug/1359284 and https://review.openstack.org/#/c/112736/4 | 18:45 |
tmckay | elmiko, hey. This horse is not quite dead ... so I set up floating ips and got a cluster (with nova). I left out port 11000, and I can't hit it from the public ip. I can hit it from private. But, that means that anyone who can launch an instance on this openstack instance can hit the port from a new machine. | 18:45 |
tmckay | (cool on the bug) | 18:45 |
tmckay | So, I just fired off a new instance, not part of the cluster, and it's on the same private network | 18:46 |
elmiko | tmckay: i would expect that. if you can put a machine on the private net, you will have access. | 18:46 |
elmiko | tmckay: that's kinda why i liked the iptables approach as a first measure | 18:46 |
alazarev | Sahara doesn’t have own requirements (we even don’t have check_requirements job) | 18:47 |
alazarev | so, I commited to global: https://review.openstack.org/115735 | 18:47 |
elmiko | alazarev: very nice | 18:47 |
tmckay | elmiko, yeah, iptables. Because, consider something big like os1 with a lot of people in the same tenant | 18:47 |
tmckay | You still may not want me perusing your swift objects | 18:48 |
elmiko | tmckay: exactly | 18:48 |
elmiko | tmckay: actually though, if we all exist in the same tenant you can't stop me from looking at your swift containers | 18:48 |
tmckay | elmiko, yeah, I just had that thought | 18:50 |
tmckay | elmiko, so same tenant gets you everybody's swift, and network access to everybody's box, unless they have layered iptables on that host specifically | 18:51 |
tmckay | elmiko, so if that's the case, ^^, then secgroup blocks access from public if you're using floating ips and an oozie server. If you're in the tenant, there is no restriction except via iptable | 18:57 |
tmckay | and even then, you're not really blocking because swift is open to you anyway | 18:57 |
tmckay | multiple people in the same tenant really seems to mean "I inherently trust you" | 18:58 |
*** miqui has joined #openstack-sahara | 18:58 | |
elmiko | tmckay: i'm not sure about the tenant-network mapping. | 19:00 |
elmiko | tmckay: about same tenant trust though, yea if you are all in the same tenant(called project in v3), you share access to many things. this can be limited with roles though, i believe. | 19:01 |
*** alazarev has quit IRC | 19:07 | |
*** mattf is now known as _mattf | 19:17 | |
*** alazarev has joined #openstack-sahara | 19:25 | |
*** tmckay has quit IRC | 19:26 | |
*** tnovacik has quit IRC | 19:35 | |
*** julienvey has joined #openstack-sahara | 19:41 | |
*** julienvey has quit IRC | 19:46 | |
elmiko | alazarev: why take out PYTHONHASHSEED in that patch? (https://review.openstack.org/#/c/112736/4) | 19:46 |
alazarev | elmiko: to make it random as recommended by tox | 19:47 |
alazarev | elmiko: tests must not rely on results of random() calls | 19:47 |
elmiko | alazarev: k, just saw that bug #1348818 | 19:47 |
elmiko | alazarev: i thought we had just put that in for a different bug | 19:47 |
alazarev | elmiko: why different? this is exactly about that | 19:48 |
alazarev | elmiko: I’ll remove #1359284 from that CR | 19:49 |
elmiko | alazarev: no, no reason to. i was mistaken | 19:49 |
elmiko | alazarev: i thought we had just put PYTHONHASHSEED=0 in to resolve a different bug. | 19:50 |
elmiko | i just didn't understand why it was being taken out | 19:50 |
openstackgerrit | Andrew Lazarev proposed a change to openstack/sahara: Do not rely on hash ordering in tests https://review.openstack.org/112736 | 19:52 |
elmiko | alazarev: that CR does fix #1359284 for me | 19:55 |
elmiko | alazarev: either way, you've got my +1 | 19:56 |
alazarev | elmiko: yes, it fixed #1359284, but we need to fix it faster because of broken Jenkins, so I commited https://review.openstack.org/115735 | 19:57 |
elmiko | alazarev: yea, i saw 115735, wasn't sure if i should +1 though | 19:57 |
elmiko | i don't have much experience with the global requirements | 19:57 |
alazarev | elmiko: I too :) | 19:58 |
*** tmckay has joined #openstack-sahara | 19:58 | |
elmiko | alazarev: lol, more than me! =) | 19:58 |
alazarev | elmiko: Sahara had own requirements before integration… so I though I just need to update it | 19:58 |
alazarev | elmiko: but now reqs automatically sync with global | 19:59 |
elmiko | alazarev: ahh, ok | 20:01 |
*** Networkn3rd has quit IRC | 20:12 | |
*** Longgeek_ has quit IRC | 20:31 | |
*** ViswaV has quit IRC | 20:34 | |
*** julienvey has joined #openstack-sahara | 20:42 | |
*** julienvey has quit IRC | 20:47 | |
*** julienve_ has joined #openstack-sahara | 20:47 | |
*** ViswaV has joined #openstack-sahara | 21:09 | |
openstackgerrit | Michael McCune proposed a change to openstack/sahara: Adding configuration and check for proxy domain https://review.openstack.org/115654 | 21:48 |
ViswaV | Hi all, quick question, does sahara require/mandate access to a PyPi/debian or other such package servers from the guest agent , in order to install applications on the guest agent or is it left to the choice of users of sahara service to provide fully baked images with everything needed on guest agent side vs providing PyPi and debian pkg servers etc? | 22:01 |
elmiko | ViswaV: currently, we use pregenerated images that can be found online or produced using the sahara-image-elements project | 22:04 |
elmiko | ViswaV: with the exception of the centos hdp images, i don't think they require access to pypi or pkg servers | 22:04 |
ViswaV | Thanks @elmiko . I believe Trove does something similar as well (pre-generated images via disk image builder) . I am trying to understand if there is a strong bias within the Openstack satellite service projects ecosystem towards pre-generated images for their guest agents (if the services have guest agent side components), and if so , are there strong reasons for that. | 22:08 |
elmiko | ViswaV: not sure if there is a greater bias across OpenStack. in the case of Sahara, i believe the pre-generated images are used to avoid issues with configuring hadoop and the other services. | 22:09 |
ViswaV | It seems assuming the availability of a standard debian pkg server or PyPi server within the Openstack infra (especially to the tenant/vm space), is a bit brittle. | 22:09 |
elmiko | ViswaV: that being said, in the case of at least 1 of our images(centos hdp) it does try to download packages from Hortonworks | 22:10 |
ViswaV | ok. got it. Thx for pointing to that case. | 22:10 |
elmiko | ViswaV: i think the reasoning for using so many pregenerated images is that you might not always be able to count on having network access to the public internet | 22:11 |
elmiko | np =) | 22:11 |
ViswaV | Yeah..that is the reason I have heard from many folks. And also security implications of providing access to public internet to the VMs.. | 22:11 |
elmiko | exactly | 22:11 |
elmiko | we had a bug about the centos hdp images not working without access to the internet. | 22:12 |
ViswaV | What did it turn out to be due to? | 22:13 |
elmiko | in the Sahara plugin for those images, there were remote ssh commands attempting to update hdp packages from Hortonworks. | 22:13 |
ViswaV | Ah. ok. | 22:14 |
elmiko | the images actually contained older versions of those components, so we were able to refine the plugin to use the installed versions if no access was available. | 22:14 |
ViswaV | ok. How do you guys manage to keep up pace with newer versions of hadoop ecosystem of applications if you rely on pre-generated images? Generating images to catchup with a new version of some hadoop tool/project is cumbersome and time consuming right? | 22:15 |
ViswaV | I guess the sahara-image-elements, probably alleviates that burden since it is automated the image creation a lot. | 22:16 |
elmiko | ViswaV: right, so we update the image-elements when needed and we have developers working on the plugins for various technologies | 22:18 |
ViswaV | Makes sense. | 22:19 |
elmiko | ViswaV: for example, we just released a 2.4.0 plugin for Hadoop and we have Storm and Spark 1.0 plugins in the pipeline. | 22:19 |
ViswaV | what does the 'plugin' mean in this context? Specifically storm/spark plugins.. Are they part of sahara-image-elements project or sahara project? | 22:20 |
elmiko | ViswaV: sorry, the plugins are part of Sahara. they interface the different platforms we wish to execute jobs on. | 22:22 |
ViswaV | Got it. thx. | 22:22 |
elmiko | i believe Hadoop was the initial target, but the project has grown since. | 22:22 |
ViswaV | Storm is more real time data processing (Stream processing) and hadoop is batch map reduce… how are these coming together under same Sahara umbrella… is the scope of sahara increasing to encompass any big data related stuff? | 22:23 |
elmiko | as i understand it, we are accepting patches from folks who wish to see various platforms integrated. the growth to Spark and Storm is just starting to take off, so we are starting to abstract the interfaces away from being Hadoop specific. | 22:25 |
elmiko | but i've heard talk about attempting to bring Mesos as an option over Oozie, for some cases. | 22:25 |
elmiko | and the Hortonworks crew maintains an image that uses Ambari and their setup on top of Hadoop | 22:26 |
ViswaV | Interesting. | 22:27 |
elmiko | yea, lots of activity going on | 22:27 |
ViswaV | Any others beyond storm and spark… like kafka, samza etc.. ? | 22:29 |
elmiko | not that i know of, i've only heard about Spark and Storm, both of which are coming along nicely. | 22:30 |
elmiko | we had a Spark 0.9 version, but recently there has been work to bring it up to 1.0 | 22:30 |
elmiko | and i think the Storm patches are under review, but don't quote me on that lol | 22:31 |
ViswaV | It's been a while since I looked at Sahara. Looks like lot of new stuff added. | 22:32 |
elmiko | we're making nice progress :) | 22:32 |
ViswaV | Where could I find more information on what other real time stream processing plugins are on the roadmap? I looked at the blueprints page and only found Storm and Spark related blueprints so far. | 22:38 |
elmiko | i think the roadmap is up to date, not sure if any other groups are working on other plugins | 22:39 |
ViswaV | ok thx. | 22:39 |
elmiko | np, glad to help | 22:39 |
*** elmiko is now known as _elmiko | 22:44 | |
*** tmckay has quit IRC | 22:48 | |
*** tmckay has joined #openstack-sahara | 22:49 | |
*** julienve_ has quit IRC | 22:59 | |
openstackgerrit | Andrew Lazarev proposed a change to openstack/sahara: Fixed wrong use of testtools.ExpectedException https://review.openstack.org/115814 | 23:04 |
*** shakamunyi has quit IRC | 23:13 | |
*** witlessb has quit IRC | 23:40 | |
*** openstackgerrit has quit IRC | 23:54 | |
alazarev | tmckay: last patch in edp_engine series: https://review.openstack.org/#/c/115823/ | 23:57 |
*** openstackgerrit has joined #openstack-sahara | 23:57 | |
alazarev | tmckay: do you have something in mind that can be moved too? | 23:58 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!