| *** tosky has quit IRC | 00:16 | |
| *** dave-mccowan has joined #openstack-sahara | 00:18 | |
| *** openstackgerrit has joined #openstack-sahara | 01:31 | |
| openstackgerrit | caishan proposed openstack/python-saharaclient master: Modify home page in setup.cfg https://review.openstack.org/532196 | 01:31 |
|---|---|---|
| *** shuyingya has joined #openstack-sahara | 01:41 | |
| openstackgerrit | Yandy Sun proposed openstack/sahara master: Update OpenStack Style https://review.openstack.org/535303 | 02:11 |
| openstackgerrit | Merged openstack/python-saharaclient master: Modify home page in setup.cfg https://review.openstack.org/532196 | 03:56 |
| *** links has joined #openstack-sahara | 04:00 | |
| *** dave-mccowan has quit IRC | 04:21 | |
| *** alinanesterova has joined #openstack-sahara | 04:24 | |
| *** pgadiya has joined #openstack-sahara | 04:39 | |
| *** alinanesterova has quit IRC | 06:30 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/sahara-dashboard master: Imported Translations from Zanata https://review.openstack.org/536230 | 06:52 |
| *** alinanesterova has joined #openstack-sahara | 06:56 | |
| *** rcernin has quit IRC | 07:17 | |
| *** pcaruana has joined #openstack-sahara | 07:52 | |
| *** tesseract has joined #openstack-sahara | 08:19 | |
| openstackgerrit | ShangXiao proposed openstack/sahara-dashboard master: Update url links in doc file of sahara-dashboard https://review.openstack.org/534641 | 08:52 |
| openstackgerrit | Merged openstack/sahara-dashboard master: Imported Translations from Zanata https://review.openstack.org/536230 | 08:53 |
| openstackgerrit | ShangXiao proposed openstack/sahara-dashboard master: Update url links in doc file of sahara-dashboard https://review.openstack.org/534641 | 08:57 |
| openstackgerrit | Shu Yingya proposed openstack/sahara-extra master: Add support to build hadoop-2.8.2 related package https://review.openstack.org/533048 | 09:02 |
| openstackgerrit | Yandy Sun proposed openstack/sahara master: Replace asseertFalse/assertTrue(a in b) https://review.openstack.org/535303 | 09:02 |
| openstackgerrit | Yandy Sun proposed openstack/sahara master: Replace asseertFalse/assertTrue(a in b) https://review.openstack.org/535303 | 09:04 |
| openstackgerrit | Yandy Sun proposed openstack/sahara master: Replace assertFalse/assertTrue(a in b) https://review.openstack.org/535303 | 09:21 |
| openstackgerrit | Merged openstack/sahara-tests master: Add .stestr to gitignore https://review.openstack.org/536158 | 09:54 |
| *** pgadiya has quit IRC | 09:58 | |
| *** pgadiya has joined #openstack-sahara | 09:59 | |
| *** tosky has joined #openstack-sahara | 10:17 | |
| openstackgerrit | Luigi Toscano proposed openstack/sahara master: DNM Test the new trustee section in compatibility mode https://review.openstack.org/529845 | 11:08 |
| openstackgerrit | Shu Yingya proposed openstack/sahara-extra master: Add support to build hadoop-2.8.2 related package https://review.openstack.org/533048 | 11:22 |
| *** links has quit IRC | 11:23 | |
| *** openstackgerrit has quit IRC | 11:33 | |
| *** links has joined #openstack-sahara | 11:35 | |
| *** caowei has joined #openstack-sahara | 11:50 | |
| *** tesseract-RH has joined #openstack-sahara | 11:55 | |
| *** tesseract has quit IRC | 11:57 | |
| *** tellesnobrega has joined #openstack-sahara | 12:01 | |
| *** alinanesterova has quit IRC | 12:04 | |
| *** caowei has quit IRC | 12:04 | |
| *** openstackgerrit has joined #openstack-sahara | 12:10 | |
| openstackgerrit | Merged openstack/sahara-dashboard master: Update url links in doc file of sahara-dashboard https://review.openstack.org/534641 | 12:10 |
| *** dave-mccowan has joined #openstack-sahara | 12:12 | |
| *** caowei has joined #openstack-sahara | 12:17 | |
| *** dave-mcc_ has joined #openstack-sahara | 12:21 | |
| *** dave-mccowan has quit IRC | 12:22 | |
| *** shuyingy_ has joined #openstack-sahara | 12:33 | |
| *** shuyingya has quit IRC | 12:37 | |
| *** shuyingy_ has quit IRC | 12:47 | |
| *** caowei has quit IRC | 12:48 | |
| *** alinanesterova has joined #openstack-sahara | 12:52 | |
| *** pgadiya has quit IRC | 13:42 | |
| openstackgerrit | Telles Mota Vidal Nóbrega proposed openstack/sahara master: Switch sahara swift to work with keystone v3 https://review.openstack.org/535915 | 13:53 |
| openstackgerrit | Merged openstack/sahara master: Replace assertFalse/assertTrue(a in b) https://review.openstack.org/535303 | 14:06 |
| *** alinanesterova has quit IRC | 14:27 | |
| *** alinanesterova has joined #openstack-sahara | 14:28 | |
| *** caowei has joined #openstack-sahara | 14:33 | |
| *** alinanesterova has quit IRC | 14:39 | |
| tellesnobrega | tosky, using all-in-one devstack the file transfer took 55 seconds with optimizations and 60 seconds without it. But this scenario is the best case ever, where sahara-engine runs on the same host as vms, and this won't happen often, I can't test with my laptop sahara right now (not at home and I don't have full control of my modem to allow remote connections), but I will keep digging when I'm home tomorrow or Wednesday | 14:45 |
| tosky | ack | 14:47 |
| tosky | tellesnobrega: is the optimization still the pipeline and no buffer? | 14:47 |
| tellesnobrega | at this point yes | 14:47 |
| tellesnobrega | I looked a little into the parameters you said | 14:48 |
| tellesnobrega | they come when starting ssh client from transport | 14:48 |
| tellesnobrega | I don't think this is the case for sahara | 14:48 |
| tosky | I see | 14:48 |
| *** caowei has quit IRC | 14:50 | |
| *** shuyingya has joined #openstack-sahara | 14:52 | |
| *** shuyingy_ has joined #openstack-sahara | 14:52 | |
| *** shuyingya has quit IRC | 14:56 | |
| *** caowei has joined #openstack-sahara | 14:58 | |
| *** caowei has quit IRC | 15:00 | |
| *** shuyingy_ has quit IRC | 15:24 | |
| *** shuyingya has joined #openstack-sahara | 15:25 | |
| *** shuyingya has quit IRC | 15:29 | |
| *** jeremyfreudberg has joined #openstack-sahara | 15:35 | |
| *** tellesnobrega has quit IRC | 15:38 | |
| *** SergeyLukjanov has quit IRC | 15:40 | |
| *** DuncanT has quit IRC | 15:40 | |
| *** zhuli has quit IRC | 15:40 | |
| *** SergeyLukjanov has joined #openstack-sahara | 15:40 | |
| *** zhuli has joined #openstack-sahara | 15:41 | |
| *** DuncanT has joined #openstack-sahara | 15:41 | |
| jeremyfreudberg | tosky, i thought depends-on did have the automagical capabilities i was looking for, but oh well | 15:45 |
| jeremyfreudberg | can i simply depend on another patch which exports LIBS_FROM_GIT in the sahara devstack plugin? or is it not that easy | 15:45 |
| tosky | jeremyfreudberg: or simply create a python-saharaclient patch on top of the others which Depends-On on the DNM sahara patch | 15:46 |
| jeremyfreudberg | that might work better, although then i have to pass the unit tests, no (you can only depend on a patch which passes tests, or did i imagine that?) | 15:47 |
| tosky | I don't think you have such limitation: the depends-on works before the execution of the tests, and it makes sure that all the patches are pulled | 15:48 |
| *** ukaynar has joined #openstack-sahara | 15:50 | |
| jeremyfreudberg | This change or one of its cross-repo dependencies was unable to be automatically merged with the current state of its repository. Please rebase the change and upload a new patchset. | 15:50 |
| jeremyfreudberg | i think if the dependent patch gets a verified=-1 first then that message will block the results of the patch with the dependency | 15:52 |
| jeremyfreudberg | could bew rong | 15:52 |
| tosky | yes, that's correct | 15:52 |
| tosky | one of the few cases where some resources are saved | 15:53 |
| openstackgerrit | Jeremy Freudberg proposed openstack/sahara-tests master: [DNM] Testing the APIv2 client https://review.openstack.org/536157 | 15:57 |
| jeremyfreudberg | well, there it is, i suppose my solution is elegant enough | 15:57 |
| openstackgerrit | Jeremy Freudberg proposed openstack/python-saharaclient master: Beginnings of APIv2 support in client https://review.openstack.org/533066 | 15:59 |
| tosky | jeremyfreudberg: do you need to rebase Monty's patch on top of it? | 15:59 |
| *** ukaynar has quit IRC | 16:01 | |
| *** ukaynar has joined #openstack-sahara | 16:01 | |
| jeremyfreudberg | tosky: theoretically yes, but monty's patch isn't complete | 16:02 |
| *** alinanesterova has joined #openstack-sahara | 16:02 | |
| tosky | ack | 16:03 |
| *** alinanesterova has quit IRC | 16:07 | |
| *** jkunigk_ has joined #openstack-sahara | 16:18 | |
| jkunigk_ | hello | 16:18 |
| jeremyfreudberg | jkunigk_, hello! | 16:19 |
| jkunigk_ | does sahara support other object storage solutions apart from swift? | 16:19 |
| jeremyfreudberg | jkunigk_, good question, we are working on support for S3-like object stores, but the feature won't be totally done till Rocky release | 16:20 |
| jkunigk_ | I see, cool | 16:20 |
| jeremyfreudberg | the current status of that feature: you can reference only job binaries in S3, but not data sources. i'm also about to finish up support for S3 on sahara images | 16:20 |
| jkunigk_ | would the final version include full incorporation of credentials into the horizon dashboards to identify the buckets for cluster-wide configuration, etc? | 16:22 |
| tosky | there is also a basic support for manila shares: https://specs.openstack.org/openstack/sahara-specs/specs/liberty/manila-as-a-data-source.html | 16:24 |
| jeremyfreudberg | yes, the final version of the feature will allow you to do everything you can already do with swift, but with s3 | 16:24 |
| jeremyfreudberg | there is manila support too, but i think we only a support a subset of the various filesystems offered | 16:25 |
| jeremyfreudberg | jkunigk_, am i right to assume that you are with cloudera? | 16:28 |
| jkunigk_ | jeez, what gave me away | 16:28 |
| jkunigk_ | ? | 16:28 |
| jeremyfreudberg | some people are lucky enough to have a unique name... i figured i would ask since it is rare that we hear from the vendors which we "support" | 16:30 |
| *** alinanesterova has joined #openstack-sahara | 16:32 | |
| *** ukaynar has quit IRC | 16:36 | |
| jeremyfreudberg | jkunigk_, do you know why navigator-keytrustee >=5.12 was moved to enterprise-only ? http://archive.cloudera.com/navigator-keytrustee5/ubuntu/trusty/amd64/navigator-keytrustee/dists/ | 16:36 |
| jkunigk_ | I wasn't aware it was ever a different story | 16:36 |
| jkunigk_ | rephrase: what do you mean by "enterprise only"? Linux-dist-wise? | 16:37 |
| jeremyfreudberg | jkunigk_, what i mean is: there aren't freely available packages for cloudera express users (like Sahara), it requires some subscription and logging into some website | 16:38 |
| jeremyfreudberg | a new change that happened at 5.12 | 16:38 |
| jkunigk_ | I see | 16:38 |
| jkunigk_ | navigator encrypt is meant for things you want to encrypt that the frameworks (map-reduce, HDFS, etc.) don't offer out-of-the box encryption support for | 16:39 |
| jkunigk_ | We use it for Kafka disks, Kudu volumes, etc. | 16:39 |
| jkunigk_ | and to that end it has never been freely available as such | 16:39 |
| jkunigk_ | you also need navigator key trustee server for navigator encrypt to manage encryption keys transparently | 16:40 |
| jkunigk_ | and that has never been freely available | 16:40 |
| jeremyfreudberg | hmm, my understanding is that key trustee was freely available at least based on the package listing which i linked | 16:41 |
| jkunigk_ | does this help? | 16:41 |
| jkunigk_ | Have you ever set it up ? | 16:42 |
| jkunigk_ | =) | 16:42 |
| jeremyfreudberg | key trustee has been included with sahara since this patch https://review.openstack.org/#/c/189373/ presumably at that time someone tested it manually. no one has tested it recently, though | 16:43 |
| jkunigk_ | I see | 16:44 |
| jkunigk_ | I cannot comment on the product strategy here, but it is generally advisable to leverage the service-internal encryption tools as much as possible | 16:45 |
| jkunigk_ | this is how we do it in the field | 16:45 |
| jeremyfreudberg | right | 16:45 |
| jkunigk_ | But we recommend storing encrpytion keys in keytrustee server | 16:46 |
| jkunigk_ | can you shoot a link to the keytrustee package you were referring to, or not anymore since it's just gone? | 16:46 |
| jeremyfreudberg | the name of the package as defined in the cloudera repos is keytrustee-keyprovider | 16:48 |
| jeremyfreudberg | the repo we used to get that is here: http://archive.cloudera.com/navigator-keytrustee5/ubuntu/xenial/amd64/navigator-keytrustee/ (replace xenial with trusty in that link if desired) | 16:48 |
| *** ukaynar has joined #openstack-sahara | 16:49 | |
| jkunigk_ | Ah, keyprovider is not keytrustee server | 16:49 |
| jkunigk_ | it's a glue layer package that provides keys to HDFS's key provider API | 16:50 |
| jkunigk_ | s/to/via | 16:50 |
| jkunigk_ | and gets them from keytrustee server | 16:50 |
| jeremyfreudberg | aha, hmm, and does that even make sense? how are we using that without using keytrustee itself? | 16:50 |
| jkunigk_ | you don't =) | 16:51 |
| *** tesseract-RH has quit IRC | 16:51 | |
| jkunigk_ | not possible | 16:52 |
| jeremyfreudberg | aha | 16:52 |
| jeremyfreudberg | strange | 16:52 |
| jeremyfreudberg | hadoop-kms seems to have something to do with it, though | 16:52 |
| *** alinanesterova has quit IRC | 16:53 | |
| jkunigk_ | keytrustee-keyprovider is effectively the hadoop-kms that works with keytrustee | 16:54 |
| jkunigk_ | kms is designed to be modular | 16:55 |
| jkunigk_ | hwx has ranger kms for example | 16:55 |
| jkunigk_ | the regular kms just uses a Java keystore... not very enterprise-y | 16:56 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_sahara master: Updated from OpenStack Ansible Tests https://review.openstack.org/536436 | 16:56 |
| jeremyfreudberg | just to check my understanding: keytrustee-keyprovider, hortonworks ranger kms, and upstream hadoop-kms are interchangable components | 16:56 |
| jeremyfreudberg | the difference is which keystore they talk to | 16:57 |
| *** ukaynar has quit IRC | 16:59 | |
| *** ukaynar has joined #openstack-sahara | 17:05 | |
| *** pcaruana has quit IRC | 17:11 | |
| jeremyfreudberg | jkunigk_, let me thank you for your help again. it's been essential in preventing the death of the cdh plugin | 17:11 |
| *** ukaynar has quit IRC | 17:28 | |
| *** ukaynar has joined #openstack-sahara | 17:29 | |
| jkunigk_ | sorry, was afk | 17:31 |
| jeremyfreudberg | no problem | 17:31 |
| jkunigk_ | that's what I like to hear (preventing death and all) | 17:31 |
| jeremyfreudberg | yes, i hoped that would be good news | 17:32 |
| jkunigk_ | w/r to your above statement: yup | 17:32 |
| jkunigk_ | absolutely. I personally appreciate all the work around the work going on in OpenStack around Hadoop very much | 17:32 |
| jkunigk_ | got 2 more questions | 17:33 |
| jkunigk_ | object storage other than swift, outside of the Sahara realm | 17:33 |
| *** ukaynar has quit IRC | 17:33 | |
| jkunigk_ | is that there already? I can't find a definitive statement, it looks more like it's not | 17:34 |
| jkunigk_ | S3 can be used by any instance obviously, but I mean integration into provisioning | 17:34 |
| jkunigk_ | (if that makes sense) | 17:34 |
| jkunigk_ | should I ask in another channel? | 17:34 |
| jeremyfreudberg | jkunigk_, what exactly do you mean by provisioning? | 17:35 |
| jeremyfreudberg | since you are talking 'outside of sahara' | 17:35 |
| jkunigk_ | well credentials setup I suppose | 17:35 |
| jkunigk_ | vendors that implement s3 compatible storage may have their own solution for this even, I suppose | 17:36 |
| jeremyfreudberg | right, on aws itself they use some magic | 17:36 |
| jkunigk_ | IAM... | 17:37 |
| jkunigk_ | =) | 17:37 |
| jkunigk_ | magic... | 17:37 |
| jeremyfreudberg | otherwise there is something called hadoop credentialprovider (which sahara does not use yet) which supports s3 stuff, i believe | 17:37 |
| jkunigk_ | ok | 17:38 |
| jeremyfreudberg | if you don't use that, from what i've seen most vendors just throw in the necessary jars and say good look setting this up | 17:38 |
| jkunigk_ | interesting | 17:38 |
| jeremyfreudberg | *luck, not look | 17:38 |
| jkunigk_ | while there is a cinder support matrix by the way, I fail to see one for swift by the way | 17:38 |
| jkunigk_ | (https://wiki.openstack.org/wiki/CinderSupportMatrix) | 17:39 |
| jkunigk_ | which begs the question: how real are enterprise storage systems for openstack object storage implementations | 17:39 |
| jeremyfreudberg | ah, you mean what kind of storage actually underlies swift | 17:40 |
| jkunigk_ | I casually drifted into a follow-up question | 17:40 |
| jeremyfreudberg | that's best asked to swift people themselves, but do be aware that many clouds have ceph pretend to be swift | 17:40 |
| jkunigk_ | but yes | 17:40 |
| jkunigk_ | is there a blueprint doc / wiki page for the s3 inclusion into Sahara you mentioned initially? | 17:43 |
| jeremyfreudberg | jkunigk_, the most complete way to see it is with a gerrit query like this https://review.openstack.org/#/q/message:s3+project:%255E.*sahara.*+after:2017-05-01 | 17:46 |
| *** hoonetorg has quit IRC | 17:50 | |
| jkunigk_ | awesome. thank you very much for all this good information | 17:52 |
| *** jkunigk_ has quit IRC | 17:52 | |
| *** tellesnobrega has joined #openstack-sahara | 17:53 | |
| *** hoonetorg has joined #openstack-sahara | 18:06 | |
| jeremyfreudberg | tosky: i'm not sure how to get the testing of the client to work | 18:15 |
| jeremyfreudberg | if you look at the scenario logs on the saharaclient patch, it doe tox -e venv -- sahara-scenario which just installs saharaclient from pip | 18:15 |
| jeremyfreudberg | which is interseting, doesn't that mean we have never really tested the new client there? | 18:15 |
| tosky | uhm, interesting | 18:15 |
| jeremyfreudberg | i could be reading it wrong | 18:16 |
| tosky | right, the scenario test does not seem ready to use the system client (in this case, the client deployed by devstack from source) | 18:17 |
| jeremyfreudberg | tosky: and is that fixed in the v3 jobs? | 18:18 |
| jeremyfreudberg | zuulv3, i mean | 18:18 |
| tosky | no, because I followed the same pattern: calling tox -e venv | 18:19 |
| tosky | but I think that it can be patched to pass use --system-site-packages | 18:19 |
| jeremyfreudberg | tosky: well, good luck | 18:21 |
| tosky | I guess I need to push another test patch | 18:22 |
| jeremyfreudberg | i guess i can run sahara scenario locally, although i'm not so happy about that | 18:22 |
| openstackgerrit | Luigi Toscano proposed openstack/sahara-tests master: Add native Zuul v3 jobs https://review.openstack.org/512058 | 18:29 |
| openstackgerrit | Luigi Toscano proposed openstack/sahara-tests master: [DNM] Testing the APIv2 client https://review.openstack.org/536157 | 18:29 |
| tosky | jeremyfreudberg: something like ^^ | 18:29 |
| tosky | maybe | 18:29 |
| tosky | hopefully | 18:29 |
| tosky | let's see | 18:29 |
| jeremyfreudberg | tosky, we'll see what happens | 18:30 |
| jeremyfreudberg | that's it for me today | 18:31 |
| *** jeremyfreudberg has quit IRC | 18:31 | |
| *** pcaruana has joined #openstack-sahara | 18:32 | |
| *** alinanesterova has joined #openstack-sahara | 18:39 | |
| *** ukaynar has joined #openstack-sahara | 18:42 | |
| *** alinanesterova has quit IRC | 18:44 | |
| *** links has quit IRC | 18:55 | |
| *** alinanesterova has joined #openstack-sahara | 19:07 | |
| *** alinanesterova has quit IRC | 19:11 | |
| *** pcaruana has quit IRC | 19:43 | |
| *** tenobreg has joined #openstack-sahara | 19:46 | |
| *** tellesnobrega has quit IRC | 19:50 | |
| *** tenobreg has quit IRC | 19:54 | |
| *** dave-mcc_ is now known as dave-mcowan | 19:59 | |
| *** dave-mcowan is now known as dave-mccowan | 19:59 | |
| *** dave-mccowan is now known as dave-mcc_ | 20:00 | |
| *** ukaynar has quit IRC | 20:01 | |
| *** ukaynar has joined #openstack-sahara | 20:03 | |
| openstackgerrit | Luigi Toscano proposed openstack/python-saharaclient master: Beginnings of APIv2 support in client https://review.openstack.org/533066 | 20:38 |
| openstackgerrit | Luigi Toscano proposed openstack/sahara-tests master: [DNM] Testing the APIv2 client https://review.openstack.org/536157 | 20:39 |
| *** dave-mcc_ has quit IRC | 21:05 | |
| *** ukaynar_ has joined #openstack-sahara | 21:22 | |
| *** ukaynar has quit IRC | 21:22 | |
| *** ukaynar has joined #openstack-sahara | 21:26 | |
| *** ukaynar_ has quit IRC | 21:26 | |
| *** dave-mccowan has joined #openstack-sahara | 21:30 | |
| *** dave-mcc_ has joined #openstack-sahara | 21:51 | |
| *** dave-mccowan has quit IRC | 21:52 | |
| *** openstackgerrit has quit IRC | 22:18 | |
| *** rcernin has joined #openstack-sahara | 22:19 | |
| *** alinanesterova has joined #openstack-sahara | 23:08 | |
| *** alinanesterova has quit IRC | 23:12 | |
| *** gokhan_ has quit IRC | 23:14 | |
| *** DuncanT has quit IRC | 23:26 | |
| *** DuncanT has joined #openstack-sahara | 23:27 | |
| *** openstack has quit IRC | 23:55 | |
| *** openstack has joined #openstack-sahara | 23:58 | |
| *** ChanServ sets mode: +o openstack | 23:58 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!