| alazarev | tmckay: the only thing that I have in mind is to add refence of oozie_engine to OozieWorkflowCreator, this should simplify some things | 00:02 |
|---|---|---|
| *** _mattf is now known as mattf | 00:08 | |
| *** ViswaV has quit IRC | 00:15 | |
| *** alazarev has quit IRC | 00:29 | |
| *** Networkn3rd has joined #openstack-sahara | 00:59 | |
| *** tmckay has left #openstack-sahara | 01:01 | |
| *** shakamunyi has joined #openstack-sahara | 01:17 | |
| *** witlessb has joined #openstack-sahara | 01:56 | |
| openstackgerrit | Lawrence Davison proposed a change to openstack/sahara: Documentation correction for RESTAPI's convert-config along with suggestions from SL. https://review.openstack.org/115144 | 02:24 |
| *** shakayumi has joined #openstack-sahara | 02:24 | |
| *** shakamunyi has quit IRC | 02:28 | |
| *** alexiz has joined #openstack-sahara | 02:30 | |
| *** shakayumi has quit IRC | 02:34 | |
| *** Networkn3rd has quit IRC | 03:10 | |
| *** shakamunyi has joined #openstack-sahara | 03:54 | |
| *** alexiz has quit IRC | 03:57 | |
| *** akuznetsov has joined #openstack-sahara | 04:15 | |
| *** ViswaV has joined #openstack-sahara | 05:13 | |
| *** ViswaV_ has joined #openstack-sahara | 05:15 | |
| *** ViswaV has quit IRC | 05:15 | |
| *** ViswaV has joined #openstack-sahara | 05:16 | |
| *** ViswaV_ has quit IRC | 05:20 | |
| openstackgerrit | OpenStack Proposal Bot proposed a change to openstack/python-saharaclient: Updated from global requirements https://review.openstack.org/115869 | 05:21 |
| openstackgerrit | OpenStack Proposal Bot proposed a change to openstack/sahara: Updated from global requirements https://review.openstack.org/115609 | 05:22 |
| openstackgerrit | OpenStack Proposal Bot proposed a change to openstack/sahara: Updated from global requirements https://review.openstack.org/115609 | 05:30 |
| *** shakamunyi has quit IRC | 05:37 | |
| openstackgerrit | OpenStack Proposal Bot proposed a change to openstack/sahara: Imported Translations from Transifex https://review.openstack.org/115877 | 06:08 |
| *** k4n0 has joined #openstack-sahara | 06:25 | |
| *** ViswaV has quit IRC | 06:39 | |
| *** shakamunyi has joined #openstack-sahara | 07:03 | |
| *** shakamunyi has quit IRC | 07:08 | |
| *** ghenriks has quit IRC | 07:13 | |
| *** ghenriks has joined #openstack-sahara | 07:14 | |
| *** skolekonov has joined #openstack-sahara | 07:23 | |
| openstackgerrit | Kazuki OIKAWA proposed a change to openstack/sahara: Add Java.EDP job type https://review.openstack.org/115884 | 07:29 |
| *** julienvey has joined #openstack-sahara | 08:03 | |
| *** shakamunyi has joined #openstack-sahara | 08:04 | |
| *** shakamunyi has quit IRC | 08:09 | |
| *** Longgeek has joined #openstack-sahara | 08:10 | |
| *** Longgeek has quit IRC | 08:11 | |
| *** Longgeek has joined #openstack-sahara | 08:12 | |
| *** Timotey has joined #openstack-sahara | 08:18 | |
| *** tnovacik has joined #openstack-sahara | 08:20 | |
| *** witlessb has joined #openstack-sahara | 08:21 | |
| *** tnovacik has quit IRC | 08:22 | |
| *** tnovacik has joined #openstack-sahara | 08:36 | |
| *** IvanBerezovskiy has joined #openstack-sahara | 09:00 | |
| *** shakamunyi has joined #openstack-sahara | 09:05 | |
| *** shakamunyi has quit IRC | 09:09 | |
| *** tosky has joined #openstack-sahara | 09:54 | |
| *** shakamunyi has joined #openstack-sahara | 10:06 | |
| *** shakamunyi has quit IRC | 10:10 | |
| *** Longgeek has quit IRC | 10:35 | |
| *** Longgeek has joined #openstack-sahara | 10:36 | |
| *** Longgeek has quit IRC | 10:40 | |
| *** Longgeek has joined #openstack-sahara | 11:06 | |
| *** shakamunyi has joined #openstack-sahara | 11:07 | |
| *** Longgeek has quit IRC | 11:09 | |
| *** Longgeek has joined #openstack-sahara | 11:09 | |
| *** shakamunyi has quit IRC | 11:11 | |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack/sahara: Fix jsonschema>=2.4.0 message assertion https://review.openstack.org/115946 | 11:39 |
| *** tosky has quit IRC | 11:58 | |
| *** witlessb_ has joined #openstack-sahara | 11:58 | |
| *** witlessb has quit IRC | 12:00 | |
| *** witlessb_ is now known as witlessb | 12:00 | |
| *** Longgeek has quit IRC | 12:03 | |
| *** shakamunyi has joined #openstack-sahara | 12:07 | |
| *** shakamunyi has quit IRC | 12:11 | |
| *** Longgeek has joined #openstack-sahara | 12:12 | |
| *** Longgeek has quit IRC | 12:13 | |
| *** Longgeek has joined #openstack-sahara | 12:14 | |
| *** tosky has joined #openstack-sahara | 12:33 | |
| *** shakamunyi has joined #openstack-sahara | 13:03 | |
| *** xianghuihui has joined #openstack-sahara | 13:06 | |
| *** _elmiko is now known as elmiko | 13:07 | |
| *** xianghui has quit IRC | 13:09 | |
| *** xianghuihui has quit IRC | 13:13 | |
| *** shakamunyi has quit IRC | 13:19 | |
| *** openstackgerrit has quit IRC | 13:21 | |
| *** shakamunyi has joined #openstack-sahara | 13:36 | |
| *** Networkn3rd has joined #openstack-sahara | 13:47 | |
| *** tmckay has joined #openstack-sahara | 13:51 | |
| *** k4n0 has quit IRC | 13:55 | |
| *** openstackgerrit has joined #openstack-sahara | 13:58 | |
| tmckay | elmiko, ping, I have an idea | 14:10 |
| elmiko | tmckay: hey | 14:11 |
| tmckay | elmiko, hey. On the soft failure theme, what if validation for the creation of job binaries and data sources checks the existence of the proxy domain at that point, and fails if the object requires the proxy but the proxy is not defined or can't be found? Error returned to the UI, "Sorry, you can't do that because the proxy is missing" | 14:13 |
| tmckay | trigger would be no user/pass in the object | 14:13 |
| elmiko | tmckay: that's kinda what i'm working towards | 14:13 |
| tmckay | elmiko, aweseome. Because at that point, you can take action if you want to, or use the old scheme | 14:14 |
| tmckay | elmiko, then of course we check again at job execution launch, and error out the job with a similar message | 14:14 |
| elmiko | so, if the user doesn't provide a username/pass, and they have requested use of the proxy, but the proxy detect fails, then error in UI | 14:14 |
| tmckay | ++ | 14:14 |
| elmiko | currently though, i'm trying to fix my devstack... :( | 14:14 |
| tmckay | great way to gracefully degrade. And in the edge case, the admin can just create the proxy at that point, problem solved | 14:15 |
| tmckay | gah, pain | 14:15 |
| tmckay | elmiko, okay +2. I can't wait to see the rest of it :) | 14:18 |
| elmiko | someone really needs to containerize devstack | 14:18 |
| elmiko | tmckay: i'm working on the proxy user creation/deletion now. when i start to plumb through the job binaries and whatnot, i'll start to get into the form validations coming from horizon. i figure that's where we'll want to put the checks in place. | 14:20 |
| elmiko | tmckay: so, if the user is creating a DataSource or JobBinary, when they attempt to create with no username/pass, sahara can quickly check the proxy_domain_name config and if it's blank throw an error | 14:20 |
| tmckay | yes, exactly | 14:23 |
| tmckay | either blank, or not created, or multi-domain (None or Exception) | 14:24 |
| tmckay | you could argue that the exception case could pass validation, because external action could still be taken, but until it is the object is unusable, and I think that would be a good time to add pressure to the admin :) | 14:25 |
| elmiko | tmckay: you think it should attempt to acquire the domain at that point to ensure it's created? | 14:25 |
| tmckay | elmiko, we probably need a different error message ^^ | 14:25 |
| tmckay | One says "no config set" and you have to restart Sahara, one says "dude, your domain is wrecked" | 14:25 |
| tmckay | elmiko, if it's not a lot of overhead, yeah | 14:26 |
| elmiko | if no domain is set, and the user attempts to create a binary without username/pass, it should just say "enter a username/pass to use this swift source" | 14:26 |
| elmiko | like, what if the stack admin purposely doesn't want to use a proxy domain | 14:27 |
| elmiko | ? | 14:27 |
| tmckay | agreed. The "None" check is really simple. | 14:28 |
| *** Networkn3rd has quit IRC | 14:28 | |
| tmckay | I suppose we could fail on job launch if the domain is not retrievable | 14:28 |
| elmiko | i think we have to assume that if proxy_domain_name is None that the admin is attempting to not use the proxy | 14:28 |
| tmckay | agreed | 14:28 |
| elmiko | so, error conditions for DataSource/JobBinary would be: | 14:29 |
| tmckay | failing on job launch still gets the message across, and if the config is non-Null, then the intent was to set it up. but something happened to ti | 14:29 |
| elmiko | 1. no username/pass + proxy_domain_name=None, "please enter username/pass" | 14:29 |
| tmckay | and the test could be done in job_execution validation | 14:29 |
| elmiko | 2. no username/pass + proxy_domain_name=Set + can't find proxy domain, "error, please talk to stack admin" | 14:29 |
| tmckay | elmiko, agreed. But I could see doing #2 only on job submission. | 14:30 |
| elmiko | i think it's a nice shortcut to do the error check when the user attempts to create a binary | 14:30 |
| elmiko | i'll bring it up in the meeting today | 14:30 |
| tmckay | okay. Is it a lot of overhead? | 14:30 |
| elmiko | it's just a rest call to the keystone server | 14:30 |
| elmiko | it would be a pain to setup a bunch of binaries, then find out the proxy is misconfigured | 14:31 |
| tmckay | probably good then to check in both places (you've got to get it on job launch anyway) | 14:31 |
| tmckay | minor point though, I think the overall approach is good | 14:32 |
| elmiko | yea, there is check by default on job execution as it will be attemting to create the proxy user | 14:32 |
| *** Networkn3rd has joined #openstack-sahara | 14:34 | |
| elmiko | tmckay: so here's a question, if the proxy domain is active and the user enters a username/pass for a binary, should sahara just silently drop the creds? | 14:53 |
| tmckay | hmmm | 14:54 |
| dmitryme | elmiko: as an option, I thnk Sahara can create proxy user and trust using these provided credentials | 14:55 |
| dmitryme | this could be useful in case current user does not have access to swift data, but provided creds do grant such access | 14:56 |
| dmitryme | or, possibly, user wants to process data stored in different tenant | 14:57 |
| dmitryme | in that case tenant_name must be part of creds | 14:57 |
| tmckay | elmiko, that's a tough one. I think I would probably drop them, leaning toward better security. Even in the case dmitryme is citing, there is no need to store them in the database | 14:59 |
| dmitryme | ok, I was talking about data sources, but the same could be applied to job binaries as well | 14:59 |
| tmckay | If the proxy domain is active, and a user can be created, the creds should be tossed | 14:59 |
| dmitryme | tmckay: agree, there is no need to store the cred | 15:00 |
| dmitryme | *creds | 15:00 |
| elmiko | using the provided credentials to create the trust does add a layer of complexity | 15:00 |
| elmiko | we would still need to store the creds temporarily, as the trust won't be created when the DataSource/JobBinaries are created | 15:00 |
| tmckay | elmiko, I would skip it first pass and revisit once it all works. for now, show them the garbage can :) | 15:00 |
| openstackgerrit | OpenStack Proposal Bot proposed a change to openstack/sahara: Updated from global requirements https://review.openstack.org/115609 | 15:00 |
| elmiko | dmitryme: also, on the topic of multi-tenant swift objects. there will need to be much work done to accomplish that. | 15:01 |
| elmiko | the hadoop-swiftfs plugin assumes the object store is within the tenant provided by the credentials, and that all the objects are in that tenant. | 15:01 |
| dmitryme | elmiko: yes, I see, for current user, we will create proxy user/trust right at the time the job is launched, right? | 15:03 |
| elmiko | dmitryme: yes | 15:06 |
| elmiko | also, when we start to allow sources from multiple tenants we will need multiple trusts as well | 15:06 |
| *** skolekonov has quit IRC | 15:08 | |
| dmitryme | elmiko: seems like it is more complicated than I thought :-) | 15:12 |
| elmiko | yea | 15:12 |
| *** mattf is now known as _mattf | 15:14 | |
| *** Networkn3rd has quit IRC | 15:35 | |
| openstackgerrit | Artem Osadchiy proposed a change to openstack/sahara: MapR FS datasource https://review.openstack.org/116017 | 15:39 |
| openstackgerrit | Artem Osadchiy proposed a change to openstack/sahara-dashboard: MapR FS datasource https://review.openstack.org/116020 | 15:49 |
| *** julienvey has quit IRC | 15:53 | |
| *** Timotey has quit IRC | 16:08 | |
| *** IvanBerezovskiy has left #openstack-sahara | 16:10 | |
| elmiko | have you guys ever seen a JSONDecodeError when using the saharaclient to do a job_executions.list() ? | 16:14 |
| elmiko | do i just have an old saharaclient? | 16:14 |
| elmiko | looks like maybe i have the service catalog misconfigured | 16:17 |
| openstackgerrit | A change was merged to openstack/sahara: Fix jsonschema>=2.4.0 message assertion https://review.openstack.org/115946 | 16:18 |
| *** Networkn3rd has joined #openstack-sahara | 16:50 | |
| *** tnovacik is now known as tnovacik|gone | 16:53 | |
| *** tnovacik|gone has quit IRC | 16:57 | |
| *** Networkn3rd has quit IRC | 17:01 | |
| *** Networkn3rd has joined #openstack-sahara | 17:01 | |
| *** julienvey has joined #openstack-sahara | 17:09 | |
| *** tosky has quit IRC | 17:25 | |
| *** tosky has joined #openstack-sahara | 17:52 | |
| *** ViswaV has joined #openstack-sahara | 17:59 | |
| *** alazarev has joined #openstack-sahara | 17:59 | |
| elmiko | we having a meeting today? | 18:00 |
| elmiko | SergeyLukjanov: ^^ | 18:00 |
| SergeyLukjanov | yup, sure | 18:01 |
| SergeyLukjanov | and it's meeting time | 18:01 |
| elmiko | =) | 18:01 |
| *** ViswaV_ has joined #openstack-sahara | 18:02 | |
| SergeyLukjanov | elmiko, it's fixed in latest client version AFAIAK | 18:02 |
| tmckay | late for the meeting! | 18:02 |
| *** ViswaV has quit IRC | 18:04 | |
| *** alazarev_ has joined #openstack-sahara | 18:07 | |
| *** ViswaV has joined #openstack-sahara | 18:09 | |
| *** ViswaV__ has joined #openstack-sahara | 18:09 | |
| *** alazarev has quit IRC | 18:10 | |
| *** alazarev_ is now known as alazarev | 18:10 | |
| *** ViswaV_ has quit IRC | 18:11 | |
| *** ViswaV has quit IRC | 18:13 | |
| *** shakamunyi has quit IRC | 18:27 | |
| *** Longgeek has quit IRC | 18:44 | |
| tosky | so, quickly | 19:02 |
| tosky | the goal was sahara/horizon tests, but he had to write also with more "building block" patches | 19:03 |
| tmckay | elmiko, even in the bob/alice scenario, is it possible for bob to have different credentials in different tenants? | 19:03 |
| tosky | as you know, the review time for horizon is quite... slow | 19:04 |
| dmitryme | tmckay: it is my understanding that one user has exactly one password | 19:04 |
| tmckay | k | 19:04 |
| elmiko | tmckay: i don't think so. a user has one set of creds, that gets auth'd by the backend, then keystone provides a list of projects that user is in | 19:04 |
| tosky | so if you know horizon reviewers, especially on the test part, and you want to help pushing his patches, which will benefit sahara, well, please check them | 19:05 |
| tosky | there is a chain of dependencies, which makes thing more complicated | 19:05 |
| tmckay | k, so proxy user is still a huge improvement, and secgroup turning off port 11000 is another huge improvement. iptables is icing. I could add a few words about it in the doc, too -- "if you want to be really crazy, do this ...." | 19:05 |
| elmiko | tosky: link again please | 19:05 |
| tosky | they are here: https://review.openstack.org/#/q/status:open+owner:%22Tom%25C3%25A1%25C5%25A1+Nov%25C3%25A1%253Fik+%253Ctnovacik%2540redhat.com%253E%22,n,z | 19:05 |
| elmiko | tosky: thanks | 19:05 |
| tosky | thank you! | 19:06 |
| elmiko | tmckay: yea, that's why i think we should start putting together a general Sahara security doc. much of what we are talking about is beyond the scope of just Sahara and falls onto the stack admin/ops. | 19:06 |
| dmitryme | elmiko: sounds like a good idea | 19:07 |
| tmckay | agreed. I mostly wanted to figure out if I needed iptables before the freeze -- I'm thinking no. Docs we can add beyond the freeze, I believe. | 19:07 |
| elmiko | tmckay: it sounds like the doc fix will get wide acceptance | 19:08 |
| tmckay | yeah, and it's easier :) | 19:08 |
| elmiko | heh true that | 19:08 |
| *** alazarev has quit IRC | 19:10 | |
| *** alazarev has joined #openstack-sahara | 19:29 | |
| openstackgerrit | Andrew Lazarev proposed a change to openstack/sahara: Implemented get_open_ports method for vanilla hadoop2 https://review.openstack.org/110518 | 19:31 |
| openstackgerrit | Andrew Lazarev proposed a change to openstack/sahara: Updated docs with security group management feature https://review.openstack.org/110517 | 19:31 |
| openstackgerrit | Andrew Lazarev proposed a change to openstack/sahara: Added ability to create security group automatically https://review.openstack.org/109394 | 19:31 |
| openstackgerrit | Andrew Lazarev proposed a change to openstack/sahara: Fix some of tests that rely on hash ordering https://review.openstack.org/115132 | 19:43 |
| openstackgerrit | Andrew Lazarev proposed a change to openstack/sahara: Do not rely on hash ordering in tests https://review.openstack.org/112736 | 19:43 |
| openstackgerrit | Andrew Lazarev proposed a change to openstack/sahara: Moved validate_edp from plugin SPI to end_engine https://review.openstack.org/115823 | 19:45 |
| openstackgerrit | Andrew Lazarev proposed a change to openstack/sahara: Moved URI getters from plugin SPI to edp_engine https://review.openstack.org/115400 | 19:45 |
| openstackgerrit | Andrew Lazarev proposed a change to openstack/sahara: Made EDP engine plugin specific https://review.openstack.org/114721 | 19:45 |
| openstackgerrit | Andrew Lazarev proposed a change to openstack/sahara: Moved get_oozie_server from plugin SPI to edp_engine https://review.openstack.org/115403 | 19:45 |
| openstackgerrit | Andrew Lazarev proposed a change to openstack/sahara: Added create_hdfs_dir method to oozie edp engine https://review.openstack.org/115115 | 19:46 |
| openstackgerrit | Andrew Lazarev proposed a change to openstack/sahara: Removed one round trip to server for HDFS put https://review.openstack.org/115120 | 19:47 |
| openstackgerrit | Michael McCune proposed a change to openstack/sahara: Adding configuration and check for proxy domain https://review.openstack.org/115654 | 20:00 |
| *** Networkn3rd has quit IRC | 20:01 | |
| *** tmckay has quit IRC | 20:12 | |
| *** ViswaV__ has quit IRC | 20:15 | |
| *** ViswaV has joined #openstack-sahara | 20:54 | |
| *** alazarev has quit IRC | 20:57 | |
| *** alazarev has joined #openstack-sahara | 20:58 | |
| *** ViswaV has quit IRC | 21:03 | |
| elmiko | if i need to create a new source file, do i just use the license text with no copyright? | 21:15 |
| *** julienvey has quit IRC | 21:22 | |
| openstackgerrit | Andrew Lazarev proposed a change to openstack/sahara-specs: Make anti affinity working via server groups https://review.openstack.org/116115 | 22:01 |
| *** elmiko is now known as _elmiko | 22:12 | |
| openstackgerrit | Andrew Lazarev proposed a change to openstack/sahara: Removed support of swift-internal prefix https://review.openstack.org/116119 | 22:15 |
| *** shakamunyi has joined #openstack-sahara | 22:37 | |
| *** shakamunyi has quit IRC | 22:41 | |
| *** shakamunyi has joined #openstack-sahara | 22:51 | |
| *** tosky has quit IRC | 23:01 | |
| openstackgerrit | Andrew Lazarev proposed a change to openstack/sahara: Removed sqlite from docs https://review.openstack.org/114623 | 23:04 |
| openstackgerrit | Lawrence Davison proposed a change to openstack/sahara: Adjust RESTAPIs convert-config w/suggests from SL https://review.openstack.org/115144 | 23:29 |
| *** alazarev has quit IRC | 23:37 | |
| *** witlessb has quit IRC | 23:44 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!