| *** zhenguo has joined #openstack-sahara | 01:10 | |
| *** masber has joined #openstack-sahara | 01:27 | |
| openstackgerrit | zhulingjie proposed openstack/sahara master: Remove the duplicated word https://review.openstack.org/576325 | 03:24 |
|---|---|---|
| *** Bhujay has joined #openstack-sahara | 04:20 | |
| *** pgadiya has joined #openstack-sahara | 04:53 | |
| *** pgadiya has quit IRC | 04:53 | |
| *** Bhujay has quit IRC | 05:08 | |
| *** links has joined #openstack-sahara | 05:33 | |
| *** Bhujay has joined #openstack-sahara | 05:42 | |
| *** Bhujay has quit IRC | 06:42 | |
| *** tesseract has joined #openstack-sahara | 07:02 | |
| *** rcernin has quit IRC | 07:05 | |
| *** tosky has joined #openstack-sahara | 07:34 | |
| *** pcaruana has joined #openstack-sahara | 07:50 | |
| *** Bhujay has joined #openstack-sahara | 08:00 | |
| *** links has quit IRC | 11:06 | |
| *** links has joined #openstack-sahara | 11:22 | |
| *** links has quit IRC | 11:39 | |
| *** links has joined #openstack-sahara | 12:05 | |
| Bhujay | tosky , i was able to launch a hdp cluster finally but now back to swift integration problem . the full log with ssl and without ssl is here http://paste.openstack.org/show/723821/ | 12:36 |
| tosky | I don't know what to say | 12:36 |
| Bhujay | however , in spark 1.6 just incorporating a domain property in core-site.xml works file . | 12:36 |
| tosky | I was able to use swift | 12:36 |
| tosky | and that's the less known of my area of knowledge | 12:37 |
| tosky | oh, uh | 12:37 |
| tosky | there was a patch | 12:37 |
| tosky | Bhujay: may be this could help, see also the comments: https://review.openstack.org/#/c/572209/ | 12:38 |
| Bhujay | sure , let me check | 12:38 |
| Bhujay | tosky , well this actually adds the domain property in the core-site.xml which i have done . and with that spark1.6 cluster is working fine ... also with hdp the first stage goes through ... if you look at the log mapreduce job has been submitted , without domain name this part will also fail ... | 12:47 |
| tosky | ack, so it matches what you did and it works, good to hear | 12:47 |
| Bhujay | but the second part looks like yarn is again calling soe other library and there it fails | 12:47 |
| Bhujay | yes absolutely | 12:47 |
| tosky | uhm uhm | 12:48 |
| tosky | right now the gates are broken due to a change in keystone which affected many services, and sahara in a special way, so we can't merge patches | 12:48 |
| tosky | but feel free to vote and comment on that patch if you have an account (otherwise we will try to merge it as soon as possible) | 12:49 |
| Bhujay | I will , the only thing is i am not sure if domain info is present in the context or not , i see the sahara-engine log shows in the context arguments domain info is normally none . Need to understand that part | 12:52 |
| Bhujay | tosky: now although that is anyways required is not solving the problem entirely ...as you can see in the paste .. we need to trace the reason , if you can provide some more hints please .. | 12:54 |
| tosky | Bhujay: so, the non-SSL parts talks about a timeout in keystone; do the keystone logs say anything? | 12:56 |
| tosky | about the SSL part, I think I already asked, sorry again: is this centos or ubuntu, and which version exactly (including updates)? | 12:56 |
| Bhujay | the hdp cluster vms are centos , openstack is running on ubuntu | 12:58 |
| Bhujay | version of sahara is 8.0.1 and openstack is stable/queen | 12:58 |
| Bhujay | in my spark cluster if i remove the domain info from core-site it will give same 504 gateway time out . I am check if i can get something from keystone .. | 12:59 |
| tosky | uhm, so, the SSL error is related to the certificate missing in the keystore used by java to validate the connection | 13:00 |
| Bhujay | that i have imported using keytool | 13:00 |
| tosky | now, I'm not sure that this case was properly handled ever, and it would be interesting to see if the new version of hadoop/swift connector handles it better | 13:00 |
| tosky | but probably it requires some magic to update the keystore | 13:00 |
| tosky | oh | 13:00 |
| Bhujay | shd not be a big issue , i have plan to incorporate it later inside swift_helper for self signed certificate .. but we need first have a clean solution for connectivity | 13:02 |
| tosky | you may want (if you didn't do it already) switch the keystone logs level to debug | 13:05 |
| Bhujay | good idea , let me check | 13:05 |
| Bhujay | tosky, quick update got something in keystone , SchemaValidationError ... its a prod set up with multiple keystone containers ... will take some time to get more info ...but thansk for the idea | 13:09 |
| tosky | oh | 13:10 |
| Bhujay | "identity" is a required property ...looks like the call is being made without proper identity values ..... | 13:13 |
| tosky | in the pastebin above you showed example of calls to hadoop distcp etc etc. Are those calls the same executed by sahara code? | 13:14 |
| Bhujay | no , i had run them manually | 13:15 |
| Bhujay | i picked up the commands soemwhere from sahara documentation | 13:16 |
| tosky | oh | 13:16 |
| tosky | but then it may be that sahara does the right call with the right parameter, and that our documentation is simply outdated | 13:17 |
| tosky | i.e. not updated for keystone v3 | 13:17 |
| Bhujay | huum , ok , i was going by my experince on the spark 1.6 cluster , but that image was downloaded from upstream. let me check with a edp job .. | 13:18 |
| Bhujay | tosky, meanwhile here is the keystone log http://paste.openstack.org/show/723825 | 13:22 |
| *** links has quit IRC | 13:22 | |
| tosky | it may really be that the hadoop/swift driver that we ship does not pass the proper credentials for v3 | 13:26 |
| tosky | and we may need to switch back to the original one | 13:26 |
| *** mtsv has joined #openstack-sahara | 13:27 | |
| Bhujay | tosky , you said this b4 and it worked once .. i tried to review the code and foudn it is true .. but ... | 13:29 |
| Bhujay | got confused , iirc in upstream i cloud not found v3 but in sahara-extra it was ... and finally with spark 1.6 i did not have to replace the jar file only putting the domain info solved the problem . | 13:31 |
| Bhujay | also you see the ssl issue ... once keytool import was done in sprk 1.6 it was all ok , here for hdp first call goes through ... | 13:32 |
| tosky | those are different, I think | 13:32 |
| Bhujay | well let me fire an EDP job , i am just correcting the core-site for non non slll keystone url .. | 13:33 |
| Bhujay | on the second node | 13:34 |
| Bhujay | tosky , you are a genius .... the EDP worked like a charm ... | 13:38 |
| tosky | I wish it was so easy to be a genius :D | 13:39 |
| Bhujay | tosky, thanks for being humble ... don't know how to thanks you | 13:41 |
| tosky | so, to recap: EDP jobs are working with that patch that sets the domain - is that correct? | 13:43 |
| Bhujay | well , i had inserted domain manually but the patch will do the same ... | 13:45 |
| Bhujay | but is domain info is there when context is being built ? | 13:45 |
| tellesnobrega | q | 13:46 |
| tellesnobrega | sorry | 13:46 |
| tosky | it should come from the settings, yes | 13:47 |
| openstackgerrit | Telles Mota Vidal Nóbrega proposed openstack/python-saharaclient master: Adding boot from volume https://review.openstack.org/572536 | 13:47 |
| tellesnobrega | tosky, can you check the inheritance on this patch ^ | 13:47 |
| tellesnobrega | see if it is what you suggested | 13:47 |
| tosky | Bhujay: before it was part of the [keystone_authtoken] section, now it should be in the [trustee] configuration section | 13:47 |
| Bhujay | tosky , are we saying domain info coming from sahara.conf [trustee] section , will that serve multidomain purpose ? | 13:49 |
| tosky | Bhujay: uhm, I don't know; trustee is used for the trust delegation, so I'm not sure that support for more domains is needed there | 13:51 |
| tosky | but then I would summon jeremyfreudberg | 13:52 |
| tosky | tellesnobrega: is saharaclient/osc/v1/utils.py removed? Shouldn't it be moved to saharaclient/osc/utils.py ? | 13:52 |
| tellesnobrega | I forgot to add the file | 13:52 |
| openstackgerrit | Telles Mota Vidal Nóbrega proposed openstack/python-saharaclient master: Adding boot from volume https://review.openstack.org/572536 | 13:53 |
| Bhujay | tosky, edp is also working with ssl url with self signed cert imported in keystore . as of now i am running a script to import it , let me show you .. | 13:55 |
| Bhujay | http://paste.openstack.org/show/723827 ... do you think it will be worth while to integrate with the ssl_cert section of swift_helper ? | 13:59 |
| tosky | we probably need something like that (more generic), yeah | 14:00 |
| Bhujay | we also need flag somewhere in the horizon plugin/ or sahara.conf to check if selfsigned cert is required or not | 14:02 |
| tosky | we need a more general logic for internal selfsigned certificates | 14:02 |
| tosky | in this case, if I'm not mistaken, this is the certificate coming assigned to swift, is that right? | 14:03 |
| *** jeremyfreudberg has joined #openstack-sahara | 14:05 | |
| Bhujay | not sure got the question correctly , the certifcate will be used by the swift java client and is not assigned specifically for swift , it is the self signed cert generated on the loadbalancer for everybody to access any openstack api | 14:08 |
| tosky | that's up to the openstack deployer | 14:10 |
| tosky | other deployers may create separate certificates for each service | 14:11 |
| Bhujay | i see the point | 14:11 |
| tosky | I know that TripleO can integrate with FreeIPA to generate the certificates (https://specs.openstack.org/openstack/tripleo-specs/specs/ocata/ssl-certmonger.html) | 14:12 |
| Bhujay | in that case we may capture a user provided url from where cert can be downloaded for this purpose | 14:12 |
| tellesnobrega | tosky, what do you think of the current structure? | 14:41 |
| tellesnobrega | jeremyfreudberg, can you take a look as well? https://review.openstack.org/#/c/572536 | 14:42 |
| tellesnobrega | did a little refactoring to avoid too much copy and paste | 14:42 |
| tosky | tellesnobrega: fine, it looks | 14:42 |
| tellesnobrega | :) | 14:42 |
| tellesnobrega | cool | 14:43 |
| tellesnobrega | I just need to figure out how to mock the api_version or add it to FakeApp directly | 14:43 |
| openstackgerrit | Jeremy Freudberg proposed openstack/sahara master: DNM Dummy change to trigger jobs https://review.openstack.org/304019 | 15:04 |
| *** Bhujay has quit IRC | 15:24 | |
| *** jeremyfreudberg has quit IRC | 15:28 | |
| *** pcaruana has quit IRC | 15:29 | |
| *** knikolla has quit IRC | 16:19 | |
| *** knikolla has joined #openstack-sahara | 16:57 | |
| *** pcaruana has joined #openstack-sahara | 17:05 | |
| *** tesseract has quit IRC | 17:08 | |
| *** jeremyfreudberg has joined #openstack-sahara | 17:47 | |
| *** jeremyfreudberg has quit IRC | 17:48 | |
| tosky | uargh https://storyboard.openstack.org/#!/story/2002617 | 18:18 |
| tosky | as if we didn't have enough issues | 18:19 |
| *** whooligan has joined #openstack-sahara | 18:22 | |
| tellesnobrega | tosky, I will try to take a look into it | 18:46 |
| *** jeremyfreudberg has joined #openstack-sahara | 18:51 | |
| openstackgerrit | Chuck Short proposed openstack/sahara master: Switch ostestr to stestr https://review.openstack.org/571468 | 18:53 |
| jeremyfreudberg | tosky: here's a quick update on the keystone situation | 18:57 |
| jeremyfreudberg | case sensitivty does not affect us, but something else does | 18:57 |
| jeremyfreudberg | basically, the change in keystone that broke us created some new "implied roles" | 18:57 |
| jeremyfreudberg | these implied roles had a bug with trusts that no one (except me) knew about | 18:58 |
| tellesnobrega | jeremyfreudberg, how did you know about that? | 18:58 |
| jeremyfreudberg | tellesnobrega: somehow i figured it out | 18:58 |
| jeremyfreudberg | anyway, this is the workaround: https://review.openstack.org/#/c/576548/ | 18:59 |
| jeremyfreudberg | and https://review.openstack.org/#/c/576610/ and https://review.openstack.org/#/c/576611/ are the actual fix | 18:59 |
| tellesnobrega | great | 19:00 |
| jeremyfreudberg | hopefully some combination of those patches are merged soon, to unblock our gate | 19:00 |
| tellesnobrega | cool | 19:01 |
| jeremyfreudberg | just wanted to keep you two updated, so you know what to look for | 19:01 |
| tellesnobrega | jeremyfreudberg, thanks :) great work | 19:01 |
| jeremyfreudberg | :) | 19:01 |
| openstackgerrit | Corey Bryant proposed openstack/sahara master: Use register_error_handler to register make_json_error https://review.openstack.org/576617 | 19:03 |
| tellesnobrega | tosky, looks like we got a fix already | 19:05 |
| openstackgerrit | Corey Bryant proposed openstack/sahara master: Use register_error_handler to register make_json_error https://review.openstack.org/576617 | 19:19 |
| tosky | oooh | 19:24 |
| tosky | thanks jeremyfreudberg | 19:25 |
| tosky | really, much appreciated | 19:26 |
| jeremyfreudberg | i had fun doing it | 19:27 |
| jeremyfreudberg | (and no one on the keystone team was really stepping up to work on it) | 19:27 |
| *** jeremyfreudberg has left #openstack-sahara | 19:27 | |
| *** jeremyfreudberg has joined #openstack-sahara | 19:27 | |
| jeremyfreudberg | tosky: we should still probably do Member->member for the proxy users thing | 20:16 |
| jeremyfreudberg | because eventually case sensitivity *will* matter (right now it doesn't) | 20:17 |
| tosky | right; I'm just not sure how and if it could impact upgrades | 20:19 |
| tosky | it should not, unless the user customized that value | 20:19 |
| jeremyfreudberg | that remains to be seen | 20:19 |
| tosky | which means that I should at least add a release note item to my WIP patch | 20:19 |
| jeremyfreudberg | yes | 20:19 |
| *** jeremyfreudberg has quit IRC | 20:27 | |
| *** pcaruana has quit IRC | 20:29 | |
| *** tellesnobrega has quit IRC | 21:42 | |
| *** tellesnobrega has joined #openstack-sahara | 21:43 | |
| *** rcernin has joined #openstack-sahara | 22:12 | |
| *** tosky has quit IRC | 23:45 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!