| *** rcernin_ has joined #openstack-sahara | 00:34 | |
| *** rcernin has quit IRC | 00:37 | |
| *** rcernin_ has quit IRC | 02:09 | |
| *** Bhujay has joined #openstack-sahara | 04:08 | |
| *** links has joined #openstack-sahara | 04:37 | |
| *** pgadiya has joined #openstack-sahara | 05:18 | |
| *** pgadiya has quit IRC | 05:18 | |
| *** rcernin has joined #openstack-sahara | 05:20 | |
| *** Bhujay has quit IRC | 06:18 | |
| *** Bhujay has joined #openstack-sahara | 06:23 | |
| *** pcaruana has joined #openstack-sahara | 06:35 | |
| *** rcernin has quit IRC | 07:01 | |
| *** tesseract has joined #openstack-sahara | 07:22 | |
| *** tosky has joined #openstack-sahara | 07:36 | |
| *** links has quit IRC | 07:38 | |
| *** links has joined #openstack-sahara | 07:55 | |
| *** links has quit IRC | 08:04 | |
| openstackgerrit | chen jiao proposed openstack/sahara master: Amend the spelling error of a word https://review.openstack.org/576050 | 08:04 |
|---|---|---|
| *** Bhujay has quit IRC | 08:09 | |
| *** links has joined #openstack-sahara | 08:18 | |
| *** Bhujay has joined #openstack-sahara | 08:31 | |
| openstackgerrit | chen jiao proposed openstack/sahara master: Amend the spelling error of a word https://review.openstack.org/576050 | 08:53 |
| tosky | apparently https://review.openstack.org/#/c/572243/ (the only thing that merged in keystone between the working and the non-working jobs) is the reason for the breakage | 09:07 |
| tosky | investigation ongoing on #openstack-keystone | 09:07 |
| openstackgerrit | Luigi Toscano proposed openstack/sahara master: DNM Dummy change to trigger jobs https://review.openstack.org/304019 | 09:12 |
| openstackgerrit | chen jiao proposed openstack/sahara master: Amend the spelling error of a word https://review.openstack.org/576078 | 10:03 |
| openstackgerrit | Luigi Toscano proposed openstack/sahara master: DNM Adapt to keystone change: use member instead of Member https://review.openstack.org/576081 | 10:15 |
| Bhujay | tosky: in last 48 hours i cloud build mapr and ambari images using sahara-image-pack , but when laucning the cluster it is trying to download more pkgs from internet and fails since from my sahara cluster vm i dont have internet access proxy access | 10:32 |
| Bhujay | during the image build shd it not install all necessary pkgs ? | 10:33 |
| tosky | Bhujay: sahara-image-pack should load all the required packages, as far as I know | 10:40 |
| tosky | oh, well | 10:40 |
| tosky | at least for mapr | 10:40 |
| tosky | for mapr on centos | 10:40 |
| tosky | for ambari, no, because the image only contains ambari and the HDP bits are downloaded dynamically | 10:41 |
| Bhujay | tosy: i see for map r the engine is trying to install scala but in the resources/images.py scala is there | 10:43 |
| Bhujay | tosy: i tried creating the images with proxy information in /etc/environment and then engine can download but at a later stage sahara fails since proxy prevents intercommunication between cluster , may be it is my knowledge issue with proxy , any suggestion | 10:46 |
| Bhujay | tosky ^ | 10:46 |
| tosky | Bhujay: scala should be already installed; can you check if scala is already there? | 10:47 |
| tosky | I fixed recently the code which download scale, which broke | 10:47 |
| Bhujay | tosky: ok let me check | 10:47 |
| tosky | which version of Sahara do you use? Maybe you don't have the fix | 10:47 |
| Bhujay | tosky: i am using openstack 17.0.4 | 10:48 |
| Bhujay | queens | 10:48 |
| tosky | Bhujay: there is no such version of openstack :) | 10:48 |
| tosky | which version of sahara? | 10:48 |
| tosky | this is the fix: http://git.openstack.org/cgit/openstack/sahara/commit/?h=stable/queens&id=a97ea07071ceb8e377001096e90504f9379e532a | 10:48 |
| Bhujay | trying to figureout the version | 10:49 |
| Bhujay | tosky: noted the fix | 10:49 |
| tosky | sorry, you already mentioned it but I forgot: how did you install it? If it's about packages, you can simply check the version of the sahara packages | 10:50 |
| Bhujay | i installed it using openstack-ansible | 10:50 |
| tosky | oh, right | 10:50 |
| tosky | uhm, I would expect that they take the tip of the relevant branch | 10:51 |
| tosky | but yeah, please check if the fix is there | 10:51 |
| Bhujay | checking | 10:52 |
| tosky | openstack-ansible-os_sahara installs the pip packages, so a pip list should show the version of sahara | 10:54 |
| Bhujay | tosky : pbr -v sahara = 3.1.1 and scala.noarch 2.12.6-1 is installed on the cluster . Now checking if the fix is there ... | 10:57 |
| tosky | uhm, 3.1.1 is the version of the pbr, not sahara | 10:59 |
| tosky | but the fix should be there | 10:59 |
| tosky | that said, there is a codepath which tries to install scala anyway during run time | 11:00 |
| tosky | I opened a story to track the issue some time ago | 11:00 |
| tosky | for now, just comment out that code | 11:00 |
| tosky | this one: https://storyboard.openstack.org/#!/story/2001965 | 11:00 |
| tosky | sadly we didn't properly test properly the "no network" case for a while | 11:00 |
| Bhujay | tosky: sorry about the version information... i open the code install_scala and the test if [ $test_only -eq 0 ]; thenis not there , so let me follow your storyboard link | 11:05 |
| tosky | Bhujay: sorry, which code did you follow? | 11:06 |
| tosky | there are two different code paths | 11:06 |
| tosky | the fix that I linked above applies to | 11:07 |
| tosky | sahara/plugins/mapr/resources/images/centos/install_scala and sahara/plugins/mapr/resources/images/ubuntu/install_scala | 11:07 |
| Bhujay | tosky: I was looking at the sahara/plugins/mapr/resources/images/centos/install_scala file | 11:07 |
| tosky | oki | 11:07 |
| tosky | can you paste its content? | 11:07 |
| Bhujay | sure | 11:08 |
| Bhujay | http://paste.openstack.org/show/723703 | 11:09 |
| tosky | Bhujay: ok, yeah, that's the code without the fix | 11:11 |
| Bhujay | tosky: i got the version 8.0.1.dev13 and for now i commented out the rpm installation and re launching another cluster | 11:28 |
| *** dave-mccowan has joined #openstack-sahara | 11:30 | |
| *** dave-mcc_ has joined #openstack-sahara | 11:33 | |
| *** dave-mccowan has quit IRC | 11:35 | |
| Bhujay | tosky: is there anyway we can prevent hdp pkg download by cluster vm in ambari plugin , like preinstalling them in image ? | 11:43 |
| Bhujay | tosky: nope , now it tries to search https://package.mapr.com , http://paste.openstack.org/show/723705. can we somehow prevent reinstallation if the pkgs are already there ? | 11:52 |
| openstackgerrit | Merged openstack/sahara-dashboard master: Fix _cls_registry type https://review.openstack.org/575437 | 11:59 |
| tosky | Bhujay: for HDP: no, because it's Ambari doing the downloading | 12:03 |
| tosky | there was a proposed patch to configure a mirror though | 12:03 |
| Bhujay | tosky : i see , | 12:03 |
| tosky | but it did not move forward | 12:03 |
| tosky | https://review.openstack.org/#/c/498087/ | 12:04 |
| Bhujay | tosky: no , not with mapr either .. it is now trying to install mapr core pkgs | 12:04 |
| tosky | that should not happen | 12:04 |
| tosky | or at least, it should not fail if the packages are already installed | 12:04 |
| Bhujay | i disabled the pkgs from image.yml in sahara engine let me see | 12:04 |
| tosky | can you please paste a longer snippet from the log? I can't get which is the exact place where it's failing | 12:05 |
| tosky | one possible solution for HDP is to create a mirror of HDP packages on a local instance, and add that repository to the sahara images | 12:05 |
| Bhujay | tosky: yeah , that sound ok, | 12:06 |
| Bhujay | tosky | 12:06 |
| Bhujay | in my env i can give internet connection to cluster vms through proxy the only problem is ... | 12:08 |
| Bhujay | while the proxy info in vm can be predefined in the image , the no_proxy also should include the cluster public ips , and i dont know how to do that | 12:09 |
| tosky | ehm, I'm not sure I understand the problem for no_proxy | 12:13 |
| Bhujay | one sec it agin failed .... | 12:13 |
| Bhujay | http://paste.openstack.org/show/723707 is a longer log showing the failure | 12:15 |
| tosky | this is from sahara/plugins/mapr/base/base_cluster_configurer.py, it seems | 12:22 |
| tosky | that code may not have been written with the use case of no-connection | 12:22 |
| openstackgerrit | Telles Mota Vidal Nóbrega proposed openstack/python-saharaclient master: Adding boot from volume https://review.openstack.org/572536 | 12:24 |
| tosky | I would try to see if at least it works with a proxy, but I'm not sure I get the problem with it | 12:26 |
| tosky | also tellesnobrega is around | 12:26 |
| tellesnobrega | hi | 12:26 |
| Bhujay | tosky , the download works with proxy that i have checked . The proxy information that i have given in the image was included in the /etc/environment.. | 12:30 |
| Bhujay | however due to this , ambari agent cant connect to ambari server since it tries to connect to proxy for any http connection , thats where no_proxy env var comes in which should list the http server for which the seesion should ignore proxy server . hope i am able to explain ... | 12:33 |
| Bhujay | however , i just figured out that the proxy info can be given in yum.com instead of /etc/environment , that shoudl solve the problem , let me try this ... | 12:33 |
| tosky | oh, ok, I didn't know that variable (no_proxy) | 12:33 |
| tosky | and yes, you should be able to configure the proxy just for yum, not for all services | 12:34 |
| Bhujay | nevertheless , in a corporate environment it is not desirable to allow the vms to access internet , then any software can be downloaded by the user from the vm vilolating the security plocies etc | 12:35 |
| Bhujay | so , if we can change the code such that during image creation all the reqd pkgs and need for internet connection is over and during deployment sahara only verifies will help enterprises adopt it more easily . | 12:37 |
| tellesnobrega | Bhujay, that is a good point, we need to see how we can do that, because iirc both ambari and cdh download stuff during cluster creation | 12:38 |
| tosky | tellesnobrega: there is a lot of mapr code which does the installation and configuration of packages, coming from MapR, that maybe could be removed if we always run the verification/libguestfs code | 12:40 |
| tosky | which is doing basically the same | 12:40 |
| tosky | see for example sahara/plugins/mapr/base/base_cluster_configurer.py | 12:41 |
| Bhujay | tellesnobrega, i understood ambari problem since ambari need the download , but for mapr we can do it first | 12:41 |
| tosky | but I'm not sure if what I wrote above is true | 12:41 |
| tosky | Bhujay: it's probably easier for ambari | 12:41 |
| tosky | because you just need to create a mirror | 12:41 |
| Bhujay | tosky , i see | 12:41 |
| tosky | ambari has a lot of code which downloads and configure | 12:41 |
| Bhujay | true that will be easier | 12:41 |
| tosky | right now we have more pressing issue (a change in keystone broke sahara), so let's see | 12:42 |
| Bhujay | local repo sounds a good solution | 12:42 |
| tellesnobrega | tosky, the Member x member patch? | 12:48 |
| tosky | tellesnobrega: you may want to read the logs of #openstack-keystone starting from earlier today | 12:49 |
| tosky | so far I'm sure that I don't understand what's going on | 12:49 |
| tellesnobrega | tosky, will do | 12:49 |
| *** jeremyfreudberg has joined #openstack-sahara | 13:01 | |
| *** jeremyfreudberg_ has joined #openstack-sahara | 13:15 | |
| *** jeremyfreudberg has quit IRC | 13:18 | |
| *** jeremyfreudberg_ is now known as jeremyfreudberg | 13:23 | |
| tellesnobrega | tosky, this error is very weird, the patch to revert that one needs to merge for us to test ? | 13:32 |
| openstackgerrit | Telles Mota Vidal Nóbrega proposed openstack/python-saharaclient master: Adding boot from volume https://review.openstack.org/572536 | 13:35 |
| tosky | tellesnobrega: in my testing, reverting https://review.openstack.org/#/c/572243/ restored the previous behavior | 13:35 |
| tosky | see the revert https://review.openstack.org/#/c/576056/ and the dummy patch for sahara on top of it: https://review.openstack.org/#/c/304019/ | 13:35 |
| tellesnobrega | ok | 13:35 |
| tellesnobrega | jeremyfreudberg, can you take a look at the patch on saharaclient on boot from volume | 13:41 |
| jeremyfreudberg | tellesnobrega: yes, i'll take a look | 13:41 |
| tellesnobrega | I added the v2 part for node group templates, I would like for you to take a look on it so I can continue working on preparing the client for v2 | 13:42 |
| jeremyfreudberg | tosky, something weird in the keystone logs is that first it says "Conflict occurred attempting to store role - Duplicate entry found with name Member".... but then... "Could not find role: Member.: RoleNotFound: Could not find role: Member." | 13:42 |
| tellesnobrega | so we are on the same page on how that should be done | 13:42 |
| jeremyfreudberg | seems to be contradictory | 13:42 |
| jeremyfreudberg | tellesnobrega: yep | 13:42 |
| tellesnobrega | about the error, can this be related to how it is stored on db? | 13:43 |
| tellesnobrega | meaning it has member there, so it fails to add the new one | 13:43 |
| tellesnobrega | and when restored it is checked for Member (case sensitive) and it says it is not there | 13:43 |
| tosky | the "Conflict occurred..." is what appears as fatal exception in heat logs | 13:44 |
| tosky | if you want to move this discussion to #openstack-keystone, it may help the people trying to reproduce the issue | 13:44 |
| Bhujay | tosky , putting the porxy in yum.conf seems to be working, i have reached 34% wait_ambari_request ... | 13:48 |
| Bhujay | tosky, got the web page for ambari , what is the user name and password ? | 13:48 |
| tellesnobrega | it should show at the cluster info on horizon | 13:49 |
| tellesnobrega | iirc the password is generated | 13:49 |
| Bhujay | ok | 13:50 |
| *** links has quit IRC | 14:09 | |
| jeremyfreudberg | so, keystone seems to store role names withOUT case sensitivity, so it makes sense that tosky's attempt failed | 14:09 |
| jeremyfreudberg | i'm not sure, what is trying to create the role, though | 14:09 |
| tosky | please comment there :) | 14:11 |
| jeremyfreudberg | where is "there"? whose review? | 14:11 |
| jeremyfreudberg | or which review | 14:11 |
| tosky | on the review, or on #openstack-keystone | 14:11 |
| jeremyfreudberg | i'll throw it on -keystone | 14:12 |
| tosky | thanks | 14:12 |
| tellesnobrega | jeremyfreudberg, not sure I follow your comment on the client path | 14:22 |
| tellesnobrega | patch | 14:22 |
| jeremyfreudberg | depending on the nature of the v1->v2 change, it may be simpler to have just one class | 14:23 |
| jeremyfreudberg | and do if-else for the difference | 14:23 |
| jeremyfreudberg | there seems to be a lot of copy-paste now, for the sake of a tiny change | 14:23 |
| tellesnobrega | hum, ok | 14:25 |
| *** elmiko has joined #openstack-sahara | 14:38 | |
| tellesnobrega | jeremyfreudberg, I'm heading out for lunch, but when I'm back (if you are still here) I have some follow up questions on the v2 implementation | 15:08 |
| *** Bhujay has quit IRC | 15:36 | |
| *** jeremyfreudberg has quit IRC | 16:11 | |
| *** tesseract has quit IRC | 16:19 | |
| *** pcaruana has quit IRC | 16:20 | |
| *** dave-mcc_ has quit IRC | 16:30 | |
| *** jeremyfreudberg has joined #openstack-sahara | 16:59 | |
| tellesnobrega | jeremyfreudberg, about v2 | 17:32 |
| tellesnobrega | do you really prefer ifs instead of a new class? under a v2 folder? | 17:32 |
| tellesnobrega | I understand that we can use the ifs, and I can make get_version under utils.py to get the selected version | 17:33 |
| tellesnobrega | but, Inheritance looks so much better than a bunch of ifs in the code | 17:33 |
| tellesnobrega | I don't know | 17:33 |
| tellesnobrega | tosky, do have an opinion on this? | 17:34 |
| jeremyfreudberg | i'm looking through again now | 17:34 |
| tellesnobrega | thanks | 17:34 |
| jeremyfreudberg | for example, inheritance looks really nice on get_parser stuff | 17:34 |
| tellesnobrega | yes | 17:34 |
| tellesnobrega | I understand that the take_action is too much copy and paste | 17:35 |
| tosky | it depends on how much code is duplicated, but technically even with inheritance there should not be much duplicated code | 17:38 |
| tosky | I mean, one of the thing about inheritance is to group the common code too | 17:38 |
| jeremyfreudberg | tosky, in the case of node group template commands, the problem is that the function bodies are almost identical, but not quite | 17:38 |
| jeremyfreudberg | hadoop_version->plugin_version | 17:38 |
| tosky | define a common _<almost_same_name> function and pass the different string as parameter? | 17:39 |
| jeremyfreudberg | there is probably a hacky way with constants and getattr that would achieve maximum inheritance and minimum copypaste, but it would be awkward to read | 17:39 |
| tellesnobrega | exactly, small changes, but they need to be there | 17:39 |
| tellesnobrega | I like a common _<almost_same_name> function idea | 17:40 |
| tellesnobrega | we can do that | 17:40 |
| jeremyfreudberg | we can try it like that, although i'm having trouble imagining the specifics in this case | 17:42 |
| tellesnobrega | I will have to take a look | 17:54 |
| tellesnobrega | but even with this case of the node group templates, where we copy and paste the take actions, I'm inclined to choose it instead of if...else | 17:55 |
| jeremyfreudberg | tosky: which user does sahara-tests use in the gate? admin? sahara? | 18:10 |
| jeremyfreudberg | tosky: found it, it's admin | 18:13 |
| tosky | yes, even if I'd like to switch to demo (a non-admin) at some point | 18:14 |
| *** dave-mccowan has joined #openstack-sahara | 18:17 | |
| jeremyfreudberg | tosky: how do i change this user easily to something else? http://logs.openstack.org/81/576081/1/check/sahara-tests-scenario/20379dc/job-output.txt.gz#_2018-06-18_10_50_16_224355 | 18:20 |
| jeremyfreudberg | i'd like to try it as someone without the admin role | 18:20 |
| *** pcaruana has joined #openstack-sahara | 18:25 | |
| tosky | jeremyfreudberg: uhm, we should be using demo already: http://git.openstack.org/cgit/openstack/sahara-tests/tree/roles/run-sahara-scenario/tasks/main.yaml | 18:30 |
| jeremyfreudberg | tosky: the job that i linked had --os-cloud devstack-admin | 18:30 |
| tosky | jeremyfreudberg: no, sorry: http://git.openstack.org/cgit/openstack/sahara-tests/tree/roles/run-sahara-scenario/defaults/main.yaml | 18:30 |
| tosky | just change the default | 18:30 |
| tosky | but it may not work, I don't remember why I switched back; probably to have the same behavior of the zuul v2 jobs | 18:31 |
| tosky | and also here http://git.openstack.org/cgit/openstack/sahara-tests/tree/roles/setup-sahara-scenario-env/defaults/main.yaml | 18:33 |
| jeremyfreudberg | tosky: where are those clouds defined? | 18:33 |
| tosky | jeremyfreudberg: devstack creates them | 18:35 |
| tosky | you can find in the logs, <blabla>/etc/openstack/clouds.yaml | 18:35 |
| *** pcaruana has quit IRC | 18:39 | |
| openstackgerrit | Jeremy Freudberg proposed openstack/sahara-tests master: DNM admin->demo https://review.openstack.org/576231 | 18:42 |
| tellesnobrega | jeremyfreudberg, do you know where the FakeApp for the openstackclient is defined? | 18:53 |
| tellesnobrega | found it | 18:54 |
| jeremyfreudberg | cool | 18:54 |
| tellesnobrega | when you said that self.app.api_version has the selected version from the user | 18:55 |
| tellesnobrega | are you sure about that? | 18:55 |
| jeremyfreudberg | 90% | 18:55 |
| tellesnobrega | hmm, because the FakeApp doesn't have api_version defined | 18:56 |
| tellesnobrega | I'm seeing it under clientmanager | 18:56 |
| tellesnobrega | https://github.com/openstack/python-openstackclient/blob/06263bd5852aad9cd03a76f50140fbbb2d0751ba/openstackclient/common/clientmanager.py#L50 | 18:56 |
| tellesnobrega | so, it could be self.app.client_manager.api_version['data_processing'] | 18:57 |
| tellesnobrega | or even, self.app.client_manager.data_processing.api_version | 18:57 |
| jeremyfreudberg | (Pdb) self.app.client_manager.api_version | 18:58 |
| jeremyfreudberg | *** AttributeError: 'ClientManager' object has no attribute 'api_version' | 18:58 |
| jeremyfreudberg | it has _api_version, though | 18:58 |
| tellesnobrega | does self.app.api_version exist? | 18:58 |
| jeremyfreudberg | (Pdb) self.app.api_version['data_processing'] | 18:59 |
| jeremyfreudberg | '1.1' | 18:59 |
| jeremyfreudberg | works great | 18:59 |
| tellesnobrega | can you try self.app.client_manage.data_processing.api_version | 18:59 |
| tellesnobrega | ? | 18:59 |
| tellesnobrega | cool, I guess I will need to update the FakeApp for tests to work | 18:59 |
| jeremyfreudberg | you mean self.app.client_manager.data_processing._api_version ... because this comes from https://github.com/openstack/python-saharaclient/blob/master/saharaclient/api/client.py#L52 | 19:00 |
| jeremyfreudberg | self.app.api_version seems like the best choice, i guess | 19:00 |
| tellesnobrega | ok | 19:00 |
| tellesnobrega | makes sense | 19:00 |
| openstackgerrit | Telles Mota Vidal Nóbrega proposed openstack/python-saharaclient master: Adding boot from volume https://review.openstack.org/572536 | 19:26 |
| tellesnobrega | jeremyfreudberg, tosky take a look at ^ and see if it makes sense (ps.: tests will fail because FakeApp doesn't have the api_version) but focus on the refactoring for inheritance | 19:26 |
| tosky | tellesnobrega: that, or inherit from a common ancestor where the common function should live | 19:30 |
| tosky | (less refactoring when the support for v1 will be dropped) | 19:30 |
| tellesnobrega | can do that too | 19:31 |
| *** dave-mccowan has quit IRC | 20:12 | |
| *** dave-mccowan has joined #openstack-sahara | 20:13 | |
| *** jeremyfreudberg has quit IRC | 20:42 | |
| *** dave-mcc_ has joined #openstack-sahara | 20:52 | |
| *** dave-mccowan has quit IRC | 20:54 | |
| *** rcernin has joined #openstack-sahara | 22:36 | |
| *** dave-mcc_ has quit IRC | 23:16 | |
| *** tosky has quit IRC | 23:20 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!