| *** rcernin has quit IRC | 01:40 | |
| *** rcernin has joined #openstack-sahara | 01:40 | |
| *** Bhujay has joined #openstack-sahara | 02:04 | |
| *** Bhujay has quit IRC | 02:12 | |
| *** Bhujay has joined #openstack-sahara | 04:58 | |
| *** Bhujay has quit IRC | 05:53 | |
| *** Bhujay has joined #openstack-sahara | 05:59 | |
| *** pcaruana has joined #openstack-sahara | 06:05 | |
| *** rcernin has quit IRC | 07:06 | |
| *** tosky has joined #openstack-sahara | 07:51 | |
| *** openstackgerrit has quit IRC | 08:22 | |
| *** Bhujay has quit IRC | 08:38 | |
| remix_tj | hi tosky! A colleague created a cluster with 12 nodes and now he cannot see the cluster status while clicking on cluster name in horizon, because i get a 504 Gateway Error. I cannot troubleshoot why, so i suggested him to use the CLI | 09:31 |
|---|---|---|
| tosky | good suggestion | 09:31 |
| remix_tj | but with the new cli (openstack dataprocessing cluster show ) he cannot get the Cloudera management url and password | 09:32 |
| *** Bhujay has joined #openstack-sahara | 09:32 | |
| tosky | 504 Gateway Error may be an issue in horizon | 09:32 |
| tosky | is it there a backtrace in the horizon logs? | 09:32 |
| remix_tj | i've seen the old one sahara cluster-show reported a field called info that contained this infos, but i cannot find it | 09:32 |
| remix_tj | tosky: i didn't found any, but maybe i was looking at the wrong file | 09:33 |
| tosky | uhm, there should be something | 09:33 |
| tosky | the "where" depends on how horizon was deployed | 09:33 |
| remix_tj | tripleo, 3 node setup. But i've stopped all httpd instances to have the traffic balanced only to a single node | 09:34 |
| tosky | but there should be some place where horizon is running | 09:35 |
| remix_tj | yes, only one node | 09:35 |
| *** Bhujay has quit IRC | 09:37 | |
| tosky | remix_tj: containerized stuff? | 09:39 |
| tosky | it should be /var/log/containers/httpd/horizon/ | 09:39 |
| remix_tj | nah, is an old setup | 09:39 |
| tosky | otherwise probably just /var/log/httpd/horizon/ | 09:40 |
| remix_tj | ok, i'll take a look more in depth | 09:40 |
| remix_tj | anyway my issue is that the openstack cli is not reporting the info field with the password for cloudera manager | 09:40 |
| tosky | about the cloudera credentials, uhm | 09:40 |
| remix_tj | but i've an old cli | 09:41 |
| tosky | then please create a story | 09:47 |
| tosky | this could affect both cloudera and ambari | 09:47 |
| remix_tj | i just tested with the latest cli from pip and the info value is available | 09:48 |
| remix_tj | my fauld | 09:48 |
| remix_tj | *fault | 09:48 |
| remix_tj | for the horizon part i'll take a look. Is there a way to increase verbosity of horizon to have more details? | 09:50 |
| tosky | good question, but I don't know | 09:58 |
| remix_tj | seems that is a haproxy timeout because the request is returned with 200 on horizon log | 09:58 |
| remix_tj | 10.5.93.45 - - [24/Sep/2018:09:51:45 +0000] "GET /dashboard/project/data_processing/clusters/cluster/586b0261-fe0b-4ce0-9209-58c25a09003e HTTP/1.1" 200 9234 "http://openstack/dashboard/project/data_processing/clusters/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36" | 09:59 |
| tosky | O.o | 09:59 |
| tosky | but isn't 200 good? | 09:59 |
| remix_tj | yes it is, i've to look at haproxy | 10:00 |
| remix_tj | i'll try skipping haproxy and going directly to the host | 10:11 |
| remix_tj | tosky: it's defintely an haproxy issue | 10:24 |
| remix_tj | the page reply takes about 5 minutes, so it goes in timeout. | 10:24 |
| tosky | happy it's not a sahara issue, not happy that there is an issue | 10:25 |
| tosky | uh, still a 5 minute page replay is a bit too much | 10:25 |
| remix_tj | it's an old setup, it might be fixed in a newer setup | 10:25 |
| remix_tj | i'll fix with a workaround after lunch | 10:25 |
| tosky | how old? :) Newton? Ocata? Pike? | 10:27 |
| *** Bhujay has joined #openstack-sahara | 10:40 | |
| remix_tj | tosky: newton | 10:58 |
| *** Bhujay has quit IRC | 11:02 | |
| *** Bhujay has joined #openstack-sahara | 11:05 | |
| *** Bhujay has quit IRC | 11:06 | |
| *** Bhujay has joined #openstack-sahara | 11:07 | |
| *** Bhujay has quit IRC | 11:08 | |
| *** Bhujay has joined #openstack-sahara | 11:08 | |
| *** Bhujay has quit IRC | 11:09 | |
| *** Bhujay has joined #openstack-sahara | 11:10 | |
| *** Bhujay has quit IRC | 11:11 | |
| *** Bhujay has joined #openstack-sahara | 11:11 | |
| *** Bhujay has quit IRC | 12:15 | |
| *** Bhujay has joined #openstack-sahara | 12:16 | |
| *** Bhujay has quit IRC | 12:39 | |
| *** Bhujay has joined #openstack-sahara | 12:40 | |
| *** Bhujay has quit IRC | 12:41 | |
| *** Bhujay has joined #openstack-sahara | 12:41 | |
| Bhujay | tosky , tanks for your review . I want to write a test but need some to learn and may need little coaching from you :) | 12:57 |
| tosky | eheh, the complicated thing about unit tests is mocking some resources (so that, in this case, you don't need to fix a real file) | 12:57 |
| Bhujay | ok | 12:59 |
| Bhujay | i will need some time to comprehend | 13:00 |
| tosky | I guess we can merge this in the meantime - up to the others | 13:01 |
| Bhujay | thanks , that will be very encouraging to me | 13:01 |
| Bhujay | but I hope to to catch up with your suggestion as well about the test , TDD is on my learning list for a long time | 13:03 |
| Bhujay | how is the plan for HDP v3 plugin ? | 13:15 |
| tosky | there is none right now - if someone works on it, sure, it will happen | 13:40 |
| Gaasmann | I'm trying to figure out something about ambari. On Pike, plugin list give me ambari version 2.3 2.4 and 2.5. sahara-image-create -h gives me 2.2.0.0 2.2.1.0 and 2.4.2.0. | 14:28 |
| Gaasmann | So I'm guessing I can use 2.4.2.0 to create a cluster ambari/2.4. | 14:28 |
| Gaasmann | But I'm not sure about getting an ambari/2.3 or 2.5 or if the images 2.2.0.0 and 2.2.1.0 can be used for something | 14:29 |
| Gaasmann | I feel I'm missing something here :-) | 14:29 |
| tosky | Gaasmann: the plugin version is not the version of ambari, but the version of HDP | 14:51 |
| tosky | until pike, when sahara-image-elements was the only way to build images, we had | 14:52 |
| tosky | ambari 2.2.1.0 for HDP 2.4, ambari 2.2.0.0 for HDP 2.3 (even if 2.2.1.0 should be able to create HDP 2.3 clusters too) | 14:52 |
| tosky | and finally ambari 2.4.2.0 for HDP 2.5 | 14:52 |
| tosky | that said, people reported that ambari 2.4.2.0 does not always work properly | 14:52 |
| tosky | from queens onward we have a new image generation method (and we really deploy ambari 2.4 for HDP 2.5, 2.4 and 2.3; it's ambari 2.6 in rocky) | 14:53 |
| *** Bhujay has quit IRC | 15:07 | |
| *** dave-mccowan has joined #openstack-sahara | 15:13 | |
| *** dave-mccowan has quit IRC | 15:17 | |
| *** dave-mccowan has joined #openstack-sahara | 15:23 | |
| *** openstackgerrit has joined #openstack-sahara | 15:41 | |
| openstackgerrit | Merged openstack/sahara master: Add template param for ambari pkg install timeout https://review.openstack.org/593598 | 15:41 |
| Gaasmann | tosky: ho ok I see. Is there a place describing which image version should be use with which plugin version? | 16:16 |
| tosky | Gaasmann: not yet, there is a work in progress patch: https://review.openstack.org/#/c/604187/ | 16:24 |
| *** dave-mccowan has quit IRC | 16:38 | |
| Gaasmann | tosky: great. thanks! | 16:53 |
| *** tosky has quit IRC | 17:30 | |
| *** dave-mccowan has joined #openstack-sahara | 17:35 | |
| *** tosky has joined #openstack-sahara | 19:11 | |
| *** pcaruana has quit IRC | 19:17 | |
| Gaasmann | https://storyboard.openstack.org/#!/story/1736631 I'm having the same issue (problem spawninb Ubuntu/HDP/Ambari cluster). Discussion was about image generation method. So I wonder, can I use sahara-image-pack instead of sahara-image-create even if I'm running Openstack/sahara version stable/pike? | 20:24 |
| tosky | Gaasmann: as I mentioned before, no | 20:29 |
| tosky | not on pike | 20:29 |
| tosky | uh | 20:29 |
| tosky | no, sorry, I think I confused the versions, let me recheck | 20:29 |
| Gaasmann | tosky: I think you're right. I was wondering because of the discussion on the bug report I paste | 20:30 |
| tosky | confirmed: no sahara-image-pack | 20:31 |
| tosky | not in pike, only from queens | 20:31 |
| tosky | unfortunately ambari 2.4.2 was added and not much tested at that time | 20:31 |
| Gaasmann | I have the same issue with all ambari version | 20:32 |
| Gaasmann | 2.2.0.0/HDP_2.3 2.2.1.0/HDP_2.4 2.4.2.0/HDP_2.5 | 20:33 |
| tosky | we recently retested Ambari 2.2.1 with HDP 2.4 on queens and it was working | 20:33 |
| tosky | tellesnobrega: ^^ | 20:33 |
| tosky | Gaasmann: did you check the logs of ambari-server? | 20:34 |
| tosky | so far unfortunately we didn't get many logs for this error | 20:34 |
| tellesnobrega | hi | 20:34 |
| tellesnobrega | Gaasmann, all ambari versions are failing to install? | 20:35 |
| Gaasmann | all those combinaison 2.2.0.0/HDP_2.3 2.2.1.0/HDP_2.4 2.4.2.0/HDP_2.5 | 20:36 |
| Gaasmann | tosky: Did you retest by using images generated with sahara-image-pack or sahara-image-create? | 20:37 |
| tosky | Gaasmann: on queens, with sahara-image-create, because we had a suspect issue with ambari and SSL | 20:38 |
| tosky | (in addition to the tests with sahara-image-pack) | 20:38 |
| Gaasmann | tosky: for the logs I stopped at sahara-engine.log as it seems sahara try to start ambari-agent and fails because it's already started | 20:39 |
| Gaasmann | ok | 20:39 |
| tellesnobrega | Gaasmann, you used images from sahara-image-create right? | 20:40 |
| Gaasmann | tellesnobrega: correct | 20:41 |
| tellesnobrega | we might need to double check but it could be an error during image creation that is failing to stop the services | 20:41 |
| tellesnobrega | ubuntu right? | 20:41 |
| tellesnobrega | can you try centos? | 20:41 |
| Gaasmann | yes and yes, I'm generating the image right now | 20:41 |
| tellesnobrega | thanks | 20:43 |
| tellesnobrega | I had a similiar issue when I was implementing sahara-image-pack scripts | 20:43 |
| tosky | Gaasmann: not the logs of sahara: I'm talking about the logs of ambari-agent and ambari-server, directly from the ambari instance | 20:43 |
| tellesnobrega | Gaasmann, tosky suggestion is very important, please take a look at the ambari logs | 20:44 |
| Gaasmann | tosky: yep, I stopped before going that far. let me see | 20:44 |
| Gaasmann | seems ok, both processes are running. The client complained at the beginning because it wasn't able to reach the server but at some point it did and initialized itself | 20:52 |
| *** rcernin has joined #openstack-sahara | 22:45 | |
| *** remixtj has joined #openstack-sahara | 22:54 | |
| *** remix_tj has quit IRC | 23:00 | |
| *** ruhe has quit IRC | 23:03 | |
| *** tosky has quit IRC | 23:03 | |
| *** ruhe has joined #openstack-sahara | 23:06 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!