| *** ukaynar has quit IRC | 00:08 | |
| *** ukaynar has joined #openstack-sahara | 00:11 | |
| *** ukaynar has quit IRC | 00:11 | |
| *** ukaynar has joined #openstack-sahara | 00:12 | |
| *** ukaynar has quit IRC | 00:26 | |
| *** esikachev has joined #openstack-sahara | 00:27 | |
| *** esikachev has quit IRC | 00:31 | |
| *** shuyingya has joined #openstack-sahara | 00:39 | |
| *** shuyingya has quit IRC | 00:44 | |
| *** tomtomtom has quit IRC | 00:45 | |
| *** dave-mccowan has joined #openstack-sahara | 01:27 | |
| *** esikachev has joined #openstack-sahara | 01:28 | |
| *** shuyingya has joined #openstack-sahara | 01:31 | |
| *** esikachev has quit IRC | 01:32 | |
| *** shuyingya has quit IRC | 01:32 | |
| *** shuyingya has joined #openstack-sahara | 01:32 | |
| openstackgerrit | lan proposed openstack/sahara-extra master: Modify a description of the parameters to function. https://review.openstack.org/490703 | 02:10 |
|---|---|---|
| *** esikachev has joined #openstack-sahara | 02:28 | |
| *** dave-mccowan has quit IRC | 02:29 | |
| *** esikachev has quit IRC | 02:32 | |
| *** http_GK1wmSU has joined #openstack-sahara | 02:53 | |
| *** http_GK1wmSU has left #openstack-sahara | 02:56 | |
| *** esikachev has joined #openstack-sahara | 03:29 | |
| *** esikachev has quit IRC | 03:34 | |
| *** esikachev has joined #openstack-sahara | 03:59 | |
| *** links has joined #openstack-sahara | 04:03 | |
| *** esikachev has quit IRC | 04:03 | |
| *** esikachev has joined #openstack-sahara | 04:06 | |
| *** esikachev has quit IRC | 04:38 | |
| *** tuanluong has joined #openstack-sahara | 04:41 | |
| *** esikachev has joined #openstack-sahara | 04:41 | |
| *** esikachev has quit IRC | 04:49 | |
| *** esikachev has joined #openstack-sahara | 05:10 | |
| *** ssmith has joined #openstack-sahara | 05:16 | |
| *** Poornima has joined #openstack-sahara | 05:16 | |
| *** Poornima has quit IRC | 05:33 | |
| *** Poornima has joined #openstack-sahara | 05:39 | |
| *** pgadiya has joined #openstack-sahara | 05:48 | |
| *** pgadiya has quit IRC | 06:17 | |
| *** pgadiya has joined #openstack-sahara | 06:18 | |
| *** rcernin has joined #openstack-sahara | 06:30 | |
| *** Poornima has quit IRC | 06:32 | |
| *** anshul has joined #openstack-sahara | 06:38 | |
| *** esikachev has quit IRC | 06:49 | |
| *** Poornima has joined #openstack-sahara | 07:05 | |
| *** tesseract has joined #openstack-sahara | 07:16 | |
| *** esikachev has joined #openstack-sahara | 07:27 | |
| *** pcaruana has joined #openstack-sahara | 07:38 | |
| *** ssmith has quit IRC | 07:42 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/sahara-dashboard master: Imported Translations from Zanata https://review.openstack.org/490797 | 08:25 |
| *** abalutoiu has joined #openstack-sahara | 08:37 | |
| *** esikachev has quit IRC | 08:44 | |
| *** esikachev has joined #openstack-sahara | 09:20 | |
| *** Poornima has quit IRC | 09:24 | |
| *** tosky has joined #openstack-sahara | 09:25 | |
| *** tosky has quit IRC | 09:44 | |
| *** pgadiya has quit IRC | 09:45 | |
| *** pgadiya has joined #openstack-sahara | 10:02 | |
| *** tuanluong has quit IRC | 10:08 | |
| *** Poornima has joined #openstack-sahara | 10:11 | |
| *** openstackgerrit has quit IRC | 10:18 | |
| *** esikachev has quit IRC | 10:34 | |
| *** esikachev has joined #openstack-sahara | 10:41 | |
| *** Poornima has quit IRC | 10:45 | |
| *** tellesnobrega has quit IRC | 10:46 | |
| *** abalutoiu has quit IRC | 10:57 | |
| *** tosky has joined #openstack-sahara | 11:02 | |
| *** Poornima has joined #openstack-sahara | 11:17 | |
| *** esikachev has quit IRC | 11:17 | |
| *** Poornima has quit IRC | 11:48 | |
| *** tellesnobrega has joined #openstack-sahara | 11:50 | |
| *** dave-mccowan has joined #openstack-sahara | 11:55 | |
| *** Poornima has joined #openstack-sahara | 11:56 | |
| *** esikachev has joined #openstack-sahara | 12:01 | |
| *** shuyingya has quit IRC | 12:33 | |
| *** esikachev has quit IRC | 12:42 | |
| *** pgadiya has quit IRC | 12:54 | |
| *** shuyingya has joined #openstack-sahara | 12:56 | |
| *** shuyingya has quit IRC | 13:00 | |
| *** Poornima has quit IRC | 13:01 | |
| *** jeremyfreudberg has joined #openstack-sahara | 13:01 | |
| *** pgadiya has joined #openstack-sahara | 13:07 | |
| *** lucasxu has joined #openstack-sahara | 13:34 | |
| *** pgadiya has quit IRC | 13:41 | |
| *** jeremyfreudberg has quit IRC | 13:46 | |
| *** ssmith has joined #openstack-sahara | 13:46 | |
| *** cpusmith has joined #openstack-sahara | 13:47 | |
| *** openstackgerrit has joined #openstack-sahara | 13:48 | |
| openstackgerrit | Luigi Toscano proposed openstack/sahara master: Restructure the documentation according the new spec https://review.openstack.org/490872 | 13:48 |
| tosky | that was tough | 13:48 |
| tellesnobrega | tosky, thanks | 13:49 |
| tosky | when the job complets you can see the preview from the results of the docs job | 13:50 |
| tellesnobrega | tosky, looking here at spark jobs, I saw that most of the time spark uses minimum core ad minimum memory. This could be something interesting for us to look into. Maybe this could be optimized to run with all available resources or something more than minimum | 13:50 |
| tosky | tellesnobrega: do you mean that we test it only with tiny instances? | 13:51 |
| *** ssmith has quit IRC | 13:51 | |
| tellesnobrega | tosky, from what I saw, when we run spark-submit (and there isn't any specification on the job) spark will allocate the minimum amount of resources to the job | 13:55 |
| tellesnobrega | so, it wouldn't matter having a 8 vcpu with 16gb RAM machine, because it would allocate only 1cpu and 1GB ram if I understand it correctly | 13:56 |
| tellesnobrega | there are some command line flags that can be used to tune that --executor-cores and --executor-memory | 13:57 |
| tellesnobrega | I have to read a little more on this, but we might be under using the resources on job submission | 13:58 |
| *** cpusmith_ has joined #openstack-sahara | 13:59 | |
| *** jeremyfreudberg has joined #openstack-sahara | 13:59 | |
| tosky | oh, so basically Sahara should do the magic and pass all the resources available on the worker node to spark? | 14:00 |
| *** cpusmith has quit IRC | 14:02 | |
| tellesnobrega | tosky, that is something to think about | 14:03 |
| tellesnobrega | not sure if it should be sahara, or if we should make this available for the user to set | 14:03 |
| tellesnobrega | jeremyfreudberg, please read a little back and give your thoughts on this matter | 14:04 |
| *** cpusmith_ has quit IRC | 14:10 | |
| *** links has quit IRC | 14:12 | |
| *** cpusmith_ has joined #openstack-sahara | 14:13 | |
| *** cpusmith has joined #openstack-sahara | 14:13 | |
| *** cpusmith_ has quit IRC | 14:17 | |
| jeremyfreudberg | tellesnobrega, reading now, i pulled up the help for spark-submit on my cluster too for reference http://paste.openstack.org/raw/617543/ | 14:18 |
| tellesnobrega | jeremyfreudberg, cool | 14:19 |
| openstackgerrit | Luigi Toscano proposed openstack/sahara master: Restructure the documentation according the new spec https://review.openstack.org/490872 | 14:19 |
| jeremyfreudberg | tellesnobrega, i guess one solution to this might be use the oozie edp engine to submit spark jobs on vanilla... | 14:23 |
| tellesnobrega | jeremyfreudberg, this isn't really a solution but more of a workaround | 14:23 |
| tellesnobrega | I'm thinking that we could provide the users the option to tune that (UI and CLI) and we append that info on the spark-submit command line | 14:24 |
| *** anshul has quit IRC | 14:26 | |
| jeremyfreudberg | tellesnobrega, that could work too | 14:33 |
| jeremyfreudberg | especially if you could grab good defaults from the flavor info | 14:34 |
| *** tellesnobrega has quit IRC | 14:36 | |
| *** shuyingya has joined #openstack-sahara | 15:12 | |
| *** rcernin has quit IRC | 15:14 | |
| *** lucasxu has quit IRC | 15:15 | |
| tosky | and the preview: http://docs-draft.openstack.org/72/490872/2/check/gate-sahara-docs-ubuntu-xenial/45e5a32//doc/build/html/ | 15:16 |
| *** shuyingya has quit IRC | 15:16 | |
| *** Adri2000 has joined #openstack-sahara | 15:23 | |
| Adri2000 | hello | 15:23 |
| Adri2000 | I can't figure out what "INFO keystonemiddleware.auth_token [-] Rejecting request" in the sahara logs is about | 15:23 |
| Adri2000 | ocata version, deployed using openstack-ansible, if that matters | 15:24 |
| jeremyfreudberg | ah, adri2000, a common question | 15:24 |
| Adri2000 | ah :) | 15:24 |
| jeremyfreudberg | so, you have something external (perhaps haproxy, nagios) talking to sahara | 15:24 |
| jeremyfreudberg | but because sahara uses keystonemiddleware for auth, any http request to it not well formed yields that log message | 15:25 |
| jeremyfreudberg | however, it's not an error and should not really affect sahara's operation | 15:25 |
| Adri2000 | ohhh, certainly haproxy keepalive pings! | 15:25 |
| jeremyfreudberg | adri2000, yes, we had another deployment report the same | 15:25 |
| jeremyfreudberg | we can try to document this issue, but as long as keystonemiddleware behaves the way it does, and haproxy talking to http port, not much we can do | 15:26 |
| * tosky <- seen haproxy | 15:26 | |
| Adri2000 | that makes sense, but I wonder why don't we notice the same thing for other components that (I guess) also use keystonemiddleware? | 15:29 |
| jeremyfreudberg | adri2000, fair point, i can look at it some | 15:30 |
| Adri2000 | anyway, thanks for you quick help and explanation! :) | 15:31 |
| jeremyfreudberg | adri2000, no problem | 15:31 |
| cpusmith | jeremyfreudberg: The Spark cluster has gone RED health overnight. Do you know where we can check for the issue? Also, when clicking on the name in Horizon we get a "Something went wrong! | 15:53 |
| cpusmith | An unexpected error has occurred." | 15:53 |
| jeremyfreudberg | cpusmith, the red is because of a health check, the default ones on spark cluster are only "all instances accessible" and "resolvconf changed" | 15:57 |
| jeremyfreudberg | for the something went wrong, usually you have a resource which refers to a different non existent resource (deleted image, disabled plugin) | 15:58 |
| jeremyfreudberg | you can check horizon logs for details (depending on your deployment, sometimes in your webserver logs) | 15:58 |
| cpusmith | Oh, I think that Tomtom turned on namespaces last night. I may have to rebuild it | 15:59 |
| *** lucasxu has joined #openstack-sahara | 16:00 | |
| jeremyfreudberg | cpusmith, that's probably it (For the "RED" status, anyway) | 16:00 |
| jeremyfreudberg | if namespaces is not working, things will be red | 16:00 |
| jeremyfreudberg | tosky, take a look at https://github.com/openstack/sahara/blob/master/sahara/config.py#L124 and https://github.com/openstack/oslo.log/blob/master/oslo_log/_options.py#L19 | 16:03 |
| jeremyfreudberg | the solution to our "rejecting request" problem | 16:03 |
| jeremyfreudberg | i will put up a patch later | 16:03 |
| openstackgerrit | Luigi Toscano proposed openstack/sahara master: Restructure the documentation according the new spec https://review.openstack.org/490872 | 16:05 |
| tosky | jeremyfreudberg: change the logging level? | 16:06 |
| jeremyfreudberg | tosky, we can actually remove most of that override log level stuff, most of it old stuff no one does anymore | 16:06 |
| tosky | oh, that too | 16:07 |
| jeremyfreudberg | the only worth keeping as override is eventlet.wsgi.server, that one being on warn is nice | 16:07 |
| jeremyfreudberg | on info it's too much stuf | 16:07 |
| *** pcaruana has quit IRC | 16:10 | |
| *** tellesnobrega has joined #openstack-sahara | 16:32 | |
| openstackgerrit | Jeremy Freudberg proposed openstack/sahara master: Better keystonemiddleware log level https://review.openstack.org/490971 | 16:33 |
| *** jeremyfreudberg has quit IRC | 16:39 | |
| *** cpusmith has quit IRC | 17:01 | |
| *** cpusmith_ has joined #openstack-sahara | 17:01 | |
| *** cpusmith has joined #openstack-sahara | 17:03 | |
| *** cpusmith_ has quit IRC | 17:06 | |
| *** tosky has quit IRC | 17:27 | |
| *** openstackstatus has quit IRC | 17:27 | |
| *** openstack has joined #openstack-sahara | 17:30 | |
| *** jeremyfreudberg has joined #openstack-sahara | 17:30 | |
| *** tosky has joined #openstack-sahara | 17:54 | |
| *** tomtomtom has joined #openstack-sahara | 17:55 | |
| tomtomtom | hello, anyone seen an error like this before: NoReverseMatch: u'data_processing.data_plugins' is not a registered namespace inside 'horizon:project' when trying to click the cluster after been created and showing in the clusters panel? The error is from the horizon log. It seems to me that horizon is complaining about not being able to find the sahara plugin | 17:56 |
| tomtomtom | is this better asked in the horizon irc? | 17:57 |
| jeremyfreudberg | tomtomtom, maybe, but first -- how was sahara ui installed? (and more importantly, any changes/upgrades that could have made it spontaneously stop working?) | 17:57 |
| tosky | I didn't see it, but I second jeremyfreudberg's question: did it work before? | 17:58 |
| tomtomtom | it was installed via openstack-ansible from the stable/newton branch | 17:59 |
| jeremyfreudberg | (tosky: it's the same deployment that has had people popping up on irc for about a week or so) | 17:59 |
| tomtomtom | There haven't been any changes, all other panels seem to work except for the one that allows you to go into a completed cluster. | 18:00 |
| jeremyfreudberg | tomtomtom, i see | 18:00 |
| tomtomtom | this is the first time we have a working cluster though. | 18:00 |
| tomtomtom | so now we noticed that the panel is broken | 18:01 |
| tosky | jeremyfreudberg: oh, I see | 18:01 |
| tomtomtom | so the Cluster panel comes up and shows that a cluster is available, but when you click it horizon throws an internal server error. | 18:02 |
| jeremyfreudberg | tomtomtom, can you ls openstack_dashboard/local/enabled for me? | 18:03 |
| tosky | I didn't see this error in the past, but of course something is happening | 18:03 |
| tosky | is it the last git revision of stable/newton? Even if I don't see specific fixes for something like that | 18:04 |
| tosky | django 1.8, right? | 18:04 |
| tomtomtom | checking | 18:09 |
| tomtomtom | @jeremyfreudber _1370_project_container_infra_panel_group.py _1371_project_container_infra_clusters_panel.pyc _1481_project_ng_loadbalancersv2_panel.py _80_admin_default_panel.py __init__.pyc _1370_project_container_infra_panel_group.pyc _1372_project_container_infra_cluster_templates_panel.py _1481_project_ng_loadbalancersv2_panel.pyc _80_admin_default_panel.pyc _1371_project_container_infra_clus | 18:10 |
| tellesnobrega | im back | 18:10 |
| tomtomtom | hmm.... just the 137* are there for clustering, in the sahara_dashboard directory there are some _18* files for the horizon dashboard | 18:11 |
| jeremyfreudberg | tomtomtom, usually those _18* files for sahara-dashboard get copied into the horizon tree as part of the install, can you do a "find" and make sure that they are all there? | 18:12 |
| jeremyfreudberg | they might have made it into openstack_dashboard/enabled instead openstack_dashboard/local/enabled, for instance, i can never remember how consistent that is across deployment methods | 18:12 |
| tomtomtom | yes they are there under openstack_dashboard/enabled, they are symlinks to sahara_dashboard/enabled | 18:12 |
| jeremyfreudberg | tomtomtom, i see | 18:13 |
| tomtomtom | @tosky it might be like 14.0.7 as opposed to 14.0.8 so I don't think it's the absolute last one. | 18:14 |
| tomtomtom | @tosky django 1.8.14 is installed in horizon venv | 18:15 |
| tosky | tomtomtom: 14 I think it's the version of horizon, and sahara-dashboard? | 18:16 |
| *** cpusmith has quit IRC | 18:16 | |
| *** cpusmith_ has joined #openstack-sahara | 18:16 | |
| jeremyfreudberg | tomtomtom, just for sanity, you have _1810, _1820, _1830, and _1840 in the dashboard enabled folder? | 18:16 |
| jeremyfreudberg | just to be super sure | 18:16 |
| jeremyfreudberg | and I echo the question of what is versioned as 14.x, not anything sahara | 18:17 |
| tomtomtom | lol well what do you know, for some reason 1830 isn't there. | 18:17 |
| jeremyfreudberg | 1830 is the one that was missing according to your error message | 18:17 |
| jeremyfreudberg | the "data processing plugins panel" | 18:17 |
| tomtomtom | yeah that 14.x is the version of openstack-ansilbe I deployed with | 18:17 |
| tomtomtom | lol ok, didn't see that thanks | 18:18 |
| jeremyfreudberg | tomtomtom, i'm trying to look at the ansible playbooks now, trying to find out if there is a legitimate reason why that file was missed | 18:19 |
| jeremyfreudberg | maybe i'm looking in the wrong place, doesn't seem like here https://github.com/openstack/openstack-ansible-os_sahara is the relevant place for dashboard things? | 18:20 |
| jeremyfreudberg | tomtomtom, care to weigh in how to find the relevant playbook? | 18:26 |
| *** openstackstatus has quit IRC | 18:26 | |
| *** openstack has joined #openstack-sahara | 18:29 | |
| *** cpusmith has joined #openstack-sahara | 18:30 | |
| *** cpusmith_ has quit IRC | 18:33 | |
| tomtomtom | sorry gimme a minute and I'll be able to answer the questions | 18:36 |
| jeremyfreudberg | tomtomtom, not a problem at all | 18:36 |
| tellesnobrega | jeremyfreudberg, back to the spark talk (sorry I had to leave abruptly). My idea is something on the line of what you said, we can suggest a config based on flavor and make it available for user tunning | 18:36 |
| jeremyfreudberg | tellesnobrega, great, it's on the etherpad, i think that's reasonable | 18:37 |
| tellesnobrega | jeremyfreudberg, thanks | 18:37 |
| tosky | yep, default to all the resources | 18:38 |
| tomtomtom | ok so the problem is with openstack-ansible horizon role. /etc/ansible/roles/os_horizon/tasks/horizon_post_install.yml, check out the image: http://imgur.com/a/bUCiR seems like each file is specifically coded in and 1830 was missed | 18:41 |
| jeremyfreudberg | tomtomtom, oh wow, luckily fixed here: https://github.com/openstack/openstack-ansible-os_horizon/commit/7fdad55c63733bb94732c1fd034d3d752d34b4ec | 18:44 |
| jeremyfreudberg | tomtomtom, it was actually backported too https://review.openstack.org/#/q/Ied57616e5641230ff3e979a44498697a30f63143 | 18:45 |
| tomtomtom | yeah, mar 7th might have been 1 or 2 months after I pulled the stable/newton branch down..... I installed back in january/february | 18:46 |
| tellesnobrega | awesome the fix is not complicated here | 18:46 |
| tomtomtom | either way my fix was to symlink the file like the others on my horizon nodes. it's already working correctly now, thanks for the help. | 18:48 |
| jeremyfreudberg | tomtomtom, no problem | 18:49 |
| *** esikachev has joined #openstack-sahara | 18:55 | |
| tosky | I think we can use it in our presentation: most of the problems related to Sahara are not Sahara problems | 19:12 |
| jeremyfreudberg | tosky, ha | 19:13 |
| tellesnobrega | true | 19:13 |
| jeremyfreudberg | it's true, sahara does a good job of revealing how bad your deployment is | 19:13 |
| tosky | (or something in a proper English) | 19:13 |
| openstackgerrit | Merged openstack/sahara-dashboard master: Imported Translations from Zanata https://review.openstack.org/490797 | 19:13 |
| openstackgerrit | Alex Schultz proposed openstack/puppet-sahara master: Bump modules to next minor with -dev https://review.openstack.org/491065 | 19:38 |
| *** jeremyfreudberg has quit IRC | 20:13 | |
| -openstackstatus- NOTICE: Gerrit is being restarted to pick up CSS changes and should be back momentarily | 20:36 | |
| *** lucasxu has quit IRC | 20:51 | |
| tomtomtom | hey we implemented use_namespaces, based on some suggestions in here, what is the exact purpose for that? is it needed? we are getting connections issues now finishing the cluster so just looking for some clarification. | 21:03 |
| openstackgerrit | Merged openstack/puppet-sahara master: Bump modules to next minor with -dev https://review.openstack.org/491065 | 21:08 |
| *** esikachev has quit IRC | 21:13 | |
| *** cpusmith has quit IRC | 21:16 | |
| openstackgerrit | Merged openstack/sahara master: Better keystonemiddleware log level https://review.openstack.org/490971 | 21:30 |
| openstackgerrit | Merged openstack/sahara master: Deprecate Spark 1.3.1 https://review.openstack.org/490630 | 21:30 |
| *** dave-mccowan has quit IRC | 21:38 | |
| *** esikachev has joined #openstack-sahara | 22:34 | |
| *** esikachev has quit IRC | 22:39 | |
| *** esikachev has joined #openstack-sahara | 22:41 | |
| *** esikachev has quit IRC | 23:26 | |
| *** esikachev has joined #openstack-sahara | 23:33 | |
| *** esikache1 has joined #openstack-sahara | 23:40 | |
| *** esikachev has quit IRC | 23:40 | |
| *** esikache1 has quit IRC | 23:46 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!