| *** iwonka has quit IRC | 00:51 | |
| *** dave-mccowan has joined #openstack-sahara | 01:08 | |
| *** tuanluong has joined #openstack-sahara | 01:19 | |
| *** dave-mccowan has quit IRC | 02:27 | |
| *** dave-mccowan has joined #openstack-sahara | 02:54 | |
| *** ukaynar has quit IRC | 02:55 | |
| *** ukaynar has joined #openstack-sahara | 03:02 | |
| *** dave-mccowan has quit IRC | 03:48 | |
| *** ukaynar has quit IRC | 04:25 | |
| *** ukaynar has joined #openstack-sahara | 04:25 | |
| *** ukaynar has quit IRC | 04:29 | |
| *** Poornima has joined #openstack-sahara | 04:49 | |
| *** Poornima has quit IRC | 05:03 | |
| *** Poornima has joined #openstack-sahara | 05:04 | |
| *** shuyingya has joined #openstack-sahara | 05:19 | |
| *** abalutoiu has quit IRC | 05:39 | |
| *** abalutoiu has joined #openstack-sahara | 05:48 | |
| *** Poornima has quit IRC | 05:55 | |
| *** rcernin has joined #openstack-sahara | 06:05 | |
| *** pgadiya has joined #openstack-sahara | 06:07 | |
| *** Poornima has joined #openstack-sahara | 06:22 | |
| *** tnovacik has quit IRC | 06:29 | |
| *** anshul has joined #openstack-sahara | 06:42 | |
| *** abalutoiu has quit IRC | 06:59 | |
| *** tesseract has joined #openstack-sahara | 07:13 | |
| *** stanchan has joined #openstack-sahara | 07:18 | |
| *** stanchan has quit IRC | 07:28 | |
| *** anshul has quit IRC | 07:47 | |
| *** anshul has joined #openstack-sahara | 07:59 | |
| *** abalutoiu has joined #openstack-sahara | 08:18 | |
| *** hoonetorg has quit IRC | 08:32 | |
| *** Poornima has quit IRC | 08:43 | |
| *** Poornima has joined #openstack-sahara | 08:43 | |
| *** hoonetorg has joined #openstack-sahara | 08:49 | |
| *** Poornima has quit IRC | 08:51 | |
| *** pgadiya has quit IRC | 08:58 | |
| *** Poornima has joined #openstack-sahara | 08:58 | |
| *** pgadiya has joined #openstack-sahara | 09:12 | |
| *** iwonka has joined #openstack-sahara | 10:07 | |
| *** Poornima has quit IRC | 10:11 | |
| *** Poornima has joined #openstack-sahara | 10:27 | |
| *** Poornima has quit IRC | 10:36 | |
| *** Poornima has joined #openstack-sahara | 11:08 | |
| *** dave-mccowan has joined #openstack-sahara | 11:43 | |
| iwonka | ping tellesnobrega | 11:52 |
|---|---|---|
| iwonka | (any ideas what should i do now?) | 11:52 |
| *** tuanluong has quit IRC | 12:06 | |
| tellesnobrega | iwonka, just got in. Not really, the step is try to figure out what is going on | 12:08 |
| tellesnobrega | maybe, you can try debugging (print or exception) on the job binary create and see the flow there to see if you missed something on your code | 12:09 |
| iwonka | i'll try with exception from handle | 12:10 |
| iwonka | it should print the stack | 12:10 |
| tellesnobrega | cool | 12:10 |
| tellesnobrega | that should gives us a hint | 12:10 |
| *** pgadiya has quit IRC | 12:25 | |
| *** shuyingya has quit IRC | 12:31 | |
| iwonka | tellesnobrega: all i got to is that for ngt import form_valid from horizon is not run | 12:33 |
| iwonka | then it goes into django | 12:33 |
| tellesnobrega | that explains why handle is not called | 12:33 |
| tellesnobrega | since it is called from form_valid | 12:33 |
| iwonka | but the name suggests that it may consider the vorm to be invalid? | 12:34 |
| iwonka | or something? | 12:34 |
| iwonka | *form | 12:34 |
| *** pgadiya has joined #openstack-sahara | 12:35 | |
| tellesnobrega | yes, maybe something on your form is wrongly formed. You could make it more simple (just a basic form) or copy a working one to see if goes | 12:37 |
| iwonka | it has one field, how can it be simpler? | 12:40 |
| iwonka | basic form so without any fields? | 12:40 |
| tellesnobrega | I'm not sure, it is simple enough, maybe copy a working one | 12:42 |
| iwonka | it's not run without any fields | 12:42 |
| iwonka | when copying one from job binaries there's an error and the form is not displayed | 12:44 |
| iwonka | but probably it's an issue with urls or something | 12:45 |
| iwonka | but since it's not run without any fields, i guess the problem is nit with the form? | 12:45 |
| tellesnobrega | maybe it has a requirement that a form needs at least one field | 12:47 |
| iwonka | with one field it doesn't work either | 12:50 |
| tellesnobrega | iwonka, but it works with your previous form? | 12:50 |
| iwonka | it never worked | 12:51 |
| iwonka | what do you mean by previous form? | 12:51 |
| tellesnobrega | the form was launched but handle was never called, or it never started? | 12:51 |
| iwonka | handle was never called for changed form | 12:52 |
| iwonka | imean, changed by me | 12:52 |
| iwonka | same problem, when it has one CharField | 12:52 |
| tellesnobrega | is the form being displayed with only one field? | 12:53 |
| iwonka | yes | 12:53 |
| *** ukaynar has joined #openstack-sahara | 12:55 | |
| *** ukaynar has quit IRC | 12:56 | |
| *** ukaynar has joined #openstack-sahara | 12:56 | |
| tellesnobrega | iwonka, thinking | 12:57 |
| *** jeremyfreudberg has joined #openstack-sahara | 12:59 | |
| *** ukaynar has quit IRC | 13:01 | |
| tellesnobrega | iwonka, maybe you could try some debugging here | 13:05 |
| tellesnobrega | https://github.com/openstack/horizon/blob/5f72d0433eaef14072544a7166a03e43aee402ec/horizon/workflows/base.py | 13:05 |
| iwonka | i don't use workflows | 13:06 |
| iwonka | i think | 13:06 |
| iwonka | i tried, but they didn't work with FileField | 13:07 |
| *** lucasxu has joined #openstack-sahara | 13:09 | |
| tellesnobrega | seems like the right approach, job_binaries don't use workflows either | 13:10 |
| *** Poornima has quit IRC | 13:21 | |
| *** tellesnobrega has left #openstack-sahara | 13:24 | |
| *** tellesnobrega has joined #openstack-sahara | 13:24 | |
| tellesnobrega | jeremyfreudberg, are you going to join the meeting today? | 13:26 |
| jeremyfreudberg | tellesnobrega, if there's something to discuss.... personally i don't have much i need to report | 13:26 |
| tellesnobrega | tosky is out today, and most of pike is done | 13:26 |
| tellesnobrega | thinking that we could skip today | 13:26 |
| jeremyfreudberg | tellesnobrega, let's skip | 13:27 |
| *** Poornima has joined #openstack-sahara | 13:33 | |
| tellesnobrega | jeremyfreudberg, sure | 13:44 |
| tellesnobrega | iwonka, have you checked what those attrs={} on the CleareableFileInput means? | 13:45 |
| tellesnobrega | I see that your code has no attrs on the widget definition, but on job_binary it sets 3 | 13:46 |
| tellesnobrega | widget=forms.ClearableFileInput( | 13:46 |
| tellesnobrega | attrs={ | 13:46 |
| tellesnobrega | 'class': 'switched', | 13:46 |
| tellesnobrega | 'data-switch-on': 'jb_internal', | 13:46 |
| tellesnobrega | 'data-jb_internal-uploadfile': _("Upload File") | 13:46 |
| tellesnobrega | })) | 13:46 |
| tellesnobrega | iwonka, can you try changing from a FileField for something else? just a charfield to see if works | 13:47 |
| *** esikachev has joined #openstack-sahara | 13:54 | |
| tellesnobrega | esikachev, we won't be having sahara meeting today, not sure if you saw the email | 13:57 |
| esikachev | ok | 13:57 |
| esikachev | thanks | 13:57 |
| jeremyfreudberg | esikachev, btw storage issue is now critical / even worse on my cloud, sorry | 13:57 |
| esikachev | :( | 13:58 |
| *** ssmith has joined #openstack-sahara | 13:58 | |
| esikachev | it's sad | 13:58 |
| jeremyfreudberg | it's pretty unfortunate, we are working with red hat ceph support to fix it today, hopefully it gets resolved soon | 13:58 |
| jeremyfreudberg | lots of unhappy users right now | 13:59 |
| *** Poornima has quit IRC | 14:00 | |
| *** Poornima has joined #openstack-sahara | 14:00 | |
| *** cpusmith has joined #openstack-sahara | 14:18 | |
| *** ssmith has quit IRC | 14:22 | |
| *** shuyingya has joined #openstack-sahara | 14:22 | |
| cpusmith | Good morning jeremyfreudberg When we go to scale the cluster we get "Error: {} is not valid under any of the given schemas Error ID: 4da54beb-92e6-40cc-a21f-9a0478b56c3d" | 14:27 |
| cpusmith | What's the issue with ceph? | 14:31 |
| jeremyfreudberg | cpusmith, something about failed rebalancing | 14:32 |
| jeremyfreudberg | cpusmith, that's a pretty strange error. can you provide some info about existing node group template and cluster template, as well as which node group you are trrying to increase/decrease the count of. also, scaling from ui? cli? | 14:32 |
| cpusmith | From UI | 14:35 |
| tellesnobrega | cpusmith, is that the only thing on the logs? | 14:37 |
| *** ukaynar has joined #openstack-sahara | 14:40 | |
| cpusmith | That appeared in a popup | 14:40 |
| cpusmith | http://imgur.com/dviQdWC | 14:40 |
| cpusmith | http://imgur.com/QMfElpJ | 14:40 |
| jeremyfreudberg | cpusmith, you should be able to see more context if you grep around / tail the sahara logs... can you confirm that during the failed scaling you were trying to increase the count of sparkslave? | 14:42 |
| cpusmith | Was adding another slave node. Maybe because the count is only one? | 14:42 |
| tellesnobrega | handler count? | 14:43 |
| tellesnobrega | did you set that? | 14:43 |
| cpusmith | Yes, in the template | 14:43 |
| jeremyfreudberg | i don't think handler count should be the problem... | 14:44 |
| tellesnobrega | I don't think either | 14:44 |
| tellesnobrega | maybe if you find more details on the logs will be easier to pin point the problem | 14:44 |
| cpusmith | OK, looking | 14:45 |
| tellesnobrega | thanks | 14:45 |
| tellesnobrega | jeremyfreudberg, did you see your patch failed the last test? | 14:47 |
| jeremyfreudberg | tellesnobrega, yes, infra issues | 14:48 |
| tellesnobrega | yeah | 14:48 |
| tellesnobrega | rechecking, got the whole patch set up, only waiting for this to land so I can cut stable/pike | 14:48 |
| jeremyfreudberg | tellesnobrega, we need to get this https://review.openstack.org/491297 in too, i think (global reqs bump which also failed the gate due to infra issues) | 14:49 |
| tellesnobrega | true | 14:50 |
| *** cpusmith has quit IRC | 14:56 | |
| *** ssmith has joined #openstack-sahara | 15:06 | |
| ssmith | http://imgur.com/4BTFi1D Says empty body | 15:06 |
| ssmith | http://imgur.com/JtMglYm | 15:08 |
| jeremyfreudberg | ssmith, hmm, well, "empty body" is the {} from your original error. not sure why that would be happening, though | 15:08 |
| tomtomtom | This is also from the same log above and may shed more light on it. http://imgur.com/a/EOXaN | 15:08 |
| ssmith | validation_error | 15:10 |
| ssmith | Describes what is wrong with one of the values provided by user. | 15:10 |
| *** pgadiya has quit IRC | 15:11 | |
| jeremyfreudberg | ssmith, tomtomtom, the body of the request is the scale_object referenced in this function: https://github.com/openstack/sahara-dashboard/blob/master/sahara_dashboard/content/data_processing/clusters/clusters/workflows/scale.py#L114 | 15:12 |
| jeremyfreudberg | eventually trickles down into a python-saharaclient call | 15:13 |
| jeremyfreudberg | of course we have a blanket `except Exception` so we can't really tell, but scale_object is only an empty body if something goes wrong | 15:14 |
| jeremyfreudberg | which is why i'm surprised you don't see the "unable to fetch cluster to scale" message | 15:14 |
| tomtomtom | Yeah where does "Validation error" come from? | 15:18 |
| jeremyfreudberg | i wonder what happens if you edit the code to stop blindly swallowing the exception | 15:18 |
| jeremyfreudberg | tomtomtom, validation error is the server-side response | 15:18 |
| tomtomtom | right but which file I mean I can see the Unable to get cluster error but not Validation, so something is happening before it gets there to stop the process? | 15:19 |
| tomtomtom | another issue that looks like it's related is: 2017-08-10 15:19:31.683 37874 INFO keystonemiddleware.auth_token [-] Rejecting request, I can't find where it's coming from, all my auth is correct in the config files, any ideas? | 15:20 |
| ssmith | def test_cluster_scaling_scheme_v_resize_ng(self, engine_version): | 15:20 |
| ssmith | self._create_object_fun = mock.Mock() | 15:20 |
| ssmith | data = { | 15:20 |
| ssmith | } | 15:20 |
| ssmith | self._assert_cluster_scaling_validation( | 15:20 |
| ssmith | data=data, | 15:20 |
| ssmith | bad_req_i=(1, 'VALIDATION_ERROR', | 15:20 |
| ssmith | u'{} is not valid under any of the given schemas') | 15:20 |
| tomtomtom | are there multiple auth sections in sahara? | 15:21 |
| jeremyfreudberg | tomtomtom, the rejecting request is not related, see https://review.openstack.org/#/c/490971/ and the referenced bug | 15:21 |
| jeremyfreudberg | ssmith, i saw that too | 15:22 |
| *** cpusmith has joined #openstack-sahara | 15:23 | |
| *** ssmith has quit IRC | 15:23 | |
| jeremyfreudberg | cpusmith, tomtomtom , i'm still pretty mystified | 15:25 |
| jeremyfreudberg | i still think playing with the except exception on https://github.com/openstack/sahara-dashboard/blame/master/sahara_dashboard/content/data_processing/clusters/clusters/workflows/scale.py#L154 is your best bet | 15:26 |
| tomtomtom | ok, I also noticed that the validation error comes from a bad request: ./tests/unit/service/validation/test_cluster_template_create_validation.py: bad_req_i=(1, "VALIDATION_ERROR", would this be an accurate assumption here? maybe the data it's expecting in the request isn't working as intended | 15:28 |
| *** cpusmith has quit IRC | 15:29 | |
| jeremyfreudberg | a request body of just {} is the bad request | 15:29 |
| *** cpusmith has joined #openstack-sahara | 15:29 | |
| *** cpusmith_ has joined #openstack-sahara | 15:30 | |
| jeremyfreudberg | tomtomtom, cpusmith , i will be afk for a while | 15:31 |
| tomtomtom | ok I'll play with the scale.py code as suggested | 15:32 |
| *** cpusmith has quit IRC | 15:34 | |
| *** esikachev has quit IRC | 15:38 | |
| cpusmith_ | Is Sahara 5.0.1 the highest that we can run on Newton? | 15:47 |
| *** Poornima has quit IRC | 15:54 | |
| *** shuyingya has quit IRC | 15:54 | |
| *** ukaynar has quit IRC | 15:55 | |
| *** shuyingya has joined #openstack-sahara | 15:55 | |
| *** esikachev has joined #openstack-sahara | 15:56 | |
| *** shuyingya has quit IRC | 15:59 | |
| jeremyfreudberg | cpusmith_, sahara 5.0.1 is the latest newton | 16:02 |
| jeremyfreudberg | for the ui there was one fix that didn't make it into 5.0.1, but i doubt it affects you https://github.com/openstack/sahara-dashboard/commit/238e8bdd4002a1f0fb0766d055dbb834657c6e32 | 16:03 |
| jeremyfreudberg | otherwise 5.0.1 is equivalent to stable/newton HEAD | 16:04 |
| jeremyfreudberg | (still afk) | 16:04 |
| *** stanchan has joined #openstack-sahara | 16:14 | |
| *** stanchan has left #openstack-sahara | 16:15 | |
| *** anshul has quit IRC | 16:20 | |
| *** esikachev has quit IRC | 16:25 | |
| *** pcaruana has quit IRC | 16:25 | |
| *** jeremyfreudberg has quit IRC | 16:26 | |
| *** lucasxu has quit IRC | 16:27 | |
| *** shuyingya has joined #openstack-sahara | 16:30 | |
| *** ukaynar has joined #openstack-sahara | 16:31 | |
| *** rcernin has quit IRC | 16:33 | |
| *** shuyingya has quit IRC | 16:35 | |
| *** jeremyfreudberg has joined #openstack-sahara | 16:45 | |
| *** cpusmith has joined #openstack-sahara | 17:01 | |
| *** cpusmith_ has quit IRC | 17:05 | |
| *** abalutoiu has quit IRC | 17:06 | |
| *** jeremyfreudberg_ has joined #openstack-sahara | 17:12 | |
| *** jeremyfreudberg has quit IRC | 17:14 | |
| *** esikachev has joined #openstack-sahara | 17:21 | |
| *** esikachev has quit IRC | 17:26 | |
| *** jeremyfreudberg_ is now known as jeremyfreudberg | 17:27 | |
| *** ukaynar has quit IRC | 17:45 | |
| *** esikachev has joined #openstack-sahara | 18:22 | |
| *** esikachev has quit IRC | 18:26 | |
| *** tellesnobrega has quit IRC | 18:27 | |
| *** tellesnobrega has joined #openstack-sahara | 18:29 | |
| *** rcernin has joined #openstack-sahara | 18:35 | |
| *** cpusmith has quit IRC | 18:36 | |
| *** jeremyfreudberg has quit IRC | 18:57 | |
| *** rcernin has quit IRC | 18:59 | |
| *** openstackgerrit has quit IRC | 19:03 | |
| *** esikachev has joined #openstack-sahara | 19:09 | |
| *** tnovacik has joined #openstack-sahara | 19:27 | |
| *** esikachev has quit IRC | 19:36 | |
| *** tnovacik has quit IRC | 19:45 | |
| *** tnovacik has joined #openstack-sahara | 19:58 | |
| *** esikachev has joined #openstack-sahara | 20:05 | |
| *** esikachev has quit IRC | 20:10 | |
| *** tnovacik has quit IRC | 20:11 | |
| *** openstackgerrit has joined #openstack-sahara | 20:33 | |
| openstackgerrit | Merged openstack/sahara master: Fully switch to keystone authtoken parameters https://review.openstack.org/485521 | 20:33 |
| iwonka | tellesnobrega: i tried charfield at some point, it didn't work | 20:53 |
| iwonka | that's why i'm pretty sire the issue is not in the field itself | 20:54 |
| iwonka | *not in the form | 20:55 |
| tellesnobrega | iwonka, hmm, I was thinking that if the form could be invalid due to the FileField | 20:55 |
| iwonka | i copied field for name from job binary | 20:58 |
| tellesnobrega | yes, I saw | 20:58 |
| tellesnobrega | except from some args in the dict there | 20:58 |
| tellesnobrega | probably not an issue | 20:58 |
| tellesnobrega | so trying other field would test this | 20:59 |
| iwonka | but it's not the problem | 21:04 |
| tellesnobrega | I believe you | 21:04 |
| iwonka | is there any way to try other parts of the code? | 21:04 |
| *** esikachev has joined #openstack-sahara | 21:06 | |
| tellesnobrega | I'm not sure, I will have to take a look into it. I haven't done a lot of work on sahara-dashboard and its been a while, you probably know than I do | 21:06 |
| iwonka | i have like no idea what to do with the code now | 21:07 |
| iwonka | i still have to fix one commit | 21:07 |
| iwonka | and do the same for cluster templates, but that should be easy… | 21:07 |
| openstackgerrit | Merged openstack/sahara master: Updated from global requirements https://review.openstack.org/491297 | 21:08 |
| *** esikachev has quit IRC | 21:10 | |
| tellesnobrega | iwonka, I don't have time to test it right now (working on some downstream troubles) but I will get to it asap | 21:11 |
| iwonka | ok, sure | 21:11 |
| tellesnobrega | but please, keep me on the loop if you figure out something or have more questions | 21:11 |
| tellesnobrega | I'm always here to help | 21:11 |
| *** cpusmith has joined #openstack-sahara | 21:15 | |
| iwonka | i'm not sure what could i do with it | 21:16 |
| iwonka | so i'll move to cluster templates i guess | 21:16 |
| iwonka | is that ok? | 21:16 |
| tellesnobrega | sounds good | 21:20 |
| *** tesseract has quit IRC | 22:02 | |
| *** esikachev has joined #openstack-sahara | 22:06 | |
| *** esikachev has quit IRC | 22:11 | |
| *** openstackgerrit has quit IRC | 22:18 | |
| *** esikachev has joined #openstack-sahara | 23:07 | |
| *** esikachev has quit IRC | 23:12 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!