| *** devlaps has joined #openstack-storlets | 04:58 | |
| openstackgerrit | Akihito Takai proposed openstack/storlets: Add functional Tests for storlet app written in python language https://review.openstack.org/400057 | 06:19 |
|---|---|---|
| openstackgerrit | Kota Tsuyuzaki proposed openstack/storlets: WIP: add test for long running tasks to find which part should be fixed https://review.openstack.org/407342 | 06:45 |
| openstackgerrit | Kota Tsuyuzaki proposed openstack/storlets: WIP: add test for long running tasks to find which part should be fixed https://review.openstack.org/407342 | 06:49 |
| openstackgerrit | Kota Tsuyuzaki proposed openstack/storlets: Fix a space which is not a problem in running https://review.openstack.org/407349 | 06:52 |
| openstackgerrit | Kota Tsuyuzaki proposed openstack/storlets: WIP: add test for long running tasks to find which part should be fixed https://review.openstack.org/407342 | 07:26 |
| kota_ | Thanks eranrom for quick review! | 07:34 |
| eranrom | kota_: welcome. | 07:35 |
| openstackgerrit | Merged openstack/storlets: Fix a space which is not a problem in running https://review.openstack.org/407349 | 07:52 |
| openstackgerrit | Kota Tsuyuzaki proposed openstack/storlets: WIP: add test for long running tasks to find which part should be fixed https://review.openstack.org/407342 | 08:02 |
| openstackgerrit | Kota Tsuyuzaki proposed openstack/storlets: WIP: add test for long running tasks to find which part should be fixed https://review.openstack.org/407342 | 08:32 |
| openstackgerrit | Kota Tsuyuzaki proposed openstack/storlets: WIP: add test for long running tasks to find which part should be fixed https://review.openstack.org/407342 | 10:21 |
| kota_ | eranrom: done for walking through the v3 patch, basically code looks great. | 10:44 |
| eranrom | kota_: thanks. any comments / questions? | 10:44 |
| kota_ | eranrom: I found some places using/setting tenant_name for the ansible | 10:44 |
| kota_ | eranrom: I added inlines for that, could you make sure that is in your intention or we need to fix them? | 10:45 |
| eranrom | Let me fix them | 10:45 |
| eranrom | thanks! | 10:46 |
| kota_ | thanks, I'm not sure for kesytone v3 but my limited understanding, tenant_name is unecessary anymore. | 10:46 |
| kota_ | yup :-) | 10:46 |
| eranrom | kota_: in fact tenant_name is now project_name | 10:46 |
| kota_ | eranrom: that fits my idea ;-) | 10:47 |
| eranrom | and I should change the internal variables naming in ansible (I think this is what you saw...) | 10:47 |
| kota_ | yup | 10:47 |
| kota_ | eranrom: if you address them today, I'll circle back to tommorow morning to give my +2 ;-) | 10:48 |
| eranrom | thanks. I will try to do this now. | 10:48 |
| eranrom | Thanks! | 10:48 |
| * kota_ is going to be back home | 10:52 | |
| openstackgerrit | Eran Rom proposed openstack/storlets: Kestone auth version to v3 https://review.openstack.org/394168 | 11:08 |
| *** devlaps has quit IRC | 14:21 | |
| openstackgerrit | Eran Rom proposed openstack/storlets: Minor Fixes https://review.openstack.org/407559 | 15:14 |
| openstackgerrit | Eran Rom proposed openstack/storlets: Minor Fixes https://review.openstack.org/407559 | 15:14 |
| *** devlaps has joined #openstack-storlets | 16:07 | |
| openstackgerrit | Eran Rom proposed openstack/storlets: Minor Fixes https://review.openstack.org/407559 | 19:05 |
| *** devlaps has quit IRC | 19:16 | |
| *** devlaps has joined #openstack-storlets | 19:17 | |
| eranrom | kota_: Regarding the test_long_run_storlet: | 19:33 |
| eranrom | First, in order to use your testing code, I needed few minor fixes (https://review.openstack.org/#/c/407559/) | 19:33 |
| eranrom | With those fixes in place I raised the storlet_timeout to 500 seconds so as to make sure that our code is not in the way. | 19:35 |
| eranrom | Interestingly, for PUT there was no problem to run the storlet with 400sec sleep, everything worked like charm! | 19:36 |
| eranrom | This was the case when invoking the storlet both on the object server as well as on the proxy. So far so good. | 19:37 |
| eranrom | Note that all through my testing the swift node_timeout is set to 120 seconds. | 19:37 |
| eranrom | Now for the GET. In the GET data path we timeout on the node_timeout which results with 503. | 19:38 |
| eranrom | Again this happens when running the storlet either in the object or in the proxy servers. I believe that the reason for having the same behaviour | 19:39 |
| eranrom | for both object and proxy is that when running on proxy we do a get form the object and then execute the storlet so from the proxy's perspective its | 19:40 |
| eranrom | as if the read from the object node was timedout. | 19:40 |
| eranrom | I believe that the actual timeout is coming from: proxy/controllers/base.py where a call to either next or read from our iterator times out. | 19:43 |
| eranrom | To summarise, I think that there are two problems we need to solve: | 19:44 |
| eranrom | 1. In PUT, allow a long running storlet without the need to have an unlimited storlet_timeout value. | 19:45 |
| eranrom | 2. In GET, allow a long running storlet, again without the need to have an unlimited node_timeout | 19:45 |
| eranrom | Here is what I think the solution for both: | 19:45 |
| eranrom | Add a keepalive mechanism to the 'protocol' with the storlet. That is give the Storlet an API call that it needs to call every now and then. | 19:47 |
| eranrom | If the storlet does not call this API in more then storlet_timeout then we consider the storlet stuck and kill it as we do today upon storlet_timeout. | 19:47 |
| eranrom | For a GET scenario, this keepalive call needs to be intercepted in the read/next methods of our iterator (storlets/gateway/common/stob.py FileDescriptorIterator) | 19:51 |
| eranrom | This will resolve the GET when the storlet is executed on the object node. | 19:55 |
| eranrom | The PUT and GET when executed on proxy is more tricky IMO, and may require changes outside of the storlet module, but I need more time to investigate. | 19:56 |
| eranrom | BTW the keepalive can be implemented using an additional pipe where the storlet side call would write to the pipe and in the read/next methods we add the read end of the pipe | 19:58 |
| eranrom | to the select fd set. | 19:58 |
| eranrom | Hope this all make sense... | 19:58 |
| *** devlaps has quit IRC | 20:10 | |
| *** devlaps has joined #openstack-storlets | 21:06 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!