| *** takashi has joined #openstack-storlets | 07:55 | |
| *** cdoron has joined #openstack-storlets | 08:36 | |
| *** takashi has quit IRC | 09:19 | |
| *** cdoron has quit IRC | 10:03 | |
| *** cdoron has joined #openstack-storlets | 10:04 | |
| *** patchbot has quit IRC | 10:08 | |
| *** patchbot has joined #openstack-storlets | 10:11 | |
| *** eranrom has joined #openstack-storlets | 11:00 | |
| *** mattoliverau has quit IRC | 11:04 | |
| *** matt6434 has joined #openstack-storlets | 11:04 | |
| eranrom | I have updated the agenda for today. | 11:07 |
|---|---|---|
| eranrom | talk to you later. | 11:07 |
| eranrom | thanks! | 11:07 |
| *** eranrom has quit IRC | 11:39 | |
| *** eranrom has joined #openstack-storlets | 11:43 | |
| *** wbhuber has quit IRC | 11:50 | |
| *** wbhuber has joined #openstack-storlets | 11:50 | |
| *** openstack has joined #openstack-storlets | 12:05 | |
| *** kota_ has quit IRC | 12:32 | |
| *** patchbot has quit IRC | 12:32 | |
| *** cdoron has quit IRC | 12:32 | |
| *** hrou_ has quit IRC | 12:32 | |
| *** openstackgerrit_ has quit IRC | 12:32 | |
| *** openstackgerrit has quit IRC | 12:32 | |
| *** patchbot has joined #openstack-storlets | 12:34 | |
| *** openstackgerrit has joined #openstack-storlets | 12:34 | |
| *** openstackgerrit_ has joined #openstack-storlets | 12:34 | |
| *** hrou has joined #openstack-storlets | 12:35 | |
| openstackgerrit | Takashi Kajinami proposed openstack/storlets: Improve handling about Timeout https://review.openstack.org/276545 | 12:49 |
| openstackgerrit | Merged openstack/storlets: Refactor validation methods about storlet/dependency upload https://review.openstack.org/280488 | 12:52 |
| eranrom | Hï | 13:00 |
| eranrom | Hï | 13:00 |
| eranrom | Hi | 13:00 |
| takashi_ | Hello | 13:00 |
| eranrom | Hi Takashi. | 13:00 |
| *** kota_ has joined #openstack-storlets | 13:01 | |
| eranrom | Hi Kota | 13:01 |
| takashi_ | It seems freenode is not so stable today :-( | 13:02 |
| kota_ | hello, eronrom. It seems I got a problem with irc bouncer so joined directly | 13:02 |
| kota_ | takashi, ah really? | 13:02 |
| takashi_ | kota_: We see so many connect/disconnect log lines | 13:02 |
| eranrom | seems to workfor me. Anyway goos that you are here. | 13:02 |
| eranrom | I mean good to have you here :-) | 13:03 |
| takashi_ | eranrom: :-) | 13:03 |
| eranrom | Got a chance to see the agenda? https://wiki.openstack.org/wiki/Meetings/Storlets | 13:03 |
| kota_ | \o/ | 13:03 |
| takashi_ | eranrom: I checked it a little before | 13:03 |
| kota_ | ok, I am missing whole logs due to bouncer trouble :/ | 13:04 |
| eranrom | Not much has been going on, but there is the eavesdrop link at the top... | 13:04 |
| eranrom | or you meant that the link suffers the same? | 13:05 |
| kota_ | enough | 13:06 |
| kota_ | ok let's get started | 13:07 |
| eranrom | sure:-) Are you ok with the landing order I have put? | 13:07 |
| kota_ | review priority? | 13:08 |
| kota_ | or agenda order, you mean? | 13:08 |
| eranrom | merge priority actually. | 13:08 |
| takashi_ | it almost looks good to me, but | 13:10 |
| *** takashi_ has quit IRC | 13:10 | |
| kota_ | looks unstable | 13:11 |
| eranrom | wrong order? | 13:11 |
| kota_ | i mean takashi_ quit | 13:12 |
| kota_ | so | 13:12 |
| kota_ | one point, | 13:12 |
| eranrom | :-) | 13:12 |
| kota_ | I like to land patch 280038 first | 13:12 |
| patchbot | kota_: https://review.openstack.org/#/c/280038/ - storlets - Change info to debug for state printing usage | 13:12 |
| kota_ | because get_vaco patch depends on it. | 13:12 |
| eranrom | ok, will change accordingly | 13:12 |
| eranrom | ok, missed that | 13:12 |
| *** takashi has joined #openstack-storlets | 13:12 | |
| takashi | sorry for my disconnection | 13:13 |
| takashi | webchat.freenot got down... :-( | 13:13 |
| eranrom | no problem, here is what you have missed: | 13:13 |
| eranrom | *** takashi_ quit (*.net *.split) | 13:13 |
| eranrom | [15:11] <kota_> looks unstable | 13:13 |
| eranrom | [15:11] <eranrom> wrong order? | 13:13 |
| eranrom | [15:12] <kota_> i mean takashi_ quit | 13:13 |
| eranrom | [15:12] <kota_> so | 13:13 |
| eranrom | [15:12] <kota_> one point, | 13:13 |
| eranrom | [15:12] <eranrom> :-) | 13:13 |
| eranrom | [15:12] <kota_> I like to land patch 280038 first | 13:13 |
| patchbot | eranrom: https://review.openstack.org/#/c/280038/ - storlets - Change info to debug for state printing usage | 13:13 |
| eranrom | [15:12] <patchbot> kota_: https://review.openstack.org/#/c/280038/ - storlets - Change info to debug for state printing usage | 13:13 |
| eranrom | [15:12] <kota_> because get_vaco patch depends on it. | 13:13 |
| kota_ | that's why it doen't land yet even though you add workflow + 1 | 13:13 |
| patchbot | eranrom: https://review.openstack.org/#/c/280038/ - storlets - Change info to debug for state printing usage | 13:13 |
| takashi | eranrom: thx | 13:13 |
| eranrom | sure, I will take care of that, my mistake... | 13:13 |
| kota_ | looks conflicted with something for now though :/ | 13:14 |
| takashi | eranrom: And one more thing. I think the first ut refactoring patch depends on the third patch (logger exception) | 13:14 |
| eranrom | sure, I will get the order right. | 13:14 |
| eranrom | otherwise, are you ok with this? | 13:15 |
| kota_ | yup, looks good to me | 13:15 |
| takashi | looks good to me, too. | 13:16 |
| eranrom | ok, before the hackathon point, do you have anything else? | 13:16 |
| takashi | nothing special from me. I'm going to restart working about Range stuff after ut refactoring get merged. | 13:17 |
| eranrom | great | 13:17 |
| kota_ | same with takashi | 13:18 |
| eranrom | ok, so here is what I am thinking: | 13:18 |
| eranrom | EC does not seem to leave well with Storlets - that is, it is currently force us to run on proxy. | 13:19 |
| eranrom | On the other hand, hadoop and spark jobs tend to break the analysis of large files into many smaller jobs. | 13:19 |
| kota_ | exactry | 13:19 |
| kota_ | s/r/l | 13:19 |
| eranrom | We could be analytics friendly, if we knew to parse the "EC scheme" and direct a hadoop/spark range requests to the appropriate node, knowing the right offset | 13:21 |
| eranrom | does this make sense? | 13:22 |
| kota_ | yup | 13:23 |
| takashi | yes | 13:23 |
| eranrom | for ranges that cross EC fragments we can work at the proxy. Still we need to make sure that an EC fragement is larger then a typical range requested from, say, spark | 13:24 |
| eranrom | I guess my initial question is whether this can be done without changes to Swift. | 13:24 |
| kota_ | thinking | 13:25 |
| kota_ | i think at least we need swift changes, just IMO. | 13:27 |
| kota_ | actually, the proxy can know the actual layout, ranges, offset. | 13:28 |
| eranrom | Can our middleware access this? | 13:28 |
| kota_ | but the difficalities comes from the randomnes for access nodes | 13:28 |
| eranrom | not sure I follow that. | 13:29 |
| kota_ | object-erver can know 'It's a request for EC due to request (or response) headers' | 13:30 |
| kota_ | and if the object sever knows some info (e.g. ec type, k, m), the range could be calculated. | 13:31 |
| kota_ | however, as I said above, | 13:31 |
| kota_ | proxy-sever can access flatly data fragments and parities. | 13:32 |
| kota_ | if proxy-sever would send requests whole nodes which have *datat* fragments, it seems great that they have partial objects | 13:33 |
| takashi | and not parity objects, right> | 13:34 |
| kota_ | however, if proxy send requests to nodes have *parity*, it's imposible to do anything. | 13:34 |
| takashi | We have to inject some policy when selecting object-server into proxy-server | 13:34 |
| takashi | right understanding> | 13:34 |
| takashi | s/>/? | 13:34 |
| eranrom | right, so I guess that without changing the existing code, the proxy does not know where are parity and where are data fragments - right? | 13:35 |
| kota_ | hehe, I have an idea which in working progress | 13:35 |
| eranrom | for the global clusters - right? | 13:35 |
| kota_ | I am supprised I didn't expect such an usage while writing a patch/ | 13:35 |
| takashi | kota_: cool! | 13:36 |
| kota_ | eranrom: part of global ec cluster work. | 13:36 |
| eranrom | :-) | 13:36 |
| kota_ | I called it as CompositeRing | 13:36 |
| eranrom | sounds like a good topic then :-) | 13:36 |
| eranrom | I mean for the hackaton. | 13:37 |
| kota_ | i hope so | 13:37 |
| kota_ | it might have some constraint but... | 13:37 |
| kota_ | so quick description of CompositeRing is... | 13:37 |
| kota_ | starting at example | 13:37 |
| kota_ | if we have 2 rings, one is 3 replicas and the other is 2 replicas. | 13:38 |
| kota_ | the composite ring will consist of 5 replicas. | 13:38 |
| kota_ | 3 primaries from the fist ring and 2 primaries from the second | 13:38 |
| kota_ | if we set EC parameters as num_data=3, num_parity=2 | 13:39 |
| kota_ | data fragments location will be tied up to the first ring. | 13:40 |
| eranrom | gotcha. | 13:40 |
| eranrom | Do you think of it as a policy hierarchy? | 13:41 |
| kota_ | and then preparing a proxy which has the 3 replica ring, it could be possible to access only data fragments. | 13:41 |
| kota_ | eranrom: sure | 13:41 |
| eranrom | nice! | 13:41 |
| kota_ | not yet deep thinking though :/ | 13:42 |
| takashi | kota_: but exciting thing | 13:42 |
| eranrom | absolutly | 13:42 |
| eranrom | Did you consider to suggest this as a topic? | 13:42 |
| kota_ | actually, the draft implementation was already pushed into gerrit :P | 13:43 |
| kota_ | https://review.openstack.org/#/c/271920/ | 13:43 |
| patchbot | kota_: https://review.openstack.org/#/c/271920/ - swift - CompositeRing Support | 13:43 |
| eranrom | I hope I will have a chance to have a deeper look. | 13:44 |
| eranrom | I think there is a topic on improving global clusters. | 13:44 |
| eranrom | I guess the idea is to replicate only the data to the remote clusrer - right? | 13:45 |
| kota_ | right, that's my first idea. | 13:45 |
| eranrom | perhaps we can refer to this also from the Storlets part, highliting the fact it can help with analytics. | 13:45 |
| kota_ | to be easy to control the location by operator. | 13:45 |
| eranrom | right | 13:46 |
| takashi | eranrom: are you going to add a link to storlet etherpad into swift etherpad? | 13:48 |
| eranrom | as soon as I get your OK :-) | 13:48 |
| eranrom | Also, how much time do you think we will need? | 13:48 |
| takashi | eranrom: I think now it is ready. | 13:49 |
| eranrom | ok thanks, I will add the link then | 13:49 |
| eranrom | anything else for today? | 13:50 |
| kota_ | nothing | 13:50 |
| takashi | one fyi | 13:51 |
| takashi | I added patch for chunked GET support on the next swift meeting agenda | 13:51 |
| eranrom | cool! | 13:51 |
| takashi | and going to ask some reviews | 13:52 |
| takashi | That's all from me. | 13:52 |
| eranrom | ok thanks! | 13:52 |
| eranrom | So talk to you later. Thanks very much for joining! | 13:52 |
| takashi | eranrom: see you tommorow in Japan :-) | 13:53 |
| kota_ | thanks | 13:53 |
| eranrom | :-) | 13:54 |
| openstackgerrit | Kota Tsuyuzaki proposed openstack/storlets: Change info to debug for state printing usage https://review.openstack.org/280038 | 14:12 |
| kota_ | rebased^^ | 14:12 |
| openstackgerrit | Kota Tsuyuzaki proposed openstack/storlets: Extract ver/acc/con/obj instead of self._get_vaco https://review.openstack.org/280077 | 14:18 |
| kota_ | rebsed too. | 14:18 |
| *** eranrom has quit IRC | 14:34 | |
| *** takashi has quit IRC | 14:39 | |
| *** eranrom has joined #openstack-storlets | 14:51 | |
| *** kota_ has quit IRC | 15:11 | |
| *** takashi has joined #openstack-storlets | 15:24 | |
| *** takashi has quit IRC | 16:25 | |
| *** matt6434 has quit IRC | 17:34 | |
| *** matt6434 has joined #openstack-storlets | 17:37 | |
| openstackgerrit | Merged openstack/storlets: Change info to debug for state printing usage https://review.openstack.org/280038 | 17:53 |
| *** eranrom has quit IRC | 17:59 | |
| *** eranrom has joined #openstack-storlets | 18:44 | |
| openstackgerrit | Merged openstack/storlets: Use logger.exception instead of original method https://review.openstack.org/276543 | 19:36 |
| *** matt6434 is now known as mattoliverau | 20:53 | |
| *** takashi has joined #openstack-storlets | 20:59 | |
| *** takashi_ has joined #openstack-storlets | 21:05 | |
| *** takashi has quit IRC | 21:05 | |
| *** takashi_ is now known as takashi | 21:09 | |
| *** eranrom has quit IRC | 21:48 | |
| *** takashi has quit IRC | 22:01 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!