Wednesday, 2016-02-17

*** takashi has joined #openstack-storlets07:55
*** cdoron has joined #openstack-storlets08:36
*** takashi has quit IRC09:19
*** cdoron has quit IRC10:03
*** cdoron has joined #openstack-storlets10:04
*** patchbot has quit IRC10:08
*** patchbot has joined #openstack-storlets10:11
*** eranrom has joined #openstack-storlets11:00
*** mattoliverau has quit IRC11:04
*** matt6434 has joined #openstack-storlets11:04
eranromI have updated the agenda for today.11:07
eranromtalk to you later.11:07
*** eranrom has quit IRC11:39
*** eranrom has joined #openstack-storlets11:43
*** wbhuber has quit IRC11:50
*** wbhuber has joined #openstack-storlets11:50
*** openstack has joined #openstack-storlets12:05
*** kota_ has quit IRC12:32
*** patchbot has quit IRC12:32
*** cdoron has quit IRC12:32
*** hrou_ has quit IRC12:32
*** openstackgerrit_ has quit IRC12:32
*** openstackgerrit has quit IRC12:32
*** patchbot has joined #openstack-storlets12:34
*** openstackgerrit has joined #openstack-storlets12:34
*** openstackgerrit_ has joined #openstack-storlets12:34
*** hrou has joined #openstack-storlets12:35
openstackgerritTakashi Kajinami proposed openstack/storlets: Improve handling about Timeout
openstackgerritMerged openstack/storlets: Refactor validation methods about storlet/dependency upload
eranromHi Takashi.13:00
*** kota_ has joined #openstack-storlets13:01
eranromHi Kota13:01
takashi_It seems freenode is not so stable today :-(13:02
kota_hello, eronrom. It seems I got a problem with irc bouncer so joined directly13:02
kota_takashi, ah really?13:02
takashi_kota_: We see so many connect/disconnect log lines13:02
eranromseems to workfor me. Anyway goos that you are here.13:02
eranromI mean good to have you here :-)13:03
takashi_eranrom: :-)13:03
eranromGot a chance to see the agenda?
takashi_eranrom: I checked it a little before13:03
kota_ok, I am missing whole logs due to bouncer trouble :/13:04
eranromNot much has been going on, but there is the eavesdrop link at the top...13:04
eranromor you meant that the link suffers the same?13:05
kota_ok let's get started13:07
eranromsure:-) Are you ok with the landing order I have put?13:07
kota_review priority?13:08
kota_or agenda order, you mean?13:08
eranrommerge priority actually.13:08
takashi_it almost looks good to me, but13:10
*** takashi_ has quit IRC13:10
kota_looks unstable13:11
eranromwrong order?13:11
kota_i mean takashi_ quit13:12
kota_one point,13:12
kota_I like to land patch 280038 first13:12
patchbotkota_: - storlets - Change info to debug for state printing usage13:12
kota_because get_vaco patch depends on it.13:12
eranromok, will change accordingly13:12
eranromok, missed that13:12
*** takashi has joined #openstack-storlets13:12
takashisorry for my disconnection13:13
takashiwebchat.freenot got down... :-(13:13
eranromno problem, here is what you have missed:13:13
eranrom*** takashi_ quit (*.net *.split)13:13
eranrom[15:11] <kota_> looks unstable13:13
eranrom[15:11] <eranrom> wrong order?13:13
eranrom[15:12] <kota_> i mean takashi_ quit13:13
eranrom[15:12] <kota_> so13:13
eranrom[15:12] <kota_> one point,13:13
eranrom[15:12] <eranrom> :-)13:13
eranrom[15:12] <kota_> I like to land patch 280038 first13:13
patchboteranrom: - storlets - Change info to debug for state printing usage13:13
eranrom[15:12] <patchbot> kota_: - storlets - Change info to debug for state printing usage13:13
eranrom[15:12] <kota_> because get_vaco patch depends on it.13:13
kota_that's why it doen't land yet even though you add workflow + 113:13
patchboteranrom: - storlets - Change info to debug for state printing usage13:13
takashieranrom: thx13:13
eranromsure, I will take care of that, my mistake...13:13
kota_looks conflicted with something for now though :/13:14
takashieranrom: And one more thing. I think the first ut refactoring patch depends on the third patch (logger exception)13:14
eranromsure, I will get the order right.13:14
eranromotherwise, are you ok with this?13:15
kota_yup, looks good to me13:15
takashilooks good to me, too.13:16
eranromok, before the hackathon point, do you have anything else?13:16
takashinothing special from me. I'm going to restart working about Range stuff after ut refactoring get merged.13:17
kota_same with takashi13:18
eranromok, so here is what I am thinking:13:18
eranromEC does not seem to leave well with Storlets - that is, it is currently force us to run on proxy.13:19
eranromOn the other hand, hadoop and spark jobs tend to break the analysis of large files into many smaller jobs.13:19
eranromWe could be analytics friendly, if we knew to parse the "EC scheme" and direct a hadoop/spark range requests to the appropriate node, knowing the right offset13:21
eranromdoes this make sense?13:22
eranromfor ranges that cross EC fragments we can work at the proxy. Still we need to make sure that an EC fragement is larger then a typical range requested from, say, spark13:24
eranromI guess my initial question is whether this can be done without changes to Swift.13:24
kota_i think at least we need swift changes, just IMO.13:27
kota_actually, the proxy can know the actual layout, ranges, offset.13:28
eranromCan our middleware access this?13:28
kota_but the difficalities comes from the randomnes for access nodes13:28
eranromnot sure I follow that.13:29
kota_object-erver can know 'It's a request for EC due to request (or response) headers'13:30
kota_and if the object sever knows some info (e.g. ec type, k, m), the range could be calculated.13:31
kota_however, as I said above,13:31
kota_proxy-sever can access flatly data fragments and parities.13:32
kota_if proxy-sever would send requests whole nodes which have *datat* fragments, it seems great that they have partial objects13:33
takashiand not parity objects, right>13:34
kota_however, if proxy send requests to nodes have *parity*, it's imposible to do anything.13:34
takashiWe have to inject some policy when selecting object-server into proxy-server13:34
takashiright understanding>13:34
eranromright, so I guess that without changing the existing code, the proxy does not know where are parity and where are data fragments - right?13:35
kota_hehe,  I have an idea which in working progress13:35
eranromfor the global clusters - right?13:35
kota_I am supprised I didn't expect such an usage while writing a patch/13:35
takashikota_: cool!13:36
kota_eranrom: part of global ec cluster work.13:36
kota_I called it as CompositeRing13:36
eranromsounds like a good topic then :-)13:36
eranromI mean for the hackaton.13:37
kota_i hope so13:37
kota_it might have some constraint but...13:37
kota_so quick description of CompositeRing is...13:37
kota_starting at example13:37
kota_if we have 2 rings, one is 3 replicas and the other is 2 replicas.13:38
kota_the composite ring will consist of 5 replicas.13:38
kota_3 primaries from the fist ring and 2 primaries from the second13:38
kota_if we set EC parameters as num_data=3, num_parity=213:39
kota_data fragments location will be tied up to the first ring.13:40
eranromDo you think of it as a policy hierarchy?13:41
kota_and then preparing a proxy which has the 3 replica ring, it could be possible to access only data fragments.13:41
kota_eranrom: sure13:41
kota_not yet deep thinking though :/13:42
takashikota_: but exciting thing13:42
eranromDid you consider to suggest this as a topic?13:42
kota_actually, the draft implementation was already pushed into gerrit :P13:43
patchbotkota_: - swift - CompositeRing Support13:43
eranromI hope I will have a chance to have a deeper look.13:44
eranromI think there is a topic on improving global clusters.13:44
eranromI guess the idea is to replicate only the data to the remote clusrer - right?13:45
kota_right, that's my first idea.13:45
eranromperhaps we can refer to this also from the Storlets part, highliting the fact it can help with analytics.13:45
kota_to be easy to control the location by operator.13:45
takashieranrom: are you going to add a link to storlet etherpad into swift etherpad?13:48
eranromas soon as I get your OK :-)13:48
eranromAlso, how much time do you think we will need?13:48
takashieranrom: I think now it is ready.13:49
eranromok thanks, I will add the link then13:49
eranromanything else for today?13:50
takashione fyi13:51
takashiI added patch for chunked GET support on the next swift meeting agenda13:51
takashiand going to ask some reviews13:52
takashiThat's all from me.13:52
eranromok thanks!13:52
eranromSo talk to you later. Thanks very much for joining!13:52
takashieranrom: see you tommorow in Japan :-)13:53
openstackgerritKota Tsuyuzaki proposed openstack/storlets: Change info to debug for state printing usage
openstackgerritKota Tsuyuzaki proposed openstack/storlets: Extract ver/acc/con/obj instead of self._get_vaco
kota_rebsed too.14:18
*** eranrom has quit IRC14:34
*** takashi has quit IRC14:39
*** eranrom has joined #openstack-storlets14:51
*** kota_ has quit IRC15:11
*** takashi has joined #openstack-storlets15:24
*** takashi has quit IRC16:25
*** matt6434 has quit IRC17:34
*** matt6434 has joined #openstack-storlets17:37
openstackgerritMerged openstack/storlets: Change info to debug for state printing usage
*** eranrom has quit IRC17:59
*** eranrom has joined #openstack-storlets18:44
openstackgerritMerged openstack/storlets: Use logger.exception instead of original method
*** matt6434 is now known as mattoliverau20:53
*** takashi has joined #openstack-storlets20:59
*** takashi_ has joined #openstack-storlets21:05
*** takashi has quit IRC21:05
*** takashi_ is now known as takashi21:09
*** eranrom has quit IRC21:48
*** takashi has quit IRC22:01

Generated by 2.14.0 by Marius Gedminas - find it at!