Monday, 2016-09-05

openstackgerritTakashi Kajinami proposed openstack/storlets: Add range support for python storlet
openstackgerritKota Tsuyuzaki proposed openstack/storlets: Prep works for Py3 gate
openstackgerritMerged openstack/storlets: Creanup redundant and README.rst
*** takashi has joined #openstack-storlets05:27
openstackgerritTakashi Kajinami proposed openstack/storlets: Change docs for Python storlet support
takashieranrom: thanks for your review about python storlet patches. I'll check you comment!06:05
eranromtakashi: sure, I am now looking at the ranges patch06:05
eranromtakashi: I went over all the patches in the blueprint. Awesome work! Thanks!06:11
eranromHere is a summary of the major comments I have:06:11
eranromin order of importance as I see it (feel free to disagree)06:12
eranrom1. Consume the fds in the docker side according to their expected order, and adjust the datagram getters accordingly. We can discuss in what layer to do that.06:12
eranrom2. Complete the necessary documentation in wiki and code. I will be happy to assist with the wiki doc.06:13
eranrom3. Adjust the java code structure accordingly (push samples and tests down the tree)06:13
eranromI can do #306:14
eranromI think that the rest are local changes.06:15
eranromI am ok with first landing and then addressing these. Your call.06:15
eranromHaving python is really exciting!06:15
eranromI am only starting to 'digest' that :-)06:16
takashieranrom: thanks for summarize things. I generally agree with that. I'll check your all remaks, and confirm whether we have other important points to be fixed before landing06:27
takashifrom my side06:28
eranromtakashi: sure!06:28
kota_looks like python supporting get good progress :-)07:30
takashikota_: :-)07:31
takashikota_: I'd also like to ask your comment about these patch sets. Especially, we need to confirm the interface design about python storlet (see patch 363303)07:37
kota_patch bot seems absent07:38
kota_what's your concern on patch 363303?07:39
takashikota_: when I create my initial implementation about python storlet, I tried to use iterator as an interface07:42
kota_what interface means? Input/OutputStreams?07:44
takashikota_: Let me show you more detailed example07:44
kota_takashi: yes please07:45
takashiIn my current patch, I defined the interface like  Input: (in_md: a list of dict, in_fds: a list of FileObject, out_md_files: a list of MetadataFileObject which has dump method, out_files: a list of FileObject, params: a dict) Output: None07:45
takashikota_: yes07:45
takashiIn that file I implemented the interface above07:45
takashibut in my first idea07:45
takashiI tried to define the interface like Input: (in_md: a list of metadata, in_iters: a list of iterators to read objects, params: a dict) Output: (out_md: a list of processed metadata, out_iters: a list of iterators to read processed object07:47
takashito totally hide FileObject interface from user applications07:47
kota_2 questions for quick look07:48
takashikota_: ok07:48
kota_why you set the different method name between out_md_files and out_files?07:49
kota_one has "dump" and the other has "write"07:49
kota_that seems not consistent07:49
takashikota_: good point07:50
kota_ah, plus, in_md[0] is just a dict?07:50
takashikota_: a dict of swift metadata07:50
takashilet me explain about the reason why I have dump method in MetadataFile07:51
takashiIn my old patches, I did't implemented it and used write method inside storlet apps07:51
takashiHowever, in that case, I need to serialize metadata (I mean, call json.dumps) before I write it to outputfile07:52
takashiIMO, json serializing is 'a protocol stuff' which should be hidden from user applications07:52
takashiand that is the reason why I created a 'dump' interface in MetadataFile, which hides that serializing from applications.07:54
kota_let me think07:55
takashiAnother possible solution is creating the interface into storlet applications to just get processed metadata. (that is related to my first concern)07:56
kota_though i may say crazy idea for that,07:57
kota_could we make... 1. make a template class for python storlet app and all storlet apps should inherit to wrap the i/o, 2. and then, in the wrap, could we take a tuple of return value from __call__ wiht like (out_md_files, out_files)07:58
kota_i know it's far from Java api though.07:59
kota_wait a sec, trying to write the code image08:00
takashiDo you mean FileObject, by out_md_files, out_files?08:00
takashiOh. paste.bin is blocked by the proxy in my company...08:01
kota_ok, will do in gist08:02
takashikota_: thx. I'll write down details about my first concern while you are writing sample.08:04
takashikota_: So, as I mentioned I tried to let storlet application returns a list of metadata and iterators in my first draft.08:15
takashikota_: However, I noticed that interface may make it difficult to implement storlet application for muti-input/multi-output08:15
takashikota_: In multi-input/multi-output case, we need to realize conncurrent reading/writing among multiple iterators08:16
*** openstackgerrit has quit IRC08:18
takashiMy concern is that, is it reasonable to let users to implement complited iterators (ex. the synchronization mechanism among multiple output iterators)?08:19
*** openstackgerrit has joined #openstack-storlets08:19
takashior is it better to expose the FileObject interface to users and let them deal with the concurrecy inside storlet apps?08:20
kota_not yet consider how we could pass the data to out_files08:26
kota_sorry, i have to leave here for a while, maybe back in 1 hour.08:27
takashikota_: thanks. wiil check that!08:29
takashikota_: np. I'll create some example implementations. I'm asking this to you because I'm afraid the interface design may cause some problems about multi-io support.08:30
eranromkota_: I think that you suggestion is in fact closer to the java API that wraps the input streams with storlet classes. For example an input stream has a get md and set md methods that hide the serialization08:41
eranromkota_: perhaps I am missing your point though08:41
eranromregarding the gist, I like the idea. The actual implementation has one problem though: We cannot dump the out_md after the invocation, because we need to send it in the request headers before the body08:43
eranromthat is written during invoke08:43
eranromthat is as long as we do not have 'footers'.08:43
eranromtakashi: I guess that your concern has to do with whether to do the multi-threading inside or outside of the storlet - right? seems like that as long as you pass our own wrappers as done in the latest patch, you can always add threading to the read, without changing the API - right?08:53
*** openstackgerrit has quit IRC09:04
*** openstackgerrit has joined #openstack-storlets09:05
takashieranrom: you are right.09:06
takashimaybe I need to arrange my concerns before discussion...09:13
takashieranrom: by the way, I checked all of your comments now, thanks09:15
takashieranrom: I found one thing 'unsafe' about cancel method in StorletDaemon patch, so will fix it.09:16
eranromtakashi: ok, sure.09:20
takashieranrom: I'll pick up your other comments as a follow-up work, but IMO there are not so big blockers, excluding that (and remaining interface dicsussion?), from my side09:20
eranromtakashi: ok, cool. I am thinking that I will start working on the big-tent thing. I can do the java tests and samples refactoring as part of that09:22
eranromat least the pushing down one level09:22
takashieranrom: makes sense09:24
takashiI'm currently thinking two things now, after landing current python storlet.  1. Refactor and add unit test for current agent codes.  2. Refactor directory strucrture for python codes, to use as centric install scripts for all things09:25
openstackgerritDoron Chen proposed openstack/storlets: make the docker registry a service
takashikota_: I created a sample storlet for each interface designs:
takashioh sorry. wring url09:49
takashikota_: ^^09:50
takashiI don't know why we need to block access to gist from the company network...09:50
takashiMy concerin is that I'm afraid returning iterator requires us to implement very complicated things in storlet application09:51
takashiwhen using multi input/output09:52
takashiI didn't run the code, so please see just a sample09:53
takashiIf we return iterators from application, we need to deal with concurrency inside iterators09:54
kota_i'm with09:55
takashiand we need a kind of syncrhonization mechanism (in that sample, I use queue for that) between output iterators09:55
kota_i think, it would be better clarify the i/o patterns09:59
kota_maybe 3 patterns i'm guessing09:59
kota_1. storlet app can read all data from read_fd and it can prepare whole data to write10:00
kota_this is too simple, any format can be used.10:00
kota_2. storelet app app can read all data from read_fd but it wants to write outgoing data streamingly10:01
kota_this may be more complex but iterator seems a possible solution.10:01
kota_3. storlet app wants to read the data streamingly and it wants also to write outgoing data streamingly10:02
kota_this is too complex rather than the others.10:02
takashikota_: yes10:02
kota_Probably, it needs something like 'hook point' during capture/sending data.10:03
kota_now, we're thinking if we could cupslelarate the outgoing data10:04
kota_I think, we could accpept a couple of formats for outgoing data10:05
kota_not sure which format is suitable though,10:05
kota_likely StringIO? Iter? or custom class which eables to implement the *hook point* easily.10:06
takashikota_: Can you show an example about hook point?10:07
kota_not soon, sorry.10:08
kota_just a drafting idea10:08
takashikota_: ok10:08
takashikota_: Do you think we can land current interface design and do that cupselarating later?10:10
kota_not sure10:11
takashikota_: ok. thank you for your time! I'll think more about cupselarating.10:15
takashimaybe I need to list up some storlet examples, based on the i/o patterns, and think how we can deal with them.10:15
kota_maybe my idea is getting concrete in my head, I have to dump and do PoC somewhere...10:15
kota_perhaps, I could take a time to do that...later half of this week...10:16
takashikota_: np10:19
openstackgerritEran Rom proposed openstack/storlets: Towards python-centric repo
eranromHi, I created a new blueprint for the security feature. Today I have added to it a link to a spec in the wiki:
eranromcomments are welcome12:25
takashieranrom: that's the detail of the blueprint you added ,right?
eranromStill no patches, just spec12:52
takashieranrom: I will check its detail later, but at a glance it makes much sense to me12:53
eranromtakashi: Thanks!12:53
eranromtakashi: its not urgent12:53
takashieranrom: At least I currently understand its usecase, and approve it. I don't know who is an appropriate approver for bp in OpenStack projects (PTL? or the other core reviewers than proposer?), but I can be.12:55
eranromtakashi: thanks. I think that we are the approvers of ourselves :-)12:56
takashieranrom: :-)12:58
flaper87See you guys have made progress, that's awesome!13:06
flaper87keep it up13:06
takashiflaper87: thanks!13:24
takashiflaper87: Do you have some time? I'd like to ask one question about copy right stuff?13:25
flaper87takashi: sure, what's up?13:26
takashiflaper87: I'd like to make soure what you mean by 'Change the copyright header to something like', written in big-tent etherpad13:26
takashiflaper87: Do you mean there are some files missing header for Apache 2.0 License?13:27
flaper87takashi: I meant to say that all copyright headers should look like:
flaper87I noticed that some had other things13:28
flaper87not sure if that was fixed already13:28
takashiflaper87: ok. I need to recheck13:29
flaper87takashi: danke :)13:29
flaper87it might be that there's nothing left to do13:29
flaper87but I thought I'd mention it anyway13:30
takashiflaper87: One thing I want to confirm here is, do we need to change the copyright to OpenStack Foundation?13:30
flaper87I think the current stand is you can either add the copyright to the openstack foundation or no copyright at all and just use the apache license header13:30
* flaper87 normally adds the copyright to the foundation13:31
takashiflaper87: I found some files which has a copyright of each company even in Nova:
takashiflaper87: There are some files still has a copyright of IBM in storlets repository. I'm wondering if we need to change them:
takashiflaper87: If we can keep it, I'll change about the license header, and would like to keep current copy right, because they are totally written by IBM.13:35
takashiflaper87: Does it make sense to you?13:35
takashiSome files added after storlet get opensourced already have OpenStack copy right in their headers13:36
takashimaybe we still need to think how we deal with copy right when we modify them... :-(13:37
takashiflaper87: sorry. I need to be offline for a while. I'll check the log later, so please add any comments here, if you have some time. :-)13:42
takashiflaper87: thank you for helping us!13:42
*** takashi has quit IRC13:42
eranromflaper87: Looking at the example you placed in the etherpad ( there is a Redhat copyright (# Copyright 2014 Red Hat, Inc.)14:17
flaper87eranrom: no no, ignore that14:17
eranromflaper87: ok14:17
flaper87eranrom: I'd recommend to update the copyrights to the openstack foundation14:18
flaper87or just skip it and use the license header only14:18
eranromflaper87: ok, being a former IBMer I am not sure I can actually delete that. don't want to mess with them :-)14:19
flaper87hahahaha, sounds good!14:19
flaper87I mena, you can keep the old one and just add OpenStack Foundation at the top14:19
flaper87I guess that's fine *shrugs*14:19
eranromflaper87: sounds right to me14:19
openstackgerritDaniele Pizzolli proposed openstack/storlets: account params can be changed in installation_vars.yml
openstackgerritDaniele Pizzolli proposed openstack/storlets: account params can be changed in installation_vars.yml
openstackgerritDaniele Pizzolli proposed openstack/storlets: account params can be changed in installation_vars.yml
openstackgerritMerged openstack/storlets: account params can be changed in installation_vars.yml
openstackgerritMerged openstack/storlets: Bump up ansible for linting
*** openstackgerrit has quit IRC18:03
*** openstackgerrit has joined #openstack-storlets18:04
openstackgerritMerged openstack/storlets: Ignore only BADFD in _safe_close

Generated by 2.14.0 by Marius Gedminas - find it at!