Wednesday, 2015-12-09

*** mattoliverau has quit IRC00:00
*** matt6434 has joined #openstack-storlets00:12
*** matt6434 has quit IRC00:17
*** matt6434 has joined #openstack-storlets00:25
*** matt6434 is now known as mattoliverau00:27
*** openstack has joined #openstack-storlets00:53
*** openstack has joined #openstack-storlets09:08
*** openstackgerrit has quit IRC09:32
*** openstackgerrit has joined #openstack-storlets09:33
*** openstackgerrit has quit IRC10:02
*** openstackgerrit has joined #openstack-storlets10:03
*** [1]eranrom has joined #openstack-storlets11:37
*** eranrom has quit IRC11:37
*** [1]eranrom is now known as eranrom11:37
*** eranrom has quit IRC11:42
*** takashi has joined #openstack-storlets12:02
*** takashi has quit IRC13:49
*** takashi has joined #openstack-storlets13:49
*** eranrom has joined #openstack-storlets13:50
kota_meeting will start in 3 minutes here, right?13:57
takashikota_: I suppose so13:59
eranromWho is on?13:59
takashiI am13:59
hrouhere !14:00
eranromDoron told me he can't make it.14:01
eranromI have updated the agenda:
eranromtopic: NTT Updates :-)14:02
eranromtakashi, stage is yours.14:02
takashieranrom: OK14:02
takashiAs I told eranrom before, we tested Storlet last month.14:03
takashiWe tested Storlets with Swift Liberty(2.3.0).14:04
kota_liberty is 2.5.0, fyi14:04
takashikota_: Thanks, 2.5.014:04
hroutakashi !  That's great did you run into some failures ? (expected as storlets on the object server do not work as of 2.4 or what not)14:05
takashihrou: yes, we have to fix proxy-server to run storlet.14:06
takashiAfter fixing that, I could run storlets with Liberty.14:06
takashiI'm going to commit that fix of proxy-server soon.14:06
hroutakashi, this fix allows storlets to run on the object-server ?14:06
kota_hrou: i think so14:07
takashihrou: Yes14:07
kota_to get start to catch up of storlets status14:07
hrouOh wow !  Very curious to see that fix.14:07
kota_at fist, we ran that in swift-all-in-one14:07
takashiusing scripts included in storlet repository14:08
kota_and, after that, he set up cluseter environment14:08
takashiwe are now running environment with 2 proxy-servers, 3 storage-servers, and 1 keystone14:08
kota_for running on the cluster environment, he have to fix some issues14:08
kota_s/have to/had to/14:09
eranromThis is great!14:09
kota_and with his fixes, currently storlets seems to work well completely14:09
kota_that's awesome.14:09
takashikota_: thank you for your following up14:09
hrou+100 :)14:10
eranromDo you want to elaborate on the fixes or just sent pathces?14:10
takashiI think I can send the patch soon. I already finished creating some tests for that.14:11
eranromCool! Looking forward14:11
hrouYea very interested in seeing the swift patch too !14:12
eranromCan you tell us a little on your demo app?14:12
kota_and now, we are working for listing what we can be fixed and what can be improved in future work.14:13
kota_eranrom: maybe, takashi?14:13
takashieranrom, kota_: I created some image processing app, like feature detection in images.14:13
takashiWe think image processing is a good starting point, and planning some PoC using that kind of apps14:15
eranromtakashi: I assume you are invoking the Storlet on GET - right?14:16
takashieranrom: Yes14:16
takashiWe store raw images in Swift, and invoke storlet on GET, and get a result of processing.14:17
eranromtakashi: gotcha. Just out of curiousity, I assume you need some non trivial dependencies are these part of the Storlet or are you using the dependency mechanism or deploy a Docker image that has it?14:18
eranromI mean the image processing engine14:18
takashieranrom: We put the engine in the Docker container image. We are now using OpenCV, which needs so many files that it is very difficult to add them using dependency mechanism...14:19
eranromtakashi: This is great, did you use the ansible 'managememnt' scripts we have?14:20
cdoronhi there. Sorry I'm late14:21
takashieranrom: Yes, I used them.14:21
eranromcdoron: Hi!14:22
hrouhey cdoron !14:22
eranromtakashi: Now I am even more excited! Did you run into issues there as well (I assume yes :-)14:22
takashieranrom: I didn't met so big issues. I think the biggest problem was the one about ssh key management. It worked well for me.14:23
eranromYEY!!! :-)14:24
hrouThat's really great to hear, I think the scripts are much improved now from the early days :)14:24
eranromSo I am really lokking forward to the patches and your ideas for future work!14:24
hrouYep the SSH key management is usually the key14:24
takashihrou: definitly14:25
takashieranrom: So, I'm going to commit patch for swift, which we need to run Storlet,14:26
kota_on master, right?14:26
takashiand also going to commit some fixes or improvements about Storlet after creating a list14:26
takashikota_: Yes, on master14:26
eranromtakashi, That's great. BTW the Swift patch is it big?14:26
takashieranrom: It's not so big14:26
hrouYou mentioned proxy server - is it contained to that ?14:27
takashihrou: yes14:27
cdorontakashi: a while ago you mentioned unit-tests. Which components are you interested in testing?14:27
hroutakashi, great !  I did talk to Sam about it back in Austin and he said as long as it wasn't huge the community may be Ok with it;  It was the EC refactoring that introduced the issue ;)14:28
takashicdoron: storlet_handler middleware14:28
eranromdo we have more updates or should we move to the next topic?14:30
eranromwhich has to do with basic administrations?14:30
takashieranrom: I think that' all.14:30
eranromtakshi: Thanks a lot!!! You have made my day :-)14:31
eranromso the question I wanted to ask has to do with how to manage feature design.14:31
eranromFor example, I really want to start working on COPY (which kota_ asked about in the summit)14:32
hrouWell and what features would be best to focus on : )14:32
hroueranrom, kota_ can you have a use case you envision for copy ?14:32
eranromhrou: Generate thumbnail of an existing photo14:33
kota_i thinks we can manage some processing applicatiton with some times...14:33
kota_like get some results and store it into swift14:33
hroueranrom, and in this scenario you want the thumbnail to persist in object storage, otherwise you could do this on get but its a lot more processing14:33
kota_afiter that, reuse the result in another apps.14:34
kota_hrou: yup, like that.14:34
kota_that's an use case, i thought.14:34
hrouNeat yea, I  could see that use case.14:35
eranromkota_: Right, I think that this generalizes to the ability to do things more "asynchronously."14:35
hroueranrom, which is interesting; akin to Amazon Lambda14:35
eranromhrou,: sort of.14:35
hrouWell someone still needs to issue that copy (whether it be middleware or ..) but ..14:35
eranromright, but that someone does not need to wait for the response and do something with it.14:36
eranromas the 'response' goes to the srtore14:36
eranromwell perhaps "asynchronously" is not the right way to describe14:37
hrouYea I think it'll cause some confusion, but I see why you call it that :) In comparison to the get/put codepath we generalyl describe.14:37
kota_i think, we can start for the usecase with synchonously14:38
kota_becaouse it may emproy asyncronous process in out of swift.14:38
kota_a tons of typos... sorry :/14:38
eranromhrou, kota_: Agree. I should not have used async here :-)14:39
eranromSo anyway do we want blueprints - which are already there or do something like Swift with a dedicated Git repo for design docs?14:39
eranromkota_: at 11PM my time I have more14:39
hroueranrom, yea that's a good question14:39
eranromkota_: That is typoes :-)14:40
hrouit seems like swift folks are kinda moving away from swiftspec ? Or only for the very large projects it seems14:40
kota_no idea, either seems to work for us :)14:40
hrouhehe, yea I kinda like swiftspecs to be honest14:40
hrouFind its useful to colleberate on a high level design that way.14:40
eranromDo you think its a headache to set it up?14:40
eranromdo we need openstack infra guys for that?14:41
eranromI mean something like swiftspec14:41
hrouMaybe just to register the project yea but its probably pretty minor, really its just any REPO but just with docs14:41
hrouWith blueprints ?  You can't really comment inline can you ?14:41
hrouIts more like you just add comments like with a bugreport or what not.14:42
eranromhrou,: Honestly, I do not know.14:42
eranromSo how about starting with an 'unregistered' repo14:42
hrouYep I think that works well too;14:42
kota_where do you think to set that?14:42
hroukota_, what does swift3 do ?14:42
kota_ah... no official docs in future work, right now :P14:43
eranromkota_,: I think IBM has a Github 'namespace' we can put it there for a start.14:43
kota_just a talk with IRC and some online conversation...14:44
hroukota_, haha :) Gotcha!  Yea probably just IRC conversations are generally sufficent.14:44
kota_it has both pros and cons.14:44
kota_i sometimes am asked 'what's going on?' from some users14:45
kota_that's cons :/14:45
eranromI can buy starting off with IRC14:46
kota_but yeah, in flight conversation is so quick to decide some plan for future work.14:46
eranromwell, we do have a log (thanks to hrou)14:46
hrouFor anything involved design wise I could see it be a bit nice to have doc we can comment on14:47
kota_hrou: agree14:47
eranromok so lets do both. If someone is interested in a doc, (s)he can put it in a repo we will setup14:47
hrouSo eranrom I think we could try the repo route if it isn't too bad, lets go ahead with that, if we run into issues then we can do just IRC for now14:47
hroueranrom, yep exactly :014:48
eranromok then14:48
eranromnext topic?14:48
eranromBug / Feature prioritization14:48
eranromso the first item there is what takashi has a fix for. Yey!14:49
eranromWe have discussed copy14:49
eranromand there is the list that Takashi and Kota will send14:49
eranromanything else at this point?14:50
takashiI have nothing more this time.14:50
eranromSo shell we end our first IRC meeting with the famous words?14:51
eranromThank you for working on Storlets!14:51
hrouThanks all !14:52
hrouWe can use the wiki page for the agenda going forward if anyone wants to discuss something as well14:52
kota_thanks eranrom for organize this :-)14:52
eranromThank you!14:53
takashia small thing14:55
takashiI updated my Storlet many times in my multi-cluster env, but faced no updating problem.14:55
takashieranrom: I think we have to examine more about the situation to cause this problem.
openstackLaunchpad bug 1495600 in storlets "Problem in Storlet cache / container updates" [Undecided,New]14:57
eranromtakashi, ok thanks! Having a second look at it.14:59
takashieranrom: I'm going to check my logs about that tommorow.15:00
eranromtakashi: ok thanks!15:00
takashinow "today" in JST15:00
hrouActually that's an interesting one;  I recall running into that in the early days and as a result had to delete the cache (i.e. the directory containing) to trigger it to update (by fetching it again)15:01
hrouI'll check15:01
eranromOne possibility that comes in to mind is that the same object was always used so somehow the Storlet was executed on the same node all the time, and there might have been other nodes where the Storlet did not get updated.15:02
eranromI agree that there is a small chance for that though15:02
hrouDo we use x-newest on that ?15:02
eranromI do not think so15:03
eranromyou mean when copying the Storlet from Swift to the node where it runs>?15:03
hrouWell that's just a GET request right using the IC15:04
takashihrou: I found that code now.15:07
eranromright. resp = client.make_request('GET', path, {'PATH_INFO': path}, [200])15:08
eranromno x-newest15:08
hrouGreat takashi, eranrom !  Yea so maybe x-newest would help a bit15:08
hrouI mean given this is swift, this is all really ok : ) expected in an eventually consistent system15:08
eranromI quoted the wrong code15:13
eranromSo we actually HEAD the storlet object to make sure the user has read access to it - this is essentially how we enforce storlet execution:15:15
eranromuser has read access to storlet container == user can invoke it15:15
eranromAnyway that request does not have the x-newest15:16
eranromwhich is definately a problem15:16
eranromgood catch!15:16
hrouI think after the head we compared the timestamp and make the decision yea.15:17
eranromLet me update the but with this thought.15:17
eranromor do you want to?15:18
hrouSure !  I can fix that, but again it's not a bug per say : )  But we may as well15:19
hrouYou had a thought though ?15:19
eranromI did not mean to actually go fix that now, just update the bug in launchpad15:20
hrouOh !! :)  Sure will do15:21
eranromeven if this is not a bug, it did affect the development work. I know yosef has spent a lot of time chasing why the Storlet is not updated.15:21
eranromok, gotta go. ttyl15:21
hrouSure thing, thanks eranrom will do !15:22
eranromthank you!15:22
*** takashi has quit IRC15:24
*** eranrom has quit IRC15:26
*** eranrom has joined #openstack-storlets19:20
*** eranrom has quit IRC19:32

Generated by 2.14.0 by Marius Gedminas - find it at!