Tuesday, 2016-08-02

openstackgerritMerged openstack/storlets: Add missing main functions for unit test codes  https://review.openstack.org/34957000:05
openstackgerritKota Tsuyuzaki proposed openstack/storlets: Add unittests and cleanup codes  https://review.openstack.org/34661401:09
openstackgerritKota Tsuyuzaki proposed openstack/storlets: Add capability for crazy things  https://review.openstack.org/34725801:15
*** takashi has joined #openstack-storlets04:09
takashikota_, eranrom: I'm sorry but I'm not available for today's weekly meeting. (I'm not sure it the schedule change has already decided)04:10
kota_takashi: no worries04:11
takashikota_: I think I need to dicuss with you about the file_manager interface. I think can take some time tommorow, if you are available.04:11
kota_takashi: thanks04:12
kota_will do04:12
takashikota_: please check the comment I just added to that patch.04:12
*** takashi has quit IRC04:14
eranromtakashi: I have changed the schedule - thought its ok. Let me know if you wish to change it from Tue 0800 UTC.05:47
eranromOtherwise, no problem about today. Let us know if you have some review order preference.05:48
kota_sorry 8:00 UTC has passed07:16
kota_I thought it's in one hour later.07:17
eranromno problem. I though its in 40 mins :-)07:17
eranromShell we start?07:18
kota_eranrom: sure07:18
eranromok, agenda is here: https://wiki.openstack.org/wiki/Meetings/Storlets07:19
eranromlet me know if you have anything else.07:19
openstackgerritKota Tsuyuzaki proposed openstack/storlets: WIP: Items to consider for FileManager  https://review.openstack.org/34983407:20
kota_that one :-)07:20
eranromok. Let me have a look while we are talking.07:21
eranromAre we ok with creating a mitaka branch on top of master?07:21
eranromanything we want to land before this?07:21
kota_I think no blocker to do that07:21
eranromok great.07:21
eranromI read your correspondence with Takashi on the dead lock comment. I was not sure what conclusion you have came to and whether you still see a problem07:22
kota_at swift <-> container conversation?07:23
eranromIt was about the writing being done in a  saperate green thread07:24
kota_ah, yes, that was comming from my question.07:24
kota_so why it causes deadlock is my initial question but storlets seems not to ensure the data draining before ack to swift, it's answer I got.07:25
eranromWhen you say storlets not ensure, you mean the middleware or the part running in the docker container?07:26
kota_to switch the thread to wait the ack in swift while IO wait, the eventlet threading is needed there.07:26
kota_i meant docker container.07:26
kota_so storlet app may not to read() for all input stream, right?07:27
eranromright. This will cause the writing thread to get stuck at some point, and since we are writing with a timeout (I think) it should be ok. Am I missing something?07:28
kota_no? it fits my idea for now :-)07:28
kota_but I and takashi_ has thought for more improvement for this07:29
eranromok. listening :-)07:29
kota_if swift gateway got ack from container w/o draining, probably we should kill coroutines which writing to the fd to container.07:30
kota_because it will not be read :/07:30
kota_and possibly, we should ensure all eventlet for writing fds to the container finished up when returning storlet response to handler.07:31
kota_that is sort of our idea.07:32
kota_I didn't make a patch and figure out how to do that yet07:32
eranromLet me see if I get this right:07:33
eranromWe know that the storlet does not drain the input stream when we get a timeout in the thread writing to the storlet - right?07:34
eranromand this is where you want to add the cleanup code? (e.g. "ensure all eventlet for writing fds to the container finished up")07:35
eranromthat is got timeout -> run cleanup07:36
kota_maybe a part of my thought.07:36
kota_my scenario is the case storlet app doesn't drain the input stream intentionally.07:37
eranromok. I guess it would be best to wait for a patch. At this point I have one comment (perhaps I have it because I do not fully understand your thought):07:39
eranromYou are writing: "we should ensure all eventlet for writing fds to the container finished up when returning storlet response to handler."07:40
eranrommy concern here is that we will start responding to the user only after all the writing has been done. I mean the idea is to have stream processing which may be very long07:40
eranromand ideally we should be able to start streaming back the response, even before we consumed all the input...07:41
eranromdoes it make sense?07:41
kota_ah... ah07:41
kota_streaming processing need to write the input stream after got response07:42
kota_good perspective.07:42
kota_probably, my original idea conflicts with that case07:43
eranromThat was my understanding. BTW it can also cause swift to timeout if the processing of the whole data takes time...07:44
kota_ok, eranrom. that's obviously good input to consider the eventlet life-cycle. thanks.07:44
eranromok. so next topic?07:44
kota_go ahread07:44
eranromTaksahi has made a good point about the docker pull causing many gateway failures.07:45
eranromThe only solution I have in mind is keeping that image in the code...07:45
eranromthis is ~65MB when zipped.07:45
eranromperhaps we can keep this in another repo, and instead of docker pull do git checkout or wget....07:46
eranromassuming git checkout / wget are better then docker pull....07:47
kota_or can we make a cache (or testing VM image) in openstack-infra?07:47
kota_I'm not sure around the infra config but... something like dsvm...07:47
eranromsounds right!07:48
eranromSo we basically need to ask infra to add a testing vm image - right?07:48
eranromsounds like the right idea to me!07:48
kota_if it could be, we can make a vm image including the docker image, right?07:48
eranromthat is the ubuntu 14.04 image yes.07:49
eranromI will add your idea to the bug description so that we do not forget this.07:49
eranromok, so next topic.07:51
eranromreview prioritisation. Its been a very busy week and you (NTT) had a really impressive progress!07:52
eranromI tried to review quickly so as not to cause too much of a delay07:53
eranromany priorities?07:53
kota_right now, I think they are not urgent...07:54
eranromok, so I will review as I see fit. Do let me know if something changes.07:54
kota_recently I was working on a thing...07:54
patchbotkota_: patch 347258 - storlets - Add capability for crazy things07:55
kota_yeah, no docs sorry so that just drafting to make sure the capability07:55
eranromoh yes. I saw it but did not get a chance to review yet.07:55
kota_sort of multi input stream to storlet app container.07:55
kota_but I know it's hard to do that in object-server side so it's a toy right now07:56
kota_I need more consideration for the efficienccy and make sure if it's really right thing.07:56
eranromok, no problem. In fact we had this in the past (before we open sourced the project). This is the use case we had:07:56
kota_any others from me are all I found/want to fix while writing the code :P07:57
kota_oh, interesting on your use case07:57
eranrommedia files. Media files are very large and their metadata is often being kept in additional .xml files. So you have one huge file and then several more .xml metadata files.07:58
eranromThe requirement (which came at the time from RAI - Radio television Italy) was that the storlet gets access to all of them07:58
eranromso we implemented the ability to consume several streams07:58
eranromthe point was that the streams where not symmetric: one stream was from the swift url (this was the media file), and the other streams were specified in07:59
eranromheaders. Thus the storlet was invoked on the object server that held the media file, and there was not much data to move around...08:00
eranromBTW we also had the ability for the storlet to write more then one object, but we can discuss this some other time :-)08:01
kota_hew, that sounds great :-)08:02
eranromok,  https://review.openstack.org/349834?08:02
kota_oh, yea08:02
kota_I didn't to ask to takashi in detail yet but it's too conflicted with my multi inputs :/08:03
kota_it's ok to conflict though08:03
kota_wait, i mean08:03
kota_patch 34956608:04
patchbotkota_: https://review.openstack.org/#/c/349566/ - storlets - Validate file_manager option in DockerStorletRequest08:04
kota_that's a follow up08:04
kota_patch 34848208:04
patchbotkota_: https://review.openstack.org/#/c/348482/ - storlets - Create FileManager class (MERGED)08:04
kota_yes, thank you patchbot08:04
eranromoh, so I already merged something that conflicts with the multi-inputs08:05
kota_no no no08:05
kota_it's OK because due to the conflict, i can find more improvement for that.08:06
kota_cuircle back to patch 34982408:06
patchbotkota_: https://review.openstack.org/#/c/349824/ - python-senlinclient - Fix typo (MERGED)08:06
kota_back to there08:07
eranromok :-)08:07
eranromSo you want me to hold that one till you fix the conflicts in multi-stream?08:07
kota_I'd like to make the file management of get_storlet/get_dependency/put_log more general08:07
eranromok gotcha08:07
eranromand this is what 3349834 is about?08:08
kota_I don't have priority for them because I know they're conflict each other and I can resolve it.08:08
kota_in the patch 34983408:09
patchbotkota_: https://review.openstack.org/#/c/349834/ - storlets - WIP: Items to consider for FileManager08:09
eranromok, If I get you right 349834 is a pre-req to multi-streams08:09
eranromis that right?08:09
eranromok, so I will look at 349834 first.08:09
eranrommaybe I will look at both, but comment on 349834 first :-)08:10
eranromsure. There is one last thing Doron has reminded me today: I think its high time we try and make the project official (specifically with all recent activity)08:11
eranromI will send a mail to the list in the coming days...08:11
kota_for openstack official?08:12
kota_yey :-)08:12
eranromThat is all I have.08:12
kota_ok, thanks. glad to hear.08:13
eranromalright, so talk to you later08:13
eranromand thanks for joining08:14
eranromA reminder to vote for the storlet related talks :-)11:15
eranromsearch for "storlet" and "crystal"11:15
openstackgerritMerged openstack/storlets: Add unittests and cleanup codes  https://review.openstack.org/34661411:17
openstackgerritTakashi Kajinami proposed openstack/storlets: Refactor dispatch_command  https://review.openstack.org/34589212:39
openstackgerritTakashi Kajinami proposed openstack/storlets: Set os.environ parameters in subprocess  https://review.openstack.org/34959712:41
openstackgerritTakashi Kajinami proposed openstack/storlets: Fix usage of account inside gateway/runtime module  https://review.openstack.org/34960713:01
openstackgerritTakashi Kajinami proposed openstack/storlets: Refactor dispatch_command  https://review.openstack.org/34589213:43
openstackgerritTakashi Kajinami proposed openstack/storlets: Set os.environ parameters in subprocess  https://review.openstack.org/34959713:45
openstackgerritEran Rom proposed openstack/storlets: New storlet for testing pushdown  https://review.openstack.org/35012515:51
openstackgerritEran Rom proposed openstack/storlets: Storlets release 0.1.0  https://review.openstack.org/35015716:43
*** openstackgerrit_ has joined #openstack-storlets20:35
*** openstackgerrit_ has quit IRC20:36
*** notmyname has quit IRC20:55

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!