| *** eranrom has joined #openstack-storlets | 01:15 | |
| *** eranrom has quit IRC | 01:19 | |
| openstackgerrit | Takashi Kajinami proposed openstack/storlets: Implement missing methods in client datagram https://review.openstack.org/352513 | 05:04 |
|---|---|---|
| openstackgerrit | Takashi Kajinami proposed openstack/storlets: Add metadata validation for python application https://review.openstack.org/356202 | 05:04 |
| openstackgerrit | Takashi Kajinami proposed openstack/storlets: Add Python sdaemon support in daemon_factory https://review.openstack.org/356224 | 05:04 |
| openstackgerrit | Takashi Kajinami proposed openstack/storlets: Implement Python SDaemon https://review.openstack.org/358252 | 05:04 |
| *** eranrom has joined #openstack-storlets | 05:16 | |
| *** takashi has joined #openstack-storlets | 05:17 | |
| *** eranrom has quit IRC | 05:21 | |
| openstackgerrit | Takashi Kajinami proposed openstack/storlets: Implement missing methods in client datagram https://review.openstack.org/352513 | 06:44 |
| openstackgerrit | Takashi Kajinami proposed openstack/storlets: Add metadata validation for python application https://review.openstack.org/356202 | 06:44 |
| openstackgerrit | Takashi Kajinami proposed openstack/storlets: Add Python sdaemon support in daemon_factory https://review.openstack.org/356224 | 06:44 |
| openstackgerrit | Takashi Kajinami proposed openstack/storlets: Implement Python SDaemon https://review.openstack.org/358252 | 06:44 |
| *** eranrom has joined #openstack-storlets | 06:51 | |
| eranrom | Hi | 08:00 |
| takashi | Hi | 08:01 |
| eranrom | Do you know if kota_ plans on joining? | 08:01 |
| takashi | eranrom: I didn't here his plan. | 08:02 |
| takashi | s/here/hear | 08:02 |
| eranrom | ok, shell we start? | 08:02 |
| takashi | eranrom: ok | 08:02 |
| eranrom | So today the tc will continue the discussion on storlets. | 08:03 |
| eranrom | I think that overall the feedback was positive | 08:03 |
| takashi | eranrom: Sorry for my absence from the last meeting :-( | 08:03 |
| eranrom | no problem. | 08:03 |
| *** openstackgerrit has quit IRC | 08:03 | |
| takashi | eranrom: I saw your proposal is now in formal vote state | 08:04 |
| eranrom | comments? | 08:04 |
| eranrom | oh, where did you see this? | 08:04 |
| *** openstackgerrit has joined #openstack-storlets | 08:04 | |
| takashi | patch 353693 | 08:04 |
| eranrom | ok, gotcha. | 08:05 |
| takashi | ttx changed its topic to formal-vote | 08:05 |
| eranrom | I see. Funny I did not see this. Do you know when? | 08:05 |
| eranrom | I mean when did he change this? | 08:05 |
| takashi | I don't know so much about its meaning, but it seems now open for voting, from foundation members. | 08:06 |
| takashi | eranrom: Aug 17 6:49 PM, as I find in the gerrit log | 08:06 |
| eranrom | ok saw this :-) | 08:06 |
| eranrom | well, so far no votes... | 08:06 |
| takashi | I think we still need some more discussion, but it seems that currently we do not get any big objection about becoming official | 08:07 |
| takashi | great! | 08:07 |
| eranrom | right. | 08:07 |
| takashi | As ttx mentioned in his comment, we need to find the way how we can get rid of symlinks in our repository | 08:08 |
| eranrom | yes, I need to look into this. | 08:08 |
| eranrom | I a not sure hat it is used for... | 08:08 |
| takashi | I don't think it is a blocker for becoming official project, but should be addressed in not so far future. | 08:08 |
| takashi | eranrom: Currently it is used for tox testing | 08:08 |
| takashi | Currently, we have two setup.py. One for tox testing, and the other for actual deployment, executed by ansible. | 08:09 |
| eranrom | I see, do you think that having something like setup.py develop is the way to tackle this? | 08:09 |
| eranrom | ok, I clearly need to look deeper into this. | 08:10 |
| takashi | eranrom: I recently started thinking about this point, but found some problems | 08:10 |
| eranrom | I see. | 08:10 |
| takashi | Some thoughts here ... | 08:11 |
| takashi | 1. As a python project, it is desireble to have one setup.py, as swift does, for example | 08:11 |
| takashi | 2. However, we have to install host package(storlet_gateway, storlet_middleware) and agent package(storlet_daemon_factory, storlet_daemon) | 08:12 |
| takashi | we install the first one on the host level, while we install the other one in docker container | 08:12 |
| takashi | 3. I talked with oslo team, to add an option into setup.py, to decide the package to be installed, but they said they don't support that and recommended me to have a separate git repository. | 08:13 |
| takashi | In openstack projects, we create setup.py using pbr, which is maintained by oslo team | 08:14 |
| eranrom | I see. Is this also true if we do the change we wanted: Have the agent code installed on the host and 'mount' it to the container? | 08:15 |
| kota_ | oops | 08:15 |
| eranrom | Hi Kota | 08:15 |
| kota_ | sorry, being late | 08:15 |
| takashi | kota_: hi | 08:15 |
| eranrom | np :-) | 08:15 |
| kota_ | hi, all | 08:15 |
| takashi | eranrom: If we install storlets code only in host layer, we can merge today's multiple packages into one | 08:16 |
| eranrom | we are discussing the symlinks and implications (which was raised by ttx) | 08:16 |
| eranrom | takashi: Does this limit the location where we install. I mean, I envision all agent code (both python and Java) installed in the same dir, and we mount it read only | 08:17 |
| eranrom | to e.g. /opt/storlets in the container | 08:17 |
| kota_ | in the repo, reading the back log. | 08:17 |
| kota_ | ok | 08:17 |
| takashi | eranrom: one possible way is adding them not as code, but as resource | 08:17 |
| takashi | resource is not a right word... I mean something like data file related to the package | 08:18 |
| eranrom | takashi: I see. so code is something that always goes to something like /usr/local/lib/python2.7/dist-packages? | 08:19 |
| eranrom | takashi: this is a setup.py thing right? | 08:19 |
| takashi | eranrom: Yes, as far as I understand | 08:19 |
| takashi | virtualenv can be a another possible solution, but I still need some more time to confirm it... | 08:20 |
| eranrom | takashi: ok, so is seems that the first step is anyway, change the storlet image build not no include agent code, and make that mount. | 08:20 |
| eranrom | we can adjust setup.py as a follow up | 08:21 |
| eranrom | takashi: ok | 08:21 |
| takashi | eranrom: I'll create a list of possible options we can take. Currently I'm not sure there are realy no other solutions. | 08:22 |
| openstackgerrit | Merged openstack/storlets: Create gateway load function under storlet_gateway https://review.openstack.org/357599 | 08:22 |
| eranrom | takashi: thanks! Do you agree that as a first step we can take the agent code out of the docker image? | 08:22 |
| eranrom | takashi: which is something we think is good on its own... | 08:23 |
| eranrom | takashi: BTW - I will be happy to do this. | 08:23 |
| takashi | eranrom: yes. but IMO that requires some long time, and want to think if there are any short-term solutions. | 08:24 |
| eranrom | takashi: ok, so will wait for your comments. thanks | 08:24 |
| takashi | eranrom: :-) | 08:24 |
| eranrom | I have another update, is it a good time? | 08:25 |
| takashi | eranrom: yes, for me. | 08:25 |
| eranrom | it is not long. We got a 65 servers cluster from OSIC to do spark pushdown performance tests. We installed a 6 proxies with 30 object nodes Swift cluster, and 25 more machines for Spark | 08:26 |
| eranrom | and we are going to compare running spark workloads with and without storlets... | 08:26 |
| takashi | eranrom: great! | 08:27 |
| kota_ | sounds nice | 08:27 |
| eranrom | we are quite excited about it :-) | 08:27 |
| eranrom | lets wait for the results though... | 08:27 |
| kota_ | wow, OSIC is trying to use storlets? | 08:27 |
| kota_ | i'm assuming OSIC is OpenStack Innovation Center wiht Intel/Rackspace | 08:28 |
| kota_ | correct me if I'm wrong. | 08:28 |
| eranrom | right, OSIC have a program to lend servers to the community. | 08:28 |
| kota_ | hew | 08:28 |
| eranrom | so its not that ther are using it. | 08:28 |
| eranrom | we made a request ti get resources and they got back with a positive answer | 08:29 |
| eranrom | so we got it for 3 weeks | 08:29 |
| eranrom | were you worried about them using storlets? :-) | 08:29 |
| takashi | eranrom: OSIC provides machine resources for OSS commmunity activity in OpenStack projects, right? | 08:30 |
| kota_ | i was not worried ;-) | 08:30 |
| eranrom | takashi: right. | 08:30 |
| kota_ | just interested | 08:30 |
| eranrom | kota_: I see. I was wondering about your "hew" :-) | 08:30 |
| kota_ | typo :/ | 08:31 |
| eranrom | np | 08:31 |
| takashi | eranrom: Are you going to make your testing result open? | 08:31 |
| eranrom | any other topics? takashithanks for working on the bugs. | 08:31 |
| eranrom | takashi: sure. The idea is to write a paper to a conference called "middleware" | 08:32 |
| takashi | eranrom: sounds great | 08:32 |
| eranrom | I will share the results, once we have something robust. | 08:32 |
| takashi | eranrom: looking forward to seeing that! | 08:33 |
| takashi | some updates from my side, about python application support | 08:33 |
| takashi | First, I filed a blueprint for that activity, as we can find easily related patches. https://blueprints.launchpad.net/storlets/+spec/python-storlet | 08:34 |
| eranrom | great | 08:34 |
| takashi | eranrom: Can you approve that blueprint? | 08:35 |
| eranrom | takashi: sure. | 08:35 |
| takashi | I mean, can you set its definition as 'approved' | 08:35 |
| takashi | I don't think it's so critial, but afaik it is the way how we do in openstack projects. | 08:36 |
| eranrom | Looking for how to do that | 08:36 |
| eranrom | done | 08:36 |
| takashi | eranrom: Thanks | 08:36 |
| eranrom | sure | 08:37 |
| takashi | I already submitted some patches to gerrit. It is still under testing, and I need to complete some todos (application loader, func tests and so on). | 08:37 |
| eranrom | Do you want me to look at anything yet? | 08:38 |
| eranrom | Perhaps start with SDaemon | 08:38 |
| eranrom | ? | 08:38 |
| takashi | You can find the patch order in the description field of the blueprint | 08:38 |
| takashi | dependency chain is like this: middleware -> sbus -> sdaemon -> daemon_factory | 08:39 |
| eranrom | great.thanks. | 08:40 |
| takashi | eranrom: I'm very glad if I can get any feedback from you. | 08:40 |
| eranrom | takashi: sure, will start today. | 08:40 |
| eranrom | that is later today :-) | 08:40 |
| takashi | Currently I don't have so special problems. I'm currently thinking what is the best way to load python module dynamically. | 08:40 |
| takashi | That is why I submitted one patch to use stevedore. | 08:41 |
| eranrom | ok, I am not familiar with stevedore. Will have a look | 08:41 |
| takashi | eranrom: it is used in other openstack projects which has driver mechanism (keystone, cinder or so on), but currently I'm not sure it is the best way. | 08:42 |
| eranrom | ok. I see. | 08:42 |
| takashi | that's almost all about python support. | 08:43 |
| eranrom | wow, so most of the required code is already there? | 08:43 |
| takashi | eranrom: excluding some todo items | 08:44 |
| eranrom | cool! | 08:44 |
| eranrom | not to say super-cool :-) | 08:44 |
| takashi | I'm currently creating my test environment to run that codes, so I need some more debugging... | 08:44 |
| eranrom | sure | 08:44 |
| takashi | but framework is there | 08:45 |
| eranrom | In the meantime I will look at the patches. | 08:45 |
| takashi | eranrom: thx! | 08:45 |
| eranrom | anything else for today? | 08:45 |
| takashi | for the another small thing, I'm working about sdaemon refactoring and unit testing | 08:45 |
| eranrom | takashi: the Java one right? | 08:46 |
| takashi | eranrom: sorry. about daemon_factory, indeed | 08:46 |
| eranrom | takashi: right. the kill_all stuff? | 08:46 |
| takashi | it does not have dependency with python support, but I think I can be a help when I work about refactoring, after I finish python storlet work | 08:46 |
| takashi | eranrom: Yes | 08:46 |
| takashi | kill functions, shutdown functions, status functions, and starting functions | 08:47 |
| eranrom | takashi: ok, I reviewed some of it earlier. So sure, lets land them first and then go to the python app support stuff. | 08:47 |
| takashi | That's all from my side today | 08:47 |
| eranrom | takashi: thanks! | 08:48 |
| kota_ | takashi: sorry, what does it meen "functions" you use | 08:48 |
| takashi | kota_: functions in daemon_factory | 08:48 |
| kota_ | ok | 08:48 |
| takashi | I'm currently thinking to cut off the common module, which can be used by both of daemon_factory and python sdaemon | 08:49 |
| takashi | after I land python sdaemon | 08:49 |
| takashi | and refactoring/unit testing is the entry point of that work | 08:50 |
| kota_ | k | 08:51 |
| takashi | eranrom: thanks. I'll update you if I finish testing. At that point we can re-consider our landing order | 08:51 |
| eranrom | takashi: ok. Trying to understand your remark on common | 08:51 |
| eranrom | Do you mean SCommon? | 08:52 |
| takashi | eranrom: It is different from SCommon, IMO | 08:52 |
| takashi | eranrom: Both of daemon_factory adn sdaemon is a callback daemon which listen on sbus interface. | 08:52 |
| eranrom | takashi: gotcha. | 08:52 |
| takashi | At that point, we can create a common framework, like we do for http server, etc. | 08:53 |
| eranrom | So you want to have a new common infra for both. | 08:53 |
| takashi | eranrom: yes. something like agent_common? | 08:53 |
| eranrom | takashi: sounds great! | 08:53 |
| takashi | eranrom: :-) | 08:54 |
| kota_ | takashi: does it still using sbus protocol? or new protocol? | 08:54 |
| takashi | kota_: it is still using sbus protocol. I didn't make change about protocols between processes. | 08:54 |
| kota_ | on your idea | 08:54 |
| kota_ | ok, gotcha | 08:55 |
| eranrom | Anything else for today? | 08:56 |
| kota_ | ah, just saying thank you for reviews eranrom, takashi, for my patch. | 08:56 |
| kota_ | patches | 08:56 |
| kota_ | sorry, I didn't track for them yet though. | 08:57 |
| eranrom | kota_: sure, I like the feature and curios regarding the use case :-) | 08:57 |
| eranrom | kota_: np | 08:57 |
| takashi | kota_: One thing I missed when I added my comment. I think you need to rebase the patch against patch 349348 | 08:57 |
| kota_ | hopefully, I like to disclose the use case if i can get approved with my boss :P | 08:57 |
| eranrom | kota_: no worries. We can land it even if you cannot disclose. I am just curious | 08:58 |
| kota_ | patch bot has died :/ | 08:58 |
| eranrom | yeah | 08:59 |
| takashi | need to paste whole of patch url... | 08:59 |
| takashi | kota_: https://review.openstack.org/#/c/349348/ | 09:00 |
| takashi | this one | 09:00 |
| kota_ | thx | 09:00 |
| takashi | Recently some changes about sbus is merged. Please ask me if you find any questions/improvoments around it. | 09:00 |
| *** eranrom has quit IRC | 09:00 | |
| *** eranrom has joined #openstack-storlets | 09:01 | |
| eranrom | Sorry, need to go. August and kids at home :-) | 09:01 |
| eranrom | Talk to you later. | 09:02 |
| kota_ | eranrom: np, enjoy! | 09:02 |
| eranrom | kota_: Thanks | 09:02 |
| eranrom | thanks all for joining | 09:02 |
| *** eranrom has quit IRC | 09:03 | |
| takashi | eranrom: thanks. have a nice day, with your family. | 09:03 |
| *** eranrom has joined #openstack-storlets | 13:49 | |
| *** eranrom has quit IRC | 13:49 | |
| *** eranrom has joined #openstack-storlets | 13:50 | |
| *** eranrom has quit IRC | 13:51 | |
| *** eranrom has joined #openstack-storlets | 13:52 | |
| *** eranrom has quit IRC | 13:52 | |
| *** eranrom has joined #openstack-storlets | 13:56 | |
| *** eranrom has quit IRC | 14:02 | |
| *** eranrom has joined #openstack-storlets | 14:02 | |
| *** eranrom has quit IRC | 14:03 | |
| *** eranrom has joined #openstack-storlets | 14:03 | |
| *** eranrom has quit IRC | 14:10 | |
| *** eranrom has joined #openstack-storlets | 14:11 | |
| *** takashi has quit IRC | 14:13 | |
| kota_ | good morning | 19:58 |
| kota_ | waking up for tc meeting, I'm not sure if i can do follow-up it though. | 19:59 |
| kota_ | oh? no meeting today for tc? | 20:03 |
| kota_ | oh, my bad, that is scheduled at Aug 30th so i can be back to bed... | 20:04 |
| eranrom | wow I would have waited forever. | 20:05 |
| eranrom | good night | 20:05 |
| eranrom | going to bed as well. | 20:05 |
| *** eranrom has quit IRC | 21:47 | |
| *** eranrom has joined #openstack-storlets | 21:48 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!