Tuesday, 2016-08-23

*** eranrom has joined #openstack-storlets01:15
*** eranrom has quit IRC01:19
openstackgerritTakashi Kajinami proposed openstack/storlets: Implement missing methods in client datagram  https://review.openstack.org/35251305:04
openstackgerritTakashi Kajinami proposed openstack/storlets: Add metadata validation for python application  https://review.openstack.org/35620205:04
openstackgerritTakashi Kajinami proposed openstack/storlets: Add Python sdaemon support in daemon_factory  https://review.openstack.org/35622405:04
openstackgerritTakashi Kajinami proposed openstack/storlets: Implement Python SDaemon  https://review.openstack.org/35825205:04
*** eranrom has joined #openstack-storlets05:16
*** takashi has joined #openstack-storlets05:17
*** eranrom has quit IRC05:21
openstackgerritTakashi Kajinami proposed openstack/storlets: Implement missing methods in client datagram  https://review.openstack.org/35251306:44
openstackgerritTakashi Kajinami proposed openstack/storlets: Add metadata validation for python application  https://review.openstack.org/35620206:44
openstackgerritTakashi Kajinami proposed openstack/storlets: Add Python sdaemon support in daemon_factory  https://review.openstack.org/35622406:44
openstackgerritTakashi Kajinami proposed openstack/storlets: Implement Python SDaemon  https://review.openstack.org/35825206:44
*** eranrom has joined #openstack-storlets06:51
eranromDo you know if kota_ plans on joining?08:01
takashieranrom: I didn't here his plan.08:02
eranromok, shell we start?08:02
takashieranrom: ok08:02
eranromSo today the tc will continue the discussion on storlets.08:03
eranromI think that overall the feedback was positive08:03
takashieranrom: Sorry for my absence from the last meeting :-(08:03
eranromno problem.08:03
*** openstackgerrit has quit IRC08:03
takashieranrom: I saw your proposal is now in formal vote state08:04
eranromoh, where did you see this?08:04
*** openstackgerrit has joined #openstack-storlets08:04
takashipatch 35369308:04
eranromok, gotcha.08:05
takashittx changed its topic to formal-vote08:05
eranromI see. Funny I did not see this. Do you know when?08:05
eranromI mean when did he change this?08:05
takashiI don't know so much about its meaning, but it seems now open for voting, from foundation members.08:06
takashieranrom: Aug 17 6:49 PM, as I find in the gerrit log08:06
eranromok saw this :-)08:06
eranromwell, so far no votes...08:06
takashiI think we still need some more discussion, but it seems that currently we do not get any big objection about becoming official08:07
takashiAs ttx mentioned in his comment, we need to find the way how we can get rid of symlinks in our repository08:08
eranromyes, I need to look into this.08:08
eranromI a not sure hat it is used for...08:08
takashiI don't think it is a blocker for becoming official project, but should be addressed in not so far future.08:08
takashieranrom: Currently it is used for tox testing08:08
takashiCurrently, we have two setup.py. One for tox testing, and the other for actual deployment, executed by ansible.08:09
eranromI see, do you think that having something like setup.py develop is the way to tackle this?08:09
eranromok, I clearly need to look deeper into this.08:10
takashieranrom: I recently started thinking about this point, but found some problems08:10
eranromI see.08:10
takashiSome thoughts here ...08:11
takashi1. As a python project, it is desireble to have one setup.py, as swift does, for example08:11
takashi2. However, we have to install host package(storlet_gateway, storlet_middleware) and agent package(storlet_daemon_factory, storlet_daemon)08:12
takashiwe install the first one on the host level, while we install the other one in docker container08:12
takashi3. I talked with oslo team, to add an option into setup.py, to decide the package to be installed, but they said they don't support that and recommended me to have a separate git repository.08:13
takashiIn openstack projects, we create setup.py using pbr, which is maintained by oslo team08:14
eranromI see. Is this also true if we do the change we wanted: Have the agent code installed on the host and 'mount' it to the container?08:15
eranromHi Kota08:15
kota_sorry, being late08:15
takashikota_: hi08:15
eranromnp :-)08:15
kota_hi, all08:15
takashieranrom: If we install storlets code only in host layer, we can merge today's multiple packages into one08:16
eranromwe are discussing the symlinks and implications (which was raised by ttx)08:16
eranromtakashi: Does this limit the location where we install. I mean, I envision all agent code (both python and Java) installed in the same dir, and we mount it read only08:17
eranromto e.g. /opt/storlets in the container08:17
kota_in the repo, reading the back log.08:17
takashieranrom: one possible way is adding them not as code, but as resource08:17
takashiresource is not a right word... I mean something like data file related to the package08:18
eranromtakashi: I see. so code is something that always goes to something like /usr/local/lib/python2.7/dist-packages?08:19
eranromtakashi: this is a setup.py thing right?08:19
takashieranrom: Yes, as far as I understand08:19
takashivirtualenv can be a another possible solution, but I still need some more time to confirm it...08:20
eranromtakashi: ok, so is seems that the first step is anyway, change the storlet image build not no include agent code, and make that mount.08:20
eranromwe can adjust setup.py as a follow up08:21
eranromtakashi: ok08:21
takashieranrom: I'll create a list of possible options we can take. Currently I'm not sure there are realy no other solutions.08:22
openstackgerritMerged openstack/storlets: Create gateway load function under storlet_gateway  https://review.openstack.org/35759908:22
eranromtakashi: thanks! Do you agree that as a first step we can take the agent code out of the docker image?08:22
eranromtakashi: which is something we think is good on its own...08:23
eranromtakashi: BTW - I will be happy to do this.08:23
takashieranrom: yes. but IMO that requires some long time, and want to think if there are any short-term solutions.08:24
eranromtakashi: ok, so will wait for your comments. thanks08:24
takashieranrom: :-)08:24
eranromI have another update, is it a good time?08:25
takashieranrom: yes, for me.08:25
eranromit is not long. We got a 65 servers cluster from OSIC to do spark pushdown performance tests. We installed a 6 proxies with 30 object nodes Swift cluster, and 25 more machines for Spark08:26
eranromand we are going to compare running spark workloads with and without storlets...08:26
takashieranrom: great!08:27
kota_sounds nice08:27
eranromwe are quite excited about it :-)08:27
eranromlets wait for the results though...08:27
kota_wow, OSIC is trying to use storlets?08:27
kota_i'm assuming OSIC is OpenStack Innovation Center wiht Intel/Rackspace08:28
kota_correct me if I'm wrong.08:28
eranromright, OSIC have a program to lend servers to the community.08:28
eranromso its not that ther are using it.08:28
eranromwe made a request ti get resources and they got back with a positive answer08:29
eranromso we got it for 3 weeks08:29
eranromwere you worried about them using storlets? :-)08:29
takashieranrom: OSIC provides machine resources for OSS commmunity activity in OpenStack projects, right?08:30
kota_i was not worried ;-)08:30
eranromtakashi: right.08:30
kota_just interested08:30
eranromkota_: I see. I was wondering about your "hew" :-)08:30
kota_typo :/08:31
takashieranrom: Are you going to make your testing result open?08:31
eranromany other topics? takashithanks for working on the bugs.08:31
eranromtakashi: sure. The idea is to write a paper to a conference called "middleware"08:32
takashieranrom: sounds great08:32
eranromI will share the results, once we have something robust.08:32
takashieranrom: looking forward to seeing that!08:33
takashisome updates from my side, about python application support08:33
takashiFirst, I filed a blueprint for that activity, as we can find easily related patches. https://blueprints.launchpad.net/storlets/+spec/python-storlet08:34
takashieranrom: Can you approve that blueprint?08:35
eranromtakashi: sure.08:35
takashiI mean, can you set its definition as 'approved'08:35
takashiI don't think it's so critial, but afaik it is the way how we do in openstack projects.08:36
eranromLooking for how to do that08:36
takashieranrom: Thanks08:36
takashiI already submitted some patches to gerrit. It is still under testing, and I need to complete some todos (application loader, func tests and so on).08:37
eranromDo you want me to look at anything yet?08:38
eranromPerhaps start with SDaemon08:38
takashiYou can find the patch order in the description field of the blueprint08:38
takashidependency chain is like this: middleware -> sbus -> sdaemon -> daemon_factory08:39
takashieranrom: I'm very glad if I can get any feedback from you.08:40
eranromtakashi: sure, will start today.08:40
eranromthat is later today :-)08:40
takashiCurrently I don't have so special problems. I'm currently thinking what is the best way to load python module dynamically.08:40
takashiThat is why I submitted one patch to use stevedore.08:41
eranromok, I am not familiar with stevedore. Will have a look08:41
takashieranrom: it is used in other openstack projects which has driver mechanism (keystone, cinder or so on), but currently I'm not sure it is the best way.08:42
eranromok. I see.08:42
takashithat's almost all about python support.08:43
eranromwow, so most of the required code is already there?08:43
takashieranrom: excluding some todo items08:44
eranromnot to say super-cool :-)08:44
takashiI'm currently creating my test environment to run that codes, so I need some more debugging...08:44
takashibut framework is there08:45
eranromIn the meantime I will look at the patches.08:45
takashieranrom: thx!08:45
eranromanything else for today?08:45
takashifor the another small thing, I'm working about sdaemon refactoring and unit testing08:45
eranromtakashi: the Java one right?08:46
takashieranrom: sorry. about daemon_factory, indeed08:46
eranromtakashi: right. the kill_all stuff?08:46
takashiit does not have dependency with python support, but I think I can be a help when I work about refactoring, after I finish python storlet work08:46
takashieranrom: Yes08:46
takashikill functions, shutdown functions, status functions, and starting functions08:47
eranromtakashi: ok, I reviewed some of it earlier. So sure, lets land them first and then go to the python app support stuff.08:47
takashiThat's all from my side today08:47
eranromtakashi: thanks!08:48
kota_takashi: sorry, what does it meen "functions" you use08:48
takashikota_: functions in daemon_factory08:48
takashiI'm currently thinking to cut off the common module, which can be used by both of daemon_factory and python sdaemon08:49
takashiafter I land python sdaemon08:49
takashiand refactoring/unit testing is the entry point of that work08:50
takashieranrom: thanks. I'll update you if I finish testing. At that point we can re-consider our landing order08:51
eranromtakashi: ok. Trying to understand your remark on common08:51
eranromDo you mean SCommon?08:52
takashieranrom: It is different from SCommon, IMO08:52
takashieranrom: Both of daemon_factory adn sdaemon is a callback daemon which listen on sbus interface.08:52
eranromtakashi: gotcha.08:52
takashiAt that point, we can create a common framework, like we do for http server, etc.08:53
eranromSo you want to have a new common infra for both.08:53
takashieranrom: yes. something like agent_common?08:53
eranromtakashi: sounds great!08:53
takashieranrom: :-)08:54
kota_takashi: does it still using sbus protocol? or new protocol?08:54
takashikota_: it is still using sbus protocol. I didn't make change about protocols between processes.08:54
kota_on your idea08:54
kota_ok, gotcha08:55
eranromAnything else for today?08:56
kota_ah, just saying thank you for reviews eranrom, takashi, for my patch.08:56
kota_sorry, I didn't track for them yet though.08:57
eranromkota_: sure, I like the feature and curios regarding the use case :-)08:57
eranromkota_: np08:57
takashikota_: One thing I missed when I added my comment. I think you need to rebase the patch against patch 34934808:57
kota_hopefully, I like to disclose the use case if i can get approved with my boss :P08:57
eranromkota_: no worries. We can land it even if you cannot disclose. I am just curious08:58
kota_patch bot has died :/08:58
takashineed to paste whole of patch url...08:59
takashikota_: https://review.openstack.org/#/c/349348/09:00
takashithis one09:00
takashiRecently some changes about sbus is merged. Please ask me if you find any questions/improvoments around it.09:00
*** eranrom has quit IRC09:00
*** eranrom has joined #openstack-storlets09:01
eranromSorry, need to go. August and kids at home :-)09:01
eranromTalk to you later.09:02
kota_eranrom: np, enjoy!09:02
eranromkota_: Thanks09:02
eranromthanks all for joining09:02
*** eranrom has quit IRC09:03
takashieranrom: thanks. have a nice day, with your family.09:03
*** eranrom has joined #openstack-storlets13:49
*** eranrom has quit IRC13:49
*** eranrom has joined #openstack-storlets13:50
*** eranrom has quit IRC13:51
*** eranrom has joined #openstack-storlets13:52
*** eranrom has quit IRC13:52
*** eranrom has joined #openstack-storlets13:56
*** eranrom has quit IRC14:02
*** eranrom has joined #openstack-storlets14:02
*** eranrom has quit IRC14:03
*** eranrom has joined #openstack-storlets14:03
*** eranrom has quit IRC14:10
*** eranrom has joined #openstack-storlets14:11
*** takashi has quit IRC14:13
kota_good morning19:58
kota_waking up for tc meeting, I'm not sure if i can do follow-up it though.19:59
kota_oh? no meeting today for tc?20:03
kota_oh, my bad, that is scheduled at Aug 30th so i can be back to bed...20:04
eranromwow I would have waited forever.20:05
eranromgood night20:05
eranromgoing to bed as well.20:05
*** eranrom has quit IRC21:47
*** eranrom has joined #openstack-storlets21:48

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!