Wednesday, 2016-08-17

*** zhurong has quit IRC00:04
*** yizhihui has joined #openstack-smaug00:22
*** zhurong has joined #openstack-smaug01:09
*** zhurong_ has joined #openstack-smaug02:19
*** zhurong has quit IRC02:23
*** zhongjun_ has quit IRC02:34
*** zhurong_ has quit IRC02:39
*** zhurong has joined #openstack-smaug02:39
*** zhongjun_ has joined #openstack-smaug02:55
openstackgerritxiangxinyong proposed openstack/smaug: Fix plan parameters verify exception  https://review.openstack.org/35619803:11
*** zhurong has quit IRC03:35
*** zhurong has joined #openstack-smaug03:36
*** zhurong has quit IRC04:14
*** zhurong has joined #openstack-smaug04:16
*** mingyu has joined #openstack-smaug05:44
*** mingyu has quit IRC05:45
*** mingyu has joined #openstack-smaug05:49
*** yinwei_computer has joined #openstack-smaug06:37
*** yuval has joined #openstack-smaug06:43
yinwei_computerhi, yuval06:52
yuvalyinwei_computer: hey :)06:52
*** yinwei_computer has quit IRC06:53
*** yinwei_computer has joined #openstack-smaug06:55
*** yinwei_computer has quit IRC06:56
*** yinwei_computer has joined #openstack-smaug06:59
*** gouthamr has quit IRC07:00
*** mingyu has quit IRC07:02
*** yinwei_computer has joined #openstack-smaug07:04
yinwei_computerping yuval07:04
yuvalping yinwei_computer07:04
yinwei_computersorry, my network seems not stable07:05
yinwei_computerdid I miss your answer?07:05
yinwei_computerI just asked is there any conclusion about the bank&db issue?07:05
yuvalI didn't get any question07:06
yuvalyinwei_computer: saggi has posted https://review.openstack.org/35601107:06
yinwei_computerthanks, yuval. I will check this patch.07:07
yinwei_computerI see what you guys argued is whether we can replace the object storage based bank to db based bank07:08
yinwei_computeris there any question about this issue?07:08
yinwei_computeryuval, if the problem is some protection plugin will store data in bank, is it possible to divide existed bank interface into two interfaces: md_bank and data_bank?07:11
yinwei_computerand each protection plugin should claim explicitly what banks they rely on in plugin conf.  We can have protection plan creation to check the match of system bank and the requirment of the protection plugin.07:11
yinwei_computerwe could also have protection plugin initialization to check if the required bank (banks) are loaded in the system.07:12
yinwei_computerpls check if this is feasible when you're available07:16
*** mingyu has joined #openstack-smaug07:19
*** mingyu has quit IRC07:23
openstackgerritxiangxinyong proposed openstack/smaug-dashboard: Rename Smaug Dashboard to Karbor  https://review.openstack.org/35627907:26
openstackgerritzengchen proposed openstack/smaug: Support only one time format for time trigger  https://review.openstack.org/35628407:31
yuvalyinwei_computer: sorry, I'm back07:38
yuvalyinwei_computer: the question was what to do in a single site deployment where you don't want additional components such as swift07:39
yuvalyinwei_computer: (considering you don't use a protection plugin that stores the data in the bank)07:39
yuvalyinwei_computer: splitting the bank interface is a good idea imo07:40
yuvalyinwei_computer: (that's what saggi has in the above patch)07:40
saggiWho is talking about me ??07:41
saggi:)07:41
yinwei_computerhi, saggi07:41
saggiyinwei_computer: How is it going? It's been a while. We don't see you in meetings anymore.07:42
yuvalyinwei_computer: making protection plugins explicitly list the supported banks is problematic. Once a bank plugin interface is implemented, karbor and the protection plugins can rely on it07:42
yinwei_computersorry, I take leaves last month.  I will try to attend meetings on time :)07:43
yinwei_computersorry yuval, I didn't see the problem to make protection plugins explicitily claim the bank interfaces they rely on.07:44
yinwei_computercould you elaborate it?07:44
openstackgerritSaggi Mizrahi proposed openstack/smaug: Bank indexing interface  https://review.openstack.org/35601107:45
yuvalyinwei_computer: lets say we have swift bank plugin, s3 bank plugin, and a protection plugin that explicitly relies on both07:45
yuvalyinwei_computer: tomorrow I release a ceph bank plugin, and you can't use it until you manually tell the protection plugin to rely on it07:45
yinwei_computerhmm, there's some misunderstanding07:46
yinwei_computerI think what protetion plugins relying on are md_bank_interface and data_bank_interface.07:46
yinwei_computerprotection plugins should not care about the implementation under md_bank_interface and data_bank_interface;07:47
saggiyinwei_computer: We can't have a way for a bank to not work with a certain bank implementation.07:47
yinwei_computersaggi, I didn't get it.07:48
yinwei_computeris it possible to have ceph as data_bank_plugin and swift as md_bank_plugin?07:48
yinwei_computerimho, it's feasible.07:49
saggiyinwei_computer: Let me just check that I'm understanding everything. You want to split the bank interface to mb_bank_interface and data_bank_interface where one is responsible for the checkpoint md and the other contains blobs.07:49
saggiright?07:49
yinwei_computerand it's also possible that there is only md_bank_plugin07:49
yinwei_computeryes07:49
yinwei_computerone for checkpoint and the other for data07:50
saggiThis is where the problem starts. If you have a protection plugin that needs the data bank it will not work. That is problematic.07:50
yinwei_computerwhy it doesn't work?07:50
saggiWe can only consider if having a data implementation is mandatory07:50
yinwei_computereach data bank should comply with the interface, right?07:50
saggisince it depends on the data_bank07:50
saggiSo you must define a data_bank07:51
saggi?07:51
yinwei_computerif it depends on the data_bank, it should claim this in plugin.conf or in its requirement. So plan creation will check its requirment.07:52
yinwei_computerit depends on the data_bank_interface.07:52
saggiIf we implement https://review.openstack.org/#/c/355956/ there is no reason why someone can not have a data_bank.07:52
yinwei_computerit only calls the interface07:52
*** mingyu has joined #openstack-smaug07:53
saggiyinwei_computer: Why would anyone not have a data_bank?07:54
yinwei_computersaggi, I think one implementation could both implement md_bank and data_bank07:54
saggiyinwei_computer: obviously, that is what we have now07:55
yinwei_computeror only implement md_bank07:55
yinwei_computeras zhonghua has told that, to adapt more user cases07:55
saggiWhat is the use case07:56
yuvalsaggi: as we discussed yesterday, single site with no swift deployment07:56
yinwei_computerafak, most users have already got their backup system, so they could backup data across their own backup system07:56
yinwei_computersmaug will manage the DP over the openstack07:57
yinwei_computerthey used to backup volumes/vms manually07:57
* saggi is thinking about it07:57
yuvalyinwei_computer: images?07:57
*** mingyu has quit IRC07:57
yinwei_computersame as images07:57
yinwei_computernow, glance backend could be shareable as well07:58
yinwei_computerthat's what openstack provided before07:58
yinwei_computerso if these users try to adopt smaug, they will find it's nice to use smaug to manage whole DR life cycle. But they don't rely on smaug to replicate/store data.07:59
yinwei_computerin this way, they would prefer a db based bank (md bank only).  And db is the on premise software they already have.08:01
saggiyinwei_computer: The thing that I don't understand is what is the point. You don't *have* to save data in the bank and it doesn't add any overhead to allow that.08:01
saggiyinwei_computer: So the main issue is having a DB as a bank08:01
saggiThis causes two issues. 1. We purposefully don't define a schema for the data since old checkpoints should be recoverable from old versions. 2. It's hard to put blobs in a DB.08:02
yinwei_computerI think we should allow a smaug with customized protection plugins to run without data-bank, where those protection plugins don't refer data-bank and the customers don't want to run any data-bank system.08:03
saggiWe should remember to start saying Karbor :)08:03
yinwei_computersaggi, sorry, what's data blob in checkpoint?08:04
yinwei_computerI might miss sth. here.08:04
saggiblob - binary large object. An image chunk for example.08:04
yuvalyinwei_computer: you could write a 'null' bank plugin08:08
saggiyinwei_computer: MySql, and Postgres have BLOB support.08:09
saggi[1] https://www.postgresql.org/docs/9.1/static/datatype-binary.html08:09
saggi[2] https://dev.mysql.com/doc/refman/5.7/en/blob.html08:09
saggiAnd if those code paths would not be used very much we can leave them inefficient.08:10
saggiSo we can have a full bank where using the data_bank is just not recommended08:10
yinwei_computerBut in the case I told before, protection plugin doesn't have such requirement to store image chunk into bank.08:11
yinwei_computersay, they already store image to some file share system.08:11
yinwei_computerso my question is: shall we allow one smaug instance run without data_bank?08:12
chenyingdivide existed bank interface into two interfaces: md_bank and data_bank. sound it is a good idear.08:12
yuvalyinwei_computer: even if we don't, you can write a 'null data bank plugin'08:13
saggiyuval, how would it behave? Would it throw and error or silently do nothing?08:14
yuvalsaggi: silently do nothing. just an example08:14
saggiThe thing I'm worried about is people misconfiguring and have it just not do what it claims to do.08:15
saggiIt's hard to avoid that but we can at least try08:15
yinwei_computerif there's a null data bank plugin, plan creation will report error if it finds the provider protection plugin requires data bank plugin but there's only a null one?08:15
yinwei_computerso the requirement is not in conf, but in code08:16
yinwei_computereach protection plugin implements its bank_requirement08:16
yinwei_computerso api layer could check it when plan creation08:16
yinwei_computeractually, we need multiple checking in plan creation to match resources and protection plugins.08:17
saggiyinwei_computer: What is the problem with implementing blobs in with the DB and just have people that don't use it just not use it?08:23
saggiIt seems simpler08:23
yinwei_computerbut not flexible :(08:24
yinwei_computerwhat if the user's on premise db doesn't support blob?08:24
yinwei_computersince the md-bank sometimes also needs to be customized08:25
yuvalyinwei_computer: in the end of the day, the user is storing a blob somewhere when he backs up an image or a volume08:25
yuvalyinwei_computer: it might be swift, ceph, cinder or anything else, but the data will be stored08:25
yinwei_computeryes, they store them with their on premise devices08:26
yinwei_computerthey only adopt smaug for dr control08:26
yinwei_computernot data08:26
yuvalgreat, so use protection plugins which don't store data in the bank08:26
yuvalbut a data bank plugin is a must. and you do have a backend in your deployment, otherwise you had no place for the image/volume backups08:27
yinwei_computeryes, but we need a check08:27
yuvalcheck what/08:27
yuval?08:27
saggiyinwei_computer: I think all supported databases all support BLOBS. And if it doesn't you can make a DB\FS hybrid easily.08:27
yinwei_computerwhat's the point if I don't use it but I must have one?08:28
yinwei_computerhave data bank plugin means I need run this backend and I need implement a data bank plugin for it08:28
yinwei_computerwhat's the point to do it if I don't need it?08:28
saggiBut if you are 90% of people and use Swift\FS\MySql\Posgres\Mongo\Couchbase\Sqlite and countless more you *will* have it implemented with zero overhead.08:29
saggiI think it's 99%08:29
yinwei_computerwhy the overhead is 0?08:30
saggiIt doesn't require another service08:30
yinwei_computerwe will provide all of those db implementations for data bank interface?08:30
saggiWe will not split the bank up.08:31
yuvalbtw, if we want to split bank into md_bank and data_bank, why not store the md in karbor's db?08:31
saggiThis will mean that you don't have to check if the user has a data_bank all around the code08:31
yinwei_computeror customer need develop their own data bank based on the db they are using?08:31
saggiand if the user sees that the bank isn't handling all the blobs it can change the bank or change the protection plugin08:31
saggiyinwei_computer: I don't think we should make all that code for the 0.1% that don't use any of the mainstream databases.08:32
yinwei_computersaggi, the problem is  customer need develop their own data bank based on the db they are using, but they don't use the data-bank at all08:32
yinwei_computerwhy would they put these efforts?08:32
yuvalhow will the bank plugin know what is data and what is metadata if we don't split the bank?08:33
yuval(what to store in blob and what not to)08:33
saggiyinwei_computer: They will probably not need to develop one since if it's relational it just uses SQL. If it's an object\document store it doesn't matter08:33
yinwei_computerwe split bank into 2 interfaces, and thus protection plugins will explicitly call data and md interfaces.  The interface will tell them to know this context @yuval08:34
saggiyinwei_computer: I agree, we should add blob specific APIs.08:35
saggiSo that the implementation can know if it's indexable or not.08:35
yinwei_computersaggi, I'm not sure if the blob related semantics are the same. But at lease they need test if the database complies with the 'generic' data bank plugin.08:36
yinwei_computerleast08:36
chenyingI also think data bank interface is optional. Verdor's plugins may use their own solution write the backup data to their backup backend. Like some backup software verdor. they don't need additional data bank interface.08:37
yinwei_computerthe point is, customer would say I don't need it, why there's no option to get rid of this. I don't want to put any effort there.08:38
saggiyinwei_computer, chenying: I'm not disagreeing with the assertion that some users don't need to store blobs in the bank. The issue I have that if we make that interface optional we need to add code that checks for it.08:38
saggiAnd I think that it's less code to implement data_bank for SQL once than to have every plugin writer to remember to check for that interface.08:39
yinwei_computerchenying, saggi's argument is there's one 'generic' data-bank-plugin for most of dbs.  my point is even if this is the case, there will be some verification effort, and customers won't put effort on things they don't use.08:39
saggiyinwei_computer: You are talking about certifying databases?08:39
saggi*specific databases08:39
chenying Verdor's plugins is thrie own solution. They may don't consider use a uniform writing backup data interface.08:39
yinwei_computerplugin writer doesn't check it, but claim it in the requirement function.08:40
*** yinwei_computer has quit IRC08:41
saggiTBH I never liked putting blobs in the bank in the first place. But this means that generic plugins can't put *small* blobs in the bank if they want to. This means we can't have any OpenStack reference implementation depend on being able to put blobs in the bank.08:41
saggiIt could just be an `recover.sql` file that contains the backup of a db08:42
yuvalputting blobs in a bank is very beneficial in cross site protect and restore08:43
*** yinwei_computer has joined #openstack-smaug08:43
*** yinwei_computer has quit IRC08:44
saggiyuval: I think we broke yinwei :)08:46
yuval:((08:46
*** yinwei_computer has joined #openstack-smaug08:46
*** yinwei_computer has quit IRC08:47
*** yinwei_computer has joined #openstack-smaug08:51
yuvalyinwei_computer: welcome back08:51
yinwei_computeralways lost08:53
yuvalputting blobs in a bank is very beneficial in cross site protect and restore08:54
*** yinwei_computer has quit IRC08:55
saggiyinwei_computer: It seems like to much to prevent any Protection Plugin to be able to add information to the bank that is not in json format. It might just want to backup a configuration file or some data dumps from an external source.08:55
saggiYou are talking about use cases where the customer is not and is unwilling to be using MySql\Postgres\Oracle\Swift\NFS08:57
*** yinwei_computer has joined #openstack-smaug08:57
saggiAnd even for DBs that don't support BLOBS we do need to be able to insert at least small binaries since Protection Plugins might need them.08:58
saggiThis can be facilitated with b64 encoded text records08:58
*** yinwei_computer has quit IRC08:58
yuvalyinwei lost again09:00
*** yinwei_computer has joined #openstack-smaug09:01
saggiyinwei_computer: It's all your fault09:03
saggichenying: If yinwei_computer is looking for use tell her that we left for lunch09:08
yinwei_computerok09:08
yinwei_computerhave a nice day09:09
*** yinwei_computer has quit IRC09:20
*** yinwei_computer has joined #openstack-smaug09:22
*** yinwei_computer has quit IRC09:23
*** yinwei_computer has joined #openstack-smaug09:25
*** openstack has joined #openstack-smaug10:16
*** yizhihui has quit IRC10:16
*** yamamoto has joined #openstack-smaug10:50
*** yamamoto has quit IRC10:56
*** yamamoto has joined #openstack-smaug10:57
*** yamamoto has quit IRC11:02
*** yamamoto has joined #openstack-smaug11:03
*** yamamoto has quit IRC11:07
*** yamamoto has joined #openstack-smaug11:07
*** yamamoto has quit IRC11:09
*** yamamoto has joined #openstack-smaug11:09
*** yamamoto has quit IRC11:40
*** yamamoto has joined #openstack-smaug11:46
*** gouthamr has joined #openstack-smaug12:04
*** yamamoto has quit IRC12:15
*** yamamoto has joined #openstack-smaug12:28
*** yamamoto has quit IRC12:32
*** yamamoto has joined #openstack-smaug12:55
openstackgerritMerged openstack/smaug-dashboard: Fix Protection Plan Creation Failed  https://review.openstack.org/35597612:59
*** zhurong has joined #openstack-smaug13:05
*** zhurong has quit IRC13:07
*** zhurong has joined #openstack-smaug13:09
*** yamamoto has quit IRC13:29
*** yamamoto has joined #openstack-smaug13:30
*** saggi has quit IRC14:05
*** saggi has joined #openstack-smaug14:08
openstackgerritYuval Brik proposed openstack/smaug: Reimplement the Protection workflow for new design  https://review.openstack.org/34816314:24
*** yamamoto has quit IRC14:27
*** yamamoto has joined #openstack-smaug14:29
*** yamamoto has quit IRC14:34
*** yuval has quit IRC14:57
*** zhurong has quit IRC14:59
*** openstackgerrit has quit IRC15:03
*** openstackgerrit has joined #openstack-smaug15:04
*** yamamoto has joined #openstack-smaug15:32
*** yamamoto has quit IRC15:37
*** saggi has quit IRC16:08
*** yamamoto has joined #openstack-smaug17:34
*** yamamoto has quit IRC17:38
*** gouthamr has quit IRC21:04
*** yamamoto has joined #openstack-smaug22:30
*** gouthamr has joined #openstack-smaug23:28
*** zhurong has joined #openstack-smaug23:51

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!