14:00:42 <szaher> #startmeeting freezer 14:00:44 <openstack> Meeting started Thu Mar 2 14:00:42 2017 UTC and is due to finish in 60 minutes. The chair is szaher. Information about MeetBot at http://wiki.debian.org/MeetBot. 14:00:45 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 14:00:48 <openstack> The meeting name has been set to 'freezer' 14:01:20 <szaher> Hello everyone 14:01:24 <yangyapeng> hello szaher 14:01:44 <raliev> hey everyone :) 14:01:48 <yangyapeng> Allen_: 14:02:56 <szaher> Let's wait 5 minutes for everyone to join 14:03:05 <slashme> Plop 14:03:32 <dstepanenko> hi everyone 14:03:32 <daemontool> hi 14:04:10 <szaher> Please update the topics here https://etherpad.openstack.org/p/freezer_meetings if you have any 14:04:16 <yangyapeng> daemontool hi :) 14:04:25 <vnogin> hi guys 14:04:48 <szaher> I think we are good to start 14:04:50 <szaher> #topic puppet manifests for freezer 14:05:28 <vnogin> yep, me and raliev going to implement it 14:05:50 <szaher> That's really good 14:06:17 <vnogin> first question - as far as I know it's not possible to run freezer api in cluster mode. Correct me if I'm wrong 14:06:34 <dstepanenko> as far as I see it's right 14:06:41 <szaher> vnogin: I don't think so 14:06:52 <szaher> We are running freezer-api behind haproxy 14:06:54 <daemontool> isn't that how you deploy it? 14:06:59 <szaher> on 3 controllers 14:07:07 <szaher> daemontool: Yes it depends on how you deploy it 14:07:25 <daemontool> like if you have the API DB replicated 14:07:35 <daemontool> and the freezer-api endpoints behind haproxy 14:07:39 <daemontool> you have a cluster 14:07:49 <daemontool> or is intended something different by cluster? 14:08:27 <vnogin> daemontool: you are right. Ok. so it seems that we can do it 14:08:56 <daemontool> but 14:09:03 <daemontool> the blocker is, using a DB 14:09:08 <daemontool> that works like that 14:09:11 <daemontool> because now elasticsearch 14:09:17 <daemontool> is not the right decision 14:09:28 <daemontool> (I've added a point to the agenda, we can discuss it later) 14:09:50 <vnogin> ok, cool. I think mysql will solve this issue 14:09:58 <daemontool> yes for the replication 14:10:02 <daemontool> for data sync 14:10:10 <daemontool> but we need to do it... :) 14:10:27 <daemontool> was this task already assigned to anyone yet? 14:10:28 <szaher> daemontool: I think you still can build a clustered elasticsearch :) but anyway mysql would be great as well 14:10:29 <dstepanenko> so, does data sync work for elasticsearch implementation? 14:10:45 <daemontool> szaher, yes, but the sincronization is slow... 14:11:01 <daemontool> so you upload the dato through the API 14:11:06 <daemontool> then query the same data 14:11:12 <daemontool> haproxy send the client 14:11:15 <daemontool> to a different 14:11:26 <daemontool> api node, that will query an elasticsearch 14:11:30 <daemontool> node that is not yet replicated 14:11:33 <daemontool> and the data is not there... 14:11:38 <daemontool> right? 14:11:52 <daemontool> that was the problem we were suffering 14:11:54 <daemontool> I remember... 14:12:00 <dstepanenko> oh, now I understand 14:12:10 <dstepanenko> thanks for clarification, Fausto 14:12:35 <vnogin> so, at the begging we will not implement cluster deployment for api, ok? 14:12:58 <vnogin> we can do it when migration to mysql will be finished 14:13:03 <daemontool> yes 14:13:08 <daemontool> (thumbsup) 14:13:11 <szaher> daemontool: well I agree with you, but there would still be some workarounds but I don't think this is the point now :) moving to mysql I think it the best for freezer-api 14:13:20 <yangyapeng> Database migration is essential 14:13:29 <daemontool> szaher, yes 14:13:44 <daemontool> vnogin, is there anyone in our Team that has that task assigned? 14:13:48 <daemontool> or not? 14:13:49 <szaher> Guys would we consider adding project_id to our db schema ? 14:13:50 <daemontool> don't remember... 14:13:59 <daemontool> szaher, yes 14:14:22 <vnogin> daemontool: afak nope 14:14:25 <daemontool> ok 14:14:29 <yangyapeng> szaher: we should add it i think 14:14:30 <vnogin> cool, it's pretty clear now. second question - are we going to develop puppet manifests under freezer umbrella or it should the part of the separate project (for instance openstack puppet or fuel, cetera) 14:14:39 <dstepanenko> Fausto, I'm working on oslo_db implementation 14:14:43 <daemontool> szaher, yangyapeng slashme do you have bandwithd for the mysql migration? 14:14:45 <szaher> even if it will be useless in the beginning but I am working on an api change to consider sending tenant ID 14:14:46 <daemontool> a ok 14:14:52 <daemontool> dstepanenko, good good :) 14:15:02 <daemontool> excellent 14:15:07 <daemontool> so we are covered 14:15:11 <vnogin> yep 14:15:19 <daemontool> szaher, we need to add that in the mysql spec? 14:15:22 <daemontool> the project_id right? 14:15:25 <daemontool> is not there? 14:15:31 <szaher> Yes we need to add it 14:15:58 <vnogin> let's discuss second question :) 14:16:25 <daemontool> ok added here 14:16:26 <daemontool> https://review.openstack.org/#/c/408014/ 14:16:30 <szaher> vnogin: I think it would be puppet-freezer repo what do you think guys ? 14:16:48 <vnogin> szaher: 100% agree 14:16:55 <daemontool> cannot we add it directly to openstack-puppet? 14:17:00 <daemontool> is not intended to do that? 14:17:09 <daemontool> not against that... just asking 14:17:15 <daemontool> other project have their own 14:17:16 <szaher> daemontool: it there any openstack-puppet repo ? 14:17:24 <szaher> I think every project has it's own repo 14:17:25 <daemontool> repo for deployment artefacts? 14:17:32 <daemontool> ah ok 14:17:41 <szaher> daemontool: https://github.com/openstack/?utf8=%E2%9C%93&q=puppet&type=&language= 14:18:06 <daemontool> yes 14:18:07 <daemontool> https://wiki.openstack.org/wiki/Puppet 14:18:19 <daemontool> good good :) 14:18:41 <daemontool> szaher, manage repos and all that is on you, now, is it? :) 14:18:55 <szaher> I think so :) 14:19:02 <szaher> I will request it 14:19:26 <daemontool> brilliant 14:19:36 <vnogin> cool 14:19:50 <daemontool> ok, next? 14:20:03 <szaher> anyone wants to add anything ? 14:20:11 <vnogin> szaher: please share then a result of your request :) 14:20:20 <szaher> vnogin: will do 14:20:25 <vnogin> szaher: tnx 14:20:33 <szaher> #topic Saad's summary of the PTG 14:20:54 <szaher> Well I was attending the first PTG summit this year in Atlanta 14:21:19 <vnogin> nice freezer logo btw :) I have one %) 14:21:27 <szaher> It was really nice to meet people from different projects and talk to them, there are some points that we might want to consider at some point 14:21:39 <szaher> vnogin: Cool! glad you like it 14:21:53 <szaher> I had a chat with cinder guys about volume backups 14:22:01 <yangyapeng> os_brick? 14:22:18 <szaher> I think using os-brick and mounting volumes to back it up would be good to do 14:22:52 <daemontool> szaher, https://review.openstack.org/#/c/430304/ 14:22:56 <yangyapeng> instead cinder mode 14:23:01 <daemontool> yep 14:23:03 <raliev> szaher, I'm working on it already (os-brick engine I mean) 14:23:10 <szaher> daemontool: yes I saw it :) I was reading it 14:23:23 <szaher> raliev: very good 14:23:35 <daemontool> then let's approve it :) 14:23:40 <daemontool> lol 14:24:08 <daemontool> szaher, any other feedback from PTG? 14:24:08 <szaher> we might have a value add here using freezer to backup cinder volumes as we can store the backup to multiple backends at the same time not like cinder which is using only one backend at a time 14:24:20 <daemontool> yes... 14:24:36 <szaher> daemontool: we need to be present more in the community we need to arrange discussions with different projects 14:24:58 <daemontool> szaher, yes 14:25:09 <szaher> daemontool: some silly stuff like the documentation, releasenotes, api-ref all these stuff needs to be done and published 14:25:23 <daemontool> couldn't agree more 14:25:32 <daemontool> we should onboard 14:25:36 <yangyapeng> szaher: good 14:25:39 <daemontool> someone 14:25:41 <daemontool> in out internal team 14:25:42 <daemontool> soon 14:25:42 <szaher> daemontool: I already did the tech work to do all this and I know how to publish it but we need some content :) 14:25:43 <daemontool> to do that 14:25:52 <daemontool> ok 14:25:55 <szaher> daemontool: thank would be really great 14:26:04 <daemontool> I think within the next 2 weeks 14:26:07 <daemontool> will join 14:26:10 <szaher> I can guide him how to add the content and I will takecare of publishing it 14:26:15 <daemontool> her, ok 14:26:21 <szaher> daemontool: :D 14:26:30 <daemontool> let her publish something also 14:26:41 <daemontool> positive 14:26:51 <daemontool> ok 14:26:55 <szaher> daemontool: it will be under her name but I need to add the gate jobs to freezer config :) 14:27:00 <daemontool> yes 14:27:01 <daemontool> ok 14:27:10 <vnogin> we need technical writer :) 14:27:13 <daemontool> yes 14:27:18 <szaher> anyway the PTG was good and I hope I can see you soon in the next PTG or summit 14:27:22 <daemontool> so the feedback from community are 14:27:27 <daemontool> os-brick for volumes 14:27:30 <daemontool> documentation 14:27:36 <daemontool> release notes 14:27:40 <daemontool> api-ref 14:27:42 <daemontool> ok 14:28:03 <szaher> daemontool: Yes, people need to use freezer but they don't know how or where to find resources and information 14:28:08 <daemontool> ok 14:28:24 <szaher> anymore comments about PTG ? 14:28:43 <szaher> #topic spec to improve nova backup 14:29:02 <daemontool> we have an engineer working on that 14:29:09 <daemontool> not sure if she's here 14:29:13 <daemontool> but we need to write the spec 14:29:18 <daemontool> on how to make that better 14:29:21 <szaher> I am working now on implementing nova backup engine, so having a spec would help me a lot I can push what I have now to gerrit so you can take a look guys 14:29:33 <daemontool> yes 14:29:37 <daemontool> please 14:29:45 <szaher> daemontool: Nice put me in touch with her that would be great :) 14:30:08 <daemontool> szaher, please write the spec when you can 14:30:18 <daemontool> as we need it 14:30:21 <daemontool> internally 14:30:22 <szaher> Ok 14:30:25 <daemontool> not only publicly 14:30:49 <daemontool> then Julia can help you 14:31:11 <daemontool> ok 14:31:26 <szaher> aftering finishing nova engine I will try to implement a cinder engine so may be coordinating with raliev would be great before doing that 14:31:55 <daemontool> szaher, I think raliev will get that done, before the nova engine is completed ;) 14:31:57 <daemontool> lol 14:32:03 <vnogin> lol)) 14:32:14 <raliev> daemontool, I'll try of course :) 14:32:16 <szaher> daemontool: :D nice 14:32:30 <daemontool> ok, 14:32:33 <daemontool> next? 14:32:43 <vnogin> raliev: == flash :) 14:33:06 <szaher> #topic spec to define minimum backup criteria 14:33:22 <daemontool> ok, 14:33:26 <daemontool> something like the following 14:33:29 <daemontool> - every backup engine should support the following feature: 14:33:30 <daemontool> - single object backup (i.e. single volume, single instance) 14:33:30 <daemontool> - all objects owned by a tenant 14:33:30 <daemontool> - all objects from all tenants (admin) 14:33:30 <daemontool> - for volumes, all volumes part of a consistency group 14:33:42 <daemontool> something that makes life easy 14:33:48 <daemontool> for people that manage operations 14:34:23 <vnogin> we can add images as well 14:34:35 <szaher> raliev: what do you think about this ? 14:35:19 <daemontool> by object I mean 14:35:24 <raliev> szaher, about what - os-brick discussion or minimum backup criteria? :) 14:35:25 <daemontool> instances or volumes 14:36:16 <szaher> about the backup criteria backing up one obj, all obj, tenant 14:37:49 <szaher> daemontool: Fausto can we have a spec about this one as well ? 14:38:00 <daemontool> yes 14:38:01 <szaher> so we can get an Idea about what should be done 14:38:13 <szaher> next ? 14:38:17 <daemontool> for cinder os-brick 14:38:21 <daemontool> I've added that i nthe spec 14:38:28 <daemontool> but I think one spec should be done 14:38:34 <daemontool> for all backup engines 14:38:44 <daemontool> ok 14:38:48 <daemontool> good for me 14:39:00 <szaher> It will be good for all 14:39:16 <raliev> good for me too 14:39:31 <szaher> next? 14:40:04 <daemontool> yep 14:40:21 <szaher> #topic Heat resources for Freezer 14:40:35 <daemontool> ok, we have this as a requirement 14:40:37 <daemontool> from the business.... 14:40:43 <daemontool> very important for us 14:41:15 <daemontool> we should onboard an engineer withing the net 2 weeks 14:41:17 <daemontool> to do that 14:41:19 <yangyapeng> we should have a template 14:41:21 <daemontool> yes 14:41:38 <szaher> daemontool: Do we know what needs to be done ? 14:41:38 <daemontool> we need to have a spec 14:41:53 <daemontool> yes 14:42:05 <daemontool> whatever can be possible to be done from the scheduler nd the agent 14:42:13 <daemontool> should be possible to be executed from a heat template 14:42:18 <daemontool> as a general rule 14:42:28 <daemontool> but I think we need to segment the task 14:42:32 <daemontool> and start with something small 14:42:45 <szaher> daemontool: agree 14:43:00 <daemontool> backup and restore execution 14:43:05 <daemontool> and loading the jobs 14:43:07 <daemontool> at minimum 14:44:00 <daemontool> ok we need a spec 14:44:03 <daemontool> anyone? 14:44:07 <daemontool> I can write it 14:44:18 <daemontool> yangyapeng, are you interested eventually? do you have time? 14:44:22 <daemontool> not to implement 14:44:26 <daemontool> to write the spec... 14:44:33 <daemontool> even implementation of course, as you want 14:45:01 <yangyapeng> daemontool: i can write the spec, 14:45:13 <daemontool> brilliant 14:45:16 <daemontool> thanks :) 14:45:20 <daemontool> szaher, are you ok? 14:45:21 <daemontool> with this? 14:45:26 <szaher> Yes :) 14:45:30 <daemontool> sorry to push for it, but we need it.... 14:45:46 <szaher> daemontool: I totally Ok with me :) 14:45:56 <szaher> yangyapeng: you might want to talk to heat guys at some point 14:46:03 <szaher> next ? 14:46:18 <daemontool> yangyapeng, we count on that please, thank you :) 14:46:21 <daemontool> yep 14:46:30 <yangyapeng> :) 14:47:09 <szaher> #topic Switch to MySQL 14:47:19 <daemontool> I think we discussed this already? 14:47:29 <szaher> Yes 14:47:52 <szaher> next? 14:47:57 <daemontool> yep 14:48:10 <szaher> #topic Add flexible options to rsync backup 14:48:40 <daemontool> this backup engine 14:48:42 <daemontool> is critical 14:48:47 <daemontool> but now it is not usable 14:48:53 <daemontool> for big volumes 14:48:58 <daemontool> or big mount of data 14:49:04 <daemontool> we need to make it more flexible 14:49:11 <daemontool> the issue is the sliding window 14:49:12 <daemontool> of 1 byte 14:49:18 <daemontool> for rolling checksum 14:49:37 <daemontool> we have 1 checksum computation for each byte 14:49:49 <daemontool> good for space efficiency 14:49:51 <daemontool> but 14:49:55 <daemontool> not for time 14:50:00 <raliev> guys, regarding rsync backup 14:50:07 <raliev> I have one update 14:50:34 <raliev> I also working on improving rsync engine 14:50:47 <daemontool> ok 14:51:06 <szaher> Ok 14:51:24 <raliev> I found a lot of problems 14:51:32 <raliev> and with rolling checksum also 14:51:54 <daemontool> raliev, when you can write few lines on etherpad 14:51:59 <raliev> long story short, performance of current implementation of rolling checksum is completely bad 14:52:00 <daemontool> just one liner per issue 14:52:05 <daemontool> raliev, yes 14:52:11 <daemontool> ah ok :) 14:52:13 <raliev> becuase it's computed on pure python 14:52:16 <daemontool> yes 14:52:22 <raliev> I fixed it already 14:52:29 <raliev> and soon I push all my fixes 14:52:36 <daemontool> did you wrote it in assembler? 14:52:38 <raliev> so you would see it 14:52:38 <daemontool> lol 14:52:52 <daemontool> ok 14:53:01 <szaher> raliev: Ok, is that all ? 14:53:17 <raliev> actually yes 14:53:18 <szaher> we have 7 minutes :) 14:53:23 <szaher> OK next topic 14:53:40 <szaher> #topic api v2 14:54:15 <szaher> I will be working on api v2 where I will try to introduce tenant_id 14:54:45 <szaher> I started that already and the new url schema will look like ip:9090/v2/project_id/jobs 14:54:52 <daemontool> ok 14:54:53 <szaher> so we can apply policy 14:55:09 <szaher> also in the PTG they introduced api microversions 14:55:09 <daemontool> should we add something like backup to identify the service? 14:55:21 <daemontool> in the url imean 14:55:35 <szaher> I don't think we need it as all os apis don't have something like that 14:55:49 <daemontool> okok 14:55:58 <szaher> back again to microversions iam not sure if we will need it or not 14:56:10 <szaher> gonna send something links about it got from the community 14:56:17 <szaher> i will send it in freezer room 14:56:26 <szaher> I think that all for me :) 14:56:32 <szaher> anyone wants to add anything ? 14:56:38 <szaher> we have less than 4 minutes 14:56:50 <Allen_> how quickly 14:57:06 <szaher> Allen_: the microversion links ? 14:57:12 <szaher> or the api change ? 14:58:43 <szaher> Ok guys we are running out of time, Let's continue this in freezer room if you still have any questions 14:58:46 <daemontool> ok 14:58:48 <daemontool> thanks :) 14:59:02 <szaher> Thank you guys :) 14:59:13 <szaher> #endmeeting