Thursday, 2016-01-21

*** dschroeder has quit IRC00:19
openstackgerritDeklan Dieterly proposed openstack/freezer: Add check for name=='__main__'  https://review.openstack.org/26655201:07
*** yangyapeng has joined #openstack-freezer02:17
yangyapengHello02:17
yangyapengI have a few new patch,please review it.02:18
yangyapengDuncanT: hi02:19
*** EinstCrazy has joined #openstack-freezer03:03
*** samuelBartel has joined #openstack-freezer08:29
*** daemontool has joined #openstack-freezer09:20
*** openstackgerrit has quit IRC10:02
*** openstackgerrit has joined #openstack-freezer10:03
*** daemontool_ has joined #openstack-freezer10:07
*** daemontool has quit IRC10:07
*** daemontool_ is now known as daemontool10:14
*** vannif_ has quit IRC10:59
*** emildi has joined #openstack-freezer11:09
*** vannif has joined #openstack-freezer11:09
daemontoolemildi, hi11:25
daemontool:)11:25
emildihi daemontool  :)11:43
daemontoolall we need to approve this https://review.openstack.org/#/c/270750/ and this https://review.openstack.org/#/c/270753/ as soon as they pass the checks12:01
*** daemontool has quit IRC13:19
*** yangyapeng has quit IRC13:46
*** EinstCrazy has quit IRC13:47
*** daemontool has joined #openstack-freezer14:05
*** ddieterly has joined #openstack-freezer15:26
openstackgerritFausto Marzi proposed openstack/freezer-web-ui: Align requirements to liberty global-requirements  https://review.openstack.org/24698115:31
*** dschroeder has joined #openstack-freezer15:35
*** EinstCrazy has joined #openstack-freezer15:45
*** pennerc has joined #openstack-freezer15:45
ddieterlydoes this meeting usually start late?16:05
ddieterlyis there a meeting today?16:08
ddieterlydaemontool: yo16:09
*** ddieterly has quit IRC16:11
*** ddieterly has joined #openstack-freezer16:11
ddieterlyhey, guys. is there a meeting today?16:12
ddieterlydschroeder: hey, david. can you see my posts?16:13
dschroederYes I can.16:14
ddieterlyok, i thought maybe i had a connectivity problem16:14
ddieterlyisn't there supposed to be a meeting right now?16:15
ddieterlydschroeder: isn't there supposed to be a meeting right now or did i get the times wrong?16:16
dschroederYou got the times right, 4pm UTC on Thursdays.16:19
ddieterlyok, thought i might have been mistaken16:19
ddieterlyanybody know where daemontool is?16:20
m3m0hey guys, sorry for the delay16:23
pennercLets assume there is no meeting today.  We do need to verify the meeting with Fausto.  Currently Fausto is not on line, which would be the reason for no meeting.16:23
m3m0today I'm really really busy16:23
pennercor maybe he is?16:23
m3m0but the meeting is off for today, sorry for the short notice16:24
ddieterlywho usually runs the meeting?16:24
m3m0me16:24
ddieterlywhy is it off?16:24
m3m0work issues16:24
ddieterlym3m0: could we please get advanced warning if the meeting is going to be cancelled?16:27
m3m0for sure, my bad, I just had bad timing today16:27
daemontoolI'm here sorry I had a last minute important meeting16:34
daemontoolso we can still run 25 min16:35
daemontoolif folks are good16:35
*** frescof has joined #openstack-freezer16:39
daemontoolI think we need to define e backup chariman for the meetings, in case the main cannot run the meeting16:42
daemontoolbut I agree with ddieterly we need to let people know this before hand16:42
ddieterlydaemontool: any idea why my restore is failing on my dev box?16:49
ddieterlylooks like gzip is not happy16:50
ddieterlydo i need a certain version of gzip?16:50
*** ddieterly has quit IRC16:50
*** ddieterly has joined #openstack-freezer16:51
daemontoolddieterly,  gzip is backward compatible between versions16:52
daemontoolI don't think the issue is gzip16:52
daemontoolbut I've been very busy with meetings and preparing the m2 release for mitaka16:52
daemontoolso I couldn't test it16:52
daemontoolvannif, ping16:52
ddieterly2016-01-20 16:17:12,370 root INFO Restore backup deklans-macbook-pro_backup-0_1453330051_016:55
ddieterly2016-01-20 16:17:12,386 root INFO [*] Pipe closed as EOF reached. Data transmitted successfully16:55
ddieterly2016-01-20 16:17:12,391 root ERROR [*] Restore error: gzip: (stdin): unexpected end of file16:55
ddieterly/usr/local/bin/gtar: Child returned status 116:55
ddieterly/usr/local/bin/gtar: Error is not recoverable: exiting now16:55
ddieterlyTraceback (most recent call last):16:55
ddieterly  File "/Users/dieterlyd/virtualenvs/freezer/lib/python2.7/site-packages/freezer/engine/tar/tar_engine.py", line 129, in restore_level16:55
ddieterly    tar_process.stdin.write(read_pipe.recv_bytes())16:55
ddieterlyEOFError16:55
ddieterly2016-01-20 16:17:12,394 root CRITICAL [*] Critical Error: failed to restore file16:55
ddieterly(i hope that is not 'flooding' the channel)16:56
daemontoolddieterly,  np16:56
daemontoolso after the parallal backup feature merge16:56
daemontoolI did tests16:57
daemontoolbut using a config file16:57
daemontoolit shoult be the same tho16:57
daemontoollet me finish the branching for the freezer-web-ui16:57
daemontoolcomponent16:57
daemontooland be ready for m2 mitaka16:57
daemontooland I'll check it16:57
daemontoolddieterly,  if you download that backup level 0 object form swift16:59
daemontoolcan you tar gunzip it_16:59
daemontool?16:59
ddieterlyi'll try it as soon as i get my devstack env back up17:00
daemontoolok17:00
daemontoolso we need to define a second chair17:00
daemontoolfor the meetings17:00
ddieterlyyea17:01
daemontoolor in general we can define that if the chair is not avail if one of the cores is online then he do it17:01
daemontoolthe thing is that Eldar and Pierre are off17:01
daemontoolvannif < pinfg -s 1380 -f17:01
daemontools/pinfg/ping/17:02
ddieterlyyea, i think that these meetings are critical from an openstack community perspective17:06
daemontoolddieterly, yes did you wanted to discuss anything in particular?17:07
daemontoolanything I can help you with_17:07
daemontool?17:07
ddieterlyjust that problem i'm having ;-)17:07
daemontoola part that error there that needs to be reproduced and debugged?17:07
daemontoolok17:07
ddieterlybut i'm not the only person interested in freezer17:07
daemontoolddieterly,  I agree, are you ok to be the backup chairman?17:08
ddieterlysure17:08
ddieterlyone question i do have is this17:08
ddieterlyhas freezer been perf tested or load tested anywhere?17:08
ddieterlyis there an env where integration, perf, and load testing can be performed?17:09
ddieterlyi'm assuming this would be inside hpe17:09
m3m0hey guys vannif is around somewhere, we are having very busy schedules17:09
ddieterlyis anyone outside of hpe using freezer?17:09
daemontoolddieterly,  I did that in the hp public cloud17:09
ddieterlyis it a permanent env that is reusable?17:10
daemontoollike I think17:10
m3m0and I'll be out of the office next week, so ddieterly you will have to be the chairman next week17:10
daemontoolddieterly,  no, the hp public cloud does not exist anymore :)17:10
daemontoolbut at least in the last 6 months17:10
daemontoolnothing about that was done17:10
daemontoolthere was a performance test that was done few monts ago17:10
daemontoolmonths ago17:10
daemontoolit should be there in the hp Jira17:11
daemontoolit wasn't published as it was specific to Helion17:11
daemontoolthereis also a page called automation_backup in the hpcloud wiki with some other peformance report17:12
daemontoolbut I think that tests should be done now with Rally17:12
daemontoolthat's why I'm improving our dsvm gate job17:12
ddieterlyok17:12
daemontoolbut like high load tests are a different thing17:12
ddieterlyanyone outside of hpe using freezer?17:13
daemontoolI know someone from groupon was using it to backup mongodb17:13
daemontoolthere's a roadmap to use it in Ericsson17:13
daemontooland couple of other Telcos are using it17:13
ddieterlyhow many devs do you plan to have at ericsson working on freezer?17:14
daemontoolI don't know17:14
daemontoolit depends on the business17:14
daemontoolas in any consultancy company17:14
ddieterlyare you wanting to remain ptl while your at ericsson?17:15
daemontooluntil April yes17:15
daemontoolit's not about the company anyway17:15
daemontoolbeing ptl17:15
daemontoolis about your contribution to the project17:16
daemontoolin April there are the votation for ptl in openstack17:16
ddieterlyfreezer is kind of your baby17:16
daemontoolon most of the services17:16
ddieterlyi thought that you might want to remain ptl17:16
daemontoolddieterly,  I was thinking about not propose myself for ptl once more17:17
daemontoolto give to others the opportunity17:17
daemontoolbut the thing is, that the work needs to be done :)17:17
daemontoolit's about doing the work and make things better, not being ptl17:17
ddieterlydo any of the ps guys have an interest in being ptl?17:17
daemontoolbeing ptl is a consequence17:17
daemontoolddieterly,   I think is better if you ask to them directly as you work on the same environmebnt17:18
daemontoolbut for sure17:18
daemontoolyou need to be core contrib to apply as ptl17:18
daemontoolotherwise it doesn't make sense17:18
daemontoolthat's the same for any os project17:19
daemontoolany core can apply for it17:19
daemontoolit's a self-application17:20
daemontoolddieterly,  I have some question too17:20
ddieterlyyea, just wondering what the interest level in contributing is from the current team members17:21
daemontoolif possible :)17:21
ddieterlywondering if people want to be more or less involved17:21
ddieterlywondering where the project is heading17:21
daemontoolI think it's a balance17:21
daemontoolof benefit for community17:21
ddieterlysure, ask away17:21
daemontooland for the business17:21
daemontoolso how many people from hp do you plan to allocate for freezer?17:22
ddieterlywell, just the team that is currently working on it so far17:22
ddieterlyand me ;-)17:22
daemontoolbut that needs to be driven by the business17:22
daemontoolso if for hp17:22
daemontoolthat make sense from the business perspective17:22
daemontoolpeople can be allocated17:22
daemontoollike for any other company17:23
daemontoolI think engineers wants to work 100% of their time on it17:23
ddieterlythere are no plans at this time to increase head count for freezer17:23
daemontoolddieterly,  it would be good to have at least the current one full time on it17:23
ddieterlyi'm not sure what the attitude of the ps guys is17:23
daemontools/one/ones/17:23
daemontoolwell that's an internal thing you need to solve internally17:24
ddieterlydo they want to work on it, or are they told to work on it17:24
ddieterlyyes, i am the current full-time guy17:24
daemontoolddieterly,  there's only one way to know it17:24
daemontool:)17:24
ddieterlyi have just parachuted in17:24
ddieterlylucky me ;-)17:24
daemontoolI think you are lucky17:24
daemontoolbecause you can work 100% of your time on upstream17:25
daemontoolpeople are not that lucky17:25
daemontooland I think that's the issue the ps is having right now17:25
ddieterlyyes, but i have to leave my beloved monasca17:26
daemontoolwell that is sad, I'm sorry17:26
daemontoolreally17:26
daemontoolbut that's an issue between you and your manager unfortunately17:26
ddieterly:-(17:26
daemontoolsorry17:27
ddieterlyjust kidding17:27
daemontoolso17:27
daemontoolwhat I think would be good17:27
daemontoolis that you organize some code walkthrough session17:27
daemontoolwith reldan17:27
daemontooland vannif17:27
daemontoolthat helps a lot17:27
ddieterlyinteresting idea17:27
daemontoolwe did that in the past17:28
daemontooland the outcome was good17:28
daemontoolfor the web ui m3m0  can hep you17:28
daemontools/hep/help/17:28
daemontoolwhat you like more?17:28
daemontoolwriting code for the api?17:29
daemontoolweb ui?17:29
daemontoolor the agent17:29
daemontoolfreezer under some aspect is not that different than monasca17:29
ddieterlyi do not do ui work ;-)17:29
daemontoolok17:30
ddieterlyonly if i have to17:30
daemontoolweb ui I think we are good17:30
daemontoolso17:30
daemontoolone thing I was working on17:30
daemontooland I could finish17:30
daemontoolis the block based incremental17:30
daemontoolbased on rsync17:30
daemontool(the rsync algorithm not the tool)17:31
daemontoolit's a though bone17:31
daemontoolare you interested working on it_17:31
daemontool?17:31
daemontoolthat is needed17:31
ddieterlysure, but anything i work on has to be in alignment with hpe plans17:31
ddieterlyi'm not free to decide17:32
daemontoolI agree17:32
m3m0sudo ethtool -i eth017:32
daemontoolI think that make sense for hp17:32
m3m0sorry wrong windows17:32
daemontoolm3m0,  sure17:32
daemontoolhaha17:32
daemontoolso17:32
daemontoolblock based backup allows openstack17:32
daemontoolto retain data for longer time17:32
ddieterlyso, am i officially the back up chair now?17:32
daemontooluse less band17:32
daemontoolddieterly, I think so yes17:33
ddieterlydo we need to vote or something like that?17:33
ddieterlyoh, question17:33
daemontoolnope I'm going to add it to the wiki17:33
ddieterlywas the logging level left out of the documentation on purpose or was that just an oversight?17:33
ddieterlysee https://review.openstack.org/#/c/266552/17:34
ddieterlyother question, why was elasticsearch chosen as db?17:35
ddieterlywhat was the compelling reason(s)17:35
daemontoolI think is wan't added to the README when the feature was implemented17:35
daemontooloversight17:35
daemontoolthere were 2 reason17:36
ddieterlyk17:36
daemontool1) the data generated was always json17:36
daemontoolso there's no additional computation required, just  the API recieve the data from the agent/scheduler17:36
daemontooland store it to the es17:36
daemontoolcause the data we manage is json mainly17:37
ddieterlyi see17:37
daemontool2) with es all the replica, ha, sharding mechanism17:37
daemontoolis done automatically17:37
ddieterlynice17:37
daemontoolyou do not need to set it up like in mysql17:37
daemontoolso lower the operational costs17:37
daemontoolbut17:37
ddieterlyis anyone else using pycharm for development?17:37
daemontoolyes all of us17:37
ddieterlywith virtual envs?17:38
daemontoolnope17:38
*** samuelBartel has quit IRC17:38
daemontoolI use a new vm with sshfs17:38
daemontoolfor most of the things17:38
daemontoolbut yes17:38
daemontoolalso many other times I use17:38
daemontoolpycharm with virtualenvs17:38
ddieterlyok17:38
daemontoolddieterly,  re db17:39
daemontoolbut after a bit17:39
daemontoolwe thought it was good to have also another backend as db17:39
daemontoollike mysql17:39
daemontoolbut no one implemented it yes17:39
daemontoolif you want to do it17:39
daemontoolor you are interested17:39
daemontoolalso that's another thing to do17:40
ddieterlyno shortage of things to do17:40
daemontoolnot at all17:40
daemontool;)17:40
ddieterlyright now, hpe is eager to get elasticsearch backed up17:40
daemontoolok17:41
daemontoolyou have lvm in helion17:41
daemontoolso you can do the backup17:42
daemontoolthe best person to ask for that is Slashme  but currently he's off too17:42
ddieterlythere is a swift plugin for es17:42
daemontoolok17:42
daemontoolyou don't need to backup an individual index right?17:42
ddieterlyright17:43
ddieterlyhttps://github.com/wikimedia/search-repository-swift17:43
daemontooljust the whole thing?17:43
ddieterlyyes17:43
daemontoolthen use the freezer-agent with lvm snapshots17:43
daemontooland that should work17:43
ddieterlyi am trying to get this plugin to work17:43
daemontoolok17:43
ddieterlydoes that snapshot and the backup17:43
daemontoolyes17:44
daemontoollvm snapshot17:44
daemontoolbackup of let's say /var/lib/elasticsearch17:44
daemontoolremove the snapshot17:44
daemontooldone17:44
daemontoolthe problem is the space17:44
ddieterlythe plugin does the snapshot and the backup17:44
daemontoolah ok17:44
daemontoolthen use that if you feel more comfortable17:45
ddieterlyif we can get it to work, then it will be best choice for now17:45
daemontoolok17:45
daemontoolwhy do you think is the best choice?17:45
daemontoolit can do incrementals?17:45
daemontoolif it can do incrementals then yes17:48
daemontoolprobably is17:48
daemontoolbut I think it depends also on the dataset size17:49
*** arunb has joined #openstack-freezer17:49
ddieterlyyes, it does incrementals17:52
daemontoolI'd check how it perform on the dataset you need to backup17:53
ddieterlyyea, that's why i'm interested in scale and perf testing17:55
daemontoolthe bottleneck there would be es itself17:56
daemontoolas you are extracting the data using es17:56
daemontoolwhile freezer backup the files on the fs17:56
ddieterlythe plugin puts the backup directly into swift17:57
daemontoolso you need to make an http request to an endpoint and it will trigger the backup?17:58
daemontoolI mean to an es endpoint17:59
daemontoolor does it follow its own schedule?18:00
daemontoolone thing to keep in mind is that putting that load on es for the backup, will affect the logging service itself18:04
openstackgerritFausto Marzi proposed openstack/freezer: Add check for name=='__main__'  https://review.openstack.org/26655218:09
daemontoolddieterly,  sorry18:09
daemontoolmy bad there18:09
openstackgerritFausto Marzi proposed openstack/freezer: Add check for name=='__main__'  https://review.openstack.org/26655218:11
daemontoolplease disregard patchset 8 there18:11
daemontooldid a git fetch and check out from the wrong commit sorry18:12
ddieterlynp18:12
ddieterlydoes the horizion ui have a way to download the back up files?18:13
ddieterlyi can download the metadata file, but the actual backup does not seem to have a way to download it thru the ui18:13
ddieterlyah, i see, you have go deep in the tree to get to the actual file18:15
ddieterlydaemontool: i was able to backup a file to swift using freezerc, then download it with the horizon ui, then untar it and look at it18:17
daemontoolok18:17
ddieterlyso, the file seems good in swift18:17
daemontoolwell you can download it with swift cli18:18
daemontoolyes18:18
daemontoolso the problem is definetly not tar or gzip18:18
daemontoolbut I need to reproduce18:18
daemontoolthe issue18:18
ddieterlyk18:19
*** arunb has quit IRC18:20
openstackgerritFausto Marzi proposed openstack/freezer-web-ui: Align requirements to liberty global-requirements  https://review.openstack.org/24698118:25
*** daemontool has quit IRC18:35
ddieterlydoes freezer api require a specific version of elasticsearch18:52
ddieterlyfreezer api devstack uses 1.4.218:53
ddieterlycurrent version of elastic search is 2.1.118:53
*** daemontool has joined #openstack-freezer19:42
daemontoolddieterly, I'm here available for ~1 hr if you need anything19:50
openstackgerritFausto Marzi proposed openstack/freezer-web-ui: Align requirements to liberty global-requirements  https://review.openstack.org/24698120:00
*** daemontool has quit IRC20:01
*** daemontool has joined #openstack-freezer20:02
*** daemontool_ has joined #openstack-freezer20:15
*** daemontool has quit IRC20:17
ddieterlydaemontool_: just show your message20:17
ddieterlyyou still there and for how long?20:17
daemontool_20 min20:27
*** daemontool_ has quit IRC20:52
*** daemontool_ has joined #openstack-freezer20:53
*** arunb has joined #openstack-freezer20:53
daemontool_Hi arunb20:57
arunbHi20:57
daemontool_how are you doing :)20:57
arunbI am good. Who is daemontool? I assume its Fausto :)20:57
daemontool_this is Fausto20:57
arunbhow are you20:57
daemontool_yes20:57
daemontool_not too bad20:57
arunbIts late in Italy. Still working?20:58
daemontool_always working20:58
openstackgerritMerged openstack/freezer: Align requirements and version to Mitaka  https://review.openstack.org/26923521:12
openstackgerritMerged openstack/freezer-api: Align requirements and version to Mitaka  https://review.openstack.org/26923721:12
*** ddieterly has quit IRC21:36
openstackgerritMerged openstack/freezer-web-ui: Align requirements to liberty global-requirements  https://review.openstack.org/24698121:56
*** pennerc has quit IRC22:51
daemontool_https://review.openstack.org/#/c/271070/22:55
daemontool_hope we'll make it for mitaka22:55
daemontool_:)22:55

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!