Monday, 2014-03-10

*** kevinconway has quit IRC00:03
*** yogesh has quit IRC00:10
openstackgerritMichael Basnight proposed a change to openstack/trove: Removes XML api from trove  https://review.openstack.org/6833300:16
*** mattgriffin has joined #openstack-trove00:35
openstackgerritAuston McReynolds proposed a change to openstack/trove: Root_on_create per datastore  https://review.openstack.org/7727500:50
openstackgerritAuston McReynolds proposed a change to openstack/trove: Root_on_create per datastore  https://review.openstack.org/7727500:56
*** Barker has joined #openstack-trove01:04
*** ramashri has joined #openstack-trove01:22
*** nosnos has joined #openstack-trove01:37
*** nosnos has quit IRC01:38
*** nosnos has joined #openstack-trove01:40
*** achampion has quit IRC01:45
*** Ranjitha has joined #openstack-trove01:46
*** rhodgin has joined #openstack-trove01:51
openstackgerritA change was merged to openstack/trove: Rename Openstack to OpenStack  https://review.openstack.org/7320901:59
*** ramashri has quit IRC02:00
*** mattgriffin has quit IRC02:04
*** Ranjitha has quit IRC02:05
*** haomaiwa_ has joined #openstack-trove02:06
*** erkules_ has joined #openstack-trove02:06
*** haomaiwang has quit IRC02:06
*** erkules has quit IRC02:09
*** erkules_ is now known as erkules02:09
openstackgerritAuston McReynolds proposed a change to openstack/python-troveclient: Adding datastore_version column to instances list table  https://review.openstack.org/7449102:13
*** achampion has joined #openstack-trove02:15
*** haomaiwang has joined #openstack-trove02:41
*** haomaiwa_ has quit IRC02:45
*** Barker has quit IRC03:04
openstackgerritAuston McReynolds proposed a change to openstack/trove: Remove Min/Max for Configuration Group Booleans  https://review.openstack.org/7849903:55
*** grapex_ has joined #openstack-trove04:14
*** grapex_ has quit IRC04:16
*** shivam_ has joined #openstack-trove04:17
*** grapex has quit IRC04:18
*** shivamshukla has quit IRC04:21
*** haomaiwa_ has joined #openstack-trove04:33
*** haomaiwang has quit IRC04:34
*** rhodgin has quit IRC04:36
openstackgerritAuston McReynolds proposed a change to openstack/python-troveclient: Fix JSON In Output for Configuration Groups  https://review.openstack.org/7923504:37
*** atomic77 has joined #openstack-trove04:48
*** saju_m has joined #openstack-trove04:59
*** grapex has joined #openstack-trove05:17
*** grapex has quit IRC05:25
*** nosnos has quit IRC05:39
*** nosnos has joined #openstack-trove05:39
*** saju_m has quit IRC05:54
*** saju_m has joined #openstack-trove05:55
*** atomic77 has quit IRC05:57
*** saju_m has quit IRC06:00
*** saju_m has joined #openstack-trove06:02
*** SushilKM has joined #openstack-trove06:07
*** 7JTAAJM33 has quit IRC06:09
*** grapex has joined #openstack-trove06:17
*** SushilKM has quit IRC06:25
*** grapex has quit IRC06:25
*** SushilKM has joined #openstack-trove06:25
*** osherov has joined #openstack-trove06:28
*** nosnos_ has joined #openstack-trove06:51
*** nosnos has quit IRC06:51
*** yogesh_ has joined #openstack-trove06:53
*** nosnos has joined #openstack-trove06:56
*** nosnos_ has quit IRC06:58
*** osherov has quit IRC07:04
*** nosnos has quit IRC07:05
*** nosnos has joined #openstack-trove07:08
*** nosnos has quit IRC07:12
*** nosnos has joined #openstack-trove07:12
*** grapex has joined #openstack-trove07:18
*** grapex has quit IRC07:22
*** SnowDust has joined #openstack-trove07:33
*** matsuhashi has joined #openstack-trove07:39
openstackgerritAuston McReynolds proposed a change to openstack/trove: Changes Volume Prefix From mysql To datastore  https://review.openstack.org/7182907:40
*** flaper87|afk is now known as flaper8708:14
*** iartarisi has joined #openstack-trove08:17
*** grapex has joined #openstack-trove08:18
*** saju_m has quit IRC08:22
*** grapex has quit IRC08:22
*** tattabbum has joined #openstack-trove08:25
*** saju_m has joined #openstack-trove08:28
*** ihrachys|afk is now known as ihrachys08:28
*** yogesh_ has quit IRC08:35
*** matsuhashi has quit IRC08:50
*** saju_m has quit IRC08:50
*** matsuhashi has joined #openstack-trove08:54
*** saju_m has joined #openstack-trove09:04
*** saju_m has quit IRC09:10
*** saju_m has joined #openstack-trove09:11
*** grapex has joined #openstack-trove09:19
*** grapex has quit IRC09:23
openstackgerritIonut Artarisi proposed a change to openstack/trove: rename and fix the db_wipe command  https://review.openstack.org/7322909:28
openstackgerritSlickNik proposed a change to openstack/python-troveclient: Get rid of XML related trove client bindings  https://review.openstack.org/6999909:34
openstackgerritGiuseppe Galeota proposed a change to openstack/trove: Trove Manual Installation guide  https://review.openstack.org/7860809:37
*** fghaas has joined #openstack-trove10:00
*** SnowDust has quit IRC10:01
fghaasamcrn: I wasn't sure whether you were still up, but seeing as you are, we might as well have our conversation here :)10:11
amcrnwell, i'm just about to hit the hay unfortunately10:12
fghaasno worries. was just about to write this:10:12
fghaasI've already been chatting with some folks about getting this set up on StackForge, and Chris Hoge and Emilien Macchi have offered to help10:13
amcrnah, excellent10:13
fghaasit would be great if you could let us know when you reckon you could push your stuff up on GitHub10:14
fghaasbut feel free to do that in the morning, by all means :)10:14
amcrnwe've got to strip out one or two internal puppet resources, and update it to handle icehouse, but it generally shouldn't be too burdensome10:14
fghaassounds great. I take it you're familiar with the process of setting up a StackForge project? if not, either I or any of the other two guys I mentioned can help10:15
amcrni've never done it personally, so i definitely wouldn't mind if you got the ball rolling on that10:16
fghaasroger that, can do — but we need an initial repo to pull from. :) hence my question of when this will be up on GitHub10:16
amcrnah, you're asking me to put it up on a personal repo as a first step10:17
amcrnalright, give me about a week, i've got some competing interests10:17
amcrnwe can circle back next monday?10:17
fghaaseither personal or one that your organization owns, yes10:18
fghaasor, if some of the commit history is confidential/proprietary, we can also set up an empty repo and you copy your code in10:19
*** grapex has joined #openstack-trove10:19
fghaaswhichever works better for you10:19
amcrnsounds good, let me do some internal housekeeping on the modules and get back to you next week10:20
fghaasthat works for me. I'll write a quick email so everyone who expressed interest is informed, does that work for you?10:20
*** IvanZ has joined #openstack-trove10:20
amcrnthat works10:21
fghaasperfect, thanks!10:22
*** grapex has quit IRC10:23
*** achampio1 has joined #openstack-trove10:32
*** achampion has quit IRC10:32
*** achampio1 has quit IRC10:36
*** achampion has joined #openstack-trove10:40
*** arist has quit IRC10:54
*** arist has joined #openstack-trove10:54
*** saju_m has quit IRC11:05
*** grapex has joined #openstack-trove11:20
tattabbumdenis_makogon: Hi denis. Does your mysql.cloudinit (https://gist.github.com/denismakogon/9410248) execute into a partial pre-build image or into a vanilla ubuntu image (like this: http://cloud-images.ubuntu.com/precise/current/precise-server-cloudimg-amd64-disk1.img)?11:22
*** grapex has quit IRC11:24
*** IvanZ has quit IRC11:34
*** fghaas has quit IRC11:35
*** kevinconway has joined #openstack-trove11:42
*** kevinconway has quit IRC11:45
*** achampion has quit IRC11:59
*** achampion has joined #openstack-trove11:59
*** kevinconway has joined #openstack-trove12:01
*** achampion has joined #openstack-trove12:01
*** kevinconway_ has joined #openstack-trove12:06
*** kevinconway has quit IRC12:06
*** kevinconway_ is now known as kevinconway12:06
*** achampion has quit IRC12:06
*** SnowDust has joined #openstack-trove12:09
*** matsuhashi has quit IRC12:11
*** fghaas has joined #openstack-trove12:11
*** matsuhashi has joined #openstack-trove12:13
*** achampion has joined #openstack-trove12:14
*** radez_g0n3 is now known as radez12:19
*** grapex has joined #openstack-trove12:20
*** grapex has quit IRC12:24
*** SnowDust has quit IRC12:28
*** SushilKM has quit IRC12:37
*** grapex has joined #openstack-trove12:47
*** grapex has quit IRC12:47
*** pdmars has joined #openstack-trove12:51
*** jcru has joined #openstack-trove13:14
*** grapex has joined #openstack-trove13:14
*** rhodgin has joined #openstack-trove13:17
*** grapex has quit IRC13:20
*** grapex has joined #openstack-trove13:21
*** rustlebee is now known as russellb13:26
*** matsuhashi has quit IRC13:32
*** Barker has joined #openstack-trove13:36
*** nosnos has quit IRC13:38
*** iartarisi has quit IRC13:52
*** iartarisi has joined #openstack-trove13:55
*** freyes has quit IRC14:16
*** mattgriffin has joined #openstack-trove14:32
*** rwsu has joined #openstack-trove14:36
*** iartaris` has joined #openstack-trove14:48
*** juice has quit IRC14:49
*** iartarisi has quit IRC14:49
cp16netamcrn: you around?14:49
fghaascp16net: I spoke with him earlier at about 2am his local time and he was just about to hit the hay — as that was only 4 hours ago, I suspect he's fast asleep now :)14:49
cp16nethah14:49
cp16netok14:49
cp16netthx14:49
*** juice has joined #openstack-trove14:51
*** iartaris` is now known as iartarisi`14:51
*** iartarisi` has quit IRC14:52
*** iartarisi` has joined #openstack-trove14:52
openstackgerritJenkins proposed a change to openstack/trove: Updated from global requirements  https://review.openstack.org/7670514:53
*** freyes has joined #openstack-trove14:53
*** demorris has joined #openstack-trove14:55
*** iartarisi` is now known as iartarisi14:57
*** thedodd has joined #openstack-trove14:58
*** datsun180b has joined #openstack-trove14:59
*** Barker has quit IRC15:24
juiceamcrn: thanks for the behemoth push this weekend to review/approve patches15:43
*** jmontemayor has joined #openstack-trove15:46
*** ViswaV has joined #openstack-trove15:47
*** ViswaV_ has joined #openstack-trove15:49
*** ViswaV_ has quit IRC15:50
*** ViswaV_ has joined #openstack-trove15:50
*** ViswaV has quit IRC15:52
*** jmontemayor has quit IRC15:56
*** Barker has joined #openstack-trove15:59
*** jmontemayor has joined #openstack-trove16:00
*** jmontemayor has quit IRC16:22
*** jmontemayor has joined #openstack-trove16:32
*** fghaas has quit IRC16:33
openstackgerritMat Lowery proposed a change to openstack/trove: Test restore full and restore incremental  https://review.openstack.org/7373616:37
*** harlowja has joined #openstack-trove16:49
*** khyati has joined #openstack-trove17:01
amcrnjuice: no worries, happy monday :)17:03
amcrncp16net: be back in a bit, will ping you17:03
*** amcrn has quit IRC17:04
*** Ranjitha has joined #openstack-trove17:11
*** michael-yu has joined #openstack-trove17:12
*** eghobo has joined #openstack-trove17:23
*** amcrn has joined #openstack-trove17:27
*** Ranjitha has quit IRC17:30
*** SnowDust has joined #openstack-trove17:31
*** demorris has quit IRC17:36
*** yogesh has joined #openstack-trove17:45
*** yidclare has joined #openstack-trove17:46
*** demorris has joined #openstack-trove17:47
*** SnowDust has quit IRC17:47
*** tanisdl has joined #openstack-trove17:55
*** robertmyers has joined #openstack-trove17:57
*** Ranjitha has joined #openstack-trove18:01
*** Ranjitha has quit IRC18:02
*** Ranjitha has joined #openstack-trove18:03
*** fghaas has joined #openstack-trove18:07
grapexo/18:11
grapexSorry, I just like making this emoticon18:12
vipul\o18:12
amcrn\o/18:12
*** SnowDust has joined #openstack-trove18:12
amcrnKicking it off with https://wiki.openstack.org/w/index.php?title=Trove/Blueprints/Trove-v1-MySQL-Replication18:12
* vipul reading18:13
vgnbkro/18:13
SlickNiko/18:13
amcrnthe key point in that blueprint is: "Phase I implementation, targeted for the Juno release, will implement Use Case A core functionality and the MySql datastore replication support. Replication support for other datastores beyond MySQL are beyond the scope of this blueprint and will be covered elsewhere."18:13
yogesho/18:13
vipulamcrn: +118:14
vgnbkrI filled in some of the other sections, but probably got some of the details wrong.   Feel free to point such deficiencies out.18:14
vipulwe shoudl first focus on just MySQL, and let other datastores follow18:14
grapexI wonder, should phase 1 be it's own blueprint? It's it possibly to nest them?18:14
vipulgrapex: I think doug was going to file a separate BP as well18:15
SlickNikI agree with the scope and use-cases mentioned in the bp.18:15
grapexvipul: Ok, I support this then if that's doug's intent18:15
amcrnvipul: i believe this *is* the separate bp18:15
vipulvgnbkr: I think there will be some configuration changes18:15
amcrn"There should be no changes to configuration files required." => this shouldn't be correct, because there was a verbal consensus that the server_id should be removed from configuration-groups and handled by trove to avoid conflicts18:15
vipulamcrn: well i mena on LP18:15
*** iartarisi has quit IRC18:15
vgnbkrgrapex: yes, it could be broken out, but my thought would be to stick with baby steps for now.18:16
amcrnah, well "configuration file" vs. "configuration group"; but anyway, my point above is still valid18:16
abramleyThe intent was to set the stage for the full feature - but then have this blueprint focus on the phase 1 mysql deliverables18:16
SnowDustvipul : focus on mysql means atleast the configs should be pluggable .. so that it may be adopted freely18:16
vipulamcrn grapex SlickNik https://blueprints.launchpad.net/trove/+spec/replication-v118:16
vgnbkramcrn: I didn't realize that server_id was already there - I thought it was to be added, and we agreed not to add it.  my mistake.18:16
vipulSnowDust: Sure, it shoudl be built with the realization that there are other datastores that will support it18:17
vipulCan we talk about Use Case 1?18:17
grapexLet's18:17
yogeshI am sorry if I missed out...has the discussion on container vs join approach done...?18:17
vipul4. When master fails, a slave can be chosen to be promoted to new master, with other slaves switched to follow new master18:17
vgnbkrvipul: Yes, I think the idea is to concentrate on MySQL, but keep the others in the back of our minds.18:17
vipuldoes this mean some promotion API? or some automated way18:17
*** yidclare has quit IRC18:18
vipulyogesh: this is the BP review meeting.. the container vs join isn't this yet18:18
grapexI feel like A4 is phase 218:18
amcrnagreed18:18
yogeshvipul: sure...thanks18:18
grapexOr should be- A1-A3 seems like a good phase 1 so we can get some traction on this18:18
vipulwell i think we should support an API call for slave promotion..18:19
amcrnphase 1 should allow you to break-off a slave so that it becomes an isolated instance18:19
vgnbkrSo is the intention that the user would use root access to do A4 manually?18:19
grapexamcrn vipul: Agreed18:19
SlickNikamcrn: I think that's explicitly called out in 518:19
amcrnah, good catch SlickNik: so yeah, i'd vote to strike 4 from phase 118:19
*** yidclare has joined #openstack-trove18:19
vipulSlickNik amcrn is that the same thing as 'promote a slave' via API ?18:20
vipul5 that is18:20
amcrnvipul: well, rds uses "promote" in this context, so yes18:20
abramleyso it sounds like move #4 out of the phase 1 deliverables18:21
vgnbkrSo we're agreed on A6 - any existing site, even with data, can be made into a master?18:21
SlickNikvipul: depends on the definition of "promote" I guess. It's RDS' definition, but not what 4 lays down as the definition of "promote" (since that entails switching other slaves over as well).18:21
amcrnif "promote" means that a slave becomes a standalone instance, then yes, we should support a promote operation for a slave18:22
vgnbkrA4 is different that RDS' 'promote'.  The latter is like A5.18:22
vipulSo to make things simpler.. if we enforced a replication depth initially == 118:22
abramleyI think we would want #5 and #6 in order to transition sites into and out of replication18:22
vgnbkrvipul: by which you mean no slaves of slaves?18:22
vipulvgnbkr: yes, then 4 isn't an issue correct?18:23
vipulor is this not a depth issue18:23
SlickNikI don't think it's a depth issue.18:23
amcrn+1 SlickNik18:23
vgnbkrA4 is still an issue, just a nomenclature issue.  It can be moved to phase II, but that means admins will need to do that manually.18:23
cp16netwhat up party people18:23
vipulok yea i think i understand18:24
SlickNikIt's talking about switching other slaves that were replicating from the busted master to now replicate from the promoted slave.18:24
amcrnvgnbkr: correct18:24
vipulSlickNik: so if we implemented only 5, the other slave would potentially be just dangling18:24
grapexRequest to cores: can we move talk of the guest agent upgrade to another time? I should be free 1:30 onward PST.18:25
SlickNikYes, the user would have to manually switch slaves over.18:25
vipulgot it18:25
vgnbkrA5 is detaching a slave - all the other slaves would continue to work.18:25
vipulvgnbkr: ok makes sense..18:25
vipul7. All slaves should be in the same zone. (is this necessary or desired?)18:26
vgnbkrIf the master was detached, or otherwise became unavailable, the slaves would be left dangling.18:26
amcrnfor #7, that just means trove remains as-is in respect to the availability_zone behavior, no changes correct?18:26
vgnbkrI take it we should disallow the master to be 'detached'/promoted?18:26
amcrnvgnbkr: yes18:26
vipulamcrn: I think we want to ideally place them on differnt AZs if multiple AZs are available18:27
SlickNikI don't think we care about 7.18:27
amcrnvipul: correct, but that should work as-is today18:27
*** tattabbum has quit IRC18:27
amcrnassuming you have multiple az's in the same nova deployment18:27
vgnbkrMultiple zones is important, but the bp is suggesting that that is a different use case.18:27
grapexSlickNik: Agreed18:27
vipulamcrn: sure.. for V1, i think we need to look at actually dong that scheduleing for the user in future revs18:27
amcrnvipul: totally agree18:28
vgnbkrI don't think the intention is to do anything to disallow multi AZ deployment, just that it's not a requirement for that use case.18:29
SlickNikSo what I'm hearing is that we're good with all points except 4, and 7 (we might address these incrementally in a future bp, but not for v1)18:29
*** denis_makogon_ has joined #openstack-trove18:29
vipulSlickNik: +118:29
amcrn+118:29
vgnbkrSo you are saying that multi AZ deployment is a requirement for phase I?18:29
vipulvgnbkr: so Trove alread supports specifying the AZ in a create call, we'd just leave it as-is18:30
vipulvgnbkr: if the user specified separate AZs for the two instances, then they get that.. if they don't.. they dont18:30
vipulso do we vote ?18:31
vgnbkrvipul: OK, but could there potentially be issues in a multi AZ deployment.  If that's a requirement, then we would need to address that.18:31
grapexSorry guys, I've got to go. Something's come up.18:31
abramleyamcrn - previously we talked about putting status of replication into the /topology requiest - do you think we need that in this phase and to specifically call it out here ?18:32
vipulgrapex please yay or nay before you go :)18:32
*** grapex has quit IRC18:32
amcrnabramley: the slight variations of schema are tbd, but the general approach is the same18:32
vgnbkrabramley: I think that's an implementation issue for the spec doc.18:32
SlickNikgrapex: no worries, we'll go through the bps and fill you in when you get back. :)18:32
amcrnone thing i don't see addressed is the level of visibility of the replication to (1) the user and/or (2) the admin18:33
amcrnso, should mgmt-show have information about replication-lag18:33
cp16netvipul: grapex said yay if not accepting A4 and A7 now18:33
abramleyamcrn - that is sort of where I was heading18:33
*** denis_makogon has quit IRC18:33
*** denis_makogon_ is now known as denis_makogon18:33
vipulcp16net: cool18:33
amcrnor should the user be able to see it? or should the user see some general level of "health" of replication in phase 1?18:33
denis_makogono/18:34
vgnbkramcrn: I thought we had discussed that and the answer was 'No'18:34
*** dmakogon_ has joined #openstack-trove18:34
amcrnvgnbkr: it's not recorded in the blueprint, so i'm confirming with everyone here ;)18:34
vipulwell ideally we would push up the replication status via guest agent heartbeats.. and use that info to affect the 'state' of the instance18:34
vgnbkramcrn: no worries - personally, I think there should be some indication, but the group seemed to think not.18:35
vipulso it might say something like 'degraded', not necessarily show you the details18:35
abramleyvgnbkr - I agree - some support for state / heartbeat seems desirable in phase 118:36
vipulor if it's broken.. then indicate so18:36
vgnbkrvipul: should that be in the bp or the spec?18:36
amcrnit's a high-level user-facing change, so some mention of it should be in the bp (imho)18:36
vipuli don't personally think we ned that level of detail in BP.. maybe jsut call it out as a 1 liner18:36
SlickNikvipul: +1.18:36
vgnbkrvipul: will do.18:36
vipul+2 approved18:37
SlickNikI've added the comments (in italics) in the bp at https://wiki.openstack.org/wiki/Trove/Blueprints/Trove-v1-MySQL-Replication#Use_Case_Requirements18:38
SlickNikI'm good with approving the bp, based on this.18:38
vgnbkrSlickNik: thanks.18:38
*** fghaas has left #openstack-trove18:38
vgnbkrSlickNik: (for the edits)18:39
abramleySlickNik - we will do a quick update to the doc and then move forward18:39
cp16netnice18:39
SlickNikabramley/vgnbkr: Feel free to reword them as you see fit.18:39
SlickNikokay moving on?18:39
SlickNikamcrn?18:40
amcrnlet's do it18:40
amcrnsince we're deferring on discussing guest-agent until tim comes back18:40
amcrnhttps://blueprints.launchpad.net/trove/+spec/dbinstance-log ?18:41
amcrnhttps://wiki.openstack.org/wiki/Trove/DBInstanceLogOperation18:41
denis_makogoni'm in18:41
denis_makogonsome pre-history18:41
amcrna long time ago, in a galaxy far away18:42
denis_makogonRDS has the same feature18:42
denis_makogonsince Trove is a Database as A service18:42
denis_makogonand there are no possible ways to log in to the instance18:42
denis_makogonand no way to take a look at database log files18:43
denis_makogondirectly18:43
denis_makogonmy suggestion was to give to the end-user and ability to pull logs from instance through Swift18:43
denis_makogoniteration one was implemented18:44
denis_makogonlinks you could see inside the BP18:44
denis_makogonnext iteration will give an ability to retrieve N lines of given log file18:44
denis_makogonit's something like polling18:45
denis_makogonthe question is how to store or not store retrieved lines from log18:45
denis_makogonin terms of iteration 218:46
amcrnwell, we can defer that conversation when phase 2's bp presents itself18:46
amcrnlet's focus on phase 118:46
denis_makogonsounds reasonable18:46
denis_makogonok18:47
*** freyes has quit IRC18:47
denis_makogonany questions about Phase 1 ?18:47
amcrndoes someone have any high-level questions about phase 1, otherwise i'll kick it off with a few18:47
vgnbkrCan I ask a point of order?  Is discussion open to everyone, or just core and stakeholders?18:47
SlickNikeveryone18:47
SlickNikgo for it vgnbkr18:47
vgnbkrOk, thanks.18:47
vgnbkrSo if the logs go into swift, how are they managed?  Who deletes them?18:48
denis_makogonvgnbkr, we don't manage them18:48
denis_makogonvgnbkr, only user responsible for those files18:49
vgnbkrSo why not just download the files to the user's browser, then?  Save blowing out the swift repository with temporal data.18:50
denis_makogonvgnbkr, user owns the container that created in Swift, so user can delete them18:50
vgnbkrSo what is the reason for putting them is Swift?18:51
vgnbkrs/is/in/18:51
robertmyersvgnbkr: the api sends an rpc message to the guest which performs the actions18:51
denis_makogonvgnbkr, if we're working through Horizon dashboard, you could go to the Containers pannel and download them manually18:51
robertmyersso the api can not respond directly with the content18:52
denis_makogonvgnbkr, Swift is a consistent storage for all possible data across cloud18:52
vgnbkrIt just seems onerous to expect the user to manage their log files.18:52
amcrndenis_makogon: is there a plan to limit the # of log files, as is done with backups?18:53
denis_makogonamcrn, i think no/18:53
denis_makogonamcrn, unless there's huge need in quotas18:54
denis_makogonamcrn, logs are not the backups18:54
amcrnlol, of course they're not18:54
amcrnthe reason i ask is that by having a limit, users are indirectly influenced to prune18:54
denis_makogonamcrn, i mean they have different signification18:54
cp16netdenis_makogon: are the files streamed to swift as they are written to or uploaded each time requested?18:54
denis_makogoncp16net, eachtime18:55
denis_makogoncp16net, they have timestamp in their name18:55
cp16netok that was my next question18:55
denis_makogoncp16net, hope i answered18:56
denis_makogonany other questions ?18:56
amcrndenis_makogon: how will the rotation of the logs be handled across datastores, and how will this be presented to the user?18:56
amcrnexample: user asks for a log, then has no idea why it doesn't include Day X18:57
cp16netor time X18:57
amcrncorrect18:57
cp16netbecause the logs get rotated too quickly..18:57
denis_makogonamcrn, didn't thought about that18:57
SlickNikdenis_makogon: Or does the user have an option to ask for a log file from time X to Y?18:58
denis_makogonamcrn, i guess that if user can enable log rotation, probably he'll know that logs will contain only the part from X to Y18:59
denis_makogonSlickNik, this could be as part of iteration 318:59
amcrndenis_makogon: well, that's not true; i believe mysql auto log rotates right now, and without an api for the user to ask what the rotation schedule is, they have no idea18:59
amcrnso unless we agree on a single rotation schedule for all datastores, then publish this in the docs, you'll have to present *something*19:00
amcrneven then, users asking for "log3.tar.gz" because they've done napkin math to figure out which one has the data they're looking for seems a bit janky19:00
SlickNikdenis_makogon: How does this work with different log locations? (eg. As a provider, I deploy a different my.cnf specifying a different log location)19:00
amcrnSlickNik: good point, was just about to bring that up. in short, i think we'll have to remove the ability to change the log location in configuration-groups19:01
denis_makogonSlickNik, for now it looks like https://review.openstack.org/#/c/64302/27/trove/dbinstance_log/models.py19:02
amcrnyeah, aka remove the ability to change it in the configuration-group19:02
denis_makogonamcrn, yes i thought about that, just wanted to ask it19:02
vgnbkrShould the feature be up-scoped to provide somethnig beyond retrieving log files?  Something like including log data in the heartbeat so that the user can see log entries in delayed-time?19:02
amcrnvgnbkr: not sure i follow19:03
denis_makogonvgnbkr, now its orientied only for logs, and i think that there's no reason for polling something else than logs19:04
vgnbkrOn every heartbeat, include changes to the long since the last heartbeat.  Save a single log for each instance in (say) Swift. User could retrieve that, or watch near-live in Horizon.19:04
vgnbkrs/long/log/19:04
denis_makogonvgnbkr, heartbeat happens each 3 sec19:05
vgnbkrdenis_makogon: yes, I'm just referring to logs.19:05
vgnbkrdenis_makogon: right, hence 'near-live'.19:05
amcrnvgnbkr: by "changes", do you mean the actual incremental content, or a summary of said content, or ?19:05
cp16netsounds like a period task or scheduled task to pump out the logs19:06
denis_makogonvgnbkr, i don't think we need to pull to server side part of log each time heartbeat happens19:06
vgnbkramcrn: Really I'm just wondering if we need something more than just retrieving log files.19:06
denis_makogonvgnbkr, i guess no19:06
vgnbkrdenis_makogon: I was thinking a push, not a pull.19:06
denis_makogonvgnbkr, feature designed to push logs to Swift, instead of Trove server-side19:07
SlickNikvgnbkr: Personally I think that might be a tad overkill. But it really depends on the use-case. denis_makogon: What's the specific use case(s) we're trying to address with this?19:08
amcrnvgnbkr: aren't you really referencing [11:54:58]  <cp16net> denis_makogon: are the files streamed to swift as they are written to or uploaded each time requested?19:08
denis_makogonSlickNik, use-case is an audit19:08
denis_makogonSlickNik, i worked with database security audit19:08
juicedenis_makogon: rather than use swift and a custom log file manager, have you considered using syslog to push the files to a remote server of the user's choice?19:08
denis_makogonSlickNik, but there saw an ability to access the instance via SSH19:09
vgnbkrMy alternate suggestion was really just an idea of the top of my head.  My real point was to suggest that just pulling log files adhoc might not be enough.19:09
juicethey could even setup another vm that is their central log server and then there they can do what they want with them and have full access to the history?19:09
vgnbkrju19:09
vgnbkrjuice: nice idea19:09
cp16neti was just thinking about if you could always have the log files in swift up to date.19:10
amcrnjuice: with swift being an option of a remote-server?19:10
juiceplus you'd get all the headache of log ration taken care of by you19:10
amcrnor in replacement of.19:10
denis_makogonamcrn, + Swift is enough19:10
juiceamcrn: well maybe from the central log server you can set that up yourself or something19:10
juiceamcrn: I was thinking in replacement of mostly19:10
cp16netsomething along the lines of scheduling the log files to append each time the task is fired19:10
juicedb instance 1-100 send your log files to syslog server 10.0.0.101.19:11
juicessh into 10.0.0.101 and go to town19:11
denis_makogoncp16net, we could talk about appending log data inside the iteration 2 or 319:11
cp16netok19:11
juiceputting log files in swift is not going to be THAT convenient for users to manage19:11
SlickNikdenis_makogon: wouldn't you want _all_ the logs in the case of an audit? Will a one-time API call to copy existing logs over work?19:12
robertmyersjuice: syslog is a good idea19:12
SlickNikdenis_makogon: And how are you going to stitch log-files together to get a complete view?19:12
robertmyersjuice: that could work with loggly too19:12
denis_makogonSlickNik, yes it is19:12
denis_makogonSlickNik, audit features allows to analyze set of files19:13
juicerobertmyers: indeed: seems to keep the options open for more robust logging in a complex deployment (ie. not just one vm)19:13
denis_makogonjuice, but for now it's too complicated19:14
robertmyersdenis_makogon: how is syslog too complicated?19:14
juicedenis_makogon: I would disagree - building our own logging manager is going to be much more so19:14
*** tattabbum has joined #openstack-trove19:14
juiceand that wheel has already been invented19:14
denis_makogonjuice, what do you mean under "custom log manager" ?19:15
juicedenis_makogon: perhaps log shipper is a better description19:15
openstackgerritRanjitha Vemula proposed a change to openstack/trove-integration: MySQL 5.6 disk-image-builder element  https://review.openstack.org/7941319:15
denis_makogonjuice, there's no log sniphers inside the trove19:15
juicedenis_makogon: aren't we talking about building one now?19:16
denis_makogonjuice, noone analyze the data19:16
denis_makogonjuice, no19:16
denis_makogonjuice, could you please take a look at BP an WIKI page19:17
SlickNikOkay, I get a feeling we're rat-holing a bit too much on this.19:17
juicedenis_makogon: the basic goal is to get the data off of the db instance since the user does not have access to it.  that has been solved with syslog already19:17
juiceslicknik: you mean rabbit hole19:17
juice:)19:18
juicecarry-on - just putting the idea out there19:18
amcrnin general, i think the real use-case needs to be explained more. taking the "audit" scenario, to SlickNik's point, wouldn't you want all of the logs all of the time? outside of an audit scenario, how often would a user actually request to see a log? without hard answers (or gut feels) for those questions, ascertaining the right approach becomes more difficult.19:19
SlickNiklol, I'm fully on-board with the intent of this bp; but I'm not convinced that the current proposed implementation addresses the use-case it's trying to solve.19:19
denis_makogonjuice, since inside the could we're only relaying only on existing services - the only way for Trove is to store logs to existed storage19:19
amcrnmy gut tells me juice's approach makes more sense19:19
SlickNikI mean, I can see that this implementation may be okay if, say I set some config param that messed something up, and I want to take a quick look at the current logs.19:20
denis_makogonamcrn, we cannot relay to "syslog" server that might exist or not19:21
denis_makogonamcrn, the only way is to use Swift and only19:21
SnowDustdenis_makogon, +119:21
juicedenis_makogon: true, you would need to setup an additional vm or pay for swift storage or both19:21
denis_makogonjuice, Swift is our everything19:22
juicedenis_makogon: swift is definitely a good repository to store old archived log files19:22
SlickNikBut if we're talking of gathering logs for a database security audit , I'm not sure this would be an effectual approach.19:22
amcrnbut current, in-flight logs, not so much19:22
juiceamcrn: exactly - perhaps the last weeks worth19:23
amcrnSlickNik: 100% agree with your assessment, so to borrow a phrase from our friend dougshelley66: what problem are we actually trying to solve19:23
*** ViswaV_ has quit IRC19:23
amcrnso denis_makogon, i think you need to update the blueprint with said information19:23
denis_makogonamcrn, the proble is that user cannot access log files of database that is the backend of his application19:23
openstackgerritRanjitha Vemula proposed a change to openstack/trove-integration: MySQL 5.6 disk-image-builder element  https://review.openstack.org/7941319:24
denis_makogonamcrn, audit is the next approach19:24
denis_makogonamcrn, perfomance tuning is the third approach (only logs could tell how database feels on currect hardware)19:25
amcrni think we can summarize that: the actual use-case(s) being solved for need to be elaborated on, including the expected access patterns (when, how often, etc.) and (2) a unified rotation strategy needs to be considered, or how to present to the user the rotation strategy in place so they can request for a log that's relevant to the timeframe they're interested in19:25
amcrnwithout those, i'm of the opinion we're at an impasse.19:26
SlickNikagreed. amcrn: +1. I'd vote that we defer the bp and revisit it later when we have more clarity around those data points.19:26
SnowDustamcrn, juice, esp : can we do a quick discussion on this https://bugs.launchpad.net/trove/+bug/128876719:27
amcrnnot right now SnowDust, no.19:28
SnowDustk .. will wait19:28
espI'll try to comment on this in a bit SnowDust :)19:28
amcrnare we good to move on to the next blueprint? does someone else believe we need more information than the aforementioned to get clarity on the dblog blueprint?19:29
SnowDustesp: welcome .. need more discussions anyway ;)19:29
denis_makogonamcrn, lets move on19:30
SlickNiksounds good to me.19:30
amcrnhttps://blueprints.launchpad.net/trove/+spec/datastore-type-version-to-backp-model19:30
denis_makogonApproach: avoid wrong restoring19:30
amcrnyeah, aka handle the various problems seen @ https://bugs.launchpad.net/trove/+bug/128587619:31
denis_makogonThe way it'll be implemented: base backup model will be extended with two fields: type and version19:32
SlickNikQuestion: We already have strategy as part of the backup model. Is that insufficient? (if so, I'm trying to understand why)19:33
denis_makogonwe should not relay on _deleted_ instances19:33
SlickNikeg. If your cassandra instance is trying to backup with an innobackup strategy, that would point to a mis-configuration, right?19:33
denis_makogonSlickNik, stragegy could be changed in conf file19:33
denis_makogonSlickNik, but old backup could have old strategy19:34
cp16neti dont see why we need both type and version when you all you really care about is the version19:34
cp16netor can you restore a different version to a type?19:34
cp16neterr the same type to a different version?19:34
denis_makogonSlickNik, example: backup (strategy: MySQLDump), but the guest works only with Xtrabackup stuff19:35
SlickNikdenis_makogon: can you clarify, I don't fully understand that example.19:36
denis_makogonSlickNik, suppose we backuped the instance with xtrabackup strategy19:37
denis_makogonSlickNik, and now we want to restore new one, but trove-api doesnt know anything about possible strategies19:37
denis_makogonSlickNik, same for the trove-taskmanager - it doesn't know is the strategy correct or not19:38
vgnbkrI don't see anything about strategies in the bp?  Is it called something else?19:38
amcrni'm a bit lost as well, i thought this bp was solving something else entirely19:38
SlickNikWhy does trove not know about the strategies? There's a mapping between the datastore and strategy types in the config file.19:39
denis_makogonamcrn, the approach to avoud incorrent provisioning-restoring19:39
denis_makogon*avoud19:39
denis_makogonavoid19:39
amcrnbased on a datastore-version incompatibility, or a strategy incompatibility, or both?19:40
denis_makogonSlickNik, by telling "config file" you mean cfg.py ?19:40
denis_makogonamcrn, i guess datastore-version19:41
SlickNikYes, cfg.py or the sample conf.19:42
denis_makogonSlickNik, we cannot relay on config file since backup_strategy is the config attribute is the part of guest config19:42
denis_makogonSlickNik, i wouldn't suggest to duplicate options through all trove services19:43
SlickNikBut if you store this in the database, wouldn't you be duplicating the info in the models instead?19:44
vgnbkrThe bp says the purpose is to prevent the user from restoring a mysql backup to mongo.  What would be wrong with that?  Or are you just saying to prevent the error?  Is the guest-agent not capable of detecting bad input?19:44
amcrnvgnbkr: as of now, it doesn't detect it and the instance is perma-stuck in BUILD19:44
denis_makogonSlickNik, we're not duplicating data inside trove backend19:45
*** Ranjitha has quit IRC19:45
vgnbkrSo wouldn't it be better to fix the guest agent?19:45
denis_makogonSlickNik, strict validation requires special attribute19:46
*** Ranjitha has joined #openstack-trove19:46
denis_makogonSlickNik, we have two options: 1) version attr in backup mode. 2) backup_strategy - inappropriate since instance provisioning will go deeper19:47
denis_makogonSlickNik, we need to block restore call at API part19:47
SlickNikdenis_makogon: Tell me this. Should I be able to restore a backup taken from mysql (using innobackup) on percona instance (using innobackup)?19:47
vgnbkrIt seems like it would be good to prevent the user from restoring to the wrong db, but there would be cases where the user validly wishes to restore to a different instance.  We would need to allow all of those cases, no?19:47
vgnbkr What SlickNik said :-)19:48
denis_makogonSlickNik, we talked about this, it's very complicated since it would not work from time to time19:49
denis_makogonSlickNik, that is why i suggested to have datastore type and version19:49
vgnbkrdenis_makogon: as long as the user got a reasonable error from the guest-agent, wouldn't that be OK?19:49
denis_makogonSlickNik, user will at least try to create percona instane from mariadb backup19:49
denis_makogonvgnbkr, we should not let the provisioning19:50
SlickNikdenis_makogon: When would it not work?19:50
vgnbkrdenis_makogon: why?19:50
denis_makogonvgnbkr, quotas19:51
SlickNikdenis_makogon: I see what you are saying about error'ing out eagerly in the API call. If we need to expose the mapping in the API conf, for this purpose, I think that's an acceptable solution.19:51
*** Ranjitha has quit IRC19:51
denis_makogonSlickNik, my guess is that it would not work when migrating from one major version to another19:51
denis_makogonSlickNik, i'd better keep trove.conf clean19:52
denis_makogonSlickNik, backup strategies is the part of the guest, not he APU19:52
denis_makogon*trove-api19:52
denis_makogonlets do not mix everything19:53
SlickNikWait so you want to error out eagerly in the API, but not have it be part of the API? :)19:54
denis_makogonSlickNik, yes19:55
denis_makogonSomething like "backup doesn't fit"19:55
SlickNikdenis_makogon: But it does fit. What we're saying is that it might be completely valid to have one datastore backup restore to a _different_ datastore type.19:56
SlickNikThe limiting factor here isn't the datastore type / version, but the strategy, and that's configurable by the operator.19:57
*** decora has joined #openstack-trove19:57
denis_makogonSlickNik, so, you say that it'll be better to have same options inside the api service and the guest19:58
denis_makogoni mean:19:58
SlickNikIf you wanted to move the map of datastore -> strategies to the db so it's not repeated in the conf, we can talk about that.19:58
SlickNikBut that's not what this bp is about.19:58
denis_makogon[datastore] backup strategy= blah-blah19:58
vgnbkrSlickNik: but with time, don't you think there might be multiple strategies per datastore?19:59
SlickNikvgnbkr: That's a good point. We probably need a one-to-many mapping instead of a one-to-one mapping.20:00
*** ViswaV has joined #openstack-trove20:00
denis_makogonSlickNik, i wanted to avoid provisioning with inappropriate backups20:01
SlickNikBut I'm still unconvinced that we need to force restores such that they can only happen on a datastore that has the same type and version as the original backup.20:01
denis_makogonSlickNik, and i'm searching the way how to o thi20:01
denis_makogon*to do this20:01
vgnbkrIf the guest-agent is physically able to restore a dataset, shouldn't we allow it?  In cases where it can't, shouldn't it be robust enough to give a reasonable error and recover itself?20:02
denis_makogonvgnbkr, guest cannot recover itself on restoring20:02
denis_makogonvgnbkr, because "i want server with _this_ data, not just the server"20:03
denis_makogonSlickNik, then lets stick to the strategies20:03
vgnbkrdenis_makogon: I'm simply saying that the guest-agent shouldn't crash or corrupt the database due to detectably bad data sent to it.20:04
denis_makogonvgnbkr, main approach to guest is to stay simple as it possible, so i'd rather block provisioning instead of getting decreased quota20:06
denis_makogonand making guest smart enough to rule the world20:06
SlickNikdenis_makogon: agreed, I think it makes sense sticking to strategies.20:06
SlickNikAs the bp is written now, I don't think it makes sense to approve it.20:07
denis_makogonSlickNik, what if backup model will have datastore_manager ?20:07
denis_makogonnevermind, strategies is enough20:08
denis_makogoni guess20:08
*** ViswaV has quit IRC20:08
*** ViswaV has joined #openstack-trove20:09
vgnbkrdenis_makogon: if `file <fn>` = 'xtradb' is pretty simple.20:09
*** ViswaV_ has joined #openstack-trove20:10
SlickNikOkay, I think that's all we have for bps for now.20:10
SlickNikAs a heads up, we'll chat about the guest upgrade bp once grapex is back. :)20:10
SlickNikI think we need to be a bit more rigid on how much time we spend on each of these bps.20:10
SnowDustok .. seems i have some time here :)20:11
SnowDustmay i ?20:11
amcrnSlickNik: agreed, some sort of timeboxing20:11
SlickNikSo perhaps for the next bp review meeting, I propose a time limit on each bp discussion?20:11
SlickNikamcrn: +120:11
SnowDusthttps://bugs.launchpad.net/trove/+bug/128876720:11
*** ViswaV__ has joined #openstack-trove20:11
amcrnSlickNik: let me think about it a bit more, and hit up you + the channel on some thoughts (on the # of blueprints, how much time per blueprint, avoiding trying to solve implementation specific details)20:12
amcrntime to grab some lunch20:12
SnowDust:D20:12
denis_makogonthere's another one20:12
SnowDusthttps://bugs.launchpad.net/trove/+bug/128876720:12
SlickNikSnowDust: yes, I'm reading it.20:13
*** Ranjitha has joined #openstack-trove20:13
*** mattgriffin has quit IRC20:13
*** ViswaV has quit IRC20:13
SlickNikDid you have a question?20:13
amcrnSnowDust: feel free to talk about your bug right now, i'll re-read the chat logs after i stuff my belly20:13
SnowDustSlickNik: tx !20:13
SnowDustso .. the reviewers are different on the commit20:13
SnowDust1st solution proposed in the bug details is favored by amcrn20:14
SnowDustthe last patchset ( approach) is after discussion with juice, esp20:14
*** ViswaV_ has quit IRC20:14
SnowDustbut juice commented on more enhancement20:14
SnowDustso i need to get to a resolution on this .. acceptable by all :)20:15
denis_makogonSnowDust, again, i don't see the problem of extending you custom trove patch with another oslo group20:16
juicedenis_makogon: do you mean a config group?20:16
denis_makogonjuice, yes20:16
yogeshdnis_makogon: i think having a fallback will make things generic from trove perspective20:17
denis_makogonjuice, SnowDust 's caught the problem of the custom trove20:17
yogeshdenis_makogon: i think having a fallback will make things generic from trove perspective20:17
espSnowDust: I think the underlying problem is that we expect to have a configuration value set (datastore.mount_point) but we did not make it clear anywhere.20:17
SnowDustuntil the implementation is in cfg.py there is no way to keep running an experimental datastore20:18
denis_makogonyogesh, fallbacks is not required, please patch you taskmanager20:18
denis_makogonSnowDust, write the new oslo group20:18
denis_makogoneasy as hell20:18
yogeshconceptually20:18
SnowDust[DEFAULT] group in conf  was so easy .. to keep this working20:18
juiceI thought someone shot down the default20:19
SnowDustjuice, u r right !20:19
yogeshif there is something required by each of the datastores, then why to have it per group, it can be within the default group20:19
denis_makogonyogesh, no, again, no, because we have N datastores that have their own mount points20:20
denis_makogonso, we don't need one mount point in whole trove deployments20:20
SnowDustdenis_makogon, cfg.py binding is a big NO too !20:20
yogeshand do we have more than one datastores available from a single api?20:20
denis_makogonyogesh, yes20:20
denis_makogonyogesh, cassandra, mongo, redis, couchbase20:21
espSnowDust: if you supply the correct configuration in your taskmgr.conf does this get you around the issue?20:21
*** grapex has joined #openstack-trove20:21
denis_makogonesp, oslo group need to be registered20:21
SnowDustesp, no .. it will only take the route to the coded datastore groups in cfg.py20:21
esphmm... ok20:21
SnowDustuntil ur experimental datastore group is part of config group in cfg.py .. u are not entertained20:22
yogeshdeni_makogon: what if 3 out of these 4 have the same mount poin config value20:22
espah ok I understand now20:22
yogeshi mean, why not get it handled like a base class20:22
juiceis there any enforcement to ensure that every custom group has this setting?20:22
denis_makogonyogesh, take more precise look20:22
yogeshdenis_makogon: yeah sure, just trying to understand batter...20:22
denis_makogonyogesh, https://github.com/openstack/trove/blob/master/trove/common/cfg.py#L286-L28820:23
denis_makogonyogesh, https://github.com/openstack/trove/blob/master/trove/common/cfg.py#L326-L32820:23
denis_makogonyogesh, https://github.com/openstack/trove/blob/master/trove/common/cfg.py#L346-L34820:23
denis_makogonyogesh, https://github.com/openstack/trove/blob/master/trove/common/cfg.py#L368-L37020:23
denis_makogonyogesh, https://github.com/openstack/trove/blob/master/trove/common/cfg.py#L388-L39020:24
espI wish I would have looked at the cfg.py changes earlier, sorry about that.20:24
*** mattgriffin has joined #openstack-trove20:24
SnowDustesp, any difference after u saw  :)20:24
denis_makogonSnowDust, register new oslo group and be happy20:24
denis_makogonSnowDust, that is so easy20:24
SnowDustdenis_makogon,  give me a solution without touching the trove checkout20:25
espto me it seems like it would have been nice to abstract the OptGroup stuff out into the property file itself rather than cluttering up the code with every datastore we might support one day.20:25
yogeshdenis_makogon: thanks for the pointers...20:25
SnowDusti had that working .. when it was config driven [DEFAULT] group option kept that running for me20:25
espnot sure if oslo config supports this tho20:25
SnowDustesp, thats the problem20:26
yogeshi am just trying to think the harm or design-flaw the default config is causing20:26
denis_makogonesp, oslo.config cannot do this20:26
SnowDusteven if u define your custom groups .. they need to be loaded by a change in cfg.py20:26
denis_makogonesp, this is how paste-deploy works20:26
yogeshit is keeping this reusable20:26
espSnowDust: and everyone, how do yo feel about allowing for a generic datastore Group in the cfg.py for the purpose of testing experimental datastores for now?20:26
*** decora has quit IRC20:26
amcrnunless you polymorphically loaded subheaders by requiring some sort of prefix20:27
SnowDustesp, no harm .. i am back running with that20:27
denis_makogonSnowDust, you're patching guestagent, then patch the trove20:27
SnowDustdenis_makogon, i dont patch20:27
SnowDusti load it as a paste filter20:27
espdenis_makogon: that but that's kind of gross to have to mess with the api-paste (if that is what you mean)20:27
SnowDustbulls eye :)20:27
denis_makogonSnowDust, you wrote your own manager i guess20:27
SnowDustthe way trove was designed20:28
SlickNikamcrn: Yeah, I think that's the gist of the problem.20:28
denis_makogonesp, oslo.config works over paste deploy lib20:28
amcrnSlickNik: should be doable, although it has downsides20:28
*** grapex has quit IRC20:29
yogeshdenis_makogon: what problem are we solving taking the setting out from default20:29
kevinconwayesp: wouldn't those simply be API/Acceptance tests that we put in tempest?20:29
*** ViswaV__ has quit IRC20:29
openstackgerritMat Lowery proposed a change to openstack/trove: Get service endpoints from catalog  https://review.openstack.org/6801520:29
*** grapex has joined #openstack-trove20:29
*** ViswaV has joined #openstack-trove20:29
amcrnif we go back to the specific problem at hand: every single datastore should have mount_point defined in its config-opts, and relying on a "default" to handle the scenario in which an operator does something silly, is in my opinion, overreaching.20:29
SnowDustamcrn: ok20:30
espkevinconway: sorta I think SnowDust wants to test a new datastore in a live env.  I could be wrong though.20:30
amcrnall of the current datastores have a defaulted value for mount_point in cfg.py20:30
*** ViswaV has quit IRC20:30
amcrnesp: right, but let's make sure we're not clumping all of these requirements together20:30
denis_makogonyogesh, we're dealing with the problem of specific parameters that are differenet fro all datastores20:30
amcrnso, to say that there's an issue with the current set of datastores as it relates to mount_point is incorrect20:31
kevinconwayesp: i mean in terms of generic tests, some thing simply executes api calls without caring about what's inside20:31
*** ViswaV has joined #openstack-trove20:31
espamcrn: k20:31
yogeshdenis_makogon: but then how about abstraction of the ones which are common?20:31
denis_makogonyogesh, they are already in default section20:31
amcrnnow if we're talking about how do you support another experimental datastore without checking out cfg.py: how are you adding a custom manager or custom logic without checkout out the code?20:31
amcrnyou can't, so i fail to see a real life scenario that requires you modifying cfg.py and nothing else20:32
espkevinconway: yeah dude, I think you are correct I was thinking of a way to allow for kind of dump datastore group that would help bypass the problem.20:32
SnowDustamcrn: conf modifications .. is the only change20:32
SnowDustamcrn: no *.py change20:32
amcrnSnowDust: what's the specific, real life example20:32
denis_makogonamcrn, ++ to impossible scenarious20:32
yogeshmount_point is needed for all datastores20:32
openstackgerritMat Lowery proposed a change to openstack/trove: Test restore full and restore incremental  https://review.openstack.org/7373620:32
amcrnyogesh: yes, it's set already as a default in cfg.py20:32
denis_makogonyogesh, we already have mount point for all supported datastores20:33
amcrni'm asking SnowDust for a real scenario in which he needs to modify cfg.py to support an experimental datastore, and absolutely modify no other .py files20:33
denis_makogonamcrn, i curious to, SnowDust please tell20:33
SnowDust?20:33
SnowDustreal scenario is mysql_foo20:33
SnowDustthats configured as default_datastore in trove.conf20:33
denis_makogonwhat is "mysql_foo"20:34
denis_makogon?20:34
SnowDustdenis_makogon which u never tried20:34
denis_makogonis it datastore manager ?20:34
SnowDustdenis_makogon, yes20:34
amcrnSnowDust: let's avoid abstractions, I'd like to hear a real life example20:34
SnowDustamcrn ?20:34
amcrnMySQL 5.6 vs. 5.5?20:34
denis_makogonSnowDust, then you need to have mysql/mysql_foo.py20:34
SnowDustamcrn its mysql_foo version 1.120:34
openstackgerritAndrew Bramley proposed a change to openstack/trove: Improve Datastore Not Implemented exceptions and remove traceback from messages  https://review.openstack.org/7805420:34
amcrnmysql_foo isn't based in reality, please explain the real problem you're trying to solve20:35
SnowDustamcrn as i said20:35
amcrnwhy do you need mysql_foo, what changed in the manager20:35
SnowDustthe datastore has the guestagent manager  written in a separate package beautifully loaded by the guest.py20:35
SnowDustbut .. fails to function as cfg.py has no config group for mysql_foo20:35
denis_makogonSnowDust, you need to keep in the codebase mysql_foo manager20:36
amcrni'm aware of your abstract; explain to me a real life example20:36
SnowDustdenis_makogon, i will never do that .. i dont want to touch trove code ( due to license)20:36
yogeshSnowdust: is it only about mount_point or other setting as well?20:36
SnowDustbut i will keep developing my extension and have right to make profit .. :)20:36
SnowDustyogesh, bare minimum mount point20:37
yogeshSnowdust: why can't it just use it from default settings?20:37
SnowDustyogesh: there is _no_ default mount point setting now20:37
SnowDustthats the point of bug20:37
amcrnas there shouldn't be20:37
espyogesh: I was under the impression that the code that is there now isn't making use of the default settings20:37
espamcrn: yeah it was removed during the refactor/intro of datastores20:38
*** ViswaV has quit IRC20:38
*** ramashri has joined #openstack-trove20:38
espand a default that would makes sense for all datastores probably doesn't exist20:38
SnowDustesp, yes !20:39
SnowDustthats my point20:39
denis_makogonSnowDust, you are not able to add new datastore manager by changing only the conf file =)20:39
denis_makogonSnowDust, send you config files and actual code to your github acc20:39
amcrnhere let me do it for you SnowDust: I need mysql_foo as a new manager, because I need to be able to deploy MySQL vanilla with the manager as 'mysql', and then mysql_foo for my Facebook fork of MySQL that requires very different configuration values. By using CONF.datastore_registry_ext, i'm able to point mysql_foo to trove.guestagent.datastore.mysql.manager.Manager, meaning I have no other .py file changes.20:40
SnowDustamcrn: slight modifications20:40
amcrnthat's the only scenario i can cook up that would necessitate your requirements20:40
amcrnwhich would have the cfg.py problem20:41
SnowDustmysql_foo is loaded from  mysql_foo: mysql_foo.abc.def.ghi.manager.Manager20:41
SnowDusta separate entity20:41
yogeshesp: isee20:41
SnowDustamcrn: i said the same thing20:41
amcrnthat's additional .py files20:41
espamcrn: nice are you doing a side project for facebook too ;)20:41
denis_makogonamcrn, ++ =))20:41
amcrnso if you added additional .py files, i don't see why modifying cfg.py is so heinous20:41
amcrnyou've already effectively forked the project20:42
denis_makogonagreed20:42
*** ViswaV has joined #openstack-trove20:42
amcrnyou could theoretically write some code to polymorphically load various [datastore-<x>] subheaders in the CONF, but that seems extremely overkill given your very unique situation20:42
*** ViswaV_ has joined #openstack-trove20:43
*** ViswaV has quit IRC20:43
amcrnplus, are you really going to deploy 'mysql' vanilla? just re-purpose mysql to use your facebook fork.20:43
SnowDustamcrn: u understood my purpose20:44
amcrnesp: nope, was just taking a sherlock stab in the dark ;)20:44
*** demorris has quit IRC20:44
*** ViswaV_ has quit IRC20:44
SnowDustalso, the changes how difficult they have made the generic solution into20:44
*** demorris has joined #openstack-trove20:44
yogeshamcrn: it can make things more flexible...polymorphically loading the config...20:44
SnowDustand just a fallback can bring this back20:44
*** ViswaV has joined #openstack-trove20:44
SnowDustits either through falling back to /var/lib/mysql as the mount point for taskamanger20:45
amcrnyogesh: that has quite a few downsides, one of which is ascertaining what parameters are required for datastores20:45
SnowDustor, from the default group mount_point20:45
*** ViswaV has quit IRC20:45
SnowDustwhich may be configured in the taskmanager conf file20:45
yogeshamcron: some of the configs are mandatory for all datastores...20:45
*** ViswaV has joined #openstack-trove20:45
yogeshamcrn: sorry for isspelling ur alias..20:46
amcrnSnowDust: you're blinded a bit by your particular scenario; having a default is bad in nearly every other circumstance. If you add the default, then if someone misconfigures a new datastore, they're going to fallback to /var/lib/mysql, which is completely nonsensical.20:46
yogeshamcrn: i agree to an extent20:46
SnowDustamcrn: thats why we have neutral values :) /var/lib/datastore20:46
SnowDustthan /var/lib/mysql20:46
amcrnlol, you're abusing a feature to get the desired result20:46
amcrnyou need to fork cfg.py, plain and simple20:46
*** ViswaV_ has joined #openstack-trove20:47
yogeshSnowdust: i think u need to think more20:47
amcrnunless someone writes a blueprint and a rough poc proving a polymorphic load would work, and wouldn't have detrimental side-effects20:47
SnowDustamcrn, if it was cfg.cfg :) i shall had with no complains20:47
amcrnand to juice point, we need to look into how to *mandate* that a particular conf has a value, vs. defaulting it and hoping the user doesn't accidentally set it to "" in the configuration file20:47
*** ViswaV_ has quit IRC20:47
*** ViswaV_ has joined #openstack-trove20:48
espamcrn: ++ SnowDust may want to fork his work20:48
SnowDustfinally: should we make cfg.py pluggable itself :D20:50
*** ViswaV has quit IRC20:50
SnowDustif thats .. to be tied to datastore implementation20:50
yogeshfor a setting which is mandatory for all the datastores...having a setting in default group gives an option to provide a single value which is applicable to all the datastores....in case those datastores are running behiand the same api...20:51
espSnowDust: hopefully we can put in some time to clean up the datastore config stuff.20:51
yogeshthats the only use..20:51
amcrnSnowDust: so for now, no defaults. as a side project, you can look into what it means to make it pluggable.20:53
SnowDustamcrn : yep .. it can be like the config templates20:54
*** radez is now known as radez_g0n320:56
*** Ranjitha has quit IRC20:57
yogeshSnowdust: best solution will be to push your datastore implementation upstream :-)20:57
*** ViswaV_ has quit IRC20:58
*** michael-yu has quit IRC20:58
yogeshi mean after all the due diligence20:58
SnowDustyogesh: let me think ;)20:58
espyep, seems reasonable.  I like the idea of config templates too. but I think folks will want to take a hard look at it.20:59
SnowDustbut i guess .. we can have a template driven datastore loading20:59
*** ramashri has quit IRC21:02
*** Ranjitha has joined #openstack-trove21:02
*** ViswaV has joined #openstack-trove21:03
SnowDustamcrn, denis_makogon : so i think settings.template shall be fine to think as alternative21:04
SnowDustthat will keep it pluggable21:04
SnowDustbut .. need to think how early it can be called :)21:05
*** amcrn has quit IRC21:06
denis_makogonIt can't be the setting template21:07
SnowDustdenis_makogon, reasons ?21:07
*** michael-yu has joined #openstack-trove21:08
denis_makogonSnowDust, because its the base confi files21:08
denis_makogonSnowDust, they cannot be template21:09
*** ramashri has joined #openstack-trove21:09
*** Barker has quit IRC21:11
*** amcrn has joined #openstack-trove21:12
*** demorris has quit IRC21:17
*** demorris has joined #openstack-trove21:20
*** tanisdl has quit IRC21:23
espdenis_makogon: would be nice someday to have template driven way to configure datastores?  not sure what consensus on that idea is but I like it :)21:24
denis_makogonesp, need more clean elaboration for that21:25
espso, we'd have json/yaml or whatever file for each datastore21:25
espsort of a serialized version of the object that would be loaded into the code21:26
espthat way, you could abstract that from living inside the code itself21:26
*** tanisdl has joined #openstack-trove21:28
espmakes sense denis_makogon?  again I'm just thinking out loud.21:28
denis_makogonesp, you could try to disgn it21:30
esplol21:30
espsure I can do a blue print or something21:31
espbut I'm open to other suggestions as well21:31
espI just don't see shoving all the datastore implementations in code as maintainable way to go21:32
*** esp has left #openstack-trove21:37
*** tanisdl has quit IRC21:37
*** esp has joined #openstack-trove21:42
*** robertmyers has quit IRC21:45
openstackgerritDan Nguyen proposed a change to openstack/trove: Removes extra initialization from config  https://review.openstack.org/7946721:48
*** demorris has quit IRC21:54
*** pdmars has quit IRC22:12
SnowDustesp, i said in similar line22:12
*** rhodgin has quit IRC22:12
espSnowDust: yep. seems like it would be a nice thing.  I'm just hesitant to say it's going to fit with the rest of the code base and all.  Probably need to see what code currently supports.22:14
SnowDust:)22:14
SnowDustwill surely give some time to that22:15
*** datsun180b has quit IRC22:16
*** SnowDust has quit IRC22:19
*** mattgriffin has quit IRC22:27
*** yogesh has quit IRC22:27
*** thedodd has quit IRC22:35
*** rhodgin has joined #openstack-trove22:36
*** yidclare has quit IRC22:43
*** denis_makogon has quit IRC22:54
*** jmontemayor has quit IRC23:01
*** Ranjitha has quit IRC23:05
*** amcrn has quit IRC23:07
*** ramashri has quit IRC23:10
*** Ranjitha has joined #openstack-trove23:10
*** amcrn has joined #openstack-trove23:10
*** yidclare has joined #openstack-trove23:12
*** yidclare has quit IRC23:14
*** michael-yu has quit IRC23:18
*** ViswaV has quit IRC23:20
*** kevinconway has quit IRC23:20
*** flaper87 is now known as flaper87|afk23:22
*** ViswaV has joined #openstack-trove23:22
*** tenaglia has quit IRC23:25
openstackgerritGiuseppe Galeota proposed a change to openstack/trove: Trove Manual Installation guide  https://review.openstack.org/7860823:27
*** tattabbum has quit IRC23:28
*** michael-yu has joined #openstack-trove23:31
*** ViswaV has quit IRC23:32
*** grapex has quit IRC23:39
*** ViswaV has joined #openstack-trove23:41
*** ViswaV_ has joined #openstack-trove23:42
*** ViswaV has quit IRC23:42
*** jcru has quit IRC23:54

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!