18:02:39 #startmeeting trove 18:02:39 Meeting started Wed Mar 19 18:02:39 2014 UTC and is due to finish in 60 minutes. The chair is hub_cap. Information about MeetBot at http://wiki.debian.org/MeetBot. 18:02:40 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 18:02:43 The meeting name has been set to 'trove' 18:02:55 o/ 18:02:56 haylo 18:03:05 o/ 18:03:05 so looks like the first thing on the agenda is the refactoring datastore options thing 18:03:07 7o7 18:03:07 o/ 18:03:18 action items? 18:03:19 #link https://wiki.openstack.org/wiki/Meetings/TroveMeeting 18:03:23 o/ 18:03:55 #link 18:03:59 lol stupid paste 18:04:07 #link http://eavesdrop.openstack.org/meetinags/trove/2014/trove.2014-03-12-17.59.txt 18:04:08 o/ 18:04:17 o/ 18:04:22 o/ 18:04:26 \o 18:04:29 o/ 18:04:40 ok so im thinking next time, dotn worru about saying yer here 18:04:42 hub_cap: 404 on that link 18:04:44 i trust everyone is here 18:04:49 o/ 18:05:09 http://eavesdrop.openstack.org/meetings/trove/2014/trove.2014-03-12-17.59.txt 18:05:14 vipul is not here 18:05:21 juice: then when i say, hey vipul 18:05:24 and he doesnt respond 18:05:27 ill know hes not here :) 18:05:28 hub_cap: But roll call is my favorite part! 18:05:38 it also logs us as there in the meeting notes 18:05:47 grapex: present SIR ! 18:05:50 :D 18:05:53 kevinconway: im not sure thats necessary 18:06:06 i get paid per lines in the meeting log 18:06:09 anyhoo lets talk about it in open discuss 18:06:14 kevinconway: lol exactly 18:06:18 kevinconway :D 18:06:21 so lets start w/ 18:06:27 hub_cap: i didn't see this topic on the agenda 18:06:33 #link refactoring-datastore-options 18:06:40 pdmars: hence my open discuss comment ;) 18:06:49 #undo 18:06:50 Removing item from minutes: 18:06:59 #topic refactoring-datastore-options 18:07:10 #link 18:07:15 ARGGGGGGGG PASTE 18:07:25 #link https://blueprints.launchpad.net/trove/+spec/refactoring-datastore-options-in-cfg 18:07:37 thx SlickNik 18:07:40 https://blueprints.launchpad.net/trove/+spec/refactoring-datastore-options-in-cfg 18:07:44 thanks .. 18:07:50 hub_cap: my linux box never has this problems. 18:08:04 So we discussed this at the bp-review on Mon. 18:08:15 cweid: my linux box is about 15 min old, so i havent fixed my keybindings :) 18:08:35 SlickNik: yes but we didnt come to a conclusion about work starting 18:08:43 ohhhhh welcome to the cool kids team =) 18:08:53 SnowDust: The main concern was that the current design / implementation allows for a complete override of the datastore config values. 18:08:59 cweid: your mind is terrible, ive been on linux for 6+mo 18:09:24 oh my bad. 18:09:27 SlickNik: as the override is from the config file or from the implementation of datastore( external package) 18:09:36 this should be a "Safe" option 18:09:43 but i am open to discuss the consequence 18:09:49 SlickNik: are we suer thats the case? 18:09:51 hub_cap: that's almost 7 months 18:10:06 The proposal was to have the design extend the default datastore configs instead of override them completely. 18:10:08 cuz teh code, if i grok correctly, only uses the options if they are in a optgrp 18:10:44 kevinconway: +:=<2wks 18:10:58 SlickNik, until an ADMIN overrided using conf ( whcih he can even do in the current code) 18:11:01 no override happens 18:11:14 in current code also 18:11:32 so this code doesnt split the conifgs out in any way 18:11:33 we may define [mysql] / [percona]/[redis]/ in conf files 18:11:46 to override datastore defaults for the datastore 18:11:55 it just moves the in memory ones to different files, but until the datastores use the optgroups 18:12:02 they will be ignored 18:12:23 iirc u need to do a cfg.mysql.mount_point to access the [mysql]...mount_point=blah 18:12:38 so in its current state im not sure what this code does :) 18:13:24 hub_cap: SlickNik: the BP is just to enhance the component nature of databases 18:13:48 my idea was to separate datastore specific config from the cfg.py to their own packages 18:14:06 as we had the design of datastore Managers being a class loaded using config 18:14:25 so its safe to load the configurations from the module itself .. from which the Managers are loaded 18:14:29 SnowDust: would that include the ability to override a global option but only in one specific datastore (like the mount_point hub_cap mentioned)? 18:14:45 SnowDust: The concern is that you'd be able to override the config values for a datastore type so that you don't load the confs for say cassandra, but the guestagent manager still had a cassandra implementation. 18:14:45 18:14:48 kevinconway: just like current code the nature remains the same for override 18:14:49 kevinconway: some code would have to be ther for that 18:15:13 snowdust: I like the modular nature/i.e. reduced clutter but on the flip side I need to piece together two files to see the complete picture 18:15:16 hub_cap: right, i'm just trying to keep up with what this BP is for 18:15:37 snowdust: not sure it is worth the additional complexiyt 18:15:39 juice: is it two files or just two sections in one file? 18:15:52 juice: its worth is for keeping datastore implementations external 18:15:58 so kevinconway i wrote this for cinders multi backend 18:16:00 kevinconway: two files as it is proposed 18:16:01 https://github.com/openstack/cinder/blob/master/cinder/volume/configuration.py 18:16:11 it lets you specify an optgroup, and if not, it uses the defaults 18:16:32 so u just do configuration.mount_point, and if it can find the "mysql" mount_point, it uses 18:16:50 kevinconway: iirc ;) 18:17:00 SlickNik: i didnot get the last one from u 18:17:43 SlickNik: as u saw cfg.py ( code snippet in the BP) loads Cassandra by default ( with its well defined defaults) 18:18:02 but override is user choice and thats why we have config variables ment 18:18:24 SnowDust: https://github.com/openstack/trove/blob/fba8cabea326527bacdeca56760a97e14cdcc18f/trove/guestagent/dbaas.py#L34-L51 lets you specify datastore managers. 18:18:38 SlickNik : right 18:18:53 ok so looking at this, SlickNik SnowDust 18:19:05 hub_cap : thanks .. 18:19:12 does it override the existing opts instead of definig them _only_ in an optgroup? 18:19:23 if thats the case this is a BIG nono 18:19:35 its does not override any existing ones .. 18:19:36 we will not, ever, overwrite opts in memory that were defined 18:19:56 ok so how do i use the opt in [mysql] mount_point SnowDust ? 18:20:12 confs just exist in ONE place .. their place of existance has been reworked .. from cfg.py ( which is a tied down approach) to the datastore module itself 18:20:24 hub_cap 18:20:28 as per the cfg.py code 18:20:38 the datastore_options dictOpt 18:21:10 SnowDust: the default managers are always loaded; similarly the default configs should also be always loaded. 18:21:12 is a mapping between datastore_manager and the datastore_manager.implementation.module.entrypoint 18:21:28 SnowDust: ok, so how do i get that option 18:21:50 after that is done .. it used oslo.config's method .. import_group 18:21:59 cuz what yer doing is using a cfg optgroup, and your code would have to look like cfg.mysql.mount_point right? 18:22:06 import_group reads the module entry point to import the opts .. 18:22:21 yeah hub_cap .. end of that 18:22:35 its CONF.mysql.mount_point .. just as rigth now 18:22:49 so you plan on changing that in all the datastores, correct? 18:22:59 hub_cap right 18:23:08 even which are external to the trove code 18:23:12 ok so what happens when i wnat to use the default? 18:23:25 SnowDust: thats not true at all, those are all over the trove guest code and other places 18:23:40 if u look at a datastore impl it needs to use conf values, right? 18:23:49 and those conf values are not changed 18:23:56 hub_cap they are also loaded in cfg.py 18:24:00 but there is also no backwards compatibility 18:24:04 so .. default are available all the time 18:24:07 how do i, SnowDust , which isthe question we are asking 18:24:12 right but i have to code it for every one 18:24:13 right? 18:24:13 when ever we import from trove.common import cfg 18:24:30 if not cfg.mysql.mount_point use cfg.mount_point 18:24:38 we have done that only in current cfg.py for all the datastores 18:24:42 if not cfg.#{datastore}.mount_point use cfg.mount_point 18:24:44 there is not common defaults now .. 18:24:59 hub_cap .. there is no cfg.mount_point now 18:25:03 default have been removed 18:25:15 ok so maybe mount 18:25:18 point is a bad example 18:25:19 i pushed a patchset on that question only .. then were suggested to go for this BP 18:25:48 hub_cap .. let me share the abandoned patchset.. which wanted to restore the cfg.mount_point but was turned off 18:25:52 no 18:25:57 lets use backup_strategy 18:26:33 what i think we want to see, is the ability to use these values but not put a bunch of if'stmts for each "default" vs a datastore specific 18:27:01 i suggest u look @ the configuration object in cinder 18:27:20 it handles it pretty transparently, but can likely be updated to be better :) 18:27:41 kevinconway: SlickNik et al do yall feel better about it assuming that 18:27:58 #link https://github.com/openstack/cinder/blob/master/cinder/volume/configuration.py 18:28:00 * if we dont find a datastore specific, we use the one in the [DEFAULT] group 18:28:11 hub_cap .. :) 18:28:24 they all turned off .. the fallback 18:28:31 amcrn, r u there? 18:29:08 SnowDust: He had to step out. 18:29:11 hub_cap: I think there was some question regarding if we still consider mysql as the default datastore impl 18:29:24 ok so we could provide datastore specific overrides/special entries and then fall back to default configs when they aren't present? 18:29:43 would that allow for config options in mongo that don't exist in mysql? 18:29:56 or do they all *have* to exist in DEFAULT? 18:29:56 sure kevinconway 18:30:01 it probably makes sense to have a default datastore 18:30:08 if they dont exist in default then we fail 18:31:15 hub_cap: Wait, are you saying we can't add datastore specific config options? 18:31:21 hub_cap: meaning if you don't specify a required config for your datastore we fail? 18:31:28 no not at all 18:31:31 this is how configs work 18:31:34 1) in memory 18:31:36 hub_cap: Ok, just checking 18:31:44 2) overrides from disk if avail 18:31:48 3) failures 18:32:07 opt groups get funky since u have to identify them 18:32:17 oslo wont "Default" for u 18:32:42 if u specify cfg.somegrp.someval, if someval is in default, it wont pull it 18:33:08 itll look for [somegrp]...someval=blah, and if it doesnt find, itll use whats in memory for that value, if any 18:33:15 sounds like something we could put a bug in oslo for? 18:33:16 only for that memory in that optgroup 18:33:22 its not a bug 18:33:30 its the design of using different optgroups 18:33:38 hub_cap: Ok- I thought you were saying oslo could be smart enough to use someval from default is cfg.somegrp.someval wadn't found 18:33:40 However 18:33:43 hub_cap : so the current BP implements the defaults 18:33:53 we could do cfg.DEFAULT if we wanted :) 18:33:53 i said when cfg is imported 18:33:57 ok 18:34:03 you could have code in a datastore that checked to see if the group specific default was not None, and if it was try the default one 18:34:10 the load_datastore_opts() 18:34:13 function is called 18:34:20 which loads the mapped datastores 18:34:25 grapex: or just put it in a wrapper object, like configuration object 18:34:35 hub_cap: Yeah 18:34:38 u pass in None, or the optgroup, like 'cassandra' 18:34:42 and then just do configuration.blah 18:34:47 instead of cfg.blah 18:35:29 So not getting caught up in the implementation details, and coming back to the bp. 18:35:38 yes :) 18:35:59 i think its ok to move the options to config groups for datastores 18:36:10 assumign we can interact w/ them in a easy way ;) 18:36:14 I think that the bp does simplify some of the datastore config value loading. 18:36:52 id like to see a 2nd patchset w/ options actually updated before i can merge p1 :) 18:37:26 as in a dependent patch set that shows the code updating, say, mysql, to use the optgrp 18:38:36 hub_cap int_tests just show that what options are being selected for mysql 18:38:48 but as it stands, im ok w/ this 18:39:52 Same here, I'm okay with the bp. 18:40:05 The implementation details might need some tweaking. 18:40:17 Ok- sorry 18:40:29 this all seems simple but I feel like I'm fuzy on some of the details we're talking about 18:40:37 And I think it'd be easier to make some of these comments in gerrit. 18:40:45 SlickNik: +1000 18:40:47 yea after looking more at the code, it looks like we are already calling cfg.get(datastore_mgr).tcp_ports 18:40:52 thats the only place we are doing it 18:40:52 just to be sure, none of us want to load all of the datastore options in the common/cfg.py right? 18:41:18 robert just pointed out on the last line here, it loads all of the config options for everything which seems a bit much: https://review.openstack.org/#/c/80061/5/trove/common/cfg.py 18:41:48 grapex: yes 18:41:51 all other openstack projects do not do this 18:41:55 grapex: i suggested in last discussion 18:41:57 robertmy_: ? 18:41:59 this may be further trimmed 18:42:06 if we want to do it from conf files only 18:42:08 everything exists in a big global cfg 18:42:09 they use the option groups in the file 18:42:17 hub_cap: no 18:42:18 SnowDust: Sorry, I guess we didn't all see that. 18:42:21 robertmy_: tahts what he says this is doing 18:42:23 so that only the one which is being enabled gets loaded 18:42:32 robertmy_: this is a change then :) 18:42:58 hub_cap: I don't get the reasoning 18:43:05 why force all the config options to load when cfg.py is parsed? 18:43:14 well wait up, i need robertmyers to explain 18:43:17 let me find an example 18:43:34 plz do, cuz last i thought, everyon grabs cfg.CONF 18:43:45 and its parsed on load (thats how nova used to work) 18:43:48 again, *used to* 18:44:07 ok so maybe we need to move this convo out of here 18:44:09 and move on 18:44:27 hub_cap: https://github.com/openstack/nova/blob/master/nova/api/auth.py#L46 18:44:35 https://github.com/openstack/cinder/blob/master/cinder/volume/drivers/san/san.py 18:44:57 so you see the option groups are defined at the point they are used 18:45:00 right its still a global CONF 18:45:02 That's what robert_my means- auth_opts are simply defined next to where they are used, and no special code exists to parse them as the root cfg.py is created 18:45:22 oh u mean just the loading.. yea that can be taken out of the global cfg stuff 18:45:32 there is no need for them to be loaded before their impls are loaded 18:45:42 hub_cap: it is a global config file, but not a global object 18:45:54 cfg.CONF ?? 18:46:11 im pretty sure that using CONF everywhere defines a global obj right? 18:46:20 well, it all reads/extends cfg.CONF 18:46:24 my python-speak may be wrong 18:46:36 everything uses CONF right? 18:46:38 technically yes 18:46:54 ok i think we are speaking the same thing 18:46:59 but, it is lazy loaded 18:46:59 i agree we dont need to "preload" Them 18:47:04 ok 18:47:12 the point of cfg is to allow u to define groups in the files they are used (or imported) 18:47:14 * grapex wipes sweat off his brow 18:47:18 i think we should change the conf files to YAML 18:47:30 kevinconway: if i only had /kick privs 18:47:31 xml 18:47:35 kevinconway: https://www.youtube.com/watch?v=4DgbUBoxa48 18:47:48 so lets remove the preload stuff SnowDust 18:47:51 and just let the app load it 18:48:05 lol 18:48:13 and id like to see you update an impl to use more than just whats defined now in the optgropu, which is tcp_ports 18:48:26 unless thats done 18:49:02 we are already doing that in places 18:49:07 so then we dont need to see that done 18:49:24 so just removing the autoload will satisfy everyone, right? 18:49:39 and we can make a different bp for the "fallback" to default 18:49:43 hub_cap: aye 18:49:58 hub_cap u mean i should not "REGISTER" the groups ? 18:50:06 +1 on the lazy loading. I have a couple other comments, I'll add them to gerrit :) 18:50:11 SnowDust: u register the opts/group _in_ the file 18:50:15 +1 18:50:16 whih i thought u were doing already 18:50:57 https://review.openstack.org/#/c/80061/5/trove/guestagent/datastore/cassandra/options.py 18:51:06 see you are already registering the group and opts 18:51:21 so if u just completely remove all the code from cfg, itll work 18:51:37 hub_cap: all the code?!? 18:51:42 ALLL 18:51:44 i dont get 18:51:53 SnowDust: offline 18:51:57 lets move on we have 9 min heh 18:52:07 until u register a group 18:52:07 how do u use the group options ? 18:52:12 SnowDust: offline 18:52:14 lets move on we have 9 min heh 18:52:27 #topic remove mockito 18:52:28 hub_cap i do register them in the options.py of datastore module 18:52:28 yes 18:52:34 #link https://blueprints.launchpad.net/trove/+spec/remove-mockito 18:52:37 but we need to import the group in cfg 18:52:41 so whos doing this? 18:52:45 so that cfg.CONF 18:52:45 can then be used to call them 18:52:45 SnowDust: offline 18:52:50 SnowDust: offline!!!! 18:52:51 using CONF.mysql.root_on_create etc 18:52:51 hub_cap ? 18:52:53 SnowDust: offline!!!! 18:52:57 hub_cap sure ! 18:52:57 hub_cap : sure ! 18:53:09 thanks ALL :) 18:53:23 SnowDust: u will have to reiterate Q offline plz 18:53:26 SlickNik: is this u? 18:53:32 yup, I added this 18:53:34 whos tackling removing mockito? 18:53:51 i can lend a hand 18:54:01 I looked into this yesterday, probably gonna be juice and me. 18:54:50 cool, anything to add to it? 18:54:52 Lots of mockito in the tests, probably gonna take 2-3 days to get it done. 18:54:56 yea for sure 18:55:05 SlickNik: is there any plan to remove mox? 18:55:25 esp: I guess mox is still in the mix 18:55:28 esp: right now it's not an issue since mox exists in the global_requirements. 18:55:29 do we use mox? 18:55:31 but we shouldn't do it 18:55:48 robertmyers: I think there was some mox in the trove code 18:55:56 robertmyers: I think there is some unit tests in python-troveclient that may use mox 18:55:57 robertmyers: I haven't seen it in many places; but I haven't looked too hard. 18:56:13 is the one from google I believe 18:56:24 let not use it 18:56:26 *it's 18:56:28 Timeline wise, I think we might have to do this before the icehouse cut. 18:56:31 robertmyers: +1 18:56:45 yes SlickNik lets focus on mockito 18:56:50 and make esp do the mox stuff 18:56:53 ;) 18:56:58 esp: thanks! :) 18:56:59 hehe 18:57:02 np 18:57:06 that's all I had. 18:58:11 i have to run - talk to you guys later 18:58:43 juice: I'll catch you offline to chat about mockito. 18:58:45 thanks! 18:59:13 ok so the last topic 18:59:14 no time 18:59:18 lol 18:59:20 who put it on? 18:59:30 we can move to #openstack-trove to discuss 18:59:35 anyone? 18:59:43 yep 19:00:06 ok so if anyone wants to discuss container vs join, goto the channel 19:00:09 #endmeeting