18:02:39 <hub_cap> #startmeeting trove
18:02:39 <openstack> Meeting started Wed Mar 19 18:02:39 2014 UTC and is due to finish in 60 minutes.  The chair is hub_cap. Information about MeetBot at http://wiki.debian.org/MeetBot.
18:02:40 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
18:02:43 <openstack> The meeting name has been set to 'trove'
18:02:55 <imsplitbit> o/
18:02:56 <cp16net> haylo
18:03:05 <SlickNik> o/
18:03:05 <hub_cap> so looks like the first thing on the agenda is the refactoring datastore options thing
18:03:07 <kevinconway> 7o7
18:03:07 <pdmars> o/
18:03:18 <cp16net> action items?
18:03:19 <hub_cap> #link https://wiki.openstack.org/wiki/Meetings/TroveMeeting
18:03:23 <grapex> o/
18:03:55 <hub_cap> #link
18:03:59 <hub_cap> lol stupid paste
18:04:07 <hub_cap> #link http://eavesdrop.openstack.org/meetinags/trove/2014/trove.2014-03-12-17.59.txt
18:04:08 <mat-lowery> o/
18:04:17 <vgnbkr> o/
18:04:22 <juice> o/
18:04:26 <k-pom> \o
18:04:29 <doug_shelley66> o/
18:04:40 <hub_cap> ok so im thinking next time, dotn worru about saying yer here
18:04:42 <kevinconway> hub_cap: 404 on that link
18:04:44 <hub_cap> i trust everyone is here
18:04:49 <glucas> o/
18:05:09 <hub_cap> http://eavesdrop.openstack.org/meetings/trove/2014/trove.2014-03-12-17.59.txt
18:05:14 <juice> vipul is not here
18:05:21 <hub_cap> juice: then when i say, hey vipul
18:05:24 <hub_cap> and he doesnt respond
18:05:27 <hub_cap> ill know hes not here :)
18:05:28 <grapex> hub_cap: But roll call is my favorite part!
18:05:38 <kevinconway> it also logs us as there in the meeting notes
18:05:47 <SnowDust> grapex: present SIR !
18:05:50 <SnowDust> :D
18:05:53 <hub_cap> kevinconway: im not sure thats necessary
18:06:06 <kevinconway> i get paid per lines in the meeting log
18:06:09 <hub_cap> anyhoo lets talk about it in open discuss
18:06:14 <hub_cap> kevinconway: lol exactly
18:06:18 <SnowDust> kevinconway :D
18:06:21 <hub_cap> so lets start w/
18:06:27 <pdmars> hub_cap: i didn't see this topic on the agenda
18:06:33 <hub_cap> #link refactoring-datastore-options
18:06:40 <hub_cap> pdmars: hence my open discuss comment ;)
18:06:49 <hub_cap> #undo
18:06:50 <openstack> Removing item from minutes: <ircmeeting.items.Link object at 0x38eddd0>
18:06:59 <hub_cap> #topic refactoring-datastore-options
18:07:10 <hub_cap> #link
18:07:15 <hub_cap> ARGGGGGGGG PASTE
18:07:25 <SlickNik> #link https://blueprints.launchpad.net/trove/+spec/refactoring-datastore-options-in-cfg
18:07:37 <hub_cap> thx SlickNik
18:07:40 <SnowDust> https://blueprints.launchpad.net/trove/+spec/refactoring-datastore-options-in-cfg
18:07:44 <SnowDust> thanks ..
18:07:50 <cweid> hub_cap: my linux box never has this problems.
18:08:04 <SlickNik> So we discussed this at the bp-review on Mon.
18:08:15 <hub_cap> cweid: my linux box is about 15 min old, so i havent fixed my keybindings :)
18:08:35 <hub_cap> SlickNik: yes but we didnt come to a conclusion about work starting
18:08:43 <cweid> ohhhhh welcome to the cool kids team =)
18:08:53 <SlickNik> SnowDust: The main concern was that the current design / implementation allows for a complete override of the datastore config values.
18:08:59 <hub_cap> cweid: your mind is terrible, ive been on linux for 6+mo
18:09:24 <cweid> oh my bad.
18:09:27 <SnowDust> SlickNik: as the override is from the config file or from the implementation of datastore( external package)
18:09:36 <SnowDust> this should be a "Safe" option
18:09:43 <SnowDust> but i am open to discuss the consequence
18:09:49 <hub_cap> SlickNik: are we suer thats the case?
18:09:51 <kevinconway> hub_cap: that's almost 7 months
18:10:06 <SlickNik> The proposal was to have the design extend the default datastore configs instead of override them completely.
18:10:08 <hub_cap> cuz teh code, if i grok correctly, only uses the options if they are in a optgrp
18:10:44 <hub_cap> kevinconway: +:=<2wks
18:10:58 <SnowDust> SlickNik, until an ADMIN overrided using conf ( whcih he can even do in the current code)
18:11:01 <SnowDust> no override happens
18:11:14 <SnowDust> in current code also
18:11:32 <hub_cap> so this code doesnt split the conifgs out in any way
18:11:33 <SnowDust> we may define [mysql] / [percona]/[redis]/ in conf files
18:11:46 <SnowDust> to override datastore defaults for the datastore
18:11:55 <hub_cap> it just moves the in memory ones to different files, but until the datastores use the optgroups
18:12:02 <hub_cap> they will be ignored
18:12:23 <hub_cap> iirc u need to do a cfg.mysql.mount_point to access the [mysql]...mount_point=blah
18:12:38 <hub_cap> so in its current state im not sure what this code does :)
18:13:24 <SnowDust> hub_cap: SlickNik:  the BP is just to enhance the component nature of databases
18:13:48 <SnowDust> my idea was to separate datastore specific config from the cfg.py to their own packages
18:14:06 <SnowDust> as we had the design of datastore Managers being a class loaded using config
18:14:25 <SnowDust> so its safe to load the configurations from the module itself .. from which the Managers are loaded
18:14:29 <kevinconway> SnowDust: would that include the ability to override a global option but only in one specific datastore (like the mount_point hub_cap mentioned)?
18:14:45 <SlickNik> SnowDust: The concern is that you'd be able to override the config values for a datastore type so that you don't load the confs for say cassandra, but the guestagent manager still had a cassandra implementation.
18:14:45 <SlickNik> 
18:14:48 <SnowDust> kevinconway: just like current code the nature remains the same for override
18:14:49 <hub_cap> kevinconway: some code would have to be ther for that
18:15:13 <juice> snowdust: I like the modular nature/i.e. reduced clutter but on the flip side I need to piece together two files to see the complete picture
18:15:16 <kevinconway> hub_cap: right, i'm just trying to keep up with what this BP is for
18:15:37 <juice> snowdust: not sure it is worth the additional complexiyt
18:15:39 <kevinconway> juice: is it two files or just two sections in one file?
18:15:52 <SnowDust> juice: its worth is for keeping datastore implementations external
18:15:58 <hub_cap> so kevinconway i wrote this for cinders multi backend
18:16:00 <juice> kevinconway: two files as it is proposed
18:16:01 <hub_cap> https://github.com/openstack/cinder/blob/master/cinder/volume/configuration.py
18:16:11 <hub_cap> it lets you specify an optgroup, and if not, it uses the defaults
18:16:32 <hub_cap> so u just do configuration.mount_point, and if it can find the "mysql" mount_point, it uses
18:16:50 <hub_cap> kevinconway: iirc ;)
18:17:00 <SnowDust> SlickNik: i didnot get the last one from u
18:17:43 <SnowDust> SlickNik: as u saw cfg.py ( code snippet in the BP) loads Cassandra by default ( with its well defined defaults)
18:18:02 <SnowDust> but override is user choice  and thats why we have config variables ment
18:18:24 <SlickNik> SnowDust: https://github.com/openstack/trove/blob/fba8cabea326527bacdeca56760a97e14cdcc18f/trove/guestagent/dbaas.py#L34-L51 lets you specify datastore managers.
18:18:38 <SnowDust> SlickNik : right
18:18:53 <hub_cap> ok so looking at this, SlickNik SnowDust
18:19:05 <SnowDust> hub_cap : thanks ..
18:19:12 <hub_cap> does it override the existing opts instead of definig them _only_ in an optgroup?
18:19:23 <hub_cap> if thats the case this is a BIG nono
18:19:35 <SnowDust> its does not override any existing ones ..
18:19:36 <hub_cap> we will not, ever, overwrite opts in memory that were defined
18:19:56 <hub_cap> ok so how do i use the opt in [mysql] mount_point SnowDust ?
18:20:12 <SnowDust> confs just exist in ONE place .. their place of existance has been reworked .. from cfg.py ( which is a tied down approach) to the datastore module itself
18:20:24 <SnowDust> hub_cap
18:20:28 <SnowDust> as per the cfg.py code
18:20:38 <SnowDust> the datastore_options  dictOpt
18:21:10 <SlickNik> SnowDust: the default managers are always loaded; similarly the default configs should also be always loaded.
18:21:12 <SnowDust> is a mapping between datastore_manager and the  datastore_manager.implementation.module.entrypoint
18:21:28 <hub_cap> SnowDust: ok, so how do i get that option
18:21:50 <SnowDust> after that is done .. it used oslo.config's method .. import_group
18:21:59 <hub_cap> cuz what yer doing is using a cfg optgroup, and your code would have to look like cfg.mysql.mount_point right?
18:22:06 <SnowDust> import_group reads the module entry point to import the opts ..
18:22:21 <SnowDust> yeah hub_cap .. end of that
18:22:35 <SnowDust> its CONF.mysql.mount_point .. just as rigth now
18:22:49 <hub_cap> so you plan on changing that in all the datastores, correct?
18:22:59 <SnowDust> hub_cap right
18:23:08 <SnowDust> even which are external to the trove code
18:23:12 <hub_cap> ok so what happens when i wnat to use the default?
18:23:25 <hub_cap> SnowDust: thats not true at all, those are all over the trove guest code and other places
18:23:40 <hub_cap> if u look at a datastore impl it needs to use conf values, right?
18:23:49 <hub_cap> and those conf values are not changed
18:23:56 <SnowDust> hub_cap they are also loaded in cfg.py
18:24:00 <hub_cap> but there is also no backwards compatibility
18:24:04 <SnowDust> so .. default are available all the time
18:24:07 <hub_cap> how do i, SnowDust , which isthe question we are asking
18:24:12 <hub_cap> right but i have to code it for every one
18:24:13 <hub_cap> right?
18:24:13 <SnowDust> when ever we import from trove.common import cfg
18:24:30 <hub_cap> if not cfg.mysql.mount_point use cfg.mount_point
18:24:38 <SnowDust> we have done that only in current cfg.py for all the datastores
18:24:42 <hub_cap> if not cfg.#{datastore}.mount_point use cfg.mount_point
18:24:44 <SnowDust> there is not common defaults now ..
18:24:59 <SnowDust> hub_cap .. there is no cfg.mount_point now
18:25:03 <SnowDust> default have been removed
18:25:15 <hub_cap> ok so maybe mount
18:25:18 <hub_cap> point is a bad example
18:25:19 <SnowDust> i pushed a patchset on that question only .. then were suggested to go for this BP
18:25:48 <SnowDust> hub_cap .. let me share the abandoned patchset.. which wanted to restore the cfg.mount_point but was turned off
18:25:52 <hub_cap> no
18:25:57 <hub_cap> lets use backup_strategy
18:26:33 <hub_cap> what i think we want to see, is the ability to use these values but not put a bunch of if'stmts for each "default" vs a datastore specific
18:27:01 <hub_cap> i suggest u look @ the configuration object in cinder
18:27:20 <hub_cap> it handles it pretty transparently, but can likely be updated to be better :)
18:27:41 <hub_cap> kevinconway: SlickNik et al do yall feel better about it assuming that
18:27:58 <SlickNik> #link https://github.com/openstack/cinder/blob/master/cinder/volume/configuration.py
18:28:00 <hub_cap> * if we dont find a datastore specific, we use the one in the [DEFAULT] group
18:28:11 <SnowDust> hub_cap .. :)
18:28:24 <SnowDust> they all turned off  .. the fallback
18:28:31 <SnowDust> amcrn, r u there?
18:29:08 <SlickNik> SnowDust: He had to step out.
18:29:11 <esp> hub_cap: I think there was some question regarding if we still consider mysql as the default datastore impl
18:29:24 <kevinconway> ok so we could provide datastore specific overrides/special entries and then fall back to default configs when they aren't present?
18:29:43 <kevinconway> would that allow for config options in mongo that don't exist in mysql?
18:29:56 <kevinconway> or do they all *have* to exist in DEFAULT?
18:29:56 <hub_cap> sure kevinconway
18:30:01 <esp> it probably makes sense to have a default datastore
18:30:08 <hub_cap> if they dont exist in default then we fail
18:31:15 <grapex> hub_cap: Wait, are you saying we can't add datastore specific config options?
18:31:21 <esp> hub_cap: meaning if you don't specify a required config for your datastore we fail?
18:31:28 <hub_cap> no not at all
18:31:31 <hub_cap> this is how configs work
18:31:34 <hub_cap> 1) in memory
18:31:36 <grapex> hub_cap: Ok, just checking
18:31:44 <hub_cap> 2) overrides from disk if avail
18:31:48 <hub_cap> 3) failures
18:32:07 <hub_cap> opt groups get funky since u have to identify them
18:32:17 <hub_cap> oslo wont "Default" for u
18:32:42 <hub_cap> if u specify cfg.somegrp.someval, if someval is in default, it wont pull it
18:33:08 <hub_cap> itll look for [somegrp]...someval=blah, and if it doesnt find, itll use whats in memory for that value, if any
18:33:15 <cp16net> sounds like something we could put a bug in oslo for?
18:33:16 <hub_cap> only for that memory in that optgroup
18:33:22 <hub_cap> its not a bug
18:33:30 <hub_cap> its the design of using different optgroups
18:33:38 <grapex> hub_cap: Ok- I thought you were saying oslo could be smart enough to use someval from default is cfg.somegrp.someval wadn't found
18:33:40 <grapex> However
18:33:43 <SnowDust> hub_cap : so the current BP implements the defaults
18:33:53 <hub_cap> we could do cfg.DEFAULT if we wanted :)
18:33:53 <SnowDust> i said when cfg is imported
18:33:57 <cp16net> ok
18:34:03 <grapex> you could have code in a datastore that checked to see if the group specific default was not None, and if it was try the default one
18:34:10 <SnowDust> the load_datastore_opts()
18:34:13 <SnowDust> function is called
18:34:20 <SnowDust> which loads the mapped datastores
18:34:25 <hub_cap> grapex: or just put it in a wrapper object, like configuration object
18:34:35 <grapex> hub_cap: Yeah
18:34:38 <hub_cap> u pass in None, or the optgroup, like 'cassandra'
18:34:42 <hub_cap> and then just do configuration.blah
18:34:47 <hub_cap> instead of cfg.blah
18:35:29 <SlickNik> So not getting caught up in the implementation details, and coming back to the bp.
18:35:38 <hub_cap> yes :)
18:35:59 <hub_cap> i think its ok to move the options to config groups for datastores
18:36:10 <hub_cap> assumign we can interact w/ them in a easy way ;)
18:36:14 <SlickNik> I think that the bp does simplify some of the datastore config value loading.
18:36:52 <hub_cap> id like to see a 2nd patchset w/ options actually updated before i can merge p1 :)
18:37:26 <hub_cap> as in a dependent patch set that shows the code updating, say, mysql, to use the optgrp
18:38:36 <SnowDust> hub_cap int_tests just show that what options are being selected for mysql
18:38:48 <hub_cap> but as it stands, im ok w/ this
18:39:52 <SlickNik> Same here, I'm okay with the bp.
18:40:05 <SlickNik> The implementation details might need some tweaking.
18:40:17 <grapex> Ok- sorry
18:40:29 <grapex> this all seems simple but I feel like I'm fuzy on some of the details we're talking about
18:40:37 <SlickNik> And I think it'd be easier to make some of these comments in gerrit.
18:40:45 <grapex> SlickNik: +1000
18:40:47 <hub_cap> yea after looking more at the code, it looks like we are already calling cfg.get(datastore_mgr).tcp_ports
18:40:52 <hub_cap> thats the only place we are doing it
18:40:52 <grapex> just to be sure, none of us want to load all of the datastore options in the common/cfg.py right?
18:41:18 <grapex> robert just pointed out on the last line here, it loads all of the config options for everything which seems a bit much: https://review.openstack.org/#/c/80061/5/trove/common/cfg.py
18:41:48 <SnowDust> grapex: yes
18:41:51 <robertmy_> all other openstack projects do not do this
18:41:55 <SnowDust> grapex: i suggested in last discussion
18:41:57 <hub_cap> robertmy_: ?
18:41:59 <SnowDust> this may be further trimmed
18:42:06 <SnowDust> if we want to do it from conf files only
18:42:08 <hub_cap> everything exists in a big global cfg
18:42:09 <robertmy_> they use the option groups in the file
18:42:17 <robertmy_> hub_cap: no
18:42:18 <grapex> SnowDust: Sorry, I guess we didn't all see that.
18:42:21 <hub_cap> robertmy_: tahts what he says this is doing
18:42:23 <SnowDust> so that only the one which is being enabled gets loaded
18:42:32 <hub_cap> robertmy_: this is a change then :)
18:42:58 <grapex> hub_cap: I don't get the reasoning
18:43:05 <grapex> why force all the config options to load when cfg.py is parsed?
18:43:14 <hub_cap> well wait up, i need robertmyers to explain
18:43:17 <robertmyers> let me find an example
18:43:34 <hub_cap> plz do, cuz last i thought, everyon grabs cfg.CONF
18:43:45 <hub_cap> and its parsed on load (thats how nova used to work)
18:43:48 <hub_cap> again, *used to*
18:44:07 <hub_cap> ok so maybe we need to move this convo out of here
18:44:09 <hub_cap> and move on
18:44:27 <grapex> hub_cap: https://github.com/openstack/nova/blob/master/nova/api/auth.py#L46
18:44:35 <robertmyers> https://github.com/openstack/cinder/blob/master/cinder/volume/drivers/san/san.py
18:44:57 <robertmyers> so you see  the option groups are defined at the point they are used
18:45:00 <hub_cap> right its still a global CONF
18:45:02 <grapex> That's what robert_my means- auth_opts are simply defined next to where they are used, and no special code exists to parse them as the root cfg.py is created
18:45:22 <hub_cap> oh u mean just the loading.. yea that can be taken out of the global cfg stuff
18:45:32 <hub_cap> there is no need for them to be loaded before their impls are loaded
18:45:42 <robertmyers> hub_cap: it is a global config file, but not a global object
18:45:54 <hub_cap> cfg.CONF ??
18:46:11 <hub_cap> im pretty sure that using CONF everywhere defines a global obj right?
18:46:20 <robertmyers> well, it all reads/extends cfg.CONF
18:46:24 <hub_cap> my python-speak may be wrong
18:46:36 <hub_cap> everything uses CONF right?
18:46:38 <robertmyers> technically yes
18:46:54 <hub_cap> ok i think we are speaking the same thing
18:46:59 <robertmyers> but, it is lazy loaded
18:46:59 <hub_cap> i agree we dont need to "preload" Them
18:47:04 <robertmyers> ok
18:47:12 <hub_cap> the point of cfg is to allow u to define groups in the files they are used (or imported)
18:47:14 * grapex wipes sweat off his brow
18:47:18 <kevinconway> i think we should change the conf files to YAML
18:47:30 <hub_cap> kevinconway: if i only had /kick privs
18:47:31 <robertmyers> xml
18:47:35 <grapex> kevinconway: https://www.youtube.com/watch?v=4DgbUBoxa48
18:47:48 <hub_cap> so lets remove the preload stuff SnowDust
18:47:51 <hub_cap> and just let the app load it
18:48:05 <pdmars> lol
18:48:13 <hub_cap> and id like to see you update an impl to use more than just whats defined now in the optgropu, which is tcp_ports
18:48:26 <hub_cap> unless thats done
18:49:02 <hub_cap> we are already doing that in places
18:49:07 <hub_cap> so then we dont need to see that done
18:49:24 <hub_cap> so just removing the autoload will satisfy everyone, right?
18:49:39 <hub_cap> and we can make a different bp for the "fallback" to default
18:49:43 <grapex> hub_cap: aye
18:49:58 <SnowDust> hub_cap u mean i should not "REGISTER" the groups ?
18:50:06 <SlickNik> +1 on the lazy loading. I have a couple other comments, I'll add them to gerrit :)
18:50:11 <hub_cap> SnowDust: u register the opts/group _in_ the file
18:50:15 <robertmyers> +1
18:50:16 <hub_cap> whih i thought u were doing already
18:50:57 <hub_cap> https://review.openstack.org/#/c/80061/5/trove/guestagent/datastore/cassandra/options.py
18:51:06 <hub_cap> see you are already registering the group and opts
18:51:21 <hub_cap> so if u just completely remove all the code from cfg, itll work
18:51:37 <kevinconway> hub_cap: all the code?!?
18:51:42 <hub_cap> ALLL
18:51:44 <SnowDust> i dont get
18:51:53 <hub_cap> SnowDust: offline
18:51:57 <hub_cap> lets move on we have 9 min heh
18:52:07 <SnowDust> until u register a group
18:52:07 <SnowDust> how do u use the group options ?
18:52:12 <hub_cap> SnowDust: offline
18:52:14 <hub_cap> lets move on we have 9 min heh
18:52:27 <hub_cap> #topic remove mockito
18:52:28 <SnowDust> hub_cap i do register them in the options.py of datastore module
18:52:28 <SnowDust> yes
18:52:34 <hub_cap> #link https://blueprints.launchpad.net/trove/+spec/remove-mockito
18:52:37 <SnowDust> but we need to import the group in cfg
18:52:41 <hub_cap> so whos doing this?
18:52:45 <SnowDust> so that cfg.CONF
18:52:45 <SnowDust> can then be used to call them
18:52:45 <hub_cap> SnowDust: offline
18:52:50 <hub_cap> SnowDust: offline!!!!
18:52:51 <SnowDust> using CONF.mysql.root_on_create etc
18:52:51 <SnowDust> hub_cap ?
18:52:53 <hub_cap> SnowDust: offline!!!!
18:52:57 <SnowDust> hub_cap sure !
18:52:57 <SnowDust> hub_cap : sure !
18:53:09 <SnowDust> thanks ALL :)
18:53:23 <hub_cap> SnowDust: u will have to reiterate Q offline plz
18:53:26 <hub_cap> SlickNik: is this u?
18:53:32 <SlickNik> yup, I added this
18:53:34 <hub_cap> whos tackling removing mockito?
18:53:51 <juice> i can lend a hand
18:54:01 <SlickNik> I looked into this yesterday, probably gonna be juice and me.
18:54:50 <hub_cap> cool, anything to add to it?
18:54:52 <SlickNik> Lots of mockito in the tests, probably gonna take 2-3 days to get it done.
18:54:56 <hub_cap> yea for sure
18:55:05 <esp> SlickNik: is there any plan to remove mox?
18:55:25 <juice> esp: I guess mox is still in the mix
18:55:28 <SlickNik> esp: right now it's not an issue since mox exists in the global_requirements.
18:55:29 <robertmyers> do we use mox?
18:55:31 <juice> but we shouldn't do it
18:55:48 <juice> robertmyers: I think there was some mox in the trove code
18:55:56 <esp> robertmyers: I think there is some unit tests in python-troveclient that may use mox
18:55:57 <SlickNik> robertmyers: I haven't seen it in many places; but I haven't looked too hard.
18:56:13 <esp> is the one from google I believe
18:56:24 <robertmyers> let not use it
18:56:26 <esp> *it's
18:56:28 <SlickNik> Timeline wise, I think we might have to do this before the icehouse cut.
18:56:31 <esp> robertmyers: +1
18:56:45 <hub_cap> yes SlickNik lets focus on mockito
18:56:50 <hub_cap> and make esp do the mox stuff
18:56:53 <hub_cap> ;)
18:56:58 <SlickNik> esp: thanks! :)
18:56:59 <esp> hehe
18:57:02 <esp> np
18:57:06 <SlickNik> that's all I had.
18:58:11 <juice> i have to run - talk to you guys later
18:58:43 <SlickNik> juice: I'll catch you offline to chat about mockito.
18:58:45 <SlickNik> thanks!
18:59:13 <hub_cap> ok so the last topic
18:59:14 <hub_cap> no time
18:59:18 <esp> lol
18:59:20 <hub_cap> who put it on?
18:59:30 <hub_cap> we can move to #openstack-trove to discuss
18:59:35 <hub_cap> anyone?
18:59:43 <SnowDust> yep
19:00:06 <hub_cap> ok so if anyone wants to discuss container vs join, goto the channel
19:00:09 <hub_cap> #endmeeting