21:01:27 <rocky_g> #startmeeting log_wg
21:01:28 <openstack> Meeting started Wed Jan  6 21:01:27 2016 UTC and is due to finish in 60 minutes.  The chair is rocky_g. Information about MeetBot at http://wiki.debian.org/MeetBot.
21:01:29 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
21:01:31 <openstack> The meeting name has been set to 'log_wg'
21:02:18 <rocky_g> Hi folks!
21:03:35 <rocky_g> anyone here for logging?
21:04:33 <rocky_g> jokke_, you here?
21:05:25 <rbradfor> rocky_g, hi
21:05:31 <rocky_g> Hey!
21:05:47 <rocky_g> T#topic default config options for logging
21:06:37 <rocky_g> I've been following the discussion, but need some clarification
21:07:18 <rbradfor> is this a ML discussion?
21:07:25 <rocky_g> supposedly the devstack config options are friendlier to humans, but do they scale for large installations?
21:07:47 <rocky_g> It is the one you started
21:07:51 <rbradfor> ok
21:09:45 <rbradfor> sure, this means devstack can change the logging_user_identity_format configuration option, to use names rather then ids
21:10:05 <rbradfor> but the context and no context format strings are returned to defaults.
21:10:13 <rocky_g> right.  but names can have duplicates, can't they?
21:11:05 <rbradfor> within a given project I hope not.
21:11:49 <rbradfor> where a user can be a duplicate in project/domain/tenant I am unsure of
21:11:53 <rocky_g> ah.  Then as long as the user_name:project_name tuple is there, it should be unique?
21:12:37 <rbradfor> the user_identity is defined as "%(user)s %(tenant)s %(domain)s %(user_domain)s %(project_domain)"   5 identifying attributes
21:13:55 <rocky_g> That seems like a better option.  More verbose, but then Ops just changes if its too much info.  But better for large installations
21:14:32 <rbradfor> what do large installations use now.  I would be interested in knowing the configuration actually used on a site.
21:15:59 <rocky_g> we should start the discussion on the openstack-ops mailing list and see what replies we get.  We can also go to the large installation ops meeting and ask there/put it on their agenda.
21:17:29 <rbradfor> I want to work with submitting a devstack patch on changing the right logging configuration variable.  This needed https://review.openstack.org/#/c/261122/ which is now merged
21:19:11 <rocky_g> The more default settings ops can use, the better as the configs get dynamically generated for new releases, then have to be reconciled during an upgrade.  And yeah, that requirements change is really important
21:20:18 <rocky_g> Any suggestions on how to stay on top of when to change that version?
21:21:10 <rbradfor> the change of oslo.log in global requirements should be already in projects. I've not verified
21:21:49 <rocky_g> It's automatic.  It's merged, so the builds all happen with the new value.  Being tested every gate run
21:22:22 <rocky_g> Unless a project has specifically set it in their own requirements, and they shouldn't
21:25:25 <rocky_g> Have you walked through the process of how the config files are generated for a build?  there is a ml discussion on that wrt Cinder happening now/last few days
21:26:08 <rbradfor> are you referring to the oslo-config-generator
21:26:13 * rbradfor looking for ML discussion
21:27:28 <rbradfor> #link http://lists.openstack.org/pipermail/openstack-dev/2016-January/083280.html
21:27:42 <rocky_g> Ah, thanks.
21:29:29 <rbradfor> coming from prior experience (and not seeing a production OS system to validate), I do not like what I've seen about configuration generation and information about the configuration options.
21:29:59 <rbradfor> I have yet to confirm, but I suspect the oslo-config-generator sets configurations options, or some of them, rather then comments of options.
21:30:14 <rbradfor> and so, an option is set, when indeed it's value is the system default.
21:30:24 <rocky_g> Nope.  The documentation is very poor and the generation has to happen via running tox
21:30:33 <rbradfor> I would (with my operator hat on), not what to see that in a config file, as it's clutter.
21:30:55 <rbradfor> the second problem is, what are the configuration options of a service running now, or when it started.
21:31:10 <rocky_g> Yup.  One of the things the log wg wants to fix.  Once we figured out the issue
21:31:19 <rbradfor> even the logging (when in DEBUG) does not log if a configuration option value is a default, or specified in a config file.
21:32:04 <rocky_g> Poor ops have to go project by project to collect the options.  But, I think there is a cli for some of the projects to show current options
21:32:07 <rbradfor> generating a configuration file from source each release is I feel a brute force approach and as you point out, likely a more difficult upgrade approach
21:32:43 <rocky_g> Yup.  The projects used to generate sample config files, but couldn't keep them up to date.
21:32:50 <rbradfor> I have no yet researched, but the oslo reports guru meditation report is how a project can obtain this.
21:33:00 <rocky_g> Oh, there is a review on the config stuff, lemme find it.
21:37:16 <rocky_g> arg!  still looking.  It's the dynamic config changes one
21:38:54 <rocky_g> oh, right.  checking wrong project.  just a sec...
21:39:51 <rocky_g> #link https://review.openstack.org/#/c/251471/
21:41:45 <rbradfor> this review is part of a larger discussion to enable reloading a config file to change just the overall logging level
21:41:54 <rocky_g> And your up to speed on it.
21:42:16 <rocky_g> not just logging, but logging is the POC.  It will expand to all mutable options
21:42:39 <rocky_g> But what a pain to have to go project by project.
21:43:21 <rocky_g> there really needs to be a global config file.  That would reduce the size of project config files tremendously
21:43:45 <rocky_g> And improve consistency across projects
21:43:49 <rbradfor> I am aware of the issues about reloading config files (ie. what's now defined and changed, and is or is not mutable), how is correctly triggered, how individual projects will need to change in order to enable this capability
21:44:46 <rbradfor> I would agree that a single common view of a "product" configuration is definitely important, but, there is a lot of issues in maintaining compatibility.
21:45:18 <rbradfor> I have not seen anybody push for using a global configuration file initially.
21:45:44 <rocky_g> Question:  if there are changes needed within the projects, do all projects need to change?  And, if so, would it make sense to propose the global config file cascading to the project configs? to get done around the same time?
21:45:47 <rbradfor> adding [some] logging options to said file may be useful.
21:46:36 <rbradfor> the issue with implementing the mutable capability is the loading/reloading of a configuration file, and maintaining backwards compatibility.
21:47:00 <rocky_g> The ops guys would love to be able to set the log format in one place.  And often the levels.  But until the levels are the same across all projects, it's hard to get the levels right.
21:47:10 <rbradfor> in my interpretration, when implemented a project must elected to enable having mutable options, and change this.
21:48:27 <rocky_g> The mutable changes will be really hard with the multiple locations of config options.  How do you know when you've got them all, or do you list the projects you want the change for?
21:48:37 <rbradfor> in one regard I would agree with you, but if the global level was INFO, in a new global config,  and you wanted just DEBUG in nova,  you need to add that now to nova config and reload.  I think in theory this is a use case, it's just how to implement it that seems complex
21:49:40 <rocky_g> Agreed.  But the implementation is hard because of the lack of federation across the projects.
21:49:53 <rbradfor> the "concept" of a mutable option needs to first be implemented in code.
21:50:11 <rbradfor> the choice of what options can then become "mutable" needs to happen across projects.
21:50:32 <rocky_g> Yeah.
21:50:44 <rbradfor> if it's say an Oslo Logging option, and Oslo logging is used by project, then that's sort of global.
21:51:48 <rbradfor> I suspect nobody may know what are the configuration options taxomony across all projects, and where can/should consolation occur.
21:51:55 <rbradfor> we are only talking about a dozen logging options.
21:52:01 <rocky_g> That's where I would like to start with global config options.  Start with all the oslo config options and defaults in one file or directory.  Then overlay the project specific options/defaults.  once we have that, we can start removing dupes in projects
21:52:58 <rocky_g> And the config options are poorly documented for many of the projects.
21:52:58 <rbradfor> has any work or discussion happened on global config files.
21:53:39 <rbradfor> I concur *developer* documentation I have observed is poor. What I saw of *operator* docs was just as confusing.
21:54:00 <rbradfor> even if the docs had a "global section", i.e. these options are in each project, would simplify the docs
21:54:02 <rocky_g> No.  Not even raised yet.  If we put together a spec and got some comments on it, we could have a summit session and target N for phase 1
21:54:45 <rocky_g> Yup.  I started that conversation with the docs ptl.  She's up for something like that.
21:55:41 <rocky_g> And providing that info would go a long way in helping the ops guys and even the devs get a handle on what is the same and what is special per project
21:57:29 <rocky_g> Question:  would starting with the Nova opts documentation as a template/talking point be a good way to approach the docs team about the global stuff?
21:58:26 <rbradfor> I'm sure getting one project to adopt improvements makes it easier to persuade other projects.
21:58:32 <rbradfor> Do you know who is responsible for Nova docs.
21:59:46 * rbradfor looking for actual link to reference
21:59:57 <rocky_g> It's the nova team.  There may be some dev folks that specifically focus on the docs, and there my be some specialists in the docs team, but I think it's project by project how docs are created.
22:00:33 <rbradfor> are you referring to http://docs.openstack.org/liberty/config-reference/content/list-of-compute-config-options.html
22:00:36 <rocky_g> Oslo has doc sprint days to add info.  It helped a lot from Kilo to Liberty.  But mostly it went into dev docs.
22:01:06 <rbradfor> I also have found http://docs.openstack.org/openstack-ops/content/logging_monitoring.html but this does not seem to refer to configuration
22:01:31 <rocky_g> yp.  the first.
22:01:44 <rocky_g> the ops stuff is stuff generated by operations people.
22:02:13 <rocky_g> sorry.  No, it's not.
22:02:27 <rocky_g> The config stuff is all supposed to be in the config-reference
22:02:32 <rbradfor> #link http://docs.openstack.org/liberty/config-reference/content/list-of-compute-config-options.html
22:02:38 <rbradfor> #link http://docs.openstack.org/liberty/config-reference/content/keystone-configuration-file.html
22:02:43 <rocky_g> You can see that there is huge variation project to project
22:03:15 <rbradfor> examples of logging_context_format_string configuration for exmaple that is duplicated in config-reference, when ideally it should be a single source.
22:03:45 <rocky_g> link http://docs.openstack.org/liberty/config-reference/content/section_volume-misc.html
22:04:05 <rocky_g> But they also have a config files section, which may have more stuff
22:04:24 <rocky_g> So, each project *should* have a section, but they're not consistent
22:05:22 <rbradfor> there needs to be a central body that need to set and enforce project consistency, looking at configuration files when I first installed a working OS showed there is no consistency (and for operators this must be very frustrating)
22:05:48 <rocky_g> Yup.
22:06:25 <rocky_g> I can maybe help drive some of it from the docs side.  But, maybe we should bring it up in cross-project meeting?
22:07:06 <rocky_g> I was also thinking that the Nova-API wg really should be the OpenStack-API wg and should apply across all projects
22:07:34 <rbradfor> I feel there needs to be a use case, i.e. a collection of production operators supporting a request to standardize.
22:08:09 <rbradfor> but I suspect one needs to first suggest to operators some examples, get feedback, and then take to the cross-project meeting, which then lead to a ML (dev) discussion
22:08:20 <rocky_g> I can put together a user-story for the prod-wg and get it in and discussed.
22:10:03 <rocky_g> The logging issues, the config issues, all trace back to the same root.
22:10:21 <rocky_g> And the Ops folks hit it first.
22:11:01 <rocky_g> The whole consistency thing is also where a real QA team would have focused on early.
22:11:46 <rbradfor> the history is well history. what is important is where to start to produce consistency, and what key projects need to be onboard.
22:12:20 <rocky_g> the core:  nova, glance, cinder, keystone, neutron (would be nice to have swift, but not likely)
22:12:25 <rbradfor> I have not investigated nova in any detail, but is this the largest driving project for improvements like this
22:13:00 <rocky_g> It's also a good internship project, discovering current state including duplications
22:13:37 <rocky_g> nova currently has a good core team, and a good number of cores who will understand the issues when presented.
22:13:46 <rocky_g> and a good ptl
22:14:04 <rocky_g> Oslo needs more bodies.
22:14:06 <rbradfor> and what operators do you feel are a good voice
22:14:17 <rocky_g> Neutron is too young and scattered.
22:14:26 <rocky_g> Keystone could also do it.
22:16:05 <rocky_g> So, I would think a good approach would be to:  get all the oslo config opts and defaults identified/documented, then approach Nova and Keystone to work together.
22:17:07 <rocky_g> There are a number of ops folks who might work.  JJ Asghar, some of the Jessies, K Lindgren...
22:17:12 <rocky_g> A few more.
22:17:23 <rocky_g> Jim Meyers might also be able to help.
22:18:32 <rocky_g> The rackspace guys, the comcast guys and the godaddy guys really would love to see improvements.
22:19:29 <rocky_g> Those teams will help with the work.  JJ, some of the infra guys and one or two other Ops guys could be the voice.
22:20:16 <rocky_g> Jeremy Stanley.  Or whoever wrote infra's log parsing rules.
22:20:17 <rbradfor> I have heard and read a lot from the "developer" side, but I would like to hear it from operators.  Our top goal should be X first, how do we get that in the next release. With no contact I do not know what X is.
22:20:53 <rocky_g> That's why you should come to the Ops Midcycle ;-)
22:21:28 <rocky_g> Oh, a great voice for ops is Tom Fifield.  He's the foundation ambassador to those folks and the community manager.
22:21:32 <rbradfor> Is it first standardizing the values of a subset of config options (i.e. logging), ensuring it's clearly documented in developer/operations (not duplicated), and confirmation based on what operators actually use in production systems today.
22:21:51 <rocky_g> And he might get you access to a working cloud or two.
22:24:06 <rocky_g> Also, Tom knows who to reach out to.
22:26:22 <rocky_g> So, I'm gonna end this meeting, just about a half hour late.  But, we have the log.
22:26:34 <rocky_g> Any last words you want recorded?
22:26:39 <rbradfor> as I get a better understanding of things it would be good to be introduced. I'm doing code cleanup and working the developer docs at present. I would like to see how to get information into the right operator docs, because operators should not need to look at deveoper docs
22:26:57 <rbradfor> I changed the wiki page to reflect the meeting time.
22:27:28 <rbradfor> #link https://wiki.openstack.org/wiki/Meetings/log-wg   now 21:00 UTC
22:27:51 <rocky_g> Thanks!
22:28:15 <rocky_g> #endmeeting