21:05:35 <rocky_g> #startmeeting log_wg
21:05:36 <openstack> Meeting started Wed Jan 27 21:05:35 2016 UTC and is due to finish in 60 minutes.  The chair is rocky_g. Information about MeetBot at http://wiki.debian.org/MeetBot.
21:05:38 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
21:05:40 <openstack> The meeting name has been set to 'log_wg'
21:05:50 <rbradfor> o/
21:06:03 <rocky_g> sorry I'm late
21:06:09 <rocky_g> jokke_, you there?
21:06:31 <jokke_> yes
21:06:33 <jokke_> o/
21:06:49 <jokke_> sorry, just got in
21:06:56 <jokke_> so I was bit late as well
21:07:23 <rocky_g> OK.  jokke_ meet rbradfor  and vice versa
21:07:48 <rbradfor> jokke_, hi,hi
21:08:18 <rocky_g> and rbradfor, monty says hi!
21:08:33 <rbradfor> monty taylor?
21:08:39 <rocky_g> yup
21:08:41 <jokke_> hi rbradfor \o
21:09:09 <rocky_g> we were discussing logging and I mentioned you.
21:09:28 <rbradfor> yep, we have known each other, almost 10 years now.
21:09:52 <rocky_g> Wow.  So, monty was like four?
21:10:29 <jokke_> :P
21:10:33 <rocky_g> #topic status updates
21:10:55 <rocky_g> rbradfor, I know you're working on config stuff.  Whaere are you and how can we help?
21:11:09 <rbradfor> From an action item of last meeting I generated Sphinx docs of configuration options for 5 of 6 listed projects (swift doesn't play ball) to see what was possible.  My demo was at http://ronaldbradford.com/tmp/openstack-opts/
21:11:17 <rbradfor> I took this one step further and integrated Glance into full docs which provided some surprises. https://review.openstack.org/#/c/270920/ has merged, you can see the full glory at http://docs.openstack.org/developer/glance/opts.html
21:11:41 <rbradfor> surprises are things we should discuss in relation to configuration reference guide
21:12:27 <rocky_g> excellent.  Get to them before the endusers see them
21:12:33 <jokke_> rbradfor: oh that was you ;)
21:12:47 <rbradfor> jokke_, ??
21:13:23 <jokke_> rbradfor: didn't realize to relate the nick and name together, that's all
21:13:37 <rocky_g> jokke_'s main job is glance core
21:13:43 <jokke_> I was looking that change but Flavio and Sabari were faster approving
21:13:55 <rbradfor> jokke_, ok, well I didn't know that either.
21:14:10 <jokke_> so, thanks for that :D
21:14:35 <rbradfor> I just picked glance at random but the interaction with Sabari was very helpful towards decisions one may make in a operator config guide
21:15:05 <rocky_g> please elaborate..
21:15:14 <rbradfor> Flavio (who I did not know is PTL) gave a nice review feedback I appreciated.
21:15:28 <rbradfor> so, some points of discussion
21:15:42 <rbradfor> 1. what do you do when a project has multiple configuration files.
21:15:55 <rbradfor> in this case I put all files into one page.
21:16:03 <rbradfor> this has advantages and disadvantages.
21:16:32 <rbradfor> the main disadvantage is if you look at just one file, you don't see all possible options.
21:16:55 <rbradfor> what you get (and it took some formatting) is specific options and a "common libraries" section
21:17:03 <jokke_> glance has actually at least 5 config files + policy and -paste.inis
21:17:05 <rbradfor> which is then linked to the end of the document.
21:17:34 <rbradfor> jokke_, I documented 5 I knew of (based on entry_points)
21:17:47 <jokke_> cool
21:18:06 <rbradfor> what I learned is that you include other namespaces, via your opt/oslo-config-generator/<file> options (something knew I learned)
21:18:36 <rbradfor> my first instinct was to include all namespaces into each file, but I could not actually do that, I ended up with duplicate DEFAULT sections (a later topic)
21:18:49 <rbradfor> and, we would end up with what we are trying to avoid, repeating the same content.
21:19:12 <jokke_> ++
21:19:15 <rbradfor> so while a disadvantage you don't see all options possible, the advantage is I did not duplicate (for example logging options)
21:20:03 <rbradfor> so, debatable if it should be one configuration file per page, and another page of "included namespaces", all linked nicely, or one page as it is now.
21:20:38 <jokke_> glance is actually one of the rare projects still carrying example configs in tree, so in that perspective it does not make that huge of a problem either way
21:20:52 <rbradfor> and how it is now, could be achieved with present sphinx directives. creating one all encompassing file would have met a oslo.config code change
21:21:22 <rbradfor> jokke_, I liked the use of the oslo-config-generator conf file, I hope other projects use that.
21:21:25 <jokke_> I personally like the idea of having the documentation all in single page so it's easy to search when you are crawling through those configs
21:21:37 <rbradfor> infact, I'd propse we create a sphinx option to read these files specifically.
21:21:47 <rocky_g> ++
21:21:54 <jokke_> rbradfor: I don't, but that's different story ... Hopefully we get the reason fixed in oslo-config-generator
21:22:06 <rbradfor> so the docs creatiion uses one conf file, and generates sample configs and matching docs.
21:22:22 <rbradfor> jokke_, whats the fix you speak of?
21:23:02 <jokke_> well there is no fix yet, but the problem is that if you change something in the config options (at the code level) you get XX% different output from oslo config generator
21:23:04 <rbradfor> as I just fixed another bug in this area, happy to hear your input while it's fresh
21:23:21 <jokke_> making config management absolute nightmare
21:23:28 <jokke_> and thus upgrade as well
21:23:56 <rbradfor> ok, well lets come back to this, because I'd like to know what it is so I can fix it
21:24:19 <rocky_g> FYI here, Nova is working to merge all their config files into a single one, with clear "sections"
21:24:23 <jokke_> IIUC Stuart McLaren opened bug against it just last week
21:24:30 <jokke_> haven't got to look into it yet
21:24:42 <jokke_> rocky_g: we are not :P
21:24:48 <rocky_g> So, I think there will be a trend in Newton to do the same in other projects.
21:25:11 <rbradfor> I've seen a lot of reviews on nova cleanup
21:25:23 <rocky_g> Yup.
21:25:30 <jokke_> there has been push for that at least for 3 cycles now, and I'm happy to keep -2ing any proposed changes for such :P
21:25:49 <rbradfor> speaking of sections, this is a problem I found, and I'm going to raise it in Austin, that is having logging not be in [DEFAULT] but in a [logging] section
21:25:54 <jokke_> same with removing the examples from tree
21:26:12 <rocky_g> Ah.  Interesting.
21:26:15 <rbradfor> jokke_, silly to remove conf examples from code.
21:26:30 <jokke_> rbradfor: that's purely because it comes from oslo_log, not from the project namespace
21:26:56 <rocky_g> I'd love to see a proposal to have a "global" config that only gets overwritten by individaul project config files.  Right now, no global
21:26:58 <jokke_> rbradfor: and tbh. I like the fact that all the config options are now under their own section, not scattered around [DEFAULT]
21:27:17 <rbradfor> yes, because it's a namespace, and a few others benefits
21:27:30 <rocky_g> So, instead of "default" how about "global" with sections?
21:27:39 <rbradfor> 1 is the generation of "common libraries". all others have there own section
21:27:42 <jokke_> rocky_g: the problem with that is that it's great idea for devstack, but kind of pointless when you deploy the services to their dedicated machines
21:28:04 <rbradfor> and also if you want to move to rocky_g goal of a higher order global config I see sections as being beneficial
21:28:59 <rocky_g> I think ops would take the config files from a repo and write a distribution script to put it on the appropriate iron.
21:29:15 <jokke_> I think the oslo-config-generator is the right way to go if we get projects not to change those defaults what oslo__log offers
21:29:44 <jokke_> that would give that global default and it would be up to deployer to change if they insist
21:31:56 <rocky_g> Well, it would be good to have a way to identify all the cross project config options separate from the project specific ones and have one place to change them...
21:32:54 <rbradfor> rocky_g, it will take some time to get standardization first across projects. the siloed approach now has some big effects in just naming things consistently.
21:33:02 <rocky_g> So far as I am aware, that's only oslo stuff at the moment.
21:33:10 <rocky_g> Yeah.  Don't I know.
21:33:47 <jokke_> rbradfor: I'm bit afraid of that naming consistently part as well ... afraid that we will get even more collisions
21:33:56 <rocky_g> The cool thing about the generator is that it might be easier to identify conflicting options settings
21:34:50 <rocky_g> could we do it for oslo stuff, though?  And instead of a "default" or "global", just have an "oslo" section?
21:34:54 <jokke_> well as long as they are in different sections I don't think it cares, nor should/can
21:35:26 <jokke_> but when template update needs stuff like this https://twitter.com/epkuva/status/691669708756631552 naming collisions becomes really painful
21:36:15 <rbradfor> rocky_g, just to digress a moment on sections
21:36:18 <rbradfor> check out http://docs-draft.openstack.org/20/270920/4/check/gate-glance-docs/0e8ef17//doc/build/html/opts.html
21:36:40 <rbradfor> which I just discovered differs from http://docs.openstack.org/developer/glance/opts.html (something else to look at)
21:36:52 <rbradfor> you can see inthe first, in the left menu, all the sections
21:37:48 <rbradfor> there is already inconsistency in section headings for oslo, so I doubt we will get a common concensus, and I don't think having oslo is needed?
21:38:18 <rocky_g> oslo is only needed if there is a change from the oslo defaults
21:39:13 <rocky_g> but, there should be links to the relavent common, included other opts
21:39:43 <jokke_> rbradfor: actually good example of that sorting issue I mentioned
21:40:29 <rbradfor> jokke_, are you talking about https://bugs.launchpad.net/oslo.config/+bug/1536899
21:40:30 <openstack> Launchpad bug 1536899 in oslo.config "config generator sorting has changed" [Undecided,Fix released] - Assigned to Ronald Bradford (ronaldbradford)
21:40:34 <jokke_> http://docs.openstack.org/developer/glance/opts.html if you look for enable_v*_api ... so we have enable_v1_api, enable_v2_api, enable_v2_registry and enable_v3_api ... but all those are scattered all over the place
21:40:49 <jokke_> rbradfor: I think that would be the one, yes
21:41:01 <rbradfor> jokke_, I fixed that, it merged a few days back
21:41:11 <jokke_> oh brilliant!!!!
21:41:12 <rbradfor> docs regenerated will go back to "source code" order
21:41:29 <jokke_> rbradfor: will that be just for the docs or for the config files as well?
21:41:42 <jokke_> I mena for the generated example configs
21:41:49 <rbradfor> I'm more concerned with the draft docs and the final ones about the left menu, and section headings and formats, something is off there.
21:42:03 <rbradfor> jokke_, it's the same code so it would fix both.
21:42:54 <jokke_> rbradfor: that's best news I've heard for a while ... if you don't mind, drop link to that fix to the bug and feel free to mark it fixed or invalid, which ever way you prefer ;)
21:43:14 <rocky_g> the option group seems to clean up a lot of the toc stuff.  It drops a lot of the options out of the display level
21:43:37 <rbradfor> jokke_, https://review.openstack.org/#/c/272289/ has merged, so launchpad should mark this as closed automatically?
21:44:08 <rocky_g> It's marked fix released
21:44:08 <jokke_> rbradfor: oh yeah, it actually has (Fix released)
21:44:54 <jokke_> makes our life so much better for the future, specially upgrades
21:45:01 <rbradfor> jokke_, yeah, that came to Oslo attention at our meeting on monday. As I was in that code on Friday is was easy to identify, and not to much to fix
21:45:16 <rbradfor> jokke_, so let's talk about upgrades a bit.
21:45:54 <rbradfor> let me start with "deprecated" because I've had a ML discussion on this.  I want all options marked as deprecated to have a reason, and that reason is to list "in what release it is to be removed", but that's just text.
21:46:20 <rbradfor> it seems to me, that new options, or changed defaults also merit applicable flagging, so it would be easier to generate docs,
21:46:45 <rbradfor> e.g. "Whats new", "Whats changed", "What's deprecated", "what's removed" sections of configuration options.
21:46:55 <rocky_g> yeah.  release note stuff at a minimum
21:46:58 <rbradfor> that IMO would help operators in upgrades
21:47:12 <rbradfor> I know there are release notes on this, but it's a human writing that.
21:47:37 <rocky_g> right.  auto generate would be ideal
21:47:42 <rbradfor> if this was meta data, then the oslo-config-generator for example could have additional info at it's disposal
21:48:02 <jokke_> rbradfor: I'd prefer instead of having "Deprecated; will be removed in release X.Y.Z" it having "Deprecated; will be removed on first release after 1.6.2016" or so
21:48:22 <jokke_> that would give clear deprecation period regardless what has been released in between
21:48:51 <rbradfor> jokke_, I could see that reason. please add to ML
21:49:13 <jokke_> rbradfor: will do tomorrow when I'm by my work mail again
21:49:21 <rbradfor> jokke_, sure
21:49:35 <rbradfor> so, back on the upgrade path.
21:50:04 <rbradfor> what's at an operators disposal to know "whats changed/new/removed". do they simply diff sample confs between releases
21:50:48 <jokke_> rbradfor: that's good one
21:51:20 <jokke_> specially when you look that bug you fixed ... good luck finding anything from those diffs
21:51:30 <jokke_> before I mean
21:52:00 <rbradfor> yes, before was death, code introduced did not retain order
21:52:00 <rocky_g> It's one of the reasons I think it takes them 3-6 months to qualify an upgrade.  They have to find all this stuff.
21:52:52 <rocky_g> The other thing is that they can't allow their config files to be overwritten, or if they do, they then have to go back in and replace their changes
21:52:58 <jokke_> rocky_g: I updated our templates, the tweet I linked was how I finally got it done and even then I missed some conditions there :(
21:53:38 <rocky_g> jokke_, here's the link to the ml thread:  #link http://lists.openstack.org/pipermail/openstack-dev/2016-January/084126.html
21:53:59 <jokke_> the change for glance-api.conf + glance-registry.conf was give or take 3.5k lines :(
21:54:12 <jokke_> thanks rocky_g
21:55:15 <rocky_g> so, the config stuff is a really big painpoint for ops
21:55:45 <jokke_> yes and rbradfor fixing that bug will ease it up Newton forwards
21:55:46 <rocky_g> there was a huge thread on it when the samples were being removed.  I'll see if I can find it...
21:56:44 <jokke_> as now folks gets version of configs from Mitaka and Newton will have just the real changes, not everything around the place
21:57:29 <rbradfor> so, if each config option had the following attributes, (released, changed, deprecated, removal)  aka the cycle (sorry names for now), it would be infinitely easy to auto-generate all the lists an operator wants between releases
21:57:56 <jokke_> makes sense
21:58:13 <jokke_> problem getting that released there at least
21:58:44 <jokke_> as the release version will be mainly defined at the point when the release is cut based on the changes merged
21:59:26 <rbradfor> yes, it's some meta data, but it's a transition project during a full cycle, it's not trivial, but if a project got behind it, and then maintained it.
21:59:39 <jokke_> and tracking such down will be too much to just throw at the release folks
21:59:40 <rbradfor> new options when first merged are marked with "released"
22:00:07 <rbradfor> then if changed, deprecated they are marked.  I'm sure the change/deprecate volume is way less
22:00:19 <jokke_> ahh ... so you did not mean that those are fields with version defined when that happened
22:00:31 <rbradfor> tracking down should be determable via git blame or stable release tages.
22:00:43 <jokke_> but rather state machine of the status of the option
22:00:46 <jokke_> ?
22:01:06 <rbradfor> it was just a thought now, let me describe it in an email with examples.
22:01:48 <jokke_> sure
22:02:06 <jokke_> good discussion folks ... unfortunately we're out of time
22:03:16 <rocky_g> oops.  you're right.  I've got a couple of mailing list links for rbradfor, then I'll call it.
22:03:35 <rocky_g> #link http://lists.openstack.org/pipermail/openstack-operators/2014-December/005658.html
22:03:54 <rocky_g> #link http://lists.openstack.org/pipermail/openstack-operators/2015-January/005981.html
22:04:22 <rocky_g> This is a thread on how operators were going to try to deal with the confg opts changes/issues/lack of sample configs
22:04:28 <rocky_g> Thansk all!
22:04:35 <rocky_g> #endmeeting