20:02:30 <rockyg> #startmeeting log_wg
20:02:31 <openstack> Meeting started Wed Aug  5 20:02:30 2015 UTC and is due to finish in 60 minutes.  The chair is rockyg. Information about MeetBot at http://wiki.debian.org/MeetBot.
20:02:32 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
20:02:34 <openstack> The meeting name has been set to 'log_wg'
20:02:43 <bknudson> hi
20:02:45 <rockyg> that looks right
20:02:52 <rockyg> Hey! bknudson
20:03:20 <jokke_> yeah
20:03:48 <rockyg> Agenda:  Old bidness; ops meetup; new bidness
20:04:16 <rockyg> #topic old business
20:05:15 <rockyg> requestID spec:  looks like it is on its final round of review before acceptance.  Time to jump in and catch anything that breaks logging sanity
20:06:16 <rockyg> Unfortunately, I'm on my own 'puter so don't have the review number, but its under openstack, no other project
20:06:47 <rockyg> Any other comments about that?
20:07:19 <jokke_> no I think it's getting there
20:07:55 <rockyg> moving on, bugs.  Any we highlighted or need to?  (or other reviews)
20:08:29 <rockyg> ok.  Moving on.
20:08:41 <rockyg> #topic ops meetup
20:09:36 <rockyg> I'll be there and will have a working session.  Plan on working on Error Code spec (hopefully just review comments from ops side if I get it the rest of the way)
20:09:46 <jokke_> good
20:10:09 <jokke_> try to get the folks joining to these meetings as well
20:10:23 <rockyg> I also hope to review and summarize past etherpads and get specs or bugs out for open items
20:10:42 <rockyg> And, yeah, recruit, recruit, recruit
20:10:44 <jokke_> nice ... that's next week?
20:11:32 <rockyg> week after.  I have a week to collect up stuff.  Anyone with bugs they want discussed, lemme know.  Or reviews for specs, bps, etc.
20:12:03 <jokke_> ok, puuh I was afraid already that time flew that badly ;)
20:12:27 <rockyg> And the meeting is day 1, so might get input written up from others for discussion day 2
20:13:14 <jokke_> sounds good ... we probably skip that week's meeting or?
20:13:22 <rockyg> I'm thinking maybe focusing on everyone's favorite error message "no valid host" for getting comments, bugs, etc
20:13:39 <rockyg> Yeah.  Most likely skip.  Good idea.
20:14:00 <rockyg> #info no meeting week of 8/17/2015
20:14:04 <jokke_> you and Doug are there
20:14:23 <rockyg> Yay!
20:14:48 <rockyg> bknudson, anything we should highlight from keystone, or your view of support on logging?
20:15:17 <bknudson> rockyg: nope... we've got lots of known problems already
20:15:29 <jokke_> :D
20:15:40 <bknudson> might be interesting to know if there's anything they're concerned about from a security perspective
20:15:45 <rockyg> any top one you want solved, or at least documented as needs to be solved?
20:16:22 <rockyg> Cool.  Security.  Yeah.  Ops have much better idea about the real security issues of logs than most
20:16:36 <bknudson> we were asked to get the request IDs in the keystone logs, so would be nice to get that fixed
20:16:40 <jokke_> was that bug tag log or logs?
20:16:41 <bknudson> also, the logs are useless in general
20:17:21 <rockyg> So, too general, or wrong message, or missing specific info?
20:17:45 <bknudson> they don't contain the right information to figure out what the problem might be.
20:18:44 <rockyg> can you shoot me a half page to page write up (actually, put it on wiki) on problems:  a) how it is now b) what would be helpful
20:19:03 <bknudson> I don't have time for that.
20:19:07 <rockyg> OK.
20:19:26 <bknudson> too busy trying to fix things
20:19:49 <rockyg> How about one instance in form or bug, or a pointer to an already existing bug?  Any chance you have that?
20:19:57 <bknudson> I don't.
20:21:30 <rockyg> ok.  If you happen to stumble on something when firefighting, cut and paste into a bug and cc me, maybe?  I'll have ops who specifically *aren't* on the front lines to help dig and maybe we can check one problem off for you.
20:22:00 <jokke_> rockyg: I think good work topic would be probe if the rest of the ops agrees and are willing to if not fix the logging issues, at least give few detailed bugs what would help keystone
20:22:15 <rockyg> Great idea
20:22:37 <rockyg> #action generate bugs that help fix keystone logs
20:23:00 <rockyg> And, those might span multiple projects, so we can also cross post
20:23:15 <jokke_> logging is such a nice topic they are easy low hanging fruits and generally straight forward to fix ... identifying the issues in details is the problematic part
20:23:36 <jokke_> and ops if any should be able to give those pointers
20:23:39 <rockyg> Yes, and until we have details, no dev will touch them
20:23:45 <bknudson> just try things yourself and see if the logs are adequate
20:24:05 <jokke_> https://bugs.launchpad.net/glance/+bug/1450209 this is nice from glance end ;)
20:24:05 <openstack> Launchpad bug 1450209 in Glance "Glance API logs a never ending stream of 404 tracebacks from glance-registry " [Medium,Confirmed] - Assigned to GB21 (geetika791)
20:24:37 <rockyg> Wow.  Tracebacks.  Jeez.
20:24:54 <rockyg> Well, I'll be well armed at least.
20:25:07 <jokke_> the registry client traces every time registry returns 404, which is for example always if the user does not have rights to view, list etc.
20:25:49 <rockyg> bknudson, is debug mode useful, or still bad for logging?
20:26:11 <rockyg> jokke_, is that in debug mode or?
20:26:19 <bknudson> well, you're not supposed to require debug mode to diagnose issues
20:26:37 <jokke_> rockyg: nope :P
20:26:38 <bknudson> on top of that, debug mode generally doesn't help much.
20:27:12 <jokke_> bknudson: ++ debug should be level needed _only_ by devs
20:27:13 <rockyg> bknudson, yeah, but if just one or a few dbug msgs turned into error msgs would help, changing levels is easy fix
20:27:51 <bknudson> that's not it. keystone needs to be rearchitected so that the info needed in the logs is available.
20:27:58 <rockyg> Ah.  The crux of the matter.  If debug can't debug, then even bigger problem.
20:28:54 <rockyg> Oh.  Gotcha.  So, it's likely a set of ops user stories, or a spec that calls out what needs to be readily available info for normal ops
20:29:03 <jokke_> we should move whole openstack to waterfall mode for 3 cycles so projects would have opportunity to rewrite the dong that has been cumulating ;)
20:29:23 <bknudson> yep, rewrite the dong.
20:29:28 <rockyg> ++
20:29:31 <rockyg> lol
20:30:14 <rockyg> so, that's an issue I'll bring up to product wg
20:31:09 <jokke_> biggest problem with agile/test driven development ... it becomes bubblegum holding together tripvire wrapped in duct tape and never possible to rewrite for cleanup :(
20:31:16 <rockyg> from what I can tell, OS could use a top to bottom rewrite with an eye to separating infrastructure management from user space management
20:31:52 <rockyg> I've been thinking of trying to get a friend to speak at a summit on agile QA.
20:32:25 <rockyg> Anyway.  Sounds like enough about meetup
20:32:35 <rockyg> #topic new business
20:32:43 <rockyg> any new business?
20:34:00 <rockyg> FYI.  I was at OSCon and ran into an Intel engineer who was escorting 7 high schoolers around
20:34:21 <rockyg> All of them were openstack committers, which is why Intel was footing their bills
20:34:50 <rockyg> I think they would be great for going through and adding the error codes to logging calls.
20:34:53 <jokke_> oh
20:34:55 <jokke_> cool
20:35:31 <rockyg> So, something to think about.
20:36:17 <jokke_> that's pretty cool on both ways. Those kids actually contributing and Intel taking the effort to bring them in
20:36:20 * jokke_ likes
20:36:58 <rockyg> Yeah.  Intel also sponsors the big science fair here in the states.  It's amazing what high school kids can do
20:37:31 <rockyg> twenty years ago they were building scanning electron microscopes in their garages
20:38:49 <jokke_> how old are your high school kids?
20:38:58 <jokke_> ;)
20:38:58 <rockyg> Anyway,  I think I should close this out.  I think the key take away is get some focus on keystone logs and glance traces
20:39:17 <bknudson> I doubt it's just keystone...
20:39:17 <rockyg> Uh, lessee, I think they were 14-18
20:39:21 <jokke_> ++ thanks both of you!
20:39:23 <bknudson> it's just that keystone is the only thing I look at.
20:39:57 <rockyg> keystone is kinda key.  Security plus laced through everything else.
20:40:35 <rockyg> Maybe I'll get the ops folks to rank the projects from best to worst on logging.  Great exercise, I think.
20:40:58 <rockyg> #action Ops to rank projects best to worst on logging.
20:41:27 <rockyg> #action also get Ops to rank highest priority to lowest
20:42:06 <rockyg> Those are two actionable user stories.  We get that, we have a test pilot project to focus improvement.
20:42:43 <rockyg> It also provides us some traction.  thanks for the brainstorming!
20:42:50 <jokke_> cheers
20:42:54 <rockyg> #endmeeting