jlk | Would be nice if various upstream projects would agree on foo=bar vs foo = bar in config file standards. | 00:00 |
---|---|---|
*** SimonChung1 has joined #openstack-operators | 00:02 | |
*** SimonChung has quit IRC | 00:02 | |
*** Marga_ has joined #openstack-operators | 00:15 | |
*** Marga_ has quit IRC | 00:15 | |
*** Marga_ has joined #openstack-operators | 00:15 | |
*** signed8bit is now known as signed8bit_ZZZzz | 00:20 | |
*** alop has quit IRC | 00:21 | |
*** emagana has quit IRC | 00:36 | |
*** Piet has quit IRC | 00:40 | |
*** emagana has joined #openstack-operators | 01:36 | |
*** emagana has quit IRC | 01:48 | |
*** jaypipes has quit IRC | 01:52 | |
*** SimonChung1 has quit IRC | 02:01 | |
*** erkules_ has joined #openstack-operators | 02:05 | |
*** erkules has quit IRC | 02:07 | |
*** blair has quit IRC | 02:28 | |
*** emagana has joined #openstack-operators | 02:43 | |
*** emagana has quit IRC | 02:48 | |
*** Marga_ has quit IRC | 02:52 | |
*** Marga_ has joined #openstack-operators | 02:53 | |
*** blair has joined #openstack-operators | 02:55 | |
*** yapeng has quit IRC | 03:00 | |
*** mdorman_ has joined #openstack-operators | 03:11 | |
*** mdorman has quit IRC | 03:12 | |
*** mdorman_ is now known as mdorman | 03:12 | |
*** VW_ has joined #openstack-operators | 03:12 | |
*** mdorman has quit IRC | 03:14 | |
*** emagana has joined #openstack-operators | 03:24 | |
*** SimonChung has joined #openstack-operators | 03:24 | |
*** harlowja is now known as harlowja_away | 03:25 | |
*** SimonChung1 has joined #openstack-operators | 03:26 | |
*** blair has quit IRC | 03:26 | |
*** blair has joined #openstack-operators | 03:26 | |
*** SimonChung has quit IRC | 03:29 | |
*** VW_ has quit IRC | 03:40 | |
*** VW_ has joined #openstack-operators | 03:43 | |
*** VW_ has quit IRC | 03:47 | |
*** blairo has joined #openstack-operators | 03:57 | |
*** blair has quit IRC | 03:57 | |
*** VW_ has joined #openstack-operators | 03:58 | |
*** VW_ has quit IRC | 04:03 | |
*** VW_ has joined #openstack-operators | 04:04 | |
*** esker has joined #openstack-operators | 04:11 | |
*** klindgren_ has quit IRC | 04:13 | |
*** blair has joined #openstack-operators | 04:19 | |
*** blairo has quit IRC | 04:22 | |
*** blair has quit IRC | 04:31 | |
*** blairo has joined #openstack-operators | 04:37 | |
*** emagana has quit IRC | 04:40 | |
*** sanjayu has joined #openstack-operators | 04:49 | |
*** sanjayu has quit IRC | 04:56 | |
*** sanjayu has joined #openstack-operators | 04:56 | |
*** esker has quit IRC | 04:58 | |
*** Marga_ has quit IRC | 05:01 | |
*** blairo has quit IRC | 05:06 | |
*** blair has joined #openstack-operators | 05:07 | |
*** VW_ has quit IRC | 05:15 | |
*** VW_ has joined #openstack-operators | 05:18 | |
*** blair has quit IRC | 05:30 | |
*** VW_ has quit IRC | 05:32 | |
*** emagana has joined #openstack-operators | 05:51 | |
*** emagana has quit IRC | 05:55 | |
*** bvandenh has joined #openstack-operators | 06:28 | |
*** avozza is now known as zz_avozza | 06:32 | |
*** bvandenh has quit IRC | 06:36 | |
*** reed has quit IRC | 06:44 | |
*** VW_ has joined #openstack-operators | 07:15 | |
*** pcaruana has joined #openstack-operators | 07:16 | |
*** VW_ has quit IRC | 07:19 | |
*** belmoreira has joined #openstack-operators | 07:21 | |
*** bvandenh has joined #openstack-operators | 07:52 | |
*** erkules_ is now known as erkules | 08:11 | |
*** pboros has joined #openstack-operators | 08:52 | |
*** blairo has joined #openstack-operators | 08:54 | |
*** blairo has quit IRC | 09:02 | |
*** blair has joined #openstack-operators | 09:15 | |
*** derekh has joined #openstack-operators | 09:16 | |
*** blairo has joined #openstack-operators | 09:20 | |
*** zz_avozza is now known as avozza | 09:23 | |
*** blair has quit IRC | 09:23 | |
*** avozza is now known as zz_avozza | 09:59 | |
*** chojny has joined #openstack-operators | 10:07 | |
*** zz_avozza is now known as avozza | 10:08 | |
*** chojny has quit IRC | 10:20 | |
*** chojny has joined #openstack-operators | 10:28 | |
*** racedo_ has joined #openstack-operators | 10:36 | |
*** racedo_ has quit IRC | 10:49 | |
*** chojny has quit IRC | 11:19 | |
*** chojny has joined #openstack-operators | 11:35 | |
*** VW_ has joined #openstack-operators | 11:53 | |
*** VW__ has joined #openstack-operators | 11:54 | |
*** jemangs has quit IRC | 11:56 | |
*** VW_ has quit IRC | 11:57 | |
*** VW__ has quit IRC | 12:03 | |
*** VW_ has joined #openstack-operators | 12:04 | |
*** pasquier-s has quit IRC | 12:05 | |
*** dmsimard_away is now known as dmsimard | 12:06 | |
*** blairo has quit IRC | 12:06 | |
*** VW_ has quit IRC | 12:30 | |
*** VW_ has joined #openstack-operators | 12:30 | |
*** jemangs has joined #openstack-operators | 12:42 | |
*** pasquier-s has joined #openstack-operators | 12:46 | |
*** pasquier-s has quit IRC | 12:58 | |
*** VW_ has quit IRC | 13:02 | |
*** VW_ has joined #openstack-operators | 13:03 | |
*** VW_ has quit IRC | 13:12 | |
*** VW_ has joined #openstack-operators | 13:13 | |
*** belmoreira has quit IRC | 13:13 | |
*** pasquier-s has joined #openstack-operators | 13:15 | |
*** pasquier-s has quit IRC | 13:16 | |
*** pasquier-s has joined #openstack-operators | 13:21 | |
*** pasquier-s has quit IRC | 13:32 | |
*** pasquier-s has joined #openstack-operators | 13:37 | |
*** pasquier-s has quit IRC | 13:39 | |
*** pasquier-s has joined #openstack-operators | 13:43 | |
*** jaypipes has joined #openstack-operators | 13:52 | |
*** esker has joined #openstack-operators | 14:03 | |
*** dmsimard is now known as dmsimard_away | 14:04 | |
*** DaSchab has joined #openstack-operators | 14:06 | |
*** signed8bit_ZZZzz is now known as signed8bit | 14:09 | |
*** pasquier-s has quit IRC | 14:09 | |
*** pasquier-s has joined #openstack-operators | 14:09 | |
*** emagana has joined #openstack-operators | 14:09 | |
*** radez_g0n3 is now known as radez | 14:11 | |
*** pasquier-s has quit IRC | 14:14 | |
*** pasquier-s has joined #openstack-operators | 14:14 | |
*** pasquier-s has quit IRC | 14:14 | |
*** pasquier-s has joined #openstack-operators | 14:14 | |
*** pasquier-s has quit IRC | 14:14 | |
*** pasquier-s has joined #openstack-operators | 14:15 | |
*** esker has quit IRC | 14:18 | |
*** esker has joined #openstack-operators | 14:19 | |
*** chojny has quit IRC | 14:22 | |
*** sanjayu has quit IRC | 14:26 | |
*** VW_ has quit IRC | 14:29 | |
*** VW_ has joined #openstack-operators | 14:29 | |
*** resker has joined #openstack-operators | 14:30 | |
*** esker has quit IRC | 14:30 | |
*** resker has quit IRC | 14:33 | |
*** pasquier-s has quit IRC | 14:39 | |
*** pasquier-s has joined #openstack-operators | 14:43 | |
*** pasquier-s has quit IRC | 14:44 | |
*** pasquier-s has joined #openstack-operators | 14:49 | |
*** pasquier-s has quit IRC | 14:49 | |
*** pcaruana has quit IRC | 14:51 | |
*** pasquier-s has joined #openstack-operators | 14:51 | |
*** pasquier-s has quit IRC | 14:52 | |
*** IPPhoneGuy has joined #openstack-operators | 14:54 | |
*** esker has joined #openstack-operators | 14:57 | |
*** resker has joined #openstack-operators | 14:58 | |
*** signed8bit has quit IRC | 15:00 | |
*** pasquier-s has joined #openstack-operators | 15:01 | |
*** esker has quit IRC | 15:01 | |
*** DaSchab has quit IRC | 15:02 | |
*** VW_ has quit IRC | 15:07 | |
*** VW_ has joined #openstack-operators | 15:13 | |
*** pasquier-s has quit IRC | 15:14 | |
*** pasquier-s has joined #openstack-operators | 15:15 | |
*** chojny has joined #openstack-operators | 15:21 | |
*** signed8bit has joined #openstack-operators | 15:25 | |
*** pasquier-s has quit IRC | 15:26 | |
*** VW_ has quit IRC | 15:30 | |
*** Piet has joined #openstack-operators | 15:33 | |
*** VW_ has joined #openstack-operators | 15:33 | |
*** pasquier-s has joined #openstack-operators | 15:35 | |
*** pasquier-s has quit IRC | 15:36 | |
*** pasquier-s has joined #openstack-operators | 15:40 | |
*** belmoreira has joined #openstack-operators | 15:41 | |
*** pasquier-s has quit IRC | 15:49 | |
*** mdorman has joined #openstack-operators | 15:52 | |
*** belmoreira has quit IRC | 15:55 | |
*** reed has joined #openstack-operators | 15:59 | |
*** pasquier-s has joined #openstack-operators | 16:02 | |
*** chojny has quit IRC | 16:02 | |
*** pasquier-s has quit IRC | 16:08 | |
Piet | If you haven't already, please complete the following the online card sort to improve the information architecture in Horizon. http://ows.io/os/0v46l867 | 16:13 |
Piet | **Participants will be automatically added to a drawing for one of three HP 7 G2 tablets.** | 16:13 |
*** pasquier-s has joined #openstack-operators | 16:15 | |
*** klindgren has joined #openstack-operators | 16:20 | |
*** pasquier-s has quit IRC | 16:29 | |
*** pasquier-s has joined #openstack-operators | 16:39 | |
*** alop has joined #openstack-operators | 16:47 | |
*** chojny has joined #openstack-operators | 16:47 | |
*** chojny has quit IRC | 16:47 | |
*** pasquier-s has quit IRC | 16:58 | |
*** Marga_ has joined #openstack-operators | 17:01 | |
*** harlowja_away is now known as harlowja | 17:01 | |
*** Piet has quit IRC | 17:04 | |
*** pasquier-s has joined #openstack-operators | 17:07 | |
*** emagana has quit IRC | 17:14 | |
*** Marga_ has quit IRC | 17:15 | |
*** Marga_ has joined #openstack-operators | 17:19 | |
*** emagana has joined #openstack-operators | 17:36 | |
*** bvandenh has quit IRC | 17:41 | |
*** avozza is now known as zz_avozza | 17:42 | |
mfisch | ops meeting starting in 2 mins in #openstack-operators | 17:58 |
mfisch | derp sorry | 17:58 |
mfisch | in #openstack-meeting-4 | 17:58 |
klindgren | hehehe | 17:59 |
*** gfa has joined #openstack-operators | 17:59 | |
*** derekh has quit IRC | 18:00 | |
harlowja | meeting room 4? | 18:02 |
harlowja | holy crap there are that many meetings rooms | 18:02 |
jlk | such openstack, very projects, much contention | 18:02 |
*** resker has quit IRC | 18:08 | |
*** Marga_ has quit IRC | 18:13 | |
*** Marga_ has joined #openstack-operators | 18:14 | |
*** Marga_ has quit IRC | 18:49 | |
*** Marga_ has joined #openstack-operators | 18:49 | |
*** vinsh has joined #openstack-operators | 18:52 | |
*** praneshp has joined #openstack-operators | 18:53 | |
xavpaice | so was there an outcome on the discussions re a stackforge of tools for operators? | 18:53 |
xavpaice | or is that generally what stackforge is anyway? | 18:54 |
*** esker has joined #openstack-operators | 18:54 | |
mdorman | i think the outcome was essentially just to table it for later | 18:54 |
alop | well that was fun | 18:55 |
mdorman | when we originally set up the osops repo, people didn’t want to do stackforge because we just wanted a quick and easy place to put stuff | 18:55 |
mdorman | without having to have all teh gating and other infrastructure that stackforge requires | 18:55 |
xavpaice | that makes sense | 18:55 |
mdorman | i don’t remember the full argument/reasons, but that’s how it happened the way it did | 18:56 |
xavpaice | and is kinda why we leave stuff on our github too | 18:56 |
mdorman | yeah | 18:56 |
xavpaice | btw, we're re-writing a bunch of those collectd plugins, there's some serious inefficiencies in them | 18:56 |
*** GeoffArnold has joined #openstack-operators | 18:57 | |
mdorman | yeah i have a few ideas in mind on those, too | 18:59 |
mdorman | just haven’t gotten around to it | 18:59 |
*** GeoffArnold has quit IRC | 19:01 | |
*** Marga_ has quit IRC | 19:01 | |
*** Marga_ has joined #openstack-operators | 19:02 | |
xavpaice | thanks for the pull request - we're using that option too, and is what triggered us looking at them more clearly | 19:02 |
xavpaice | I've re-done the cinder plugin, but one of our devs pointed out something more helpful so I'll have another bash at it before calling it done | 19:03 |
* xavpaice gtr | 19:03 | |
mdorman | cool | 19:03 |
*** VW_ has quit IRC | 19:25 | |
*** VW_ has joined #openstack-operators | 19:25 | |
*** fifieldt_ has quit IRC | 19:30 | |
klindgren | who was it that sent out the email yeaterday about logs in openstack - that had a spec and the like | 19:38 |
klindgren | I have an example of logs that it would be nice if they were more consistent across projects | 19:39 |
klindgren | but I cant seem to find that email | 19:39 |
klindgren | ? | 19:39 |
mfisch | klindgren: want me to forward it? | 19:43 |
mfisch | Rocheelle Grober | 19:43 |
mfisch | has anyone seen where after a live-migration qemu doesn't stop on the orginal host? | 19:43 |
mfisch | I now have 2 qemus running | 19:44 |
klindgren | mfisch, please - I did some work to have logstash grab all of the api responses/timing stuff | 19:44 |
klindgren | and to parse the differences between the wsgi logs between all the projects is crazy | 19:44 |
klindgren | mfisch, I haven't done a live migrate - but is it asking you to accept the migrated vm? | 19:45 |
mfisch | you dont need that with live | 19:45 |
mfisch | I didnt do the migrate though let me sanity check | 19:46 |
*** fifieldt_ has joined #openstack-operators | 19:46 | |
mfisch | we've created a new nova command | 19:46 |
mfisch | nova reproduce | 19:46 |
klindgren | does that go through a different code path? IF so - care on sharing :-) | 19:48 |
mfisch | all I know is that you dont have to accept it or whatever like a resize or cold migrate | 19:48 |
mfisch | but its not supposed to leave the old one | 19:48 |
klindgren | kk | 19:48 |
mfisch | now I'm not sure which one is really responding to pings | 19:48 |
mfisch | you need shared storage to do this | 19:49 |
klindgren | wha wha wha... | 19:49 |
mfisch | most vendors sales guys say YES IT WORKS GREAT | 19:49 |
mfisch | the tech guys not so much | 19:49 |
klindgren | yea personal prefrence... not to run root volumes off shared storage | 19:50 |
klindgren | Due to past experience | 19:50 |
*** reed has quit IRC | 20:04 | |
*** reed has joined #openstack-operators | 20:21 | |
*** Piet has joined #openstack-operators | 20:28 | |
*** blair has joined #openstack-operators | 21:00 | |
*** pasquier-s has quit IRC | 21:04 | |
*** pasquier-s has joined #openstack-operators | 21:11 | |
*** Marga_ has quit IRC | 21:21 | |
*** pasquier-s has quit IRC | 21:25 | |
*** VW__ has joined #openstack-operators | 21:26 | |
*** VW_ has quit IRC | 21:26 | |
*** VW__ has quit IRC | 21:30 | |
*** Rockyg has joined #openstack-operators | 21:32 | |
jlk | I don't suppose anybody is around that has done the libvirt/qemu upgrade on ubuntu? | 21:38 |
jlk | my stuff isn't coming back right, even a simple virsh | 21:38 |
klindgren | I haven't done it - but I saw some rumblings about it a while back? | 21:39 |
jlk | yeah I've been following the thread | 21:39 |
jlk | I'm running into something they weren't though. | 21:39 |
klindgren | I can't help - sorry - I am RHEL based :-/ | 21:40 |
jlk | I wish I were | 21:40 |
*** blair has quit IRC | 21:43 | |
klindgren | honestly - I kinda wish we were ubuntu based on the compute nodes | 21:44 |
*** blair has joined #openstack-operators | 21:44 | |
klindgren | mainly due to newer kernels and versions of libvirt/kvm/qemu | 21:45 |
jlk | what the hell | 21:48 |
*** blair has quit IRC | 21:48 | |
jlk | backign rev, then forwarding rev again worked. | 21:48 |
*** blair has joined #openstack-operators | 21:49 | |
*** blairo has joined #openstack-operators | 21:53 | |
*** blair has quit IRC | 21:56 | |
*** pasquier-s has joined #openstack-operators | 22:01 | |
*** blairo has quit IRC | 22:05 | |
*** Marga_ has joined #openstack-operators | 22:08 | |
*** pasquier-s has quit IRC | 22:10 | |
*** esker has quit IRC | 22:14 | |
*** pasquier-s has joined #openstack-operators | 22:18 | |
*** blairo has joined #openstack-operators | 22:21 | |
*** jaypipes has quit IRC | 22:26 | |
*** zz_avozza is now known as avozza | 22:28 | |
*** pasquier-s has quit IRC | 22:40 | |
*** blair has joined #openstack-operators | 22:48 | |
*** b1air has joined #openstack-operators | 22:49 | |
*** blairo has quit IRC | 22:51 | |
*** pboros has quit IRC | 22:52 | |
*** blair has quit IRC | 22:53 | |
*** Rockyg has quit IRC | 23:03 | |
*** Rockyg has joined #openstack-operators | 23:03 | |
Rockyg | Hey! | 23:08 |
*** praneshp has quit IRC | 23:09 | |
klindgren | hi - so logging | 23:09 |
Rockyg | A question for operators: there is a config option for having logs output in syslog format. It only works if the message generator uses oslo.log, but it supposedly works. anyone using it, aware of it, etc? | 23:10 |
klindgren | was parsing the wsgi stuff and noticed that the wsgi logs between all the services is pretty different | 23:10 |
Rockyg | Yeah. I'm looking at your email. Pretty ugly. | 23:11 |
klindgren | I know a few people are doing logs -> syslog | 23:11 |
Rockyg | Do you need to put that in every projects' config file, or is it a set once globally? | 23:11 |
klindgren | its in every project | 23:12 |
Rockyg | UGLY | 23:12 |
Rockyg | OK. good to know. | 23:12 |
Rockyg | So, theoretically, if it were set in each of the projects you use, you'd get a consistent format (but only outside the payload, I'd expect) | 23:13 |
Rockyg | Is wsgi logging a separate logging library/process from the project(s)? Do you know? I'm just digging in on this, so I've lots to learn ridiculously fast. | 23:14 |
klindgren | I honestly dont know | 23:15 |
Rockyg | But, I know how bad it is to have inconsistent logs and know that is the case right now for OpenStack. | 23:15 |
Rockyg | that's ok. the logging working group has both devs and ops in it. the devs will know. | 23:16 |
klindgren | afaik this issue is more payload logging being different | 23:16 |
Rockyg | Yeah. That was what I thought and the meetings at the summit reinforced that idea. So, we have at least two problems. message format around the payload and the payloads themselves. | 23:17 |
klindgren | outside of possibly the name of the module being logged | 23:17 |
xavpaice | it's the multiline stuff that messes with syslog | 23:17 |
Rockyg | Oh, yeah. that would. | 23:17 |
Rockyg | I'm getting the log rationalization etherpad back up so I can add these questions to it. Need to collect up all the questions to make the devs work hard to explain why logging is so messed up | 23:20 |
*** blairo has joined #openstack-operators | 23:20 | |
Rockyg | https://etherpad.openstack.org/p/Log-Rationalization | 23:21 |
*** signed8bit has quit IRC | 23:21 | |
Rockyg | I'll transcribe this stuff into docs, agenda items, etc. | 23:22 |
*** b1air has quit IRC | 23:22 | |
Rockyg | There is one cross project spec out on log guidelines, and we need to get more operator input on it, but there are a couple more that are out there and not yet identified as crossproject. | 23:24 |
Rockyg | That set is to get requestIDs passed in the log messages so that a glance info log message would have the nova requestID for matching | 23:25 |
Rockyg | We also plan to put a cros project spec out for error codes | 23:26 |
Rockyg | reqid is in payload, but error code is outside | 23:27 |
Rockyg | I think mid to longer term is how we get the payloads more standard. We need to come up with a proposal that the devs can stomach, and get them to enforce it. Then we can get the old messages fixed during refoactors, bug fixes, etc. | 23:28 |
Rockyg | klindgren: of all the log messages you sent me, is there one that has the info needed but is better/easier to read/parse/script for? | 23:30 |
Rockyg | Oh, I see your suggestion at the bottom. Great! | 23:30 |
klindgren | sorry - got a call - catching up | 23:30 |
klindgren | yea - the suggestion at the bottom seems to be fairly close to what most projects were doing | 23:31 |
klindgren | so other people can see what we are talking about: https://gist.github.com/krislindgren/586d289e477f02db09d9 | 23:32 |
Rockyg | Cool. this is something we could likely make standard fairly quickly. Especially if it can use oslo.log to do the wsgi logging. | 23:33 |
Rockyg | There are development projects that are hitting the same issues. so this is getting a little more traction. | 23:35 |
*** praneshp has joined #openstack-operators | 23:35 | |
*** pasquier-s has joined #openstack-operators | 23:35 | |
Rockyg | Another question I have: if the log format for a project is set to syslog (or systemd format), do the wsgi log message contents get rearranged? | 23:37 |
klindgren | thats a good question - I do not know | 23:37 |
*** praneshp_ has joined #openstack-operators | 23:38 | |
Rockyg | Part of my problem is that there are so many areas where logging can be improved, that I'm having a hard time focusing on high value, low hanging fruit | 23:38 |
klindgren | If someone is using syslog can you pull some example logs from from services of the wsgi stuff and see if its rearanged - the examples I gave above are from icehouse | 23:38 |
*** avozza is now known as zz_avozza | 23:39 | |
*** praneshp has quit IRC | 23:39 | |
*** praneshp_ is now known as praneshp | 23:39 | |
Rockyg | The other thing I notice about the timestamps is that the nested one is a totally different format from the main one. | 23:40 |
Rockyg | On the keystone message, the eventlet.wsgi.server, could you file a bug about it saying eventlet rather than keystone or keystone.eventlet? Tag it with "log" or "logging" and subcribe rockyg | 23:42 |
Rockyg | I want to wade through the bug and identify the log bugs. I want as many bugs recorded as possible against bad/unclear/useless/wrong log outputs. | 23:43 |
Rockyg | I figured out what the [-] means. It means that either there is no request id, or more likely, request id has not been implemented there yet. | 23:46 |
klindgren | right but what about: [1377447c-beb0-4916-8fda-62062ec1c00f 02d775v09ngA fc4d336616d944ee86d04766d099fa3f - - -] | 23:48 |
klindgren | [reqid user tenant/project ? ? ?] | 23:49 |
Rockyg | Yeah. gotta track that one down. the reqid one was easy because I've been reading the specs/bps/patches | 23:49 |
Rockyg | nova, cinder, glance have them for nova logs | 23:50 |
Rockyg | http://specs.openstack.org/openstack/nova-specs/specs/juno/approved/log-request-id-mappings.html | 23:54 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!