| JohnnyOSA | Hi all. I have a problem on 2 of 3 infra nodes in an HA setup with the gnocchi container where a "OSError: [Errno 13] Permission denied" is found in the gnocchi-apache-error.log. | 00:41 |
|---|---|---|
| JohnnyOSA | (http://paste.openstack.org/show/690633/) Those gnocchi containers appear to not be reporting metrics due to this problem. | 00:41 |
| JohnnyOSA | This error started after I manually ran a gnocchi-upgrade cmd due to a prior "Unable to detect the number of storage sack" error on those same gnocchi containers which fixed that problem but led to this one. | 00:41 |
| JohnnyOSA | Anyone know how to troubleshoot or fix this? | 00:41 |
| JohnnyOSA | One of the infra nodes never had these problems and is operating fine. | 00:41 |
| JohnnyOSA | (be back in 30m) | 00:42 |
| larsks | JohnnyOSA: I'll bet you ran gnocchi-upgrade as root instead of as the gnocchi user. | 00:42 |
| larsks | Check the permissions on the gnocchi log files. | 00:42 |
| JohnnyOSA | Hi larsks, thanks for the fast feedback! Checking... | 00:43 |
| JohnnyOSA | larsks: I do recall running the command as root. In the /var/log/gnocchi dir, 2 of the 3 log files (access and apache-error) are root:root, the other is gnocchi:gnocchi. | 00:44 |
| JohnnyOSA | larsks: those ownership perms appear to be the same as the working infra node which I didn't run the gnocchi-upgrade command manually on. | 00:46 |
| *** dims has quit IRC | 00:47 | |
| JohnnyOSA | larsks: so probably there are files somewhere else where that had the perms/ownership botched up then... Have to step away; will start hunting a little later on what/how to fix. | 00:48 |
| *** fabian has joined #openstack-telemetry | 00:48 | |
| larsks | Good luck! That was the only idea I had off the | 00:59 |
| larsks | Top of my head | 00:59 |
| *** fabian has quit IRC | 01:06 | |
| *** rwsu has joined #openstack-telemetry | 01:11 | |
| JohnnyOSA | larsks: Cool -- that gives me a place to start! If I can't figure it out, I'll post again in another day or so with a few more details on where I get to in case anyone else has thoughts. | 01:29 |
| *** masber has joined #openstack-telemetry | 01:40 | |
| *** germs has quit IRC | 01:42 | |
| *** germs has joined #openstack-telemetry | 01:42 | |
| *** germs has quit IRC | 01:42 | |
| *** germs has joined #openstack-telemetry | 01:42 | |
| *** germs has quit IRC | 01:44 | |
| *** germs has joined #openstack-telemetry | 01:45 | |
| *** germs has quit IRC | 02:11 | |
| *** germs_ has joined #openstack-telemetry | 02:11 | |
| *** fabian has joined #openstack-telemetry | 02:31 | |
| *** fabian is now known as chenyb4 | 02:31 | |
| JohnnyOSA | larsks: Ok, got that fixed up. The /var/lib/gnocchi/ dir had to have ownership recursively set back to gnocchi:gnocchi, then apache2 service restarted. | 03:15 |
| larsks | JohnnyOSA: yay! | 03:15 |
| JohnnyOSA | :) | 03:15 |
| larsks | Always run maintenance commands as the service user. | 03:15 |
| JohnnyOSA | Ok, good to know. My newbiness is shining through ;) | 03:16 |
| JohnnyOSA | Onto my next puzzle: I'm finding that when I query a metric multiple times (each call going to a different gnocchi container via HAproxy, I'm getting different results lists returned. I would expect, prior to reading further, that each call should return the same list regardless of which gnocchi container responds to the call. | 03:17 |
| JohnnyOSA | but, could be that I'm just not understanding the architecture. | 03:18 |
| JohnnyOSA | Looks like this may have to do with ceilometer, which is logging messages of the type: "Skip pollster hardware.<metric_type>, no resources found this cycle" | 04:17 |
| larsks | JohnnyOSA: if you're stilla round: which storage backend are you using for gnocchi? | 04:38 |
| larsks | You can only run multiple gnocchi instances if you're using a shared storage backend (swift, ceph, etc). If you're using the file backend, you can only run a single instance. | 04:39 |
| JohnnyOSA | larsks: that helps, thanks. The deploy was done for PoC with OpenStack ansible using their standard HA deploy. | 04:47 |
| JohnnyOSA | larsks: in the gnocchi.conf file, the indexer refers to the mysql galera cluster, and for the file storage section, | 04:47 |
| larsks | JohnnyOSA: if there are in fact multiple gnocchi instances running, then it sounds as if the openstack-ansible folks don't really understand the telemetry services. | 04:48 |
| JohnnyOSA | there is a coordination_url set to the mysql galera cluster as well. There are some local storage paths set, but that looks like it might be for temporary storage only. I may need to dig deeper. | 04:48 |
| larsks | If your deploy already includes swift, the easiest solution is probably just to point gnocchi at it. | 04:48 |
| larsks | (or you can use shared storage -- like nfs -- with the file storage backend) | 04:49 |
| JohnnyOSA | ok. Ceph is installed. No swift. I'll look into switching. I think a ceilometer bug may also be preventing metrics that should be regularly polled to getting to gnocchi as well. I'm guessing I'll be spending another few to several hours working at understanding what's going on. | 04:53 |
| *** chenyb4 has quit IRC | 05:05 | |
| JohnnyOSA | More likely that I botched something in the OSA install than they have a bug in the playbook/roles. I see ceph tags and checks in the gnocchi playbook. | 05:06 |
| *** germs_ has quit IRC | 05:07 | |
| *** germs has joined #openstack-telemetry | 05:07 | |
| *** germs has quit IRC | 05:07 | |
| *** germs has joined #openstack-telemetry | 05:07 | |
| larsks | JohnnyOSA: I have to turn in for the evening, but I'm generally online during the week (us business hours). I've just spent the past few weeks digging into openstack telemetry, so feel free to ping me with questions. | 05:08 |
| *** germs_ has joined #openstack-telemetry | 05:08 | |
| JohnnyOSA | Thank you! Have a great night! | 05:09 |
| *** germs has quit IRC | 05:12 | |
| *** swamireddy has quit IRC | 05:42 | |
| *** germs_ has quit IRC | 06:39 | |
| *** germs has joined #openstack-telemetry | 06:39 | |
| *** germs has quit IRC | 06:39 | |
| *** germs has joined #openstack-telemetry | 06:39 | |
| *** germs has quit IRC | 06:44 | |
| *** nijaba has quit IRC | 07:03 | |
| *** nijaba has joined #openstack-telemetry | 07:06 | |
| *** dims has joined #openstack-telemetry | 07:10 | |
| *** masuberu has joined #openstack-telemetry | 07:58 | |
| *** masuberu has quit IRC | 07:58 | |
| *** masber has quit IRC | 08:01 | |
| *** germs has joined #openstack-telemetry | 08:40 | |
| *** germs has quit IRC | 08:40 | |
| *** germs has joined #openstack-telemetry | 08:40 | |
| *** germs has quit IRC | 08:44 | |
| *** swamireddy has joined #openstack-telemetry | 09:05 | |
| *** Tom-Tom_ has quit IRC | 10:23 | |
| *** germs has joined #openstack-telemetry | 10:40 | |
| *** germs has quit IRC | 10:40 | |
| *** germs has joined #openstack-telemetry | 10:40 | |
| *** germs has quit IRC | 10:45 | |
| *** Tom-Tom has joined #openstack-telemetry | 11:05 | |
| *** gongysh has joined #openstack-telemetry | 12:15 | |
| *** gongysh has quit IRC | 12:17 | |
| *** gongysh has joined #openstack-telemetry | 12:20 | |
| *** fabian has joined #openstack-telemetry | 12:23 | |
| *** fabian is now known as chenyb4 | 12:23 | |
| *** gongysh has quit IRC | 12:35 | |
| *** germs has joined #openstack-telemetry | 12:42 | |
| *** germs has quit IRC | 12:42 | |
| *** germs has joined #openstack-telemetry | 12:42 | |
| *** germs has quit IRC | 12:46 | |
| *** germs has joined #openstack-telemetry | 13:22 | |
| *** germs has quit IRC | 13:22 | |
| *** germs has joined #openstack-telemetry | 13:22 | |
| *** chenyb4 has quit IRC | 13:26 | |
| *** AlexeyAbashkin has joined #openstack-telemetry | 13:38 | |
| *** AlexeyAbashkin has quit IRC | 13:55 | |
| *** nijaba has quit IRC | 14:02 | |
| *** nijaba has joined #openstack-telemetry | 14:10 | |
| *** AlexeyAbashkin has joined #openstack-telemetry | 14:53 | |
| *** jmlowe has joined #openstack-telemetry | 14:56 | |
| *** AlexeyAbashkin has quit IRC | 14:57 | |
| *** gongysh has joined #openstack-telemetry | 15:04 | |
| *** germs has quit IRC | 15:21 | |
| *** pcaruana has quit IRC | 15:32 | |
| *** pcaruana has joined #openstack-telemetry | 15:35 | |
| *** gongysh has quit IRC | 15:36 | |
| *** AlexeyAbashkin has joined #openstack-telemetry | 15:55 | |
| *** AlexeyAbashkin has quit IRC | 15:59 | |
| *** AlexeyAbashkin has joined #openstack-telemetry | 16:53 | |
| *** AlexeyAbashkin has quit IRC | 16:58 | |
| *** jmlowe has quit IRC | 17:44 | |
| JohnnyOSA | Hi all... Digging into why ceilometer polling metrics which should be coming in every 5 minutes for nova instances don't. I see info messages in the ceilometer logs of: "Skip pollster hardware.cpu.util (or other metric), no resources found this cycle". Anyone familiar with this? | 18:11 |
| JohnnyOSA | Also -- regarding my comments from several hours ago about trying to get gnocchi centralized into Ceph storage in an OSA deployment: I needed to add a few extra params to the user_variables.yml file in the OSA deploy to get it to install with that option (see OSA channel for anyone interested). | 18:13 |
| *** AlexeyAbashkin has joined #openstack-telemetry | 18:52 | |
| *** AlexeyAbashkin has quit IRC | 18:57 | |
| larsks | JohnnyOSA: The hardware.* aren't nova instance metrics. Those are host metrics that are collected via IPMI or SNMP. There's a good chance those metrics aren't going to be configured out of the box. Metrics available for Nova instances are these: https://docs.openstack.org/ceilometer/latest/admin/telemetry-measurements.html#openstack-compute | 19:10 |
| JohnnyOSA | Larsks: thanks! Yup, you are right -- looking at snmpd config now. | 19:11 |
| *** AlexeyAbashkin has joined #openstack-telemetry | 19:53 | |
| *** AlexeyAbashkin has quit IRC | 19:57 | |
| *** jmlowe has joined #openstack-telemetry | 20:46 | |
| *** germs has joined #openstack-telemetry | 21:06 | |
| *** germs has quit IRC | 21:06 | |
| *** germs has joined #openstack-telemetry | 21:06 | |
| JohnnyOSA | Anyone know an easy way to change the archive_policy for gnocchi metrics that already exist? I'd like to change from the default of low to medium. Changed in the ceilometer.conf, but looks like that will only affect newly created metrics. | 21:08 |
| *** germs has quit IRC | 21:10 | |
| *** germs has joined #openstack-telemetry | 21:11 | |
| *** germs has quit IRC | 21:16 | |
| *** threestrands has joined #openstack-telemetry | 21:17 | |
| *** germs has joined #openstack-telemetry | 21:42 | |
| *** germs has quit IRC | 21:42 | |
| *** germs has joined #openstack-telemetry | 21:42 | |
| *** pcaruana has quit IRC | 22:17 | |
| *** rcernin has joined #openstack-telemetry | 22:28 | |
| *** AlexeyAbashkin has joined #openstack-telemetry | 22:52 | |
| *** AlexeyAbashkin has quit IRC | 22:57 | |
| *** germs has quit IRC | 23:12 | |
| *** germs has joined #openstack-telemetry | 23:13 | |
| *** germs has quit IRC | 23:13 | |
| *** germs has joined #openstack-telemetry | 23:13 | |
| *** germs has quit IRC | 23:18 | |
| *** vint_bra has joined #openstack-telemetry | 23:25 | |
| *** vint_bra has quit IRC | 23:25 | |
| *** AlexeyAbashkin has joined #openstack-telemetry | 23:53 | |
| *** AlexeyAbashkin has quit IRC | 23:57 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!