| fungi | could it be really terrible block alignment? | 00:01 |
|---|---|---|
| fungi | in our guest filesystems i mean | 00:01 |
| donnyd | its possible... | 00:01 |
| fungi | though even that shouldn't cause abysmal performance, just less than optimal | 00:02 |
| donnyd | but it should do the same on other clouds | 00:02 |
| fungi | right | 00:02 |
| donnyd | I know ceph isn't the best for db workloads at my cluster size (6 nodes) | 00:03 |
| donnyd | after the tempest results come back, i hopefully will know which way to turn | 00:04 |
| donnyd | It's pretty depressing when you get all new equipment and it does worse than the "old" gear | 00:05 |
| donnyd | LOL | 00:05 |
| *** slaweq has joined #openstack-infra | 00:11 | |
| *** slaweq has quit IRC | 00:16 | |
| dmsimard | donnyd: a quick win for ephemeral data in ceph is to tone down the amount of replicas | 00:20 |
| dmsimard | i.e, from 3 to 2 | 00:20 |
| *** roman_g has quit IRC | 00:21 | |
| donnyd | I will give that a spin | 00:22 |
| *** lseki has quit IRC | 00:27 | |
| *** verdurin has quit IRC | 00:33 | |
| *** verdurin has joined #openstack-infra | 00:34 | |
| *** yamamoto has quit IRC | 00:34 | |
| hogepodge | clarkb: I’m back and can discuss Monday | 00:36 |
| *** irclogbot_3 has quit IRC | 00:39 | |
| *** irclogbot_0 has joined #openstack-infra | 00:44 | |
| donnyd | I'm out for today. Thanks everyone for the help in getting this far. :) | 00:48 |
| fungi | thanks for working on it, donnyd! | 00:51 |
| *** imacdonn has quit IRC | 01:10 | |
| *** imacdonn has joined #openstack-infra | 01:10 | |
| *** zhurong has quit IRC | 01:29 | |
| *** Lucas_Gray has quit IRC | 01:53 | |
| *** apetrich has quit IRC | 01:58 | |
| *** pleia2 has quit IRC | 02:03 | |
| *** yamamoto has joined #openstack-infra | 02:11 | |
| *** slaweq has joined #openstack-infra | 02:11 | |
| *** slaweq has quit IRC | 02:16 | |
| *** pleia2 has joined #openstack-infra | 02:19 | |
| *** yamamoto has quit IRC | 02:22 | |
| *** mriedem has quit IRC | 02:37 | |
| *** rosmaita has left #openstack-infra | 02:49 | |
| *** bhavikdbavishi has joined #openstack-infra | 02:56 | |
| *** dklyle has quit IRC | 03:01 | |
| *** yamamoto has joined #openstack-infra | 03:02 | |
| *** yamamoto has quit IRC | 03:09 | |
| *** dciabrin has joined #openstack-infra | 03:15 | |
| *** dciabrin_ has quit IRC | 03:16 | |
| *** bhavikdbavishi1 has joined #openstack-infra | 03:44 | |
| *** bhavikdbavishi has quit IRC | 03:45 | |
| *** bhavikdbavishi1 is now known as bhavikdbavishi | 03:45 | |
| *** bhavikdbavishi has quit IRC | 03:51 | |
| *** pfallenop has quit IRC | 04:08 | |
| *** slaweq has joined #openstack-infra | 04:11 | |
| *** rh-jelabarre has quit IRC | 04:16 | |
| *** slaweq has quit IRC | 04:16 | |
| *** hwoarang has quit IRC | 04:26 | |
| *** kjackal has joined #openstack-infra | 04:27 | |
| *** hwoarang has joined #openstack-infra | 04:28 | |
| *** yamamoto has joined #openstack-infra | 05:19 | |
| *** yamamoto has quit IRC | 05:23 | |
| *** yamamoto has joined #openstack-infra | 05:34 | |
| *** factor has quit IRC | 05:46 | |
| *** factor has joined #openstack-infra | 05:46 | |
| *** slaweq has joined #openstack-infra | 06:11 | |
| *** yamamoto has quit IRC | 06:15 | |
| *** slaweq has quit IRC | 06:16 | |
| *** igordc has quit IRC | 06:30 | |
| *** kjackal has quit IRC | 06:33 | |
| yoctozepto | ask.o.o down | 06:37 |
| *** yamamoto has joined #openstack-infra | 06:46 | |
| *** yamamoto has quit IRC | 06:56 | |
| *** yamamoto has joined #openstack-infra | 07:12 | |
| *** pcaruana has joined #openstack-infra | 07:16 | |
| *** yikun has joined #openstack-infra | 07:18 | |
| *** kjackal has joined #openstack-infra | 07:18 | |
| *** yamamoto has quit IRC | 07:19 | |
| *** yamamoto has joined #openstack-infra | 07:30 | |
| *** Lucas_Gray has joined #openstack-infra | 07:37 | |
| *** yamamoto has quit IRC | 07:39 | |
| *** yamamoto has joined #openstack-infra | 07:46 | |
| *** yamamoto has quit IRC | 07:46 | |
| *** pcaruana has quit IRC | 07:48 | |
| *** yamamoto has joined #openstack-infra | 07:49 | |
| *** xek has joined #openstack-infra | 07:57 | |
| *** Lucas_Gray has quit IRC | 08:08 | |
| *** slaweq has joined #openstack-infra | 08:11 | |
| *** needscoffee is now known as kmalloc | 08:11 | |
| *** slaweq has quit IRC | 08:16 | |
| *** raissa has quit IRC | 08:17 | |
| *** kjackal has quit IRC | 08:46 | |
| *** ykarel|away has joined #openstack-infra | 09:02 | |
| *** ykarel|away has quit IRC | 09:08 | |
| *** ricolin has joined #openstack-infra | 09:14 | |
| *** altlogbot_2 has quit IRC | 09:15 | |
| *** altlogbot_3 has joined #openstack-infra | 09:16 | |
| *** slaweq has joined #openstack-infra | 09:23 | |
| *** yamamoto has quit IRC | 09:28 | |
| *** yamamoto has joined #openstack-infra | 09:29 | |
| *** yamamoto has quit IRC | 09:30 | |
| *** tosky has joined #openstack-infra | 09:36 | |
| *** ricolin has quit IRC | 09:48 | |
| *** whoami-rajat has joined #openstack-infra | 09:48 | |
| *** Lucas_Gray has joined #openstack-infra | 10:06 | |
| *** yamamoto has joined #openstack-infra | 10:22 | |
| *** yamamoto has quit IRC | 10:30 | |
| *** yamamoto has joined #openstack-infra | 10:30 | |
| *** xek has quit IRC | 10:48 | |
| *** pcaruana has joined #openstack-infra | 10:48 | |
| *** slaweq has quit IRC | 11:00 | |
| *** emine__ has joined #openstack-infra | 11:12 | |
| *** lpetrut has joined #openstack-infra | 11:14 | |
| fungi | checking | 11:23 |
| fungi | server itself is up and hasn't rebooted in over a month (when it was last rebuilt) | 11:23 |
| fungi | the site itself seems to be up at the moment too | 11:24 |
| fungi | it seems like something around the time the cron.daily jobs kick off (06:25 utc) causes the site to become unresponsive | 11:25 |
| fungi | could maybe be apache cleanup or log rotation | 11:26 |
| fungi | not likely apache htcacheclean because /etc/default/apache-htcacheclean sets HTCACHECLEAN_MODE=daemon instead of =cron, so that should be a no-op | 11:28 |
| fungi | could be redis-server log rotation | 11:31 |
| fungi | though doesn't look like it. last modification time on the most recently located log was sunday | 11:33 |
| fungi | logs under /srv/askbot-site/log/ were rotated today at 06:10-06:20 looks like. maybe that sends the service into some sort of busywait condition for a while? | 11:35 |
| fungi | and it doesn't recover immediately? | 11:35 |
| fungi | oh, it may be apache... this is from /var/log/apache2/error.log | 11:39 |
| fungi | [Sat Jun 29 06:25:05.694858 2019] [mpm_event:notice] [pid 27046:tid 139818062194560] AH00493: SIGUSR1 received. Doing graceful restart | 11:39 |
| fungi | [Sat Jun 29 06:25:05.960022 2019] [core:notice] [pid 27046] AH00060: seg fault or similar nasty error detected in the parent process | 11:39 |
| fungi | [Sat Jun 29 06:54:55.757293 2019] [core:warn] [pid 8411:tid 140671236716416] AH00098: pid file /var/run/apache2/apache2.pid overwritten -- Unclean shutdown of previous Apache run? | 11:39 |
| fungi | there's a notable ~30 minute gap | 11:39 |
| *** brett-soric has joined #openstack-infra | 11:40 | |
| *** yamamoto has quit IRC | 11:49 | |
| *** zbr has quit IRC | 11:50 | |
| *** kjackal has joined #openstack-infra | 11:52 | |
| yoctozepto | fungi: yeah, it seemed to reject connections - as in port open but nothing listening | 12:00 |
| *** whoami-rajat has quit IRC | 12:08 | |
| *** yamamoto has joined #openstack-infra | 12:08 | |
| *** lpetrut has quit IRC | 12:09 | |
| *** ab-a has quit IRC | 12:18 | |
| *** ab-a has joined #openstack-infra | 12:18 | |
| *** brett-soric has left #openstack-infra | 12:26 | |
| *** sshnaidm|off has quit IRC | 12:31 | |
| *** bhavikdbavishi has joined #openstack-infra | 12:38 | |
| *** yamamoto has quit IRC | 12:40 | |
| *** bhavikdbavishi has quit IRC | 12:42 | |
| fungi | does definitely sound like something about askbot is causing apache to occasionally freak out during log rotation then | 12:43 |
| *** yamamoto has joined #openstack-infra | 12:49 | |
| openstackgerrit | Ghanshyam Mann proposed openstack/os-performance-tools master: Update python 3 unit test job https://review.opendev.org/637759 | 12:53 |
| *** raissa has joined #openstack-infra | 12:53 | |
| openstackgerrit | Ghanshyam Mann proposed openstack/os-performance-tools master: Update python 3 unit test job https://review.opendev.org/637759 | 12:58 |
| openstackgerrit | Ghanshyam Mann proposed openstack/os-performance-tools master: Update python 3 unit test job https://review.opendev.org/637759 | 12:59 |
| openstackgerrit | Merged openstack/os-performance-tools master: Update python 3 unit test job https://review.opendev.org/637759 | 13:13 |
| *** hwoarang has quit IRC | 13:28 | |
| *** hwoarang has joined #openstack-infra | 13:29 | |
| *** pfallenop has joined #openstack-infra | 13:41 | |
| *** pfallenop has quit IRC | 13:53 | |
| *** hamzy has quit IRC | 14:07 | |
| *** tosky has quit IRC | 14:12 | |
| *** yamamoto has quit IRC | 14:18 | |
| *** bobh has joined #openstack-infra | 14:28 | |
| *** hamzy has joined #openstack-infra | 14:32 | |
| *** bobh has quit IRC | 14:42 | |
| *** kjackal has quit IRC | 14:42 | |
| *** factor has quit IRC | 15:03 | |
| *** hamzy has quit IRC | 15:04 | |
| *** factor has joined #openstack-infra | 15:04 | |
| *** Lucas_Gray has quit IRC | 15:04 | |
| *** sshnaidm has joined #openstack-infra | 15:09 | |
| *** factor has quit IRC | 15:16 | |
| *** factor has joined #openstack-infra | 15:16 | |
| *** hamzy has joined #openstack-infra | 15:16 | |
| *** ykarel|away has joined #openstack-infra | 15:17 | |
| *** whoami-rajat has joined #openstack-infra | 15:20 | |
| *** rosmaita has joined #openstack-infra | 15:20 | |
| *** factor has quit IRC | 15:21 | |
| *** icarusfactor has joined #openstack-infra | 15:21 | |
| *** yamamoto has joined #openstack-infra | 15:24 | |
| *** ykarel|away has quit IRC | 15:28 | |
| AJaeger | the gentoo test is the only failing one on corvus' change https://review.opendev.org/668249 . Anybody an idea? prometheanfire ? See that openvwswitch fails to install, see http://logs.openstack.org/49/668249/1/check/openstack-infra-multinode-integration-gentoo-17-0-systemd/e767d00/ara-report/result/d6cb44d5-a7ec-46d9-89b1-72b6e31acae7/ | 15:29 |
| *** yamamoto has quit IRC | 15:35 | |
| *** liuyulong has quit IRC | 15:37 | |
| *** sshnaidm has quit IRC | 15:39 | |
| *** sshnaidm_ has joined #openstack-infra | 15:39 | |
| *** slaweq has joined #openstack-infra | 15:48 | |
| *** ralonsoh has joined #openstack-infra | 15:58 | |
| *** Lucas_Gray has joined #openstack-infra | 15:59 | |
| *** Lucas_Gray has quit IRC | 16:04 | |
| *** igordc has joined #openstack-infra | 16:15 | |
| *** zbr has joined #openstack-infra | 16:31 | |
| mmedvede | AJaeger: with regards to ciwatch.mmedvede.net, I have some time and would try to fix it, plus change the info there so people may contact me directly. | 16:49 |
| prometheanfire | AJaeger: ok, I just stablized a new version, didn't do anything to require caps support though | 17:06 |
| prometheanfire | it's a simple fix at least (just add a file to package.use | 17:06 |
| *** goldyfruit has quit IRC | 17:09 | |
| *** tosky has joined #openstack-infra | 17:18 | |
| *** whoami-rajat has quit IRC | 17:28 | |
| *** goldyfruit has joined #openstack-infra | 17:41 | |
| AJaeger | prometheanfire: Thanks! do we need to merge a change - or just need to wait (and how long)? | 17:41 |
| prometheanfire | AJaeger: probably a dib change, not sure how to add ad-hoc package.use | 17:45 |
| prometheanfire | AJaeger: generally we do it here https://github.com/openstack/diskimage-builder/blob/master/diskimage_builder/elements/gentoo/pre-install.d/02-gentoo-02-flags#L28 | 17:45 |
| prometheanfire | actually, whatever installs openvswitch can do it | 17:47 |
| prometheanfire | AJaeger: I don't see openvswitch defined in system-config project-config or nodepool | 17:51 |
| prometheanfire | ok, openstack-zuul-jobs/tests/multi-node-bridge.yaml | 17:53 |
| openstackgerrit | Matthew Thode proposed openstack/openstack-zuul-jobs master: set package.use values for ovs on Gentoo https://review.opendev.org/668303 | 18:02 |
| prometheanfire | AJaeger: ^ something like that, I took it from the work I did in OSA | 18:02 |
| prometheanfire | actually, taken from zuul-jobs/roles/multi-node-bridge/tasks/common.yaml | 18:09 |
| *** igordc has quit IRC | 18:11 | |
| *** tosky has quit IRC | 18:13 | |
| *** sshnaidm_ has quit IRC | 18:27 | |
| *** kjackal has joined #openstack-infra | 18:34 | |
| donnyd | clarkb: After some testing it looks like I need to redeploy. I will let you know when its back up | 18:58 |
| *** goldyfruit has quit IRC | 19:01 | |
| AJaeger | thanks, prometheanfire | 19:04 |
| prometheanfire | AJaeger: doesn't look like it took :| | 19:07 |
| prometheanfire | it doesn't look like it added the line in file.... | 19:09 |
| prometheanfire | it added the other two | 19:11 |
| prometheanfire | AJaeger: so, if you don't mind doublechecking my ansible... | 19:11 |
| *** emine__ has quit IRC | 19:17 | |
| *** emine__ has joined #openstack-infra | 19:28 | |
| *** igordc has joined #openstack-infra | 19:32 | |
| *** witek has joined #openstack-infra | 19:38 | |
| *** emine__ has quit IRC | 19:46 | |
| * AJaeger checked but is puzzled why 668303 still fails ;( | 20:08 | |
| *** auristor has quit IRC | 20:17 | |
| *** hamzy has quit IRC | 20:23 | |
| *** auristor has joined #openstack-infra | 20:28 | |
| *** igordc has quit IRC | 20:30 | |
| *** hwoarang has quit IRC | 20:38 | |
| *** ralonsoh has quit IRC | 20:38 | |
| *** auristor has quit IRC | 20:40 | |
| *** hwoarang has joined #openstack-infra | 20:43 | |
| prometheanfire | ya, same | 20:44 |
| *** bdodd has quit IRC | 20:46 | |
| *** hamzy has joined #openstack-infra | 20:47 | |
| *** witek has quit IRC | 20:48 | |
| *** sshnaidm_ has joined #openstack-infra | 20:49 | |
| *** auristor has joined #openstack-infra | 20:59 | |
| *** kjackal has quit IRC | 21:38 | |
| *** slaweq has quit IRC | 22:08 | |
| *** tristanC has quit IRC | 22:14 | |
| *** tristanC has joined #openstack-infra | 22:15 | |
| *** dklyle has joined #openstack-infra | 22:26 | |
| *** dklyle has quit IRC | 22:32 | |
| *** goldyfruit has joined #openstack-infra | 22:43 | |
| *** roman_g has joined #openstack-infra | 22:43 | |
| *** efried has quit IRC | 22:52 | |
| *** yamamoto has joined #openstack-infra | 23:32 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!