| opendevreview | Ian Wienand proposed openstack/openstack-zuul-jobs master: [dnm] grafana fail https://review.opendev.org/c/openstack/openstack-zuul-jobs/+/848039 | 00:06 |
|---|---|---|
| opendevreview | Ian Wienand proposed openstack/openstack-zuul-jobs master: [dnm] grafana fail https://review.opendev.org/c/openstack/openstack-zuul-jobs/+/848039 | 00:08 |
| opendevreview | Ian Wienand proposed openstack/openstack-zuul-jobs master: [dnm] grafana fail https://review.opendev.org/c/openstack/openstack-zuul-jobs/+/848039 | 00:15 |
| opendevreview | OpenStack Proposal Bot proposed openstack/project-config master: Normalize projects.yaml https://review.opendev.org/c/openstack/project-config/+/847872 | 02:33 |
| opendevreview | Ian Wienand proposed openstack/project-config master: grafana: Add infra deployment overview https://review.opendev.org/c/openstack/project-config/+/833213 | 04:43 |
| opendevreview | Ian Wienand proposed openstack/project-config master: grafana: add helper script https://review.opendev.org/c/openstack/project-config/+/833214 | 04:43 |
| opendevreview | Ian Wienand proposed openstack/project-config master: grafana: Add infra deployment overview https://review.opendev.org/c/openstack/project-config/+/833213 | 04:46 |
| opendevreview | Ian Wienand proposed openstack/project-config master: grafana: add helper script https://review.opendev.org/c/openstack/project-config/+/833214 | 04:46 |
| *** akahat|out is now known as akahat|ruck | 05:15 | |
| opendevreview | Ian Wienand proposed openstack/openstack-zuul-jobs master: project-config-grafana: collect container logs https://review.opendev.org/c/openstack/openstack-zuul-jobs/+/848073 | 05:16 |
| opendevreview | Ian Wienand proposed openstack/project-config master: grafana: add helper script https://review.opendev.org/c/openstack/project-config/+/833214 | 05:23 |
| opendevreview | Ian Wienand proposed openstack/project-config master: grafana: Add infra deployment overview https://review.opendev.org/c/openstack/project-config/+/833213 | 05:29 |
| opendevreview | Ian Wienand proposed openstack/project-config master: grafana: add helper script https://review.opendev.org/c/openstack/project-config/+/833214 | 05:29 |
| dpawlik | dansmith: did you compare opensearch results with performance.json file for that job? | 06:16 |
| *** jpena|off is now known as jpena | 07:39 | |
| opendevreview | Merged openstack/project-config master: Normalize projects.yaml https://review.opendev.org/c/openstack/project-config/+/847872 | 07:52 |
| *** rlandy|out is now known as rlandy | 09:33 | |
| *** dviroel|out is now known as dviroel | 11:20 | |
| *** blarnath is now known as d34dh0r53 | 12:18 | |
| *** dasm|afk is now known as dasm | 14:03 | |
| dansmith | dpawlik: yes, for example this performance.json https://d2cb11988ad7ed825b5a-e765881c248de8d2cac9a744a80e425c.ssl.cf2.rackcdn.com/847877/2/check/tempest-integrated-storage-import-standalone/f8f0348/controller/logs/performance.json | 14:15 |
| dansmith | shows nova_cell1, DELETE, 613 | 14:15 |
| dansmith | but the index shows only a key of nova_cell1 and a value of zero | 14:15 |
| dansmith | and _all_ the records in opensearch show the same, so I think it's a schema problem or something (not sure how this actually works) | 14:15 |
| dpawlik | dansmith: ack. Will check tomorrow | 14:32 |
| *** dviroel is now known as dviroel|lunch | 15:24 | |
| *** rlandy is now known as rlandy|biab | 15:51 | |
| *** rlandy|biab is now known as rlandy | 16:27 | |
| *** jpena is now known as jpena|off | 16:34 | |
| *** dviroel|lunch is now known as dviroel | 16:37 | |
| *** dviroel is now known as dviroel|biab | 20:30 | |
| *** timburke__ is now known as timburke | 21:01 | |
| *** timburke is now known as timburke__ | 21:39 | |
| *** timburke__ is now known as timburke | 21:40 | |
| *** dviroel|biab is now known as dviroel | 22:09 | |
| *** dasm is now known as dasm|off | 22:15 | |
| *** rlandy is now known as rlandy|bbl | 22:25 | |
| *** dviroel is now known as dviroel|out | 22:28 | |
| pmatulis | anyone else experiencing grief on https://docs.openstack.org/ ? | 22:40 |
| fungi | what manner of grief? | 22:40 |
| clarkb | https doesn't seem to load. Doesn't for https://static.opendev.org either. However I can ssh to static and it has apache's running | 22:42 |
| clarkb | afs appears mounted and with content too | 22:42 |
| fungi | yeah, i can browse afs through it | 22:42 |
| fungi | i wonder if apache is overloaded? though system load is negligible | 22:42 |
| fungi | tried multiple sites served from that apache instance, and all of them are timing out | 22:44 |
| clarkb | I wonder if all the apache workers are full up | 22:44 |
| clarkb | everything else seems fine which makes me think it may be apache just not doing any work | 22:45 |
| fungi | last request apache logged for docs.openstack.org was at 22:34:52 | 22:45 |
| fungi | so it's probably been "hung" for roughly 10 minutes | 22:45 |
| fungi | trying to check server-status now | 22:46 |
| fungi | unfortunately, getting content from that would require apache to respond to me, so not holding out much hope | 22:46 |
| fungi | since there's nothing out of the ordinary being logged by the kernel i think the problem must be a userspace one | 22:47 |
| fungi | and i don't think it's anything network-related, because trying to pull server-status over the loopback interface is also hanging indefinitely | 22:47 |
| fungi | i'll try to strace a worker | 22:48 |
| clarkb | judging by the number of processes and the mpm_event config I think we may be maxing out the threads | 22:48 |
| fungi | they all seem to be blocked reading on fd 7 | 22:49 |
| clarkb | all of the log files were last written to at ~22:34 too | 22:50 |
| clarkb | so its not like one vhost is somehow monopolizing all the workers and everyone else is stuck. Everyone seems stuck | 22:50 |
| fungi | i can't think of anything else to check, i'm leaning toward restarting apache | 22:50 |
| *** gce108__ is now known as gce108 | 22:51 | |
| clarkb | fungi: that fd appeas to be a pipe shared by the workers. So maybe some sort of control issue from the parent process? | 22:51 |
| clarkb | ya I think restarting apache is what we should do | 22:51 |
| fungi | the parent is in a select loop | 22:51 |
| clarkb | and take it from there | 22:51 |
| pmatulis | fungi, sorry i should have been much clearer | 22:52 |
| fungi | #status log Restarted apache2 on static.opendev.org since all sites seem to have been hung and timing out requests as of 22:35 UTC | 22:53 |
| opendevstatus | fungi: finished logging | 22:53 |
| fungi | pmatulis: no worries, thanks for the heads up! | 22:53 |
| fungi | i'm mainly just annoyed that i couldn't find any obvious reason for it to be stuck like that | 22:54 |
| fungi | i'll try to keep a closer eye on it in case the problem comes back today | 22:55 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!