| -@gerrit:opendev.org- Dmitriy Rabotyagov proposed: [openstack/project-config] 978084: Stop publishing collection to PyPi https://review.opendev.org/c/openstack/project-config/+/978084 | 14:47 | |
| @fungicide:matrix.org | as we discussed yesterday, a limnoria restart after 16z today should be safe, so i'll go ahead and recheck 975321 since it'll need at least an hour to get back through check and gate anyway | 15:15 |
|---|---|---|
| -@gerrit:opendev.org- Jeremy Stanley https://matrix.to/#/@fungicide:matrix.org proposed: [opendev/system-config] 978087: Revert "Cap django-health-check for ara installations" https://review.opendev.org/c/opendev/system-config/+/978087 | 15:19 | |
| @clarkb:matrix.org | fungi: sounds good re limnoria | 15:59 |
| @clarkb:matrix.org | I went ahead and approved the django-health-check cap revert since there is a good ara report in the test jobs for that change | 16:03 |
| @clarkb:matrix.org | No reason to hang onto that extra bit of info for any longer | 16:03 |
| @fungicide:matrix.org | thanks | 16:12 |
| -@gerrit:opendev.org- Zuul merged on behalf of Clark Boylan: [opendev/system-config] 975321: Upgrade limnoria and bump its container to trixie https://review.opendev.org/c/opendev/system-config/+/975321 | 16:33 | |
| -@gerrit:opendev.org- Zuul merged on behalf of Jeremy Stanley https://matrix.to/#/@fungicide:matrix.org: [opendev/system-config] 978087: Revert "Cap django-health-check for ara installations" https://review.opendev.org/c/opendev/system-config/+/978087 | 16:33 | |
| @fungicide:matrix.org | infra-prod-service-eavesdrop is running now | 16:36 |
| @fungicide:matrix.org | welcome shrews! long time no see | 16:37 |
| @shrews:matrix.org | ohai! | 16:37 |
| @shrews:matrix.org | I actually am here with purpose | 16:37 |
| @shrews:matrix.org | I my try to fix something in bindep (https://storyboard.openstack.org/#!/story/2008049) if I can find a few moments, but wanted input from you most excellent of folks. | 16:38 |
| @shrews:matrix.org | * I may try to fix something in bindep (https://storyboard.openstack.org/#!/story/2008049) if I can find a few moments, but wanted input from you most excellent of folks. | 16:38 |
| @fungicide:matrix.org | opendevmeet rejoined irc channels after the container restart, so that went well | 16:38 |
| @shrews:matrix.org | Any specific concern with changing bindep to return additional error codes on bindep.txt parsing errors? | 16:39 |
| @shrews:matrix.org | Ansible has a product that would benefit from it | 16:39 |
| @shrews:matrix.org | and also, hi fungi and Clark and corvus and anyone else I may have missed :) | 16:40 |
| @fungicide:matrix.org | shrews: seems useful, i would also make sure that any error message output like tracebacks go to stderr rather than stdout (only package names should be written to stdout during non-prettified operation) | 16:41 |
| @fungicide:matrix.org | i think the main bug identified in story 2008049 is that it's writing error messages to stdout | 16:42 |
| @shrews:matrix.org | well, two bugs: traceback on stdout, and no able to distiquish an error by return code. but yeah, I can totally make sure we keep stdout clean. | 16:43 |
| @shrews:matrix.org | thx for the feedback fungi. i'll see about regenerating my gerrit skills first. hope you all have been well! | 16:45 |
| @jim:acmegating.com | hi shrews ! | 16:46 |
| @fungicide:matrix.org | been doing great! | 16:46 |
| @shrews:matrix.org | ossum | 16:46 |
| @clarkb:matrix.org | oh hey! | 16:55 |
| @clarkb:matrix.org | fungi's feedback all seem great to me | 16:55 |
| @clarkb:matrix.org | fungi: re eavesdrop I'll drop a message in #openstack-infra and double check channel logging | 16:56 |
| @clarkb:matrix.org | https://meetings.opendev.org/irclogs/%23openstack-infra/%23openstack-infra.2026-02-26.log and that looks good | 16:56 |
| @clarkb:matrix.org | corvus: I think that is most if not all of the images opendev is building running on trixie now. If that helps with planning for switching zuul over | 16:57 |
| @jim:acmegating.com | Clark: thanks! i think we were just waiting for the dust to settle, so i reckon we can just do that any time now? | 16:57 |
| @clarkb:matrix.org | I suspect so. We haven't really seen any issues on the opendev side either | 17:00 |
| @clarkb:matrix.org | I guess one thing that makes zuul different is the afs stuff | 17:00 |
| @clarkb:matrix.org | so that would be something to check potentially | 17:01 |
| -@gerrit:opendev.org- Jeremy Stanley https://matrix.to/#/@fungicide:matrix.org proposed: | 17:33 | |
| - [opendev/system-config] 978110: Add WAF rules to more static sites https://review.opendev.org/c/opendev/system-config/+/978110 | ||
| - [opendev/system-config] 978111: Test that WAF rules share a common block pool https://review.opendev.org/c/opendev/system-config/+/978111 | ||
| @clarkb:matrix.org | fungi: that change isn't adding the html sed rewrite stuff only the robots.txt. I'm fine with that but wanted to make sure this was intentional | 17:36 |
| @fungicide:matrix.org | yes, as previously discussed, well-behaved crawlers are known to cache robots.txt files, so we'd want to give them a chance to notice those sites grew some | 17:41 |
| @fungicide:matrix.org | the mod_sed directives will be coming in a separate change that we can merge a few days later | 17:42 |
| @fungicide:matrix.org | i'm also working on robots.txt additions to openstack/openstack-manuals and openstack/api-site before adding waf rules to those vhosts | 17:43 |
| -@gerrit:opendev.org- Joaci Otaviano de Morais proposed wip: [openstack/project-config] 978113: Add Netapp Storage App to StarlingX https://review.opendev.org/c/openstack/project-config/+/978113 | 17:54 | |
| -@gerrit:opendev.org- Jeremy Stanley https://matrix.to/#/@fungicide:matrix.org proposed: [opendev/system-config] 978118: Add WAF rules for developer and docs.openstack.org https://review.opendev.org/c/opendev/system-config/+/978118 | 18:06 | |
| @fungicide:matrix.org | i honestly did not expect 978111 to work, so that's an excellent feature bonus | 18:21 |
| @clarkb:matrix.org | Oh cool. I had no idea if it would but agree that is a nice bit of unexpected functionality | 18:26 |
| @fungicide:matrix.org | https://zuul.opendev.org/t/openstack/build/d5c12bb7dab4416b984ce4d45f2b8052/log/job-output.txt#10993-11207 is worrisome, i would not expect 403 responses there, we don't even have waf rules set up for the site until the change depending on that deploys | 18:30 |
| -@gerrit:opendev.org- Leonardo Fagundes Luz Serrano proposed: [openstack/project-config] 978124: Add meta-packages repo to StarlingX https://review.opendev.org/c/openstack/project-config/+/978124 | 18:34 | |
| @clarkb:matrix.org | fungi: is the manuals build doing a crawl itself and then tripping over it? | 18:35 |
| @clarkb:matrix.org | fungi: maybe we need to check the logs on static for the ip address belonging to that build node? | 18:35 |
| @fungicide:matrix.org | that's what i wondered, but i can't find any evidence that it follows links in robots.txt, and also we don't block it for that site yet | 18:35 |
| @fungicide:matrix.org | so far only docs.opendev.org has the waf rules, and the only hit i find in its access log is a wget from one of my own machines trying it out | 18:36 |
| @clarkb:matrix.org | if I access the url I get content and a 200 | 18:36 |
| @fungicide:matrix.org | same | 18:37 |
| @fungicide:matrix.org | the other possibility is that something changed in our ua filters? | 18:37 |
| @clarkb:matrix.org | oh yes maybe it is the UA filter | 18:37 |
| @fungicide:matrix.org | though that job succeeded 2 days ago | 18:37 |
| @fungicide:matrix.org | huh, but earlier on tuesday it errored the same exact way | 18:38 |
| @clarkb:matrix.org | I did a grep against all `*_error.log` files in the apache log dir on static and that ip doesn't show up. We should record the waf rule entries in that file. Then there is only one entry in `*_access.log` with a 200 response | 18:39 |
| @clarkb:matrix.org | almost like it is talking to something else | 18:40 |
| @fungicide:matrix.org | it's not provider-specific. of the three most recent builds, there was a failure and a success in ovh-gra1, and a failure in rax-ord | 18:41 |
| @fungicide:matrix.org | with both failures looking the same | 18:42 |
| -@gerrit:opendev.org- Joaci Otaviano de Morais proposed wip: [openstack/project-config] 978113: Add Netapp Storage App to StarlingX https://review.opendev.org/c/openstack/project-config/+/978113 | 18:46 | |
| @clarkb:matrix.org | is it possibly using some external service or proxy to make the requests? I don't see evidence of that in the log though | 18:46 |
| @clarkb:matrix.org | oh maybe it is using v6 | 18:47 |
| @fungicide:matrix.org | yeah, i can't find anything that sets something like that up in the job | 18:47 |
| @fungicide:matrix.org | well, we have a failure and success from the same provider though | 18:48 |
| @clarkb:matrix.org | fungi: `(110)Connection timed out: [client 2001:4801:7825:103:be76:4eff:fe10:cab:58112] AH00036: access to /swift/latest/index.html failed (filesystem path '/afs/openstack.org/docs')` | 18:48 |
| @clarkb:matrix.org | it was ipv6 and I read that as a failure to find the content on disk not due to our waf or ua rules | 18:48 |
| @clarkb:matrix.org | that is from /var/log/apache2/docs.openstack.org_error.log | 18:48 |
| @fungicide:matrix.org | odd that a timeout would result in a 403 error | 18:48 |
| @clarkb:matrix.org | dmesg -T reports `[Thu Feb 26 16:41:20 2026] afs: Waiting for busy volume 536870992 () in cell openstack.org` | 18:49 |
| @clarkb:matrix.org | I can ls /afs/openstack.org/docs though | 18:50 |
| @fungicide:matrix.org | usually those are fairly short-lived incidents when you just see one mention in dmesg | 18:50 |
| @clarkb:matrix.org | yes looks like that was the only one from today and occurred over an hour before the 403 errors | 18:51 |
| @clarkb:matrix.org | the afs cache lives on /opt and is only 63% used | 18:51 |
| @fungicide:matrix.org | i've rechecked the change for now | 18:52 |
| @fungicide:matrix.org | i wonder if the workers were all busy or something | 18:53 |
| @fungicide:matrix.org | the apache worker processes on static02 | 18:55 |
| @clarkb:matrix.org | that may explain the timeout | 18:57 |
| @clarkb:matrix.org | and maybe the path and filesystem mapping is just additional info but not directly relevant | 18:59 |
| @clarkb:matrix.org | fungi: also if that tool is making a new connection rather than using a requests session or similar then it could be contributing to going over the limits | 18:59 |
| @clarkb:matrix.org | * fungi: also if that tool is making a new connection for each path request rather than using a requests session or similar then it could be contributing to going over the limits | 18:59 |
| @fungicide:matrix.org | the test checking those urls just seems to be doing a `requests.head()` and then looking at the `status_code` so requests was at least claiming a http/403 status | 19:00 |
| @fungicide:matrix.org | but yeah, it looks like an individual session per url: https://opendev.org/openstack/openstack-manuals/src/branch/master/tools/www-generator.py#L186 | 19:00 |
| @clarkb:matrix.org | maybe we update that test/tool to use a session for the head calls and see if that helps | 19:00 |
| @fungicide:matrix.org | rechecking the change did indeed succeed | 19:03 |
| @fungicide:matrix.org | unrelated, the wiki server's dmesg is flooded with `nf_conntrack: table full, dropping packet` and i'm getting a mediawiki error page served up saying it can't access its database | 19:07 |
| -@gerrit:opendev.org- Leonardo Fagundes Luz Serrano proposed: [openstack/project-config] 978124: Add meta-packages repo to StarlingX https://review.opendev.org/c/openstack/project-config/+/978124 | 19:10 | |
| -@gerrit:opendev.org- Joaci Otaviano de Morais proposed wip: [openstack/project-config] 978113: Add Netapp Storage App to StarlingX https://review.opendev.org/c/openstack/project-config/+/978113 | 19:31 | |
| -@gerrit:opendev.org- Leonardo Fagundes Luz Serrano proposed: [openstack/project-config] 978124: Add meta-packages repo to StarlingX https://review.opendev.org/c/openstack/project-config/+/978124 | 19:38 | |
| @fungicide:matrix.org | #status log Rebooted wiki.openstack.org restoring it to working order after an unrecoverable database disconnect | 19:54 |
| @status:opendev.org | @fungicide:matrix.org: finished logging | 19:54 |
| -@gerrit:opendev.org- Joaci Otaviano de Morais proposed wip: [openstack/project-config] 978113: Add Netapp Storage App to StarlingX https://review.opendev.org/c/openstack/project-config/+/978113 | 20:01 | |
| -@gerrit:opendev.org- Leonardo Fagundes Luz Serrano proposed: [openstack/project-config] 978124: Add metapackages repo to StarlingX https://review.opendev.org/c/openstack/project-config/+/978124 | 20:13 | |
| -@gerrit:opendev.org- Leonardo Fagundes Luz Serrano proposed: [openstack/project-config] 978124: Add metapackages repo to StarlingX https://review.opendev.org/c/openstack/project-config/+/978124 | 20:21 | |
Generated by irclog2html.py 4.1.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!