| @vurmil:matrix.org | review.opendev.org | 07:38 |
|---|---|---|
| Service Unavailable | ||
| The server is temporarily unable to service your request due to maintenance downtime or capacity problems. Please try again later. | ||
| Apache/2.4.58 (Ubuntu) Server at review.opendev.org Port 443 | ||
| @mnasiadka:matrix.org | I just started Gerrit | 07:41 |
| @mnasiadka:matrix.org | #status log Gerrit VM was found in SHUTOFF state, it has been started and reindexing is in progress | 07:43 |
| @status:opendev.org | @mnasiadka:matrix.org: finished logging | 07:43 |
| -@gerrit:opendev.org- hushuai proposed: [zuul/zuul-jobs] 993882: add-build-sshkey: switch to ansible copy for Windows https://review.opendev.org/c/zuul/zuul-jobs/+/993882 | 08:26 | |
| -@gerrit:opendev.org- hushuai proposed: [zuul/zuul-jobs] 993882: add-build-sshkey: switch to ansible copy for Windows https://review.opendev.org/c/zuul/zuul-jobs/+/993882 | 08:38 | |
| @arnaudm:matrix.org | hey, the team said that the CDN issue for .well-known has been fixed yesterday! | 09:13 |
| -@gerrit:opendev.org- Guillaume Boutry proposed on behalf of James Page: [openstack/project-config] 903666: sunbeam: retire all single charm repositories https://review.opendev.org/c/openstack/project-config/+/903666 | 11:37 | |
| @fungicide:matrix.org | thanks arnaudm! | 13:32 |
| @fungicide:matrix.org | and thanks mnasiadka for restarting the gerrit vm. i think we got confirmation a while back that this was definitely a recurring oom condition on the hypervisor in vexxhost, and mnaser or others there were going to look into possible mitigations | 13:34 |
| -@gerrit:opendev.org- Guillaume Boutry proposed on behalf of James Page: [openstack/project-config] 903666: sunbeam: retire all single charm repositories https://review.opendev.org/c/openstack/project-config/+/903666 | 14:19 | |
| @priteau:matrix.org | Are the static web servers suffering again? | 14:42 |
| @fungicide:matrix.org | Pierre Riteau: yes, load average on static03 is up around 2000 again | 14:45 |
| @fungicide:matrix.org | Pierre Riteau: i can take a look after my next meeting, but this has recurred enough this week that our next step is probably going to be spinning up a larger server and shifting at least docs.openstack.org traffic there | 14:54 |
| @clarkb:matrix.org | there is part of me that wonders if we should just turn it off and encourage people to view the docs locally out of git checkouts. Not actually a solution, but if the Internet has decided that "either I get everything at the cost of everyone else or no one gets anything" then maybe just skipping to no one gets anything is reasonable | 14:57 |
| @fungicide:matrix.org | when it was happening earlier in the week, the user agents involved looked like they were spoofing legitimate current browser versions, so filtering by those would likely also block a lot of actual browser users | 14:59 |
| @clarkb:matrix.org | fungi: based on log file sizes docs.openstack and static.opendev are the bulk of the traffic | 14:59 |
| @clarkb:matrix.org | maybe we should turn off static.opendev since that is a utility vhost right? | 15:00 |
| @fungicide:matrix.org | it's also possible we could get better performance by increasing the size of the openafs cache on static03? | 15:00 |
| @fungicide:matrix.org | yeah, disabling the static.opendev.org vhost seems low-risk | 15:00 |
| @clarkb:matrix.org | I think part of the problem here is that the crawlers break caches | 15:00 |
| @clarkb:matrix.org | because they are always grabbing the next thing which isn't cached. So increasing caches may help slightly but is unlikely to be a proper solution | 15:01 |
| @fungicide:matrix.org | they do, but if our "cache" is large enough to hold a complete copy of all the data they're crawling then we won't be hammering afs as often | 15:01 |
| @clarkb:matrix.org | true | 15:01 |
| @clarkb:matrix.org | fwiw static appears to be crawling all of our mirror content | 15:01 |
| @clarkb:matrix.org | so ya I think step 0 here may be disabling the static vhost | 15:01 |
| @clarkb:matrix.org | just get that load off the system. Then evaluate where that puts us | 15:02 |
| @fungicide:matrix.org | i'm simply noting that the server doesn't actually seem to be out of cpu or memory capacity, so the bottleneck is elsewhere (maybe afs) | 15:02 |
| @clarkb:matrix.org | cacheinfo says the cache size is `50000000` and mounted at /opt/cache/openafs. `du -hs openafs` says 43G | 15:08 |
| @fungicide:matrix.org | looking quickly at the vhost, we can't simply disable it because it's the default vhost where we define `AFS_ROOT` and possibly do other things that affect the rest of the server | 15:08 |
| @clarkb:matrix.org | so I think we're not using all of the cache, but I'm not sure if that is because there are internal limits as you approach the cache size limit or if we really don't need all of the cache | 15:08 |
| @fungicide:matrix.org | ah | 15:09 |
| @clarkb:matrix.org | fungi: in theory we can comment out the vhosts in that file? | 15:09 |
| @clarkb:matrix.org | leave the variable definition? | 15:09 |
| @fungicide:matrix.org | i think that would be fine, yes | 15:09 |
| @clarkb:matrix.org | or use an if condition and set a var at the top to toggle it on and off? | 15:09 |
| @fungicide:matrix.org | we'll want to disable ansible deployment to that server temporarily if we only edit it locally on the fs | 15:10 |
| @clarkb:matrix.org | Define DisableStatic True then whatever the if check is to check the variable value wrapped around the vhosts | 15:10 |
| @mordred:waterwanders.com | I don't suppose any of our public cloud providers have CDNs on their swift installs that would let us offload docs to an object storage? (I love the AFS setup, but fighting the crawlers seems like it's turned into a multiple full time jobs) | 15:12 |
| @clarkb:matrix.org | on rackspace classic as far as I know and I think we're trying to moev away from that | 15:13 |
| @mordred:waterwanders.com | nod | 15:14 |
| @fungicide:matrix.org | i think we'd need to redesign the site publication jobs to stick data in swift, and swap out old files for new ones (maybe new container each time?) | 15:14 |
| @clarkb:matrix.org | in theory you can overwrite the files in swift just like you do on a regular fs | 15:15 |
| @mordred:waterwanders.com | yeah. it wouldn't be a this-morning switch to flip. just thinking out loud with the constant flood of crawlers | 15:16 |
| @clarkb:matrix.org | fungi: I edited /etc/apache2/sites-enabled/00-static.opendev.org.conf if you want to take a look | 15:16 |
| @clarkb:matrix.org | I have not restarted apache on that edit | 15:16 |
| @fungicide:matrix.org | juggling a meeting right now, but will look in a sec | 15:17 |
| @clarkb:matrix.org | fungi: it should just be the three lines. The define, the if and the close if | 15:17 |
| @fungicide:matrix.org | Clark: try `sudo apache2ctl configtest` and you'll see an error | 15:18 |
| @clarkb:matrix.org | ack thanks | 15:19 |
| @clarkb:matrix.org | so do I need to use SetEnv insetad of Define? Or maybe configtest isn't checking the Defines first? | 15:20 |
| @clarkb:matrix.org | I'll switch to SetEnv and retest | 15:20 |
| @clarkb:matrix.org | no that didn't help so its probably a parsing order of operations problem? my variable isn't dfinedwhen its checking the if conditions | 15:22 |
| @fungicide:matrix.org | is it possible to test a falsy literal instead of a variable that evaluates false? | 15:23 |
| @fungicide:matrix.org | maybe a null/empty string? | 15:24 |
| @clarkb:matrix.org | yes probably. Let me try that | 15:24 |
| @clarkb:matrix.org | heh yes that seems to have fixed it but given us a new error | 15:25 |
| @clarkb:matrix.org | fungi: ok latest version passes config test | 15:27 |
| @clarkb:matrix.org | mordred: fwiw from my perspective I'm to the point where I don't think simply scaling up to allow the crawlers to crawl harder is appropriate | 15:28 |
| @clarkb:matrix.org | what they are doing is detrimental to not just us but to their own goals (the service being down means no one can see its data). They should be actively pushed against | 15:29 |
| @clarkb:matrix.org | we can actively see them lying about their user agents. Likely operating out of illegal bot nets. They know what they are doing is wrong | 15:30 |
| @mordred:waterwanders.com | oh, for sure. they're a giant pile of horrible. but - their output is increasingly being used, and excluding access to our otherwise publically available docs just means that people using those tools will stop having those tools understand the software we're publishing. I really hate the situation and the tradeoff here | 15:32 |
| @clarkb:matrix.org | and so ya you can switch to cloudflare or some other CDN and let them further the problem by extorting the legit bots | 15:32 |
| @clarkb:matrix.org | but that just feels like caving into a system that has destroyed itself in the name of so called progress | 15:32 |
| @mordred:waterwanders.com | yah, for sure. I feel like we're in a "pick your evil" place | 15:33 |
| @clarkb:matrix.org | My favorite are the bots that request the same resources over and over and over in tight loop as if they have changed recently | 15:34 |
| @mordred:waterwanders.com | because the options are a) burn out the already oversubscribed opendev team trying to fight the bot swarm b) Clark's earlier "just let people read the docs out of git" c) use a CDN and an amount of complicity d) farm goats? | 15:34 |
| @clarkb:matrix.org | its like they are so concerned they will miss an update that rather than check first we'll just redownload it a million times a minute | 15:34 |
| @clarkb:matrix.org | I think things may be improving but I didn't restart apache yet. It doesn't look like fungi did either (but maybe my check was bad) so maybe whatever was crawling us got bored | 15:36 |
| @clarkb:matrix.org | but I still think we can drop that static vhost. I'll propose a change with what is currently staged on the host | 15:37 |
| -@gerrit:opendev.org- Clark Boylan proposed: [opendev/system-config] 993965: Disable the static afs browsing vhost https://review.opendev.org/c/opendev/system-config/+/993965 | 15:39 | |
| @fungicide:matrix.org | my favorite are the ones that are just making up urls out of randomized combinations of existing path components, presumably in order to try to identify isolated islands of content | 15:42 |
| @clarkb:matrix.org | oh I've just realized we probably have testinfra test cases that change will break /me looks closer | 15:55 |
| -@gerrit:opendev.org- Clark Boylan proposed: [opendev/system-config] 993965: Disable the static afs browsing vhost https://review.opendev.org/c/opendev/system-config/+/993965 | 15:57 | |
| @mordred:waterwanders.com | ugh. did gerrit go sideways too? | 15:58 |
| @fungicide:matrix.org | earlier today the hyopervisor oom'd it again | 15:59 |
| @fungicide:matrix.org | maybe that has recurred? | 15:59 |
| @mordred:waterwanders.com | clark's latest patch is not loading for me | 15:59 |
| @mnasiadka:matrix.org | Works for me | 15:59 |
| @clarkb:matrix.org | yes it loads for me | 16:02 |
| @mordred:waterwanders.com | weird. must be on my side | 16:03 |
| @fungicide:matrix.org | fwiw, the load average on static03 is back down in the single digits again now. the bursts this week have been short-lived so far | 16:10 |
| @clarkb:matrix.org | fungi: yup I think whatever it was let up. I think we should still disable that vhost though | 16:13 |
| @fungicide:matrix.org | yeah, at least for a while | 16:16 |
| @clarkb:matrix.org | https://review.opendev.org/c/opendev/system-config/+/993965 passes testing now if we want to approve that. Then my local edits to the vhost will be synced up and I don't have to clean up after myself manually | 16:24 |
| -@gerrit:opendev.org- Zuul merged on behalf of Clark Boylan: [opendev/system-config] 993965: Disable the static afs browsing vhost https://review.opendev.org/c/opendev/system-config/+/993965 | 16:50 | |
| @fungicide:matrix.org | it's taken effect now | 16:54 |
| @fungicide:matrix.org | looks like developer.openstack.org became the default vhost, so attempts to browse https://static.opendev.org/ now get the ssl cert for that instead, and if it's ignored then a 301 permanent redirect is served to https://developer.openstack.org/ | 16:58 |
| @fungicide:matrix.org | a little messy, but probably okay for now | 16:58 |
| @clarkb:matrix.org | fungi: hrm maybe we should've created a default static.o.o vhost that onlyserves a robots txt or something? rather than disable teh entire vhost? | 17:03 |
| @fungicide:matrix.org | yeah, i mean ultimately all we need it to avoid doing is serving the afs root | 17:04 |
| @clarkb:matrix.org | ya do you want to take a stab at that or should I (I wrote the half broken thing that exists now so I'm hapyp to do it though have started juggling a couple of other things | 17:05 |
| @fungicide:matrix.org | i can take a look at it in a moment, sure | 17:07 |
| -@gerrit:opendev.org- Jeremy Stanley https://matrix.to/#/@fungicide:matrix.org proposed: [opendev/system-config] 993971: Reenable static vhost but block all content https://review.opendev.org/c/opendev/system-config/+/993971 | 17:17 | |
| @fungicide:matrix.org | i manually applied that on static03 just to make sure it works as intended | 17:18 |
| @clarkb:matrix.org | fungi: do you want to fetch robots.txt in the test case since that should still work? | 17:19 |
| @clarkb:matrix.org | basically check the block and the robots.txt both work as expected | 17:19 |
| @clarkb:matrix.org | I +2'd with taht suggestion | 17:21 |
| @fungicide:matrix.org | oh, yeah i can do that. i missed that we serve one for static.o.o | 17:24 |
| -@gerrit:opendev.org- Jeremy Stanley https://matrix.to/#/@fungicide:matrix.org proposed: [opendev/system-config] 993971: Reenable static vhost but block all content https://review.opendev.org/c/opendev/system-config/+/993971 | 17:28 | |
| @clarkb:matrix.org | +2 thanks | 17:32 |
| @jim:acmegating.com | do we not want to test the 403 too? | 17:33 |
| @fungicide:matrix.org | oh, we for sure can. just a sec | 17:33 |
| -@gerrit:opendev.org- Jeremy Stanley https://matrix.to/#/@fungicide:matrix.org proposed: [opendev/system-config] 993971: Reenable static vhost but block all content https://review.opendev.org/c/opendev/system-config/+/993971 | 17:35 | |
| @fungicide:matrix.org | also realized that my commit message still said we were doing that | 17:35 |
| @clarkb:matrix.org | I have approved it | 17:57 |
| @fungicide:matrix.org | thanks! | 17:58 |
| @clarkb:matrix.org | fungi: if you find time the etherpad 3.3.1 upgrade change and held node may be worthy of review https://review.opendev.org/c/opendev/system-config/+/993783 | 18:02 |
| @fungicide:matrix.org | yes thanks, i can pivot to that for a bit | 18:07 |
| @fungicide:matrix.org | still have nearly an hour until my next meeting | 18:07 |
| @jim:acmegating.com | I put an item on the agenda for the next meeting to discuss larger vm sizes (related to the zuul python 3.14 work) | 18:26 |
| @fungicide:matrix.org | held etherpad 3.3.1 node at 146.20.60.66 looks good to me | 18:28 |
| @fungicide:matrix.org | i've single-core approved the upgrade, and will keep tabs on the deployment process | 18:28 |
| @clarkb:matrix.org | Thanks I'll keep an eye out too | 18:29 |
| -@gerrit:opendev.org- Zuul merged on behalf of Jeremy Stanley https://matrix.to/#/@fungicide:matrix.org: [opendev/system-config] 993971: Reenable static vhost but block all content https://review.opendev.org/c/opendev/system-config/+/993971 | 18:47 | |
| @clarkb:matrix.org | a quick check indicates ^ has applied successfully | 18:54 |
| @fungicide:matrix.org | load average on static03 has climbed but is falling again, so likely just momentary impact from the apache restart | 18:55 |
| -@gerrit:opendev.org- Zuul merged on behalf of Clark Boylan: [opendev/system-config] 993783: Upgrade Etherpad to 3.3.1 https://review.opendev.org/c/opendev/system-config/+/993783 | 19:22 | |
| @fungicide:matrix.org | infra-prod-service-etherpad is running | 19:27 |
| @fungicide:matrix.org | wrapping up even, so upgrade done | 19:27 |
| @clarkb:matrix.org | docker reports it is healthy and running for about a minute | 19:28 |
| @clarkb:matrix.org | https://etherpad.opendev.org/p/gerrit-upgrade-3.13 loads for me | 19:28 |
| @clarkb:matrix.org | I think it is working | 19:31 |
| @fungicide:matrix.org | yeah, looks good to me | 19:47 |
| @clarkb:matrix.org | The prometheus, greptimedb, and node exporter stack at https://review.opendev.org/c/opendev/system-config/+/988310 lgtm now. Note this doesn't actually set up infra-prod deployment jobs but as mentioned in some of my prior reviews I think this ok. We can add that stuff when we add nodes to the inventory. As is these updates are exercised by the system-config-run jobs in CI only | 20:58 |
| @clarkb:matrix.org | if other infra-root have time to review those that would be great | 20:58 |
| @clarkb:matrix.org | corvus: for the larger node size issue with vcpus do you think it is worth testing flavor v3-starter-8 in vexxhost? That is 16gb memory and 8vcpus. But I don't know how those get oversubscribed vs the standard flavors | 21:02 |
| @jim:acmegating.com | happy to do so! i can't remember what we were asked to use or not use on the various clouds so i tried not to stray far from what we do | 21:03 |
| @clarkb:matrix.org | I don't know that vexxhost ever said? its possible mnaser has thoughts on that. But considering its a switch from standard to starter I would think that is in the less impactful direction? | 21:04 |
| @clarkb:matrix.org | so I think in this case its probably fine to test with | 21:04 |
| @jim:acmegating.com | qq on the prom stack: does one of the changes open port 9100? do we test connecting to that from another host? | 21:04 |
| @clarkb:matrix.org | then in the case of rax classic they do have a compute1-15 with 15GB memory and 8 vcpus. THe problem there is that they appear to have no disk so would be boot from volume? | 21:05 |
| @clarkb:matrix.org | I'm thinking that may be more difficult to test with as a result | 21:05 |
| @jim:acmegating.com | > <@clarkb:matrix.org> then in the case of rax classic they do have a compute1-15 with 15GB memory and 8 vcpus. THe problem there is that they appear to have no disk so would be boot from volume? | 21:05 |
| yes, i likewise didn't want to open that can of worms unless we thought it was worth it | ||
| @clarkb:matrix.org | ya I suspect that will be a lot of trouble for potentially minimal gain compared to the proposal of using performance2-30 | 21:06 |
| @jim:acmegating.com | i left the 9100 question on https://review.opendev.org/980994 | 21:08 |
| @clarkb:matrix.org | I think you are right. In the existing test jobs the prometheus node is exporting to itself which works with our default firewall rules | 21:10 |
| @clarkb:matrix.org | In a previous comment I noted that we should update the iptables rules similar to those for snmp | 21:10 |
| @clarkb:matrix.org | https://review.opendev.org/c/opendev/system-config/+/980994/comments/0027afaa_c37047c7 this comment cc mnasiadka | 21:10 |
| @clarkb:matrix.org | on paste I ran `borg-mount backup03.ca-ymq-1.vexxhost.opendev.org` then navigated through `/opt/backups/paste02-filesystem-2026-06-18T13:26:02` looking at /var/log/containers/docker-lodgeit.log and confirming the timestamps in that file seem to match up with the backup timestamp. I then checked `/opt/backups/paste02-mariadb-2026-06-18T13:26:27/mariadb`'s head and tail and it looks like a mysqldump | 22:06 |
| @clarkb:matrix.org | This is not a comprehensive check but I think backups to the newer backup server are working. fungi I think that means the last step before clearing out the old backup server is getting wiki (and I guess any other stragglers though I don't know of any) backing up to the new server | 22:07 |
| @clarkb:matrix.org | then we can remove the old server and attach the old volume to the new server | 22:07 |
| -@gerrit:opendev.org- Clark Boylan proposed: [openstack/diskimage-builder] 994017: Revert "Disable centos/10-stream-build-succeeds functest" https://review.opendev.org/c/openstack/diskimage-builder/+/994017 | 22:21 | |
| @mordred:waterwanders.com | Clark, corvus if you have a sec, https://review.opendev.org/c/opendev/base-jobs/+/990828 and https://review.opendev.org/c/zuul/zuul-jobs/+/990826 are both good to go. I've demonstrated them working in https://review.opendev.org/c/drizzle/drizzle/+/990830 | 23:31 |
Generated by irclog2html.py 4.1.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!