| opendevreview | James E. Blair proposed zuul/zuul-jobs master: Add mirror-container-images role and job https://review.opendev.org/c/zuul/zuul-jobs/+/935574 | 04:20 |
|---|---|---|
| *** liuxie is now known as liushy | 07:04 | |
| opendevreview | Karolina Kula proposed zuul/zuul-jobs master: DNM Switch to KVM https://review.opendev.org/c/zuul/zuul-jobs/+/936023 | 11:57 |
| opendevreview | Joel Capitao proposed zuul/zuul-jobs master: DNM Switch to KVM https://review.opendev.org/c/zuul/zuul-jobs/+/936023 | 13:10 |
| *** elodilles is now known as elodilles_afk | 13:18 | |
| opendevreview | Joel Capitao proposed openstack/diskimage-builder master: DNM Testing on KVM https://review.opendev.org/c/openstack/diskimage-builder/+/936024 | 14:44 |
| *** elodilles_afk is now known as elodilles | 15:36 | |
| opendevreview | Merged opendev/zone-zuul-ci.org master: Add Bluesky Verification Record for Zuul Account https://review.opendev.org/c/opendev/zone-zuul-ci.org/+/935977 | 15:48 |
| fungi | oof, load average on the wiki server is around 70-80 for some time, pretty much all apache processes. bad enough that openid logins have started failing again | 16:10 |
| fungi | i'm stopping apache for a bit and will start it back up again and see if things get back to a semi-working state | 16:11 |
| fungi | load average is dropping like a stone, unsurprisingly | 16:11 |
| clarkb | fungi: do the logs show anything that might indicate we need to update our UA filter list? | 16:15 |
| clarkb | fungi: and/or do we have a robots.txt in there with a crawldelay? | 16:15 |
| fungi | i've started apache again, load average got to sub-1.0 | 16:16 |
| fungi | haven't had time to analyze logs on it, but no we don't have any sort of working robots.txt on that server, the url just gets redirected to a nonexistent wiki article | 16:17 |
| clarkb | looks like maybe if we do have a robots.txt it gets redirected improperly so we don't serve it properly | 16:17 |
| clarkb | I wonder if fixing that and setting a crawldelay might be helpful | 16:17 |
| fungi | yeah, one of the two | 16:17 |
| fungi | i agree, i can probably take a look after lunch | 16:18 |
| fungi | though i also want to spend some time testing mailman database queries for the upcoming domain change | 16:18 |
| clarkb | ya reading the plan etherpad is high on my list for today | 16:19 |
| fungi | i need to flesh out the db queries, the steps are pretty high-level for the moment | 16:19 |
| clarkb | I'm also going to work on the meeting agenda for today | 16:20 |
| fungi | but then i'll do a run-through on a held test node (which should hopefully appear while i'm at lunch) | 16:20 |
| clarkb | I'll put mm3 vhost changes on the agenda and remove some older things | 16:20 |
| fungi | thanks, once we discuss it in tomorrow's meeting i'll also send a maintenance announcement to the service-announce ml | 16:21 |
| clarkb | fungi: also did we say we wanted to send https://review.opendev.org/c/opendev/system-config/+/933700 and child in or do we want ot wait for after the holiday week at this point | 16:21 |
| fungi | unfortunately the apache restart did not seem to fix wiki openid logins | 16:22 |
| fungi | the last time this happened a full reboot fixed it, i can try that after i get back in about an hour | 16:22 |
| clarkb | ack | 16:22 |
| clarkb | looks like intermediate registry daily pruning change landed so I'll followup and check on the logs for that too | 16:22 |
| fungi | i'm good with going ahead and merging the backup changes | 16:22 |
| clarkb | fungi: did you want to hit +A or should I? | 16:23 |
| fungi | just wanted to be sure we were able to pay attention after they land | 16:23 |
| clarkb | I should be able to today and tomorrow. Becomes more difficult wednesday and beyond | 16:23 |
| fungi | approved both just now | 16:23 |
| clarkb | so ya now is probably a good time | 16:23 |
| fungi | will be back in an hour-ish | 16:23 |
| clarkb | I'm also going to clean up autoholds | 16:24 |
| clarkb | tonyb: you have an autohold for the gerrit point release update change. Can that be cleaned up? | 16:26 |
| clarkb | frickler: you have an osc build image hold too. Not sure if that is still needed. I'm keeping my gerrit 3.9 -> 3.10 autoholds and fungi has an mm3 autohold that looks relevant to the discussion about about renaming vhosts | 16:27 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Fix registry container name in pruning cron job https://review.opendev.org/c/opendev/system-config/+/936173 | 16:31 |
| clarkb | infra-root corvus ^ fyi seems the container name format changed on us making the old name in the revert invalid | 16:31 |
| corvus | ++ | 16:35 |
| clarkb | fungi: service-announce got an uncaught bounce notification for someone sending spam to hte list. I wonder if mm3 has an option for "don't bother doing this for non list members being rejected" | 16:37 |
| frickler | I must admit I don't even remember what that hold was good for, deleted now | 16:38 |
| clarkb | just thinking the primary benefit is for cleaning membership lists if valid messages can't be delivered, but bouncing back and forth notifications for people who aren't members just sending spam seems like maybe unnecessary and potentially a problem? | 16:38 |
| clarkb | frickler: thanks! | 16:38 |
| clarkb | infra-root 935712 and parent are another stack that may be interesting. Its a small bug fix to lodgeit that the person running into the problem fixed when I said I would be unable to fix it myself in the near future. I'd like to encourage more people to jump in and help us make things better by rolling up their sleeves and pushing patches so review on that one wouldbe great | 16:43 |
| clarkb | jcapitao[m]: left a note on 936023 not sure if that will help you in your debugging | 16:47 |
| opendevreview | Merged opendev/system-config master: backups: add retirement and purge lists https://review.opendev.org/c/opendev/system-config/+/933700 | 16:54 |
| opendevreview | Clark Boylan proposed opendev/zone-zuul-ci.org master: Fix bluesky atproto verification record https://review.opendev.org/c/opendev/zone-zuul-ci.org/+/936179 | 16:56 |
| clarkb | fungi: ^ fyi I went ahead and followed up on that from allison since you are eating lunch | 16:56 |
| opendevreview | Merged opendev/zone-zuul-ci.org master: Fix bluesky atproto verification record https://review.opendev.org/c/opendev/zone-zuul-ci.org/+/936179 | 17:00 |
| jcapitao[m] | clarkb: thanks for the headsup. So initially we set Haswell as CPU MODEL and then we went with the passthrough to go with host directly, just for testing. We'll go back with "custom" model to be have explicit CPU configuration | 17:06 |
| clarkb | jcapitao[m]: and not sure if you saw my comment the other day that I don't know how valuable this path is. I would generally recommend against relying on nested virt capability unless you actually need it and none of the dib functional tests have needed ti so far | 17:08 |
| clarkb | the suggestion for using the nested virt labels was only to limit which cpus you were given not an indication that you should also use nested virt | 17:08 |
| clarkb | I think I just realized https://review.opendev.org/c/opendev/system-config/+/933700/23/playbooks/roles/borg-backup-server/files/prune-borg-backups.sh needs to have a skip for purged repos similar to the verifier. Not super urgent since we run pruning manually but I'll go ahead and work on a patch now | 17:13 |
| clarkb | fungi: ^ fyi | 17:13 |
| clarkb | also I'll go ahead and touch the .retired file for ethercalc and we'll test with one of the other ones I think? | 17:13 |
| clarkb | oh no we have to retire ethercalc so that the other server gets handled too. So ya I think we can just go through the steps with ethercalc and then also fix the pruning script to skip purged repos | 17:14 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Skip purged borg backups during backup pruning https://review.opendev.org/c/opendev/system-config/+/936184 | 17:26 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Retire borg-ethercalc02 on the vexxhost backup server https://review.opendev.org/c/opendev/system-config/+/936185 | 17:26 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Purge the borg-ethercalc02 backups on our vexxhost backup server https://review.opendev.org/c/opendev/system-config/+/936186 | 17:26 |
| clarkb | there we go. I don't think any of those are particularly urgent (prune script should be landed before we next prune) but will actually start to exercise some of the new code | 17:27 |
| JayF | Goodness, the mailing list admin address exploded :D I'm guessing the bounce processing is working and removing folks? | 17:37 |
| clarkb | JayF: fungi reported nearly 300 removed from openstack-discuss over the weekend | 17:37 |
| JayF | that matches the unread count in my inbox more or less lol :) | 17:38 |
| clarkb | the other bounce processing stuff is due to mm3 dot trusting verp and instead trying to parse messages and failing. I think fungi intends on compiling a list generating those bounces and manually trimming the membership | 17:38 |
| clarkb | *not trusting | 17:38 |
| JayF | Those are filtered already, this was just a new exciting inbox-filler :D | 17:39 |
| opendevreview | Merged opendev/system-config master: Update backup verifier to handle purged repos https://review.opendev.org/c/opendev/system-config/+/934768 | 17:43 |
| opendevreview | Merged opendev/system-config master: Fix registry container name in pruning cron job https://review.opendev.org/c/opendev/system-config/+/936173 | 17:43 |
| fungi | okay, back and catching up, i see a lot of highlights | 17:58 |
| fungi | yeah, the mm3 autohold was one i set just before i left for lunch. i think my gertty is having trouble un-abandoning 855292 for that though, so it's not caught a failure yet, but i've restored it through the webui instead | 18:03 |
| fungi | clarkb: i haven't seen the uncaught bounce notification for service-announce (maybe i skipped past and deleted it earlier in my day?), though more generally speaking the uncaught bounce notifications mailman sends are because it decides it can't parse the details of the message to the list's bounce address sufficiently to determine how to deal with it | 18:11 |
| fungi | i believe there is a separate toggle in the config for whether toi send uncaught bounce notifications to owners or just be quiet about what bounces it's ignored | 18:11 |
| clarkb | fungi: it was spam using a what I assume was a spoofed address then that destination saying the inbox is full so couldn't delivery the you're being moderated email | 18:16 |
| clarkb | its not a big deal to keep getting them for now. If it becomes a problem we can modify | 18:17 |
| fungi | JayF: clarkb: yes, i wanted to see how the autoremovals fared before i follow up manually handling any other subscriptions that are resulting in continued bounces mailman hasn't been comfortable processing itself | 18:18 |
| clarkb | ++ definitely want to see what the automated tooling does | 18:22 |
| ildikov | Hi Y'All, I'm having some issues logging into wiki.openstack.org. Has anyone else run into that too? | 18:24 |
| fungi | ildikov: i noticed it earlier as well. i tried restarting apache which didn't help, so i'm rebooting the server now (which fixed it last time this came up) | 18:24 |
| fungi | i think the server is probably getting overwhelmed with llm training crawlers | 18:25 |
| ildikov | fungi: Yeah, it was hiccup-y last week too, but I could do what I needed on it. Thank you for looking into it! | 18:25 |
| ildikov | I can see that, since the poor thing has been slow and I don't think our community uses it *that* much anymore... | 18:26 |
| fungi | tonyb has a rebuild waiting in the wings, we just need to finish getting it swapped in | 18:27 |
| ildikov | nice | 18:27 |
| fungi | in the meantime we're trying to limp the old server along by hand | 18:27 |
| ildikov | I hope it'll hold up for a little bit longer than! | 18:28 |
| ildikov | *then | 18:28 |
| fungi | oof, load average is up to 112 after rebooting | 18:28 |
| fungi | and still climbing | 18:29 |
| ildikov | for a second I thought it's my karma, as whatever I've touched in the past one week ended up broken :/ | 18:29 |
| fungi | #status log Rebooted wiki.openstack.org in order to resolve OpenId login failures (again) | 18:29 |
| fungi | ildikov: ^ try again, though be patient, it's veeeeery slow to respond. i'll see what can be done about that next | 18:30 |
| fungi | speaking of slow, our opendevstatus bot seems to be slacking off? | 18:30 |
| ildikov | fungi: I just need to update two small things, fast or slow :) | 18:30 |
| fungi | 5-minute load average on wiki crested around 115 and is starting to fall again | 18:31 |
| ildikov | hmm, I better go hide before my karma breaks anything else :D | 18:31 |
| opendevstatus | fungi: finished logging | 18:32 |
| fungi | there is goes. wow, it does island time better than i do | 18:32 |
| ildikov | lol | 18:32 |
| ildikov | well, I managed to log in this time, but I'm not sure I get as far as editing it | 18:33 |
| ildikov | ok, I managed to get my updates in, so the wiki is not fast, but still holding up! | 18:38 |
| ildikov | thanks fungi! | 18:38 |
| fungi | i'm working on getting a robots.txt implemented now, which *might* help... we'll see | 18:38 |
| JayF | I'm just thinking about all the out of date information training that LLM :P | 18:39 |
| JayF | we have some pretty ancient information deep in that wiki in places | 18:39 |
| fungi | yes, but then consider the internet as a whole | 18:39 |
| fungi | just think of all those lingering geocities pages! | 18:39 |
| JayF | those are the only good things remaining on the internet | 18:40 |
| JayF | <blink><marquee>WELCOME TO MY WEBSITE</blink></marquee> | 18:40 |
| fungi | spinning skulls, flaming text, and barbed wire horizontal rules | 18:42 |
| fungi | ye internete of olde | 18:42 |
| fungi | clarkb: is https://wiki.openstack.org/robots.txt what you had in mind? | 18:43 |
| fungi | tonyb: ^ | 18:43 |
| opendevreview | Merged opendev/system-config master: Skip purged borg backups during backup pruning https://review.opendev.org/c/opendev/system-config/+/936184 | 18:44 |
| opendevreview | Merged opendev/system-config master: Retire borg-ethercalc02 on the vexxhost backup server https://review.opendev.org/c/opendev/system-config/+/936185 | 18:44 |
| fungi | it's mapped to /srv/mediawiki/robots.txt on disk | 18:45 |
| fungi | the robots.txt hasn't helped yet. 5-minute load average on wiki.o.o is hovering around 120 | 18:59 |
| corvus | fungi: perhaps a short downtime would break craws in-progress and cause them to re-fetch robots.txt? i have no reason to think that would be so, but it's an idea. | 19:02 |
| fungi | worth a try. i'm doing a little log analysis first | 19:03 |
| corvus | s/craws/crawls/. sorry, my inner corvid is showing. | 19:03 |
| fungi | birds of a feather... | 19:03 |
| Clark[m] | fungi: ya that robots.txt lgtm. Sorry stepped away to get some stuff done around the house. We still have a few apples to clear out and they just picked up the last load of leaves and compost | 19:06 |
| fungi | time for cidermaking? | 19:06 |
| fungi | do you have access to a cider press? | 19:07 |
| fungi | most common ua in today's access log is this: "Mozilla/5.0 (compatible; ChatGLM-Spider/1.0; +https://chatglm.cn/)" | 19:08 |
| fungi | it's an order of magnitude more active than the next most common ua | 19:08 |
| fungi | seems to confirm my suspicions at least | 19:09 |
| fungi | i've added it to our block list in the vhost config and am stopping apache for a minute too per corvus's suggestion | 19:12 |
| fungi | and starting it again now, 5-minute load average has dropped into the single digits already | 19:14 |
| Clark[m] | We used a juicer one year and ended up with vinegar. If we do it again we need to use Campden tablets to sterilize the juice before adding yeast | 19:22 |
| Clark[m] | Today they are just going in the municipal compost pickup bin | 19:22 |
| fungi | well, apple vinegar is also good, just not for the same purposes | 19:23 |
| Clark[m] | Ya I used what I made on the weeds :) | 19:23 |
| corvus | it's pretty good for salad dressing | 19:24 |
| fungi | but yeah, the presses i remember were about 4' square and you'd load them up several feet deep with apples and then the weights would be lowered down to press them into a mat a few inches thick and collect all the cider into vats underneath | 19:24 |
| fungi | doing it with a consumer grade juicer is hard to compare | 19:25 |
| fungi | it would be heat-sterilized to kill the bacteria, then either bottled directly or fermented | 19:26 |
| fungi | load average on wiki.o.o is definitely lower but still not low, fluctuating wildly between 20-50. next most common user agents look like they're probably lies, aside from "Mozilla/5.0 (Linux; Android 7.0;) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; PanguBot;pangubot@huawei.com)" | 19:32 |
| JayF | pangubot https://arxiv.org/abs/2203.17090 | 19:38 |
| JayF | > PanGu-Bot: Efficient Generative Dialogue Pre-training from Pre-trained Language Model | 19:39 |
| JayF | Oh, I missed the 'aside from' in that comment | 19:39 |
| fungi | yeah, that one is at least identifying itself, even if it's making rather a lot of requests | 19:39 |
| clarkb | if bots don't respect crawl delay I feel like thati s grounds for blockign them... | 19:40 |
| clarkb | and unfortunately it seems like rather a lot of them may not | 19:41 |
| clarkb | fungi: this year all we managed to do with the apples was make pie filling for three pies. Two pies have been consumed and we may do the third for Thursday | 19:43 |
| clarkb | we have a worm problem (part of why I'm trying to collect them aggressively as apparently that helps remove the worms and reduce the problemf or next season) and also our tree definitely does the cyclical on and off year thng and this was an off year in terms of quality but not quantity (lots of smaller meh wormy apples) | 19:44 |
| clarkb | fungi: /opt/backup/borg-ethercalc02/.retired exists now after the retirement change above landed | 19:46 |
| clarkb | I suspect we can proceed and "purge" it too at this point and then we may want to run a manual prune and see that the script is happy with the status at that point. But again not urgent I think you can focus on wiki and mm3 | 19:47 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Upgrade gitea to 1.22.4 https://review.opendev.org/c/opendev/system-config/+/936198 | 19:51 |
| fungi | 213.32.74.174 is the held lists99 test node i'll be working with to test database updates against an imported production dump | 19:58 |
| opendevreview | Merged opendev/system-config master: Purge the borg-ethercalc02 backups on our vexxhost backup server https://review.opendev.org/c/opendev/system-config/+/936186 | 20:05 |
| corvus | clarkb: you can make vinegar from apple scraps too. weaker flavor than whole apples, but still good and useful. we did that with a lot of wormy apple bits and also the peels from making pie fillings | 20:06 |
| clarkb | corvus: do you just add water to the bits or do you have to press/juice them too? | 20:11 |
| fungi | you should be able to just directly ferment the scraps, no further processing necessary other than making sure they have sufficient water content, though seeding them with some unpasteurized vinegar helps insure you don't end up with other helpers | 20:13 |
| fungi | and obviously don't seal them tightly, weight the solids so they stay below the liquid surface, and periodically skim off any growth on top | 20:15 |
| corvus | yep -- exactly as fungi said -- we just filled a mason jar most of the way up, topped it with water, put in a weight, covered with cheesecloth, and left it. skimmed it periodically. a few days later we had weak booze, and a few days after that, vinegar. final result 2.9 - 3.0 pH. we managed to do it without a starter culture this time. | 20:23 |
| corvus | very low effort :) | 20:24 |
| corvus | appropriate for scraps i think | 20:24 |
| fungi | yeah, if you do it regularly, you can just add a dash of the old to the next batch and keep the culture going | 20:26 |
| clarkb | I don't see anything odd about that backup purge | 21:12 |
| clarkb | so I guess the next step is pushing changes to retire the rest of the nodes I identified. I'm working on that next | 21:12 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Retire more backups on the vexxhost backup server https://review.opendev.org/c/opendev/system-config/+/936203 | 21:17 |
| clarkb | fungi: you don't happen to have al ink to your planning doc for the mm3 vhost name migration do you? I'm editing the meeting agenda now and can add it | 21:21 |
| clarkb | I thought I had it in email somewhere but not seeing it there | 21:21 |
| clarkb | nevermind found it in scrollback here | 21:23 |
| clarkb | and agenda is updated. Anything else to add/remove/edit? | 21:24 |
| fungi | yeah, i'll have the sql queries and other cut-n-paste commands fleshed out in that pad tomorrow morning | 21:46 |
| corvus | i'm going to restart the zuul schedulers to pick up a revert that merged recently. opendev hasn't seen the issue, but i'd like to avoid it. | 21:47 |
| clarkb | thanks | 21:50 |
| fungi | sounds good | 21:54 |
| corvus | #status log restarted zuul schedulers and web | 22:03 |
| corvus | TheJulia JayF you may notice the zuul status page has a better memory of expanded sections now (in addition to new overall toggles) | 22:04 |
| opendevstatus | corvus: finished logging | 22:05 |
| corvus | that's with this weekend's restart, so would need a reload for any tabs left open over the weekend | 22:05 |
| TheJulia | Feels like the text size got a little smaller, but LGTM | 22:06 |
| JayF | much better, tyvm | 22:10 |
| opendevreview | Jay Faulkner proposed openstack/diskimage-builder master: Followup: Ensure devuser-created dir has sane perms https://review.opendev.org/c/openstack/diskimage-builder/+/936206 | 22:54 |
| opendevreview | Merged openstack/diskimage-builder master: [gentoo] Fix+Update CI for 23.0 profile https://review.opendev.org/c/openstack/diskimage-builder/+/923985 | 23:17 |
| opendevreview | Jay Faulkner proposed openstack/diskimage-builder master: Update default Ubuntu to noble (latest LTS) https://review.opendev.org/c/openstack/diskimage-builder/+/936209 | 23:36 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!