| opendevreview | Merged zuul/zuul-jobs master: Rewrite upload-pypi test-playbook https://review.opendev.org/c/zuul/zuul-jobs/+/972252 | 00:40 |
|---|---|---|
| *** darmach3 is now known as darmach | 12:55 | |
| clarkb | looks like the honeypot change has been approved | 15:45 |
| clarkb | I'm going to check on intermediate registry pruning shortly as well | 15:45 |
| clarkb | the smaller backup server needs pruning as well. I can probably get to that if someone doesn't beat me to it | 15:46 |
| clarkb | the prune log doesn't end in a traceback and the last blob path it looks at has a fff prefix so I think it very likely it managed to iterate through the entire list this time around | 15:52 |
| clarkb | I'm going to recheck the change from mordred which kicked off the debugging of this issue now as the images it depends on should still exist after pruning | 15:53 |
| clarkb | https://review.opendev.org/c/opendev/system-config/+/970451 this change to be specific | 15:53 |
| opendevreview | Michal Nasiadka proposed openstack/project-config master: kolla: Introduce a version of Ironic core two tier structure https://review.opendev.org/c/openstack/project-config/+/972508 | 15:58 |
| opendevreview | Merged opendev/system-config master: Add modsecurity waf rules to docs.opendev.org https://review.opendev.org/c/opendev/system-config/+/970674 | 16:09 |
| opendevreview | Michal Nasiadka proposed openstack/project-config master: kolla: Introduce a version of Ironic core two tier structure https://review.opendev.org/c/openstack/project-config/+/972508 | 16:10 |
| opendevreview | Jeremy Stanley proposed opendev/system-config master: Update documentation for Matrix move https://review.opendev.org/c/opendev/system-config/+/972510 | 16:15 |
| clarkb | the honeypot change records successful deployment | 16:15 |
| fungi | infra-root: ^ we can merge that in advance, or wait until closer to the cut-over and i'll wip it in the meantime | 16:16 |
| clarkb | I can still reach https://docs.opendev.org/opendev/system-config/latest/ so we haven't broken things terribly | 16:16 |
| fungi | yeah, i'm going to try blocking one of my machines in a moment | 16:21 |
| fungi | getting 'ERROR 404: Not Found.' from the honeypot url and the client is still able to fetch normal page content | 16:23 |
| fungi | the client is 162.242.225.143 | 16:24 |
| clarkb | fungi: we don't log requests to the initial honeypot url that add to the table (but maybe we should) but we should log subsequent requests that get denied | 16:25 |
| fungi | oh, it's only for the docs vhost, let me try again | 16:26 |
| clarkb | fungi: yes only for docs.opendev.org | 16:26 |
| clarkb | (I wanted to limit the scope for the initial testing) | 16:26 |
| fungi | makes sense, sure | 16:26 |
| clarkb | also normal apache request logging is in place. The logs I talk about are specific to mod security actions | 16:27 |
| fungi | ERROR 403: Forbidden. | 16:27 |
| opendevreview | Michal Nasiadka proposed openstack/project-config master: kolla: Introduce a version of Ironic core two tier structure https://review.opendev.org/c/openstack/project-config/+/972508 | 16:27 |
| clarkb | fungi: excellent. So I think it is operating as expected. The main question now is whether or not any of the crawlers will abuse robots.txt to fetch things | 16:28 |
| fungi | and now that client gets 403 for any subsequent docs.opendev.org url | 16:28 |
| clarkb | and we may have to wait a few days to get data on that | 16:28 |
| fungi | right | 16:28 |
| fungi | so far so good | 16:28 |
| clarkb | fungi: can you make a note to test fetches to docs.opendev.org from that IP address 24 hours from now? | 16:29 |
| clarkb | fungi: in theory the table entries are only good for 24 hours | 16:29 |
| fungi | i also (inadvertently) tested triggering it with docs.openstack.org and security.openstack.org and it didn't happen there, so all working as intended | 16:29 |
| fungi | and the blocked client can still get to those | 16:29 |
| fungi | yeah i'll try non-honeypot urls from docs.opendev.org again after 16:30 utc tomorrow | 16:30 |
| clarkb | thanks | 16:30 |
| clarkb | hrm looks like the existing images for that gerrit build recheck may have already been pruned based on logs. So I need to recheck the whole stack to build new data then recheck again in a couple of days? I'll start that process now | 16:49 |
| clarkb | (the latest run of pruning got "Unknown manifest" logged which is what happens when there isn't any manifest data in the location due to a 404) | 16:50 |
| clarkb | but the pruning pass also indicates it wants to keep the manifest so I think the problem is due to the old bug | 16:50 |
| clarkb | fungi: do we want to proceed with https://review.opendev.org/c/opendev/system-config/+/970919 and https://review.opendev.org/c/opendev/system-config/+/970920 today as well to finish up the project renaming followups? I think the logging change is basically safe. The homedir perms change should be safe but has more potential for unexpected impacts | 16:55 |
| fungi | yeah, would be good to get those out of the way before we forget about them | 16:56 |
| clarkb | maybe we do the permissions change when we switch gerrit to java 21 just in case there is some podman interaction with bind mount perms changing while the container is running? Though I think it should just act like normal file perms updates and in this case we're giving the running process more access so should be safe | 16:56 |
| clarkb | I approved the log collection update change as I believe it to be completely safe (it makes the one job act like all the others) | 16:59 |
| clarkb | it also had two +2s | 16:59 |
| opendevreview | Merged opendev/system-config master: Stop collecting manage-projects logs as regular zuul job logs https://review.opendev.org/c/opendev/system-config/+/970920 | 17:08 |
| fungi | deploy succeeded | 17:14 |
| clarkb | fungi: I think it was a noop though as the manage-projects job didn't run | 17:21 |
| clarkb | fungi: if we land mnasiadka's kolla-reviewers group addition we'll exercise it though | 17:22 |
| clarkb | mnasiadka: is kolla ready for that? | 17:22 |
| fungi | ah yeah | 17:22 |
| clarkb | I'm going to prune the backup server now | 17:24 |
| mnasiadka | clarkb: sure, that’s only an additonal group | 17:25 |
| mnasiadka | But the job that is checking acl normalization is failing | 17:27 |
| mnasiadka | So if you need me to update it - then it’s going to be tomorrow | 17:27 |
| clarkb | mnasiadka: ack. I'm not sure its urgent but I can take a look and push an update if the problem has an easy solution | 17:27 |
| clarkb | backup pruning is in progress now | 17:27 |
| mnasiadka | clarkb: not urgent, but would be nice to get it in this week | 17:28 |
| clarkb | mnasiadka: did you see JayF's comment? I won't address that and will only fix the acl test job | 17:30 |
| clarkb | I'll defer to you on whether you want to take that suggestion | 17:30 |
| mnasiadka | Clark[m]: we’re fine with the main group staying as is, I can respond in the patch | 17:31 |
| opendevreview | Clark Boylan proposed openstack/project-config master: kolla: Introduce a version of Ironic core two tier structure https://review.opendev.org/c/openstack/project-config/+/972508 | 17:31 |
| clarkb | mnasiadka: I think the diff issue is that the formatting wants the rules in alphabetical order so I swapped the -reviewers rules to come after the -core rules | 17:31 |
| mnasiadka | Ah right, my alphabet flipped on my first day of work this year, I still don’t know what I’m doing - thanks :) | 17:33 |
| slittle1_ | I'm trying to update starlingx feature branch f/portable-dc from it's parent branch via merge across all our gits. A fair number are failing in zuul, even though zuul was happy with the same code when merged to master prior to the Xmas break. https://review.opendev.org/q/topic:%22merge-master-as-of-20260106T051000Z%22+status:open | 17:37 |
| slittle1_ | has anything changed on the back end? | 17:37 |
| clarkb | slittle1_: these look like job specific errors: `sysinv.common.exception.SysinvException: Invalid metadata.yaml: version should be a valid version string.` and `ModuleNotFoundError: No module named 'cgcs_patch'` for example | 17:38 |
| clarkb | possibly related to the merges themselves if not done correctly (eg using cgcs_patch module which is called somethen else on the target side of the merge) | 17:39 |
| clarkb | slittle1_: I think you need to look at the end result of the merges and see if they are valid. These don't look like zuul problems to me | 17:40 |
| slittle1_ | cgcs_patch errors I might understand, others like https://zuul.opendev.org/t/openstack/build/307dc5b5b47c4efe9c967405815ba7d4 baffle me ... zero code delta vs master | 17:41 |
| clarkb | `/home/zuul/src/opendev.org/starlingx/update/patch-scripts/start-scripts/24.09.400/pre-start.sh:42:1: E004 File did not end with a newline` is another | 17:41 |
| clarkb | slittle1_: https://review.opendev.org/c/starlingx/update/+/972405/1//MERGE_LIST this is the merge list for that one which is quite large. Is there really zero code delta? | 17:42 |
| clarkb | but again those are errors in your choice of tools as applied to the code base. I don't think zuul has done anything wrong there | 17:42 |
| clarkb | it is also a non voting job so it may be the case that those errors have been there all along and just ignored | 17:43 |
| slittle1_ | i'll try to sort cgcs_patch ... which did hit master .. just thought they had it resolved | 17:46 |
| slittle1_ | I'll see whats left after that. | 17:46 |
| clarkb | backup pruning is done. Got us back down to 68% usage. Logs are in the typical location and exit code waas 0 | 18:30 |
| clarkb | fungi: stephenfin looks like setuptools never made that end of october release (last release is from may) | 19:13 |
| clarkb | that new pep fungi pointed out made me curious | 19:14 |
| clarkb | infra-root as a reminder 23.253.20.107 is the held etherpad 2.6.0 node and there is a clarkb-test pad on it if we want to manually check https://review.opendev.org/c/opendev/system-config/+/972001 to upgrade to etherpad 2.6.0 | 19:15 |
| clarkb | as mentioned I couldn't figure out the session transfer feature and wonder if we have to use the other skin for that. But am curious if anyone else figures it out | 19:15 |
| fungi | it might be covered in the api docs if those are being kept up to date at all these days | 19:20 |
| clarkb | I only see api methods for create, delete, get, and list sessions | 22:07 |
| clarkb | does make me wonder if this is only half baked at this point. The changelog does hint at bigger changes to come | 22:07 |
| clarkb | for some reason I thought gitea was upgraded already. Probably beacuse we did a coupel upgrades in quick succession in december https://review.opendev.org/c/opendev/system-config/+/971469 is another one I should clear off my backlog (a minor gitea bugfix update) | 22:15 |
| clarkb | fwiw fungi's test is the only thing that has hit the honeypot so far today | 22:42 |
| clarkb | infra-root ^ it is the error_log not the access log that you need to check for mod security logs just fyi (the access log should show any 403s too) | 22:42 |
| fungi | maybe the ai scourge has finally come to a screeching halt | 22:43 |
| clarkb | it does seem to be cyclical | 22:43 |
| clarkb | things will be fine for a time then the next horde arrives | 22:44 |
Generated by irclog2html.py 4.0.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!