| frickler | infra-root: seems rtd trigger jobs have been failing for a month or so, slaweq tried to submit some creds in https://review.opendev.org/c/x/tobiko/+/933248 but it looks like that may be the wrong approach (cf. release channel). can anyone help with this? | 13:12 |
|---|---|---|
| fungi | frickler: it looks like the current trigger-readthedocs-webhook job (usually added via the docs-on-readthedocs template) expects projects to add openstackci as an admin for each project on rtd and then supply a rtd_webhook_id number taken from that project's webhook url | 13:37 |
| fungi | the rtd_credentials secret hasn't changed since 2018 when rtd changed their api to require authentication | 13:38 |
| fungi | that https://github.com/readthedocs/readthedocs.org/pull/11083 change looks more like deferred cleanup from 6 years ago when they started requiring authentication | 13:40 |
| frickler | fungi: yes, seems something else has changed recently then. slaweq mentioned that manually testing with the token he got from the webpage would work. maybe our token simply expired? do we have that recorded on bridge so we could test it? | 13:42 |
| fungi | i don't think what we've been authenticating with is a token, but rather a username+password | 13:43 |
| fungi | maybe they stopped allowing normal account authentication on the webhook? | 13:44 |
| fungi | https://opendev.org/zuul/zuul-jobs/src/branch/master/roles/trigger-readthedocs/tasks/main.yaml#L20-L27 | 13:45 |
| fungi | so we're using http basic auth with a username and password | 13:45 |
| fungi | oh, though there's also token support just below that | 13:46 |
| fungi | so when rtd_credentials.username is defined we do basic auth | 13:47 |
| fungi | when rtd_credentials.integration_token is defined and rtd_credentials.username is not defined we do token auth | 13:47 |
| fungi | from a timing perspective, the job last ran successfully 2024-09-19 11:08:21 utc | 13:53 |
| frickler | I see only username and password on bridge. I can run some tests after PTG session | 13:55 |
| fungi | fwiw, https://codesearch.opendev.org/?q=integration_token doesn't turn up any examples of projects on opendev using tokens, but the support has been there since 2018 when authentication was implemented | 13:57 |
| slaweq | fungi yes, seems like projects are not using token currently and actually this job seems to be failing for all projects which uses it :) | 14:01 |
| slaweq | and this token is not something what openstackci user would use before (as you said) - it is generated in the project's integration tab for each webhook created for the project in rtd | 14:02 |
| fungi | right, looking at that rtd pr, i think when they say "a secret" they explicitly mean "a token" and not merely an authenticated session | 14:02 |
| slaweq | so it's per webhook, no per user | 14:02 |
| slaweq | yes, and this "rtd_credentials.integration_token" thing in the zuul role is exactly what we need to set to make it working | 14:03 |
| slaweq | this needs to be set, same as rtd_webhook_id per project | 14:03 |
| slaweq | that's how I understand it at least | 14:03 |
| fungi | so probably the trigger-readthedocs-webhook job is going to need a rewrite to drop the existing secret and then projects are going to have to start passing their own individual secrets as an integration_token but they'll probably need pass-to-parent enabled since normally you can't supply secrets to a job defined in another repository | 14:05 |
| slaweq | fungi you mean something like in this job for example: https://opendev.org/x/tobiko/src/branch/master/zuul.d/others.yaml#L22 ? | 14:07 |
| fungi | basically, because the current rtd_credentials.username is being set unconditionally in trigger-readthedocs-webhook the role always chooses to do basic auth rather than token auth, so can't be overridden by a child or variant | 14:08 |
| slaweq | ok, I can try to prepare something maybe in next few days, but I will probably be bothering you with many questions in the meantime :) | 14:09 |
| slaweq | but if you think it is simple change and you can do it quickly, feel free to do so - I will be happy to just test it from the tobiko project if it works fine :) | 14:09 |
| fungi | yeah, similar to your tobiko-upload-git-mirror job, but the parent trigger-readthedocs-webhook will need to be changed in openstack/project-config to accommodate that | 14:09 |
| slaweq | ++ | 14:10 |
| fungi | basically in order to work similarly to the upload-git-mirror parent | 14:10 |
| slaweq | yeah, I will look at that upload-git-mirror job and will try to do it in the similar way | 14:11 |
| fungi | but yeah, after reviewing that rtd pr, i agree that was probably the change that broke the job once it rolled into production for them | 14:11 |
| fungi | it looks like it switched the v2 api webhook to only work when a token is supplied | 14:12 |
| opendevreview | wu.chunyang proposed openstack/diskimage-builder master: Remove the usage of pkg_resource https://review.opendev.org/c/openstack/diskimage-builder/+/933324 | 14:34 |
| clarkb | projects can also configure rtd to rebuild on a timer. Sometimes I wonder if that would be simpler | 14:38 |
| clarkb | (since rtd seems to break their api every couple of years) | 14:39 |
| clarkb | looking at the ptg schedule for tomorrow I think we can remove meetpad nodes from the emergency file after 1700UTC. The meetpad servers won't be touched by ansible unless we manually merge a change to run those jobs, run ansible ourselves, or wait for 0200 UTC ish daily runs saturday morning | 14:46 |
| fungi | i concur | 14:46 |
| cardoe | who owns the sync opendev/openstack -> github/openstack ? | 15:10 |
| clarkb | I'm trying to get 933162 landed this morning then I will enqueue centos 9 stream image builds to hopefully address the issue mnasiadka has been working on with ansible and locales | 15:10 |
| clarkb | cardoe: its a zuul job that openstack projects are expeced to configure. I think the job lives in openstack/project-config so cores of that project can help review fixes to the sync job | 15:10 |
| cardoe | thanks. just noticed that codegenerator isn't syncing | 15:11 |
| clarkb | cardoe: https://opendev.org/openstack/openstack-zuul-jobs/src/branch/master/zuul.d/project-templates.yaml#L3629-L3643 you add this template to your zuul config | 15:18 |
| cardoe | thanks! | 15:24 |
| fungi | and then once your next commit merges to the repo, the job should get triggered and all its contents will be mirrored to github | 15:27 |
| clarkb | oh hey https://review.opendev.org/c/opendev/system-config/+/933155 passes so openafs on arm is happily building again | 15:34 |
| fungi | excellent | 16:31 |
| timburke | frickler, thanks for the node-hold -- i think i'm good now. got cpu flags to compare and found the line down in liberasurecode that was tripping the SIGILL | 16:39 |
| frickler | timburke: ok, cool, dropping the node now. | 16:41 |
| clarkb | the nodepool image updated. The hourly run to update the image should occur in the next half hour or so then I will request an image rebuild | 16:55 |
| clarkb | centos-9-stream is building now | 17:23 |
| clarkb | https://nb02.opendev.org/centos-9-stream-b13b7adf6f3a4db4a094d57bf133d2e9.log is the log | 17:50 |
| frickler | ah, still in progress, I was about to trigger a recheck in kolla. maybe I can have another look later | 18:09 |
| clarkb | I've checked the vexxhost backup server and borg-ethercalc02's shell is still set to nologin and no new backup dir was created | 18:18 |
| clarkb | so this seems to be a workable extra permanent pruning method for these old servers if we're comfortable proceeding further. | 18:18 |
| fungi | seems reasonable to me still | 18:37 |
| clarkb | considering it is backups I'm willing to wait and see if anyone else has an opinion on it before applying it more broadly. Let me know what you think! | 18:43 |
| clarkb | can also write a blurb up in our docs for backups if we're sticking with this | 18:44 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Document process for removing old service/server backups https://review.opendev.org/c/opendev/system-config/+/933354 | 19:08 |
| clarkb | there we can chime in via this doc update. The process documented there is the one used for borg-ethercalc02 on the vexxhost server | 19:09 |
| clarkb | frickler: the image just finished building but now it must upload to the various clouds | 19:11 |
| opendevreview | Lars Kellogg-Stedman proposed openstack/diskimage-builder master: Support Fedora cloud images for Fedora 40 and later https://review.opendev.org/c/openstack/diskimage-builder/+/933361 | 19:25 |
| frickler | clarkb: thx, if you want you could recheck https://review.opendev.org/c/openstack/kolla-ansible/+/933140 once the uploads are done and see if the centos9s jobs fails early again or now. otherwise I'll take a look tomorrow | 19:26 |
| clarkb | I should be able to do that later today | 19:27 |
| opendevreview | Lars Kellogg-Stedman proposed openstack/diskimage-builder master: Support Fedora cloud images for Fedora 40 and later https://review.opendev.org/c/openstack/diskimage-builder/+/933361 | 19:30 |
| clarkb | I think two more clouds need that image. While I'm waiting I'm going to get a bike ride in before the atmospheric river/pineapple express arrives tomorrow | 20:29 |
| fungi | mmmpineapples | 20:33 |
| JayF | I shouldn't have looked at the weather /me hates the rain | 20:50 |
| corvus | can has rain plz? | 21:19 |
| fungi | ship it south a ways, yeah | 21:26 |
| JayF | I know in my brain it's good for rain to happen; but there's something hugely depressing and ominous about the 10 day forecast on weather.com just showing rain trailing off into the distance with no end in sight. I need a "SUNNY" at the end of the tunnel :laugh: | 21:27 |
| Clark[m] | It's always rough when we go into that first long rainy stretch but then you accept it and do things on the less bad days | 22:20 |
| JayF | I've been living here for a good 7 years and I've never hit any points of acceptance lol | 23:03 |
| clarkb | JayF: its one of those things where you think it will be worse than it is but once you take the leap you realize its not so bad and has its own charm. At least for me thats typically how it goes | 23:13 |
| clarkb | its looking like the kolla job is happier now if I'm reading the log correctly. It didn't blow up in the first script that runs | 23:13 |
| JayF | dead serious, I went from "miserable during fall" to "unhappy during fall" simply by giving in and buying a really, really nice rain jacket | 23:15 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!