| @traylenator:matrix.org | Hi, I am unable to login to review.opendev.org , after Ubuntu login I am always redirected to https://review.opendev.org/SignInFailure,SIGN_IN,Contact+site+administrator , email steve.traylen@cern.ch | 05:49 |
|---|---|---|
| @traylenator:matrix.org | * Hi, I am unable to login to review.opendev.org , after Ubuntu login I am always redirected to https://review.opendev.org/SignInFailure,SIGN_IN,Contact+site+administrator , my email is steve.traylen@cern.ch | 05:53 |
| -@gerrit:opendev.org- YeongHoon Choe proposed: [opendev/glean] 985217: Fix fd garbage collection race in Lock context manager https://review.opendev.org/c/opendev/glean/+/985217 | 09:34 | |
| -@gerrit:opendev.org- YeongHoon Choe proposed: [opendev/glean] 985217: Fix fd GC race in Lock context manager https://review.opendev.org/c/opendev/glean/+/985217 | 09:36 | |
| @harbott.osism.tech:regio.chat | looks like log uploads are 100% broken now, I don't have time to check until later today | 14:16 |
| @harbott.osism.tech:regio.chat | confirmed in executor logs both rax and ovh uploads are failing ... so ... not much we can do I guess :-( | 14:42 |
| @tafkamax:matrix.org | Aha those are the post_failures? | 15:08 |
| @clarkb:matrix.org | If both ovh and rax are affected then maybe the openstacksdk release from the 13th is at fault? We would've just upgraded executors to that release overnight. Maybe that was the issue with launcher image builds uploading to sjc3 as well. | 15:56 |
| @clarkb:matrix.org | Unfortunately, I am in no position to debug this today as I'm still dealing with some medical stuff today. But that is where I would start looking | 15:56 |
| @clarkb:matrix.org | > <@traylenator:matrix.org> Hi, I am unable to login to review.opendev.org , after Ubuntu login I am always redirected to https://review.opendev.org/SignInFailure,SIGN_IN,Contact+site+administrator , my email is steve.traylen@cern.ch | 15:58 |
| Chances are that this is due to you attempting to login with a new Ubuntu one openid using the same email address as an old Gerrit account with a different openid. Again I can't really check now but another infra sysadmin can confirm by looking in gerrit's error_log | ||
| @clarkb:matrix.org | If you know that this has happened one solution is to go back to the old openid if possible. Otherwise we'll need to check Gerrit notedb contents and see if there is another way forward (like disabling the old account and removing conflicting info and allowing a new account to be created) | 15:59 |
| @traylenator:matrix.org | Thanks for the reply. That sounds plausible. I tried a few weeks back and could not get in for a different error. One of the things I tried then was deleting and re-creating in U.1. | 16:25 |
| Deleting/disabling/updating the old id in gerrit sounds correct . Whenever you or someone gets a chance. | ||
| @fungicide:matrix.org | `Ansible output: b'TASK [upload-logs-swift : Upload logs to swift]` | 16:54 |
| @fungicide:matrix.org | `Ansible output: b'fatal: [localhost]: FAILED! => {"censored": "the output has been hidden due to the fact that \'no_log: true\' was specified for this result", "changed": false}'` | 16:54 |
| @fungicide:matrix.org | the last successful build to start was at 2026-04-18 08:11:58 | 16:55 |
| @fungicide:matrix.org | my first guess is that's about when our weekly automated zuul upgrade completed | 16:56 |
| @fungicide:matrix.org | Clark's guess makes sense, openstacksdk 4.11.0 happened on monday, after our last zuul upgrade | 16:58 |
| @fungicide:matrix.org | entirely possible it regressed swift api interactions for whatever version of swift we have in ovh and rackspace classic | 16:59 |
| @fungicide:matrix.org | looks like there are no new release notes for openstacksdk | 17:01 |
| @fungicide:matrix.org | https://paste.opendev.org/show/buEsGOEeUlWnvyxkyadj/ is a quick list of the changes that merged for it since 4.10.0 in february | 17:03 |
| @clarkb:matrix.org | Fungi the use objects from * changes make me wonder if it is a python version compatibility issue those libs have stuff in them that may be newer than python 3.11 iirc | 17:06 |
| @fungicide:matrix.org | unlikely since openstacksdk is currently tested against python 3.11 | 17:08 |
| @fungicide:matrix.org | https://zuul.opendev.org/t/openstack/build/3e907a59c89c4c76a2687370e4c14505 | 17:08 |
| @fungicide:matrix.org | that's a successful py311 unit test run yesterday | 17:08 |
| @fungicide:matrix.org | (on master, so where 4.11.0 was tagged) | 17:09 |
| @clarkb:matrix.org | Ack | 17:09 |
| @fungicide:matrix.org | though there are a lot of new changes merged since 4.11.0 too: https://paste.opendev.org/show/bIai1SvGLZmOL7Txo48X/ | 17:10 |
| @fungicide:matrix.org | maybe https://review.opendev.org/c/openstack/openstacksdk/+/984595 is quietly fixing a regression on clouds with keystone older than the v3 api? | 17:12 |
| @fungicide:matrix.org | though i don't know if that affects both of the clouds in question | 17:13 |
| @fungicide:matrix.org | i'm not where i can look closer at this just yet, but if someone can test using the openstacksdk in our latest zuul images to authenticate to and list containers in one of those clouds, that would be a big help | 17:14 |
| @mordred:waterwanders.com | WELL- if I could get a working clouds.yaml for my ovh account I'd totallyhelp | 17:44 |
| @mordred:waterwanders.com | of course, it's been almost 6 years since I ran openstacksdk and our openstack clouds are STILL not providing a clouds.yaml download link or *any* sane "here are the important things you need to use the API" :( So sad | 17:45 |
| @mordred:waterwanders.com | fungi: ok, I got a working clouds.yaml login, validated on 4.10 that it works to at least list containers, and then updated to master and it still worked | 17:56 |
| @mordred:waterwanders.com | I can double-check other calls if you can tell me what is failing | 17:57 |
| @fungicide:matrix.org | mordred: what about 4.11 though? or did you mean 4.11 works? | 18:00 |
| @clarkb:matrix.org | Ya if master has bugfixes it may still be that 4.11 is broken https://opendev.org/zuul/zuul-jobs/src/commit/fc4af6d44718d2e648cf6396f3e96bb46a55117a/roles/upload-logs-base/library/zuul_swift_upload.py is the code that breaks I think but with no log we don't know where | 18:01 |
| @fungicide:matrix.org | at the moment it's hard to know what calls aren't working because they're being run under ansible with `no_log-true` for the task that calls the script | 18:01 |
| @clarkb:matrix.org | But in theory if we run that script to upload something with 4.11 we should see where it fails. Run it manually I mean | 18:07 |
| @mordred:waterwanders.com | fungi: 4.11 also worked | 18:12 |
| @mordred:waterwanders.com | but - I didn't try to upload anything | 18:12 |
| @mordred:waterwanders.com | so - conn.list_containers() works on 4.10, 4.11 and master | 18:14 |
| @mordred:waterwanders.com | fungi: Clark ok, I just used that script to upload a file to OVH on 4.11 and it worked | 19:44 |
| @mordred:waterwanders.com | and validated that on python 3.11 | 19:48 |
| @mordred:waterwanders.com | just to be sure | 19:48 |
| @mordred:waterwanders.com | so - doesn't seem to be cloud API related | 19:48 |
| @mordred:waterwanders.com | one thing I did have to deal with locally, and I haven't looked at the zuul_swift_upload install sequence closely enough ... | 19:49 |
| @mordred:waterwanders.com | but zuul_swift_upload uses requestsexceptions, and openstacksdk *used* to depend on it, but since that has been deprecated/removed, it's _possible_ that the script is erroring out on requestsexceptions not being present. | 19:50 |
| @mordred:waterwanders.com | it errored on that for me after I installed ansible and openstacksdk into a venv so I could run it | 19:51 |
| @mordred:waterwanders.com | I just removed requestsexceptions from the script and it worked | 19:51 |
| @mordred:waterwanders.com | Clark: yeah, doing a quick codesearch I don't see anything that would actually install requestsexceptions if it's no longer being pulled in transitively | 19:53 |
| -@gerrit:opendev.org- Monty Taylor https://matrix.to/#/@mordred:inaugust.com proposed: [zuul/zuul-jobs] 985244: Remove use of requestsexceptions https://review.opendev.org/c/zuul/zuul-jobs/+/985244 | 19:55 | |
| @mordred:waterwanders.com | Clark, fungi ^^ that's my current hunch. I can't prove it at all | 19:56 |
| @mordred:waterwanders.com | but now I need to go take some things to storage :) | 19:56 |
| @clarkb:matrix.org | Thank you for looking into that | 20:16 |
| @mordred:waterwanders.com | Sure nuff - actually rolling out a fix is going to be ... fun :) | 21:22 |
| @mordred:waterwanders.com | oh - funny: https://review.opendev.org/c/zuul/zuul-jobs/+/979814 was already up and green | 21:23 |
| @mordred:waterwanders.com | I've abandoned mine | 21:23 |
Generated by irclog2html.py 4.1.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!