| opendevreview | Dr. Jens Harbott proposed openstack/openstack-zuul-jobs master: Remove bionic nodeset from unit and functional test jobs https://review.opendev.org/c/openstack/openstack-zuul-jobs/+/958668 | 08:56 |
|---|---|---|
| frickler | clarkb: maybe in order to finally make progress with the legacy cleanup, we could create a base job that always fails and reference that for the remainders that cannot yet be deleted? https://review.opendev.org/c/openstack/openstack-zuul-jobs/+/965402 | 09:13 |
| opendevreview | Dr. Jens Harbott proposed openstack/openstack-zuul-jobs master: Add jobs for Python 3.14 https://review.opendev.org/c/openstack/openstack-zuul-jobs/+/968510 | 09:37 |
| opendevreview | Dr. Jens Harbott proposed openstack/openstack-zuul-jobs master: Add jobs for Python 3.14 https://review.opendev.org/c/openstack/openstack-zuul-jobs/+/968510 | 09:39 |
| opendevreview | Matt Crees proposed openstack/openstack-zuul-jobs master: Add openstack-python3-jobs-nova template https://review.opendev.org/c/openstack/openstack-zuul-jobs/+/973217 | 14:32 |
| *** haleyb_ is now known as haleyb | 15:10 | |
| timburke | clarkb, password got rotated again, so i should be good to share the one that needed escaping: aU"HL]nX@7$&j!z | 16:48 |
| clarkb | timburke: cool so now I think we can throw that in a test change applied to that role and see how it breaks with logging enabled | 16:48 |
| timburke | i sprinkled in some backslashes pretty liberally; not sure *all* of them were strictly necessary, but i could get the `docker login` part to work by encrypting aU\"HL]nX\@7\$\&j\!z | 16:49 |
| clarkb | pretty sure we have tests that cover all of this in zuul/zuul-jobs. I just need to find the right place to replace the password there then also drop the no_log: true lines | 16:49 |
| timburke | new password definitely seems like it should be friendlier for ansible & json, but the promote job still fails when getting the JWT :-/ | 16:50 |
| timburke | yet something like `curl -v https://hub.docker.com/v2/users/login/ -H Content-Type:application/json -d '{"username": "screamingfrenzy", "password": "'"$(cat /tmp/secret)"'"}'` comes back 200... | 16:52 |
| clarkb | timburke: remote: https://review.opendev.org/c/zuul/zuul-jobs/+/974753 DNM Testing docker roles with fun characters in passwords | 17:01 |
| clarkb | timburke: in theory that should exercise this end to end with the fake docker hub registry in the test job. Then we can address quoting once we get a concerete error message | 17:02 |
| timburke | 👍 | 17:02 |
| clarkb | timburke: https://zuul.opendev.org/t/zuul/build/0c224ed70bf141f1ae8466ecb349524d/log/job-output.txt#1149-1173 | 17:09 |
| clarkb | timburke: I think it could be the " something is expecting it to have a closing " and treating this as a string? | 17:10 |
| timburke | yeah, seems like it's trouble using shlex to parse the command string from ansible -- maybe just need a couple `| quote` for https://opendev.org/zuul/zuul-jobs/src/branch/master/roles/upload-docker-image/tasks/main.yaml#L12 ? | 17:16 |
| clarkb | timburke: yup just pushed that | 17:17 |
| fungi | timburke: have you gotten far enough to confirm the new simpler password/secret is fully working then? | 17:20 |
| timburke | fungi, no, it's not working yet -- JWT retrieval still doesn't work :( | 17:21 |
| fungi | hrm | 17:21 |
| timburke | yeah, it's weird. like i said, doing it manually with curl seems to work just fine... | 17:22 |
| clarkb | I got one job to pass with the | quote so now I suspect it is a matter of tracking down all the locations that need the escaping | 17:28 |
| clarkb | interseting the consistent failure now is doing a push to the registry. I wonder if I need more quoting there to get it to treat the password correctly | 17:39 |
| clarkb | weird the jobs that use buildx work and those that don't fail but they both use the same docker login task | 17:53 |
| clarkb | I'm going to try and debug on the podman side to see if that gives any better indication of what might be unhappy | 17:53 |
| clarkb | oh except podman also works. weird | 17:54 |
| timburke | should i maybe try using a different base job for swift's promote job? | 17:56 |
| clarkb | timburke: there is the container instead of docker base jobs. But tehy do basically the same thing and are failing the same way in my testing | 17:58 |
| clarkb | the container jobs are a bit more generic and can work with podman or docker | 17:58 |
| clarkb | when paired with podman in my testing change they work now. But docker consistently fails except in the buildx case for some reason. I think the problem needs to be fixed in these jobs and switching base jobs isn't really a solution. The workaround is the one we've discussed which is to use a password that doesn't have these problems | 17:59 |
| clarkb | as for why the current password isn't working are we sure there asn't a trailing newline or similar? | 17:59 |
| clarkb | that is usually the problem when it seems to work when testing manually. Bsaically problem in the encryption serialization process rather than with the credential itself | 17:59 |
| timburke | good thought -- i can try encrypting again; maybe there's a newline that gets ignored for the `docker login` bit but not the JWT... | 18:01 |
| timburke | should the ansible maybe strip trailing newlines there? or maybe find a way to use the same login mechanism for both upload and promote? | 18:04 |
| fungi | the upload jobs push images and (i think) create tags, while the promote jobs create and delete tags. does dockerhub maybe have some rbac scheme and the account in question lacks some but not all of the necessary permissions? | 18:08 |
| clarkb | quay does that but I don't think docker hub does | 18:09 |
| timburke | it's the same account that was in use prior to re-keying everything; the jobs both used to work | 18:09 |
| clarkb | as far as stripping trailing newlines I think we've avoided doing that as we'd have to do it everywhere rather than simply encoding the data correctly | 18:10 |
| clarkb | and in theory the test jobs I'm trying to make happy would ensure this can be done regardless of the password value consistently across the tooling. But that clearly doesn't work right now | 18:10 |
| clarkb | I think the failures may be due to localhost resolving to ::1 and docker only treating 127.0.0.1/8 as insecure by default | 18:19 |
| clarkb | so I need to explicitly list localhost as an acceptable insecure registry in the test setup? | 18:19 |
| clarkb | I suspect that docker updates have chagned some subtle behavior here in a way that our CI jobs didn't notice until just now. I wonder if these pass even without the password chnanges (i'm guessing no) | 18:35 |
| opendevreview | Merged openstack/project-config master: Update description for Freezer projects https://review.opendev.org/c/openstack/project-config/+/974005 | 19:19 |
| opendevreview | Merged openstack/project-config master: Update description for retired roles https://review.opendev.org/c/openstack/project-config/+/962925 | 19:22 |
| clarkb | timburke: fwiw the issues I'm hitting have to do with how the test job sets up a docker registry using http instead of https. I don't think these issues imply further problems with the roles in production | 19:23 |
| clarkb | I think once I sort out the http problem that the variable quoting should stop being an issue with the | quote fix | 19:23 |
| timburke | sounds good | 19:26 |
| fungi | noonedeadpunk: see my (post-approval) comment on that ^ change (962925), but if they're really retired then they missed switching the acl, i think? | 19:33 |
| noonedeadpunk | fungi: eh.... there's been a long story of retirement and un-retirement, and then retirement again I think when monasca got retired... | 20:01 |
| noonedeadpunk | But you're pretty much right | 20:01 |
| noonedeadpunk | I will go and check the status of retirement | 20:02 |
| fungi | thanks, not urgent it was just juping out at me as a potential inconsistency | 20:03 |
| noonedeadpunk | yeah, good catch, I didn't spot that when reviewing | 20:04 |
| fungi | one of the benefits of the retired acl is that it prevents people from submitting changes for review | 20:04 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/project-config master: Properly re-retire OSA Monasca roles https://review.opendev.org/c/openstack/project-config/+/974773 | 20:22 |
| fungi | thanks! | 20:22 |
| timburke | clarkb, looks like you were right: it was an issue with a trailing newline in the secret. promote works again now! | 21:10 |
| timburke | thanks for all the help clarkb and fungi! | 21:10 |
| clarkb | timburke: great and the stack of changes I wrote for the zuul-jobs roles looks happy now so I think we can avoid this problem going forward | 21:10 |
| fungi | yeah, since it was being injected into a command string and not as the final argument, i can see where a stray newline could pose a potential problem | 21:17 |
| timburke | fungi, not where it had trouble! it was in fetching the JWT, at https://opendev.org/zuul/zuul-jobs/src/branch/master/roles/promote-docker-image/tasks/main.yaml#L13-L18 -- where presumably it was sending a body like ..."password": "<secret>\n"} | 22:13 |
| fungi | ah, strange | 22:18 |
| opendevreview | Merged openstack/project-config master: Simplify Zuul tenant config for providers https://review.opendev.org/c/openstack/project-config/+/974462 | 22:59 |
Generated by irclog2html.py 4.0.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!