| -@gerrit:opendev.org- Zuul merged on behalf of James E. Blair https://matrix.to/#/@jim:acmegating.com: | 00:28 | |
| - [zuul/zuul] 944162: AWS: Add subnet-id https://review.opendev.org/c/zuul/zuul/+/944162 | ||
| - [zuul/zuul] 944163: AWS: Add ipv6 support https://review.opendev.org/c/zuul/zuul/+/944163 | ||
| -@gerrit:opendev.org- Zuul merged on behalf of James E. Blair https://matrix.to/#/@jim:acmegating.com: [zuul/zuul] 944164: AWS: Add userdata support https://review.opendev.org/c/zuul/zuul/+/944164 | 00:30 | |
| -@gerrit:opendev.org- Ruisi Jian proposed: [zuul/zuul] 945175: WIP: fix file comment line mapping https://review.opendev.org/c/zuul/zuul/+/945175 | 00:56 | |
| -@gerrit:opendev.org- Ruisi Jian proposed: [zuul/zuul] 945175: WIP: fix file comment line mapping https://review.opendev.org/c/zuul/zuul/+/945175 | 01:34 | |
| -@gerrit:opendev.org- Ruisi Jian proposed: [zuul/zuul] 945175: WIP: fix file comment line mapping https://review.opendev.org/c/zuul/zuul/+/945175 | 01:45 | |
| -@gerrit:opendev.org- Ruisi Jian proposed: [zuul/zuul] 945175: WIP: fix file comment line mapping https://review.opendev.org/c/zuul/zuul/+/945175 | 02:48 | |
| -@gerrit:opendev.org- Zuul merged on behalf of James E. Blair https://matrix.to/#/@jim:acmegating.com: | 03:12 | |
| - [zuul/zuul] 944165: AWS: add iam-instance-profile support https://review.opendev.org/c/zuul/zuul/+/944165 | ||
| - [zuul/zuul] 944177: AWS: add image-format option https://review.opendev.org/c/zuul/zuul/+/944177 | ||
| -@gerrit:opendev.org- Zuul merged on behalf of James E. Blair https://matrix.to/#/@jim:acmegating.com: | 03:12 | |
| - [zuul/zuul] 944166: Revert "Pin boto and friends" https://review.opendev.org/c/zuul/zuul/+/944166 | ||
| - [zuul/zuul] 944178: Openstack: add AZ support https://review.opendev.org/c/zuul/zuul/+/944178 | ||
| -@gerrit:opendev.org- Ruisi Jian proposed: [zuul/zuul] 945175: WIP: fix file comment line mapping https://review.opendev.org/c/zuul/zuul/+/945175 | 04:14 | |
| @joao15130:matrix.org | Any advice appreciated. Thanks! | 06:56 |
|---|---|---|
| -@gerrit:opendev.org- Albin Vass proposed: [zuul/zuul-jobs] 728912: Revert "Revert "tox: update lint regex to not require column"" https://review.opendev.org/c/zuul/zuul-jobs/+/728912 | 07:04 | |
| -@gerrit:opendev.org- Ruisi Jian proposed: [zuul/zuul] 945175: WIP: fix file comment line mapping https://review.opendev.org/c/zuul/zuul/+/945175 | 07:19 | |
| -@gerrit:opendev.org- Ruisi Jian proposed: [zuul/zuul] 945175: WIP: fix file comment line mapping https://review.opendev.org/c/zuul/zuul/+/945175 | 08:05 | |
| -@gerrit:opendev.org- Ruisi Jian proposed: [zuul/zuul] 945175: WIP: fix file comment line mapping https://review.opendev.org/c/zuul/zuul/+/945175 | 09:50 | |
| -@gerrit:opendev.org- Ruisi Jian proposed: [zuul/zuul] 945175: WIP: fix file comment line mapping https://review.opendev.org/c/zuul/zuul/+/945175 | 10:31 | |
| -@gerrit:opendev.org- Ruisi Jian proposed: [zuul/zuul] 945175: WIP: fix file comment line mapping https://review.opendev.org/c/zuul/zuul/+/945175 | 11:09 | |
| -@gerrit:opendev.org- Ruisi Jian proposed: [zuul/zuul] 945175: WIP: fix file comment line mapping https://review.opendev.org/c/zuul/zuul/+/945175 | 11:46 | |
| -@gerrit:opendev.org- Benjamin Schanzel proposed: [zuul/zuul] 940379: web: Upgrade nodejs to latest v23 https://review.opendev.org/c/zuul/zuul/+/940379 | 12:06 | |
| -@gerrit:opendev.org- Benjamin Schanzel proposed: [zuul/zuul] 940379: web: Upgrade nodejs to latest v23 https://review.opendev.org/c/zuul/zuul/+/940379 | 12:06 | |
| -@gerrit:opendev.org- Benjamin Schanzel proposed: [zuul/zuul] 940379: web: Upgrade nodejs to latest v23 https://review.opendev.org/c/zuul/zuul/+/940379 | 12:14 | |
| -@gerrit:opendev.org- Benjamin Schanzel proposed: [zuul/zuul] 940379: web: Upgrade nodejs to latest v23 https://review.opendev.org/c/zuul/zuul/+/940379 | 12:42 | |
| -@gerrit:opendev.org- Benjamin Schanzel proposed: [zuul/zuul] 940379: web: Upgrade nodejs to latest v23 https://review.opendev.org/c/zuul/zuul/+/940379 | 12:51 | |
| -@gerrit:opendev.org- Benjamin Schanzel proposed: [zuul/nodepool] 945211: Replace assertDictContainsSubset in tests https://review.opendev.org/c/zuul/nodepool/+/945211 | 13:30 | |
| -@gerrit:opendev.org- Ruisi Jian proposed: [zuul/zuul] 945175: fix(merger): fix file comment line mapping https://review.opendev.org/c/zuul/zuul/+/945175 | 13:52 | |
| -@gerrit:opendev.org- Axel Andersson proposed: [zuul/zuul] 945220: Allow log output from FQCN tasks https://review.opendev.org/c/zuul/zuul/+/945220 | 14:19 | |
| -@gerrit:opendev.org- Axel Andersson proposed: [zuul/zuul] 945220: Allow log output from FQCN tasks https://review.opendev.org/c/zuul/zuul/+/945220 | 14:19 | |
| -@gerrit:opendev.org- Axel Andersson proposed: [zuul/zuul] 945220: Allow log output from FQCN tasks https://review.opendev.org/c/zuul/zuul/+/945220 | 14:21 | |
| -@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed: [zuul/zuul-jobs] 944813: Add upload-image-s3 role https://review.opendev.org/c/zuul/zuul-jobs/+/944813 | 14:51 | |
| -@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed: [zuul/zuul-jobs] 944813: Add upload-image-s3 role https://review.opendev.org/c/zuul/zuul-jobs/+/944813 | 15:06 | |
| -@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed: | 15:39 | |
| - [zuul/zuul] 944947: Support mariadb statement timeouts https://review.opendev.org/c/zuul/zuul/+/944947 | ||
| - [zuul/zuul] 944948: Use mysql query hint regardless of project https://review.opendev.org/c/zuul/zuul/+/944948 | ||
| -@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed: | 15:49 | |
| - [zuul/zuul] 944947: Support mariadb statement timeouts https://review.opendev.org/c/zuul/zuul/+/944947 | ||
| - [zuul/zuul] 944948: Use mysql query hint regardless of project https://review.opendev.org/c/zuul/zuul/+/944948 | ||
| -@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed: | 15:51 | |
| - [zuul/zuul] 944947: Support mariadb statement timeouts https://review.opendev.org/c/zuul/zuul/+/944947 | ||
| - [zuul/zuul] 944948: Use mysql query hint regardless of project https://review.opendev.org/c/zuul/zuul/+/944948 | ||
| @mnaser:matrix.org | We have several tenants in our Zuul environment and I'm running into this common pattern where we need to run a job that uses credentials in a pre-review pipeline, so we define it in another config-project and use it. | 15:59 |
| @mnaser:matrix.org | However.. since the credentials are encrypted using the tenant, I'm not actually able to share that job across all the tenants | 16:00 |
| @mnaser:matrix.org | I end up having to create a base job without secrets, and then duplicate the job into each tenant with the same secret (encrypted using that tenant) | 16:00 |
| @mnaser:matrix.org | My assumption was that if I was to encrypt a secret into repo A using tenant A, and loaded it in tenant B.. it just wouldn't "work" since it couldn't encrypt it | 16:01 |
| @fungicide:matrix.org | correct, the encryption keys are unique per tenant+project combo | 16:02 |
| @fungicide:matrix.org | considering it from a security standpoint, if they weren't then a sneaky user with control of projects for one tenant could copy another tenant's secrets and decrypt them in a job under their control | 16:03 |
| -@gerrit:opendev.org- Zuul merged on behalf of James E. Blair https://matrix.to/#/@jim:acmegating.com: [zuul/zuul] 944167: AWS: Remove block device mapping from template https://review.opendev.org/c/zuul/zuul/+/944167 | 16:03 | |
| @mnaser:matrix.org | yeah, i am in total agreement of the security model in that sense, but i'm just wondering if there is any better pattern than define a base job ina common repo without secrets and then the jobs with the secrets in the tenant-specific repo | 16:04 |
| @clarkb:matrix.org | how does that work for opendev's log uploads? | 16:06 |
| @fungicide:matrix.org | each tenant has their own base config repo | 16:06 |
| @clarkb:matrix.org | but the log uploads are defined once I think | 16:07 |
| @mnaser:matrix.org | https://opendev.org/zuul/zuul-base-jobs/src/branch/master/zuul.yaml#L1-L19 | 16:07 |
| @mnaser:matrix.org | yeah good question | 16:08 |
| @clarkb:matrix.org | it might be the shadow directive in our tenant config | 16:08 |
| @clarkb:matrix.org | opendev/base-jobs shadows other base jobs | 16:08 |
| @mnaser:matrix.org | but technically that comes from another tenant | 16:09 |
| @jim:acmegating.com | https://zuul-ci.org/docs/zuul/latest/project-config.html#encryption | 16:09 |
| @jim:acmegating.com | ``` | 16:09 |
| Each project in Zuul has its own automatically generated RSA keypair which can be used by anyone to encrypt a secret and only Zuul is able to decrypt it. Zuul serves each project’s public key using its build-in webserver. They can be fetched at the path /api/tenant/<tenant>/key/<project>.pub where <project> is the canonical name of a project and <tenant> is the name of a tenant with that project. | ||
| ``` | ||
| @jim:acmegating.com | * > Each project in Zuul has its own automatically generated RSA keypair which can be used by anyone to encrypt a secret and only Zuul is able to decrypt it. Zuul serves each project’s public key using its build-in webserver. They can be fetched at the path /api/tenant/<tenant>/key/<project>.pub where <project> is the canonical name of a project and <tenant> is the name of a tenant with that project. | 16:10 |
| @jim:acmegating.com | the secret is encrypted with the *project's key* and the *project's key* is obtained via the url of any tenant with that project in it | 16:11 |
| @mnaser:matrix.org | aaaaaaaaaaaaaaah | 16:11 |
| @mnaser:matrix.org | so this thing could have worked from the get go, the tenant is just a way to expose it, in the implemetnation it based on the project | 16:11 |
| @jim:acmegating.com | in other words, the initial assertion that the encryption is tenant-scoped is not correct; the encryption is project scoped regardless of tenant. | 16:11 |
| @jim:acmegating.com | yes, and that is the recommended way to set up logging as Clark suggested | 16:12 |
| @mnaser:matrix.org | funny now that you say that i realize that we have logging setup for all the tenants in the same way too and that is working... | 16:12 |
| @mnaser:matrix.org | that should have clicked, lol | 16:12 |
| @fungicide:matrix.org | aha, so the safeguard is that you don't allow tenant b to merge changes for the project under the control of tenant a | 16:13 |
| @fungicide:matrix.org | and then tenant b's project maintainers can't alter the job in order to exfiltrate the decrypted secret | 16:14 |
| @mnaser:matrix.org | for non-config repos i guess the assumption is you'll be in the same tenant, for config repos, the assumption is it'll be managed by some sort of "admin-y" person | 16:14 |
| @mnaser:matrix.org | ok, cool, this simplifies my life a lot | 16:15 |
| @fungicide:matrix.org | at most they can inherit from that job in other jobs, so of course the parent job needs to make sure not to leave decrypted copies of any secret lying around for playbooks from the child job to access | 16:17 |
| -@gerrit:opendev.org- Clark Boylan proposed: [zuul/zuul] 945234: Try to use mariadb in unittest again https://review.opendev.org/c/zuul/zuul/+/945234 | 16:18 | |
| -@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed: | 16:21 | |
| - [zuul/zuul] 944947: Support mariadb statement timeouts https://review.opendev.org/c/zuul/zuul/+/944947 | ||
| - [zuul/zuul] 944948: Use mysql query hint regardless of project https://review.opendev.org/c/zuul/zuul/+/944948 | ||
| -@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed on behalf of Clark Boylan: [zuul/zuul] 945234: Try to use mariadb in unittest again https://review.opendev.org/c/zuul/zuul/+/945234 | 16:21 | |
| -@gerrit:opendev.org- Zuul merged on behalf of James E. Blair https://matrix.to/#/@jim:acmegating.com: [zuul/zuul] 944179: OpenStack: move schema attributes to match aws https://review.opendev.org/c/zuul/zuul/+/944179 | 16:41 | |
| @clarkb:matrix.org | corvus: 945234 is breaking on unsupported dialects in db migrations | 16:44 |
| @clarkb:matrix.org | corvus: that may be something we need to address before landing 944947 as new installs would break (I think existing ones would be ok until the next db migration?) | 16:44 |
| @clarkb:matrix.org | the two changes are in the gate but have a ways to go so I won't -W them yet | 16:46 |
| @clarkb:matrix.org | ah it is just the bundel refactor migration | 16:47 |
| @clarkb:matrix.org | I think that makes this safer as it isn't every migration. I'll push an update to my chagne that can be incorporated in the parents instead | 16:48 |
| -@gerrit:opendev.org- Clark Boylan proposed: [zuul/zuul] 945234: Try to use mariadb in unittest again https://review.opendev.org/c/zuul/zuul/+/945234 | 16:51 | |
| @jim:acmegating.com | yeah, i think we should incorporate the fix into 947 | 16:51 |
| @jim:acmegating.com | it's not like critical, but it belongs there i think. | 16:52 |
| @clarkb:matrix.org | ++ do you want to do that or should I? | 16:52 |
| @jim:acmegating.com | i can | 16:52 |
| @clarkb:matrix.org | ack | 16:52 |
| -@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed: | 16:56 | |
| - [zuul/zuul] 944947: Support mariadb statement timeouts https://review.opendev.org/c/zuul/zuul/+/944947 | ||
| - [zuul/zuul] 944948: Use mysql query hint regardless of project https://review.opendev.org/c/zuul/zuul/+/944948 | ||
| -@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed on behalf of Clark Boylan: [zuul/zuul] 945234: Try to use mariadb in unittest again https://review.opendev.org/c/zuul/zuul/+/945234 | 16:56 | |
| @jim:acmegating.com | Clark: ^ | 16:56 |
| @clarkb:matrix.org | looking | 16:56 |
| @clarkb:matrix.org | I guess the only other question I have is if the mariadb dialect changes behavior of that migration sufficiently when compared to mysql. My hunch is that it won't but... | 16:57 |
| @clarkb:matrix.org | do we want to wait for 945234 to run to the db migration test before approving? | 16:58 |
| @jim:acmegating.com | i agree with the hunch and also that we should wait | 16:58 |
| @clarkb:matrix.org | particularly since so many of those statements are explicit and not going through sqlalchemy's object model stuff | 17:00 |
| @clarkb:matrix.org | we'll run the same statements before and after I suspect and the dialect is just a way to select the right statement for mysql/mariadb or postgres | 17:00 |
| @jim:acmegating.com | exactly; a surprise is more likely to come from one of the more sqlalchemy/alembic heavy migrations | 17:04 |
| @jim:acmegating.com | Clark: `2025-03-21 17:52:48.448432 | ubuntu-jammy | pymysql.err.OperationalError: (1130, "Host 'localhost' is not allowed to connect to this MariaDB server")` from https://zuul.opendev.org/t/zuul/stream/e0876f628344401d8d1c2b897c1b14b3?logfile=console.log is ringing a bell | 18:13 |
| @jim:acmegating.com | i think maybe that was the weird error we couldn't figure out before. but that's not a zuul problem, that's a mariadb/mariadb config problem. | 18:14 |
| @jim:acmegating.com | anyway, one of the jobs is pre-failing on that, but on a test that is not db related. | 18:14 |
| @clarkb:matrix.org | yup iirc it had to do with not setting up the user and per test database access properly | 18:15 |
| @clarkb:matrix.org | I agree that is probably fine to ignore and treat as preexisting problems if the bulk of the test cases are happy | 18:15 |
| @clarkb:matrix.org | I want to say it has to do with something holding a lock preventing the updates from going through | 18:15 |
| @jim:acmegating.com | maybe it is something that cold be fixed in the fixtures (but weird it's not needed for mysql; it seems racy. like maybe two threads inside of mariadb racing?) | 18:15 |
| @jim:acmegating.com | oh yeah, that's similar | 18:15 |
| @clarkb:matrix.org | I spent a bit of time with mariadb lock listings and traced it back to something along those lines (from memory) | 18:16 |
| @jim:acmegating.com | so maybe something like if the fixtures tried, verified, retried that might do it. | 18:16 |
| @clarkb:matrix.org | but I coudln't figure out how to fix it | 18:16 |
| @clarkb:matrix.org | ya maybe we just brute force it | 18:16 |
| @jim:acmegating.com | (that might eat up some cpu though; we'll have to be careful) | 18:16 |
| @jim:acmegating.com | (it's only setup, but still doing that every test) | 18:17 |
| @clarkb:matrix.org | maybe we can ask zzzeek for some time if we narrow it down again | 18:17 |
| @clarkb:matrix.org | corvus: re buggy mariadb possibility I think jammy has mariadb 10.6 and noble is 10.11. The 3.11 job runs on jammy and 3.12 on noble. Bookworm has 3.11 and mariadb 10.11. One idea is we could just punt on solving this if mariadb 10.11 is more stable and use bookworm to run the unittests? | 18:26 |
| @clarkb:matrix.org | If that doesn't sound terrible I can update my change to do that once we get results posted from the current run | 18:26 |
| @clarkb:matrix.org | then we can do lots of rechecks | 18:26 |
| @jim:acmegating.com | i like that idea -- or -- switch to using the containers... (test-setup-docker) | 18:27 |
| @jim:acmegating.com | using the containers gets us the ability to test 3 things, so that's a good long-term plan too | 18:28 |
| @clarkb:matrix.org | ya I think switching to containers would be a good followup if 10.11 works. This is just a quick way to sanity check 10.11 | 18:28 |
| @jim:acmegating.com | sounds good | 18:28 |
| @jim:acmegating.com | Clark: both finished with only the 3.11 nonrelated failure | 18:29 |
| @jim:acmegating.com | i will approve the 2 main code changes | 18:29 |
| @clarkb:matrix.org | ack | 18:30 |
| -@gerrit:opendev.org- Clark Boylan proposed: [zuul/zuul] 945234: Try to use mariadb in unittest again https://review.opendev.org/c/zuul/zuul/+/945234 | 18:38 | |
| -@gerrit:opendev.org- Clark Boylan proposed: [zuul/zuul] 945234: Try to use mariadb in unittest again https://review.opendev.org/c/zuul/zuul/+/945234 | 19:04 | |
| -@gerrit:opendev.org- Clark Boylan proposed: [zuul/zuul] 945253: DNM run lots of unittests to check mariadb instead of mysql https://review.opendev.org/c/zuul/zuul/+/945253 | 19:19 | |
| @clarkb:matrix.org | 945253 tripped over the cannot connect issue on noble with mariadb 10.11 | 19:39 |
| @clarkb:matrix.org | implying that newer mariadb isn't a workaround | 19:39 |
| @jim:acmegating.com | :( | 19:43 |
| @clarkb:matrix.org | I wonder if Openstack has seen similar. I think they do per test db schemas and maybe test with mariadb | 19:45 |
| @clarkb:matrix.org | But I think digging in properly isn't something I'll do today. It requires a level of shutting out the world that will be difficult | 19:48 |
| -@gerrit:opendev.org- Zuul merged on behalf of James E. Blair https://matrix.to/#/@jim:acmegating.com: [zuul/zuul] 944947: Support mariadb statement timeouts https://review.opendev.org/c/zuul/zuul/+/944947 | 19:52 | |
| @clarkb:matrix.org | corvus: looks like the second change after ^ failed? | 19:53 |
| @jim:acmegating.com | yep i'll recheck | 20:04 |
| @clarkb:matrix.org | corvus: looks like you unapproved https://review.opendev.org/c/zuul/zuul/+/944948? | 20:49 |
| @clarkb:matrix.org | did you see something in the test that indicate a problem? | 20:49 |
| @jim:acmegating.com | oh heh, no just task-switched right in the middle of that. :) | 20:56 |
| @clarkb:matrix.org | I'm looking at the two mariadb 10.11 localhost connection errors and they both fail when attempting to get estimated build times. Reading the test logs prior to that I think we were successfully connecting to the db before hand. I wonder/suspect if this is a database connection limit error (seems an odd way to record it, but that may explain why it is intermittent and why things can connect earlier in the test) | 21:01 |
| @clarkb:matrix.org | alternatively it could be another test case updating the schemas and user tables with locks that prevent new connections from occuring | 21:02 |
| @clarkb:matrix.org | I think test A failing to create new db connection because test B holds necessary db table locks or test A failing to create new db connections because we have hit the connection limit are both potential explanations of this behavior | 21:03 |
| @clarkb:matrix.org | I think the next step is grabbing the mariadb server logs as that shoudl record the connection limit occurence if that is the case | 21:03 |
| -@gerrit:opendev.org- Zuul merged on behalf of James E. Blair https://matrix.to/#/@jim:acmegating.com: [zuul/zuul] 944180: OpenStack: remove some unimplemented functionality https://review.opendev.org/c/zuul/zuul/+/944180 | 21:12 | |
| -@gerrit:opendev.org- Zuul merged on behalf of James E. Blair https://matrix.to/#/@jim:acmegating.com: [zuul/zuul] 944181: OpenStack: remove key-name comments https://review.opendev.org/c/zuul/zuul/+/944181 | 21:13 | |
| -@gerrit:opendev.org- Clark Boylan proposed: [zuul/zuul] 945253: DNM run lots of unittests to check mariadb instead of mysql https://review.opendev.org/c/zuul/zuul/+/945253 | 21:14 | |
| @clarkb:matrix.org | interestingly every debian python3.11 job passed but 2 of the python3.12 on ubuntu jobs failed. | 21:22 |
| @clarkb:matrix.org | The underlying filesystem should be the same for both (ext4) but the kernels would be different versions. Could also have different mariadb configurations in place | 21:22 |
| @clarkb:matrix.org | corvus: I rechecekd it again. Looked like another unrelated failure | 22:42 |
| -@gerrit:opendev.org- Clark Boylan proposed: [zuul/zuul] 945253: DNM run lots of unittests to check mariadb instead of mysql https://review.opendev.org/c/zuul/zuul/+/945253 | 23:21 | |
| @clarkb:matrix.org | zuul-nox-py312 on the latest patchset hit the error so I've gone ahead and put an autohold in place for that one | 23:56 |
| @clarkb:matrix.org | looks like the second mariadb fixup change is failing again :( | 23:56 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!