| *** dviroel|afk is now known as dviroel|out | 00:37 | |
| *** rlandy is now known as rlandy|out | 01:09 | |
| *** ysandeep|out is now known as ysandeep | 05:00 | |
| sahid | Morning all, is it possible to "block" the machine running for https://review.opendev.org/c/openstack/neutron/+/839479/21 so I can connect myself to it and debug CI job failing for neutron-ovs-grenade-dvr-multinode ? | 06:30 |
|---|---|---|
| sahid | I'm completly stuck trying to fix it, and reproducing the env is not quiet easy | 06:31 |
| *** ysandeep is now known as ysandeep|lunch | 07:35 | |
| dpawlik | clarkb: so "message" field got a json content | 08:29 |
| dpawlik | clarkb: other fields are not needed, but if someone want to create a visualization in Opensearch Dashboard instead of Grafana, it will have all informations | 08:30 |
| noonedeadpunk | hi there! I tried to re-use this publish job https://opendev.org/openstack/ansible-collections-openstack/src/branch/master/.zuul.yaml#L355-L360 here https://opendev.org/openstack/ansible-config_template/src/branch/master/zuul.d/project.yaml#L35 As a result, during relese elodilles catched 'Unable to freeze job graph: Project openstack/ansible-config_template is not allowed to run job ansible-collections-openstack-release' | 09:48 |
| noonedeadpunk | what am I missing? | 09:49 |
| noonedeadpunk | Should pipelines be somehow allowed for specific projects, or? | 09:50 |
| noonedeadpunk | k, so there's allowed_projects defined somewhere | 09:55 |
| noonedeadpunk | ah, ok, so it's basically because of the secret | 09:58 |
| noonedeadpunk | `If a job with secrets is defined in an untrusted-project, allowed-projects is automatically set to that project only, and can not be overridden` | 09:59 |
| *** ysandeep|lunch is now known as ysandeep | 09:59 | |
| noonedeadpunk | So basically, if I want/need to re-use secret, then I need to define job not inside project, but in project-config? | 10:00 |
| noonedeadpunk | Or any other suggestion on how to sort things out? As basically the goal is to publish collection under openstack namespace to ansible-galaxy, that is currently owned by ansible-collections-openstack SIG | 10:02 |
| noonedeadpunk | As it's sounds, like moving this secret to project-config and restricting with allowed-projects might be a good alternative, but dunno | 10:08 |
| noonedeadpunk | Also higly likely that I can jsut copy/paste job and secret, as secret is encrypted per tenant, not per project iirc? | 10:11 |
| noonedeadpunk | And I guess pypi releasing is solved with releases holding secret.... | 10:13 |
| noonedeadpunk | (or not) | 10:14 |
| noonedeadpunk | it's in project-config indeed | 10:14 |
| noonedeadpunk | so wdyt of moving job and secret from ansible-collections-openstack to project-config? Or there're some legal things involved as it's basically SIG and ansible stuff is licensed differently? | 10:16 |
| *** rlandy|out is now known as rlandy | 10:32 | |
| *** ysandeep is now known as ysandeep|afk | 10:53 | |
| *** dviroel|out is now known as dviroel | 11:24 | |
| fungi | noonedeadpunk: the secret is encrypted to the project which contains it | 11:30 |
| fungi | and needs to live in the same repo with the playbook which uses it | 11:30 |
| fungi | not sure what you mean by "legal things" and "basically sig" | 11:36 |
| fungi | but yes, usually if a job needs to use a secret and will be run by multiple projects, putting it in a trusted config repo like openstack/project-config is the most popular solution | 11:37 |
| fungi | the key points are documented here: https://zuul-ci.org/docs/zuul/latest/config/secret.html#secret | 11:40 |
| noonedeadpunk | fungi: well, ansible is licensed under GPL which puts some restricitions. I can recall quite big ML thread regarding licenses used there, which resulted in ansible-collections-openstack being not under governance/releasing and managed by SIG | 12:51 |
| noonedeadpunk | fungi: but I'm not sure if secret should be re-encrypted if move it between projeccts? As it's encrypted with zuul tenant RSA key, doesn't it? | 12:52 |
| noonedeadpunk | Just trying to understand how we should go forward with that | 12:53 |
| noonedeadpunk | will likely go and ask them... | 12:54 |
| fungi | it's not a per-tenant key. zuul generates a distinct key per project so that one project can't expose a secret for another even within the same tenant | 12:54 |
| fungi | so yes, if you want to move the key (and the job which uses it) into project-config, it will need to be encrypted to the project-config key rather than the ansible-collections-openstack key | 12:55 |
| fungi | alternatively, there may be a way via allowed-projects to give ansible-config_template permission to run a job defined in ansible-collections-openstack | 12:59 |
| fungi | https://zuul-ci.org/docs/zuul/latest/config/job.html#attr-job.allowed-projects | 13:00 |
| fungi | nope, i was wrong. "If a job.secrets is used in a job definition in an untrusted-project, allowed-projects is automatically set to the current project only, and can not be overridden." | 13:02 |
| fungi | though you might be able to encrypt the same credentials individually for each separate project (e.g. ansible-config_template) and then use https://zuul-ci.org/docs/zuul/latest/config/job.html#attr-job.secrets.pass-to-parent to hand that project's copy to the central job in ansible-collections-openstack | 13:04 |
| fungi | but i don't see a good reason not to just put the job and secret in project-config instead | 13:04 |
| *** dasm|off is now known as dasm|ruck | 13:05 | |
| noonedeadpunk | Nah, I already posted that among my older posts that allowed-projects is not an option right now... | 13:19 |
| noonedeadpunk | I just wonder who has these credentials in an unecrypted way | 13:19 |
| *** ysandeep|afk is now known as ysandeep | 13:20 | |
| noonedeadpunk | So who can re-encrypt and push... likely sshnaidm | 13:20 |
| fungi | yeah, reminder here that the admins have a policy not to decrypt secrets themselves: https://docs.opendev.org/opendev/infra-manual/latest/testing.html#handling-zuul-secrets | 13:22 |
| noonedeadpunk | yeah, that's out of quiestion now :) | 13:23 |
| noonedeadpunk | was trying to understand if you see any blockers why this can't be moved to system-config | 13:24 |
| fungi | project-config (confusingly, system-config is not a trusted config repo, it's where we keep our systems deployment configs not our zuul configs) | 13:26 |
| fungi | they used to be part of the same repo, but circa 2014 we decided we wanted a separate (larger) core review team for job configs so we split them apart | 13:27 |
| fungi | it used to just be called "config" back in the long-long ago, in the beforetime | 13:27 |
| sshnaidm | noonedeadpunk, what is it about? | 13:39 |
| noonedeadpunk | sshnaidm: well, I tried to re-use your a-u-c job for publishing collection on galaxy and it's in fact not posisble, as secrets can't be read by any other project, if they're stored in untrusted zuul project | 13:40 |
| sshnaidm | noonedeadpunk, I see, let's talk in 20min, brb | 13:41 |
| noonedeadpunk | sshnaidm: so basically either secrets and job should be moved to project-config repo, where all openstack secrets are (possibly limit repos that can use this job with allow-projects?) or we should just copy-paste stuff or create another namespace which could be also an option | 13:43 |
| sshnaidm | noonedeadpunk, I don't see a problem to move it to project-config, but definitely should be restricted to a limited repos. I don't think it's a good idea to allow everyone to push random things there | 14:14 |
| sshnaidm | It's weird we can't reuse it from current repo.. | 14:15 |
| noonedeadpunk | yeah, I restricting to specific repos sounds reasonable to me as well | 14:16 |
| noonedeadpunk | well, I was surprised as well, but that kind of makes sense from other side | 14:17 |
| sshnaidm | I don't think though you need to copy job content if you just want to encrypt secret. AFAIK you just define your own secret with same name and can use the job | 14:17 |
| sshnaidm | but maybe I missed something, or it's different zuul configurations.. | 14:18 |
| noonedeadpunk | well, maybe, parenting would work. given that secret would be local to the repo | 14:18 |
| noonedeadpunk | we can try that out ofc if you prefer this way | 14:19 |
| sshnaidm | noonedeadpunk, I think a job in project-config should be fine | 14:20 |
| sshnaidm | mind to make a patch? | 14:20 |
| noonedeadpunk | ofc I can. Except you would need to paste re-encrypted secret there :) | 14:20 |
| sshnaidm | noonedeadpunk, sure | 14:21 |
| sshnaidm | noonedeadpunk, is there a way to limit patches there without our reviews or kind of? Just in case someone wants to push there without coordination | 14:28 |
| sshnaidm | noonedeadpunk, actually, is it possible to have only job in project-configs, but secrets in each repo? | 14:29 |
| fungi | yes, you can use pass-to-parent for that | 14:31 |
| fungi | an example is the git repo sync some projects use to mirror their repos to various github orgs | 14:31 |
| fungi | zuul doesn't allow projects to run secret-using jobs from other non-config projects for valid security reasons related to the ability of non-config projects to exercise proposed job changes before being reviewed, so anyone who can push a change to gerrit could expose the secret if zuul didn't restrict it | 14:33 |
| sshnaidm | fungi, thanks | 14:33 |
| sshnaidm | noonedeadpunk, seems like the best solution? ^^ | 14:33 |
| fungi | we used to allow it years ago, and then discovered there was no way to secure the secrets if that was allowed | 14:33 |
| fungi | i linked the pass-to-parent option documentation above, if you want to see the usage and caveats | 14:34 |
| sshnaidm | fungi, yeah, sounds logical | 14:34 |
| sshnaidm | fungi, ack | 14:34 |
| fungi | reusable secret-using jobs in a config project are safer, because config projects explicitly don't support pre-merge applicaiton of job changes | 14:35 |
| sshnaidm | noonedeadpunk, we can actuall make a very general job that will allow push collections to galaxy to any namespace, "just define your secrets and params" | 14:35 |
| sshnaidm | fungi, yeah, right | 14:35 |
| fungi | that's a similar model to other publication jobs we already have | 14:35 |
| noonedeadpunk | well, yes, we can do that | 14:35 |
| noonedeadpunk | sshnaidm: eventually it's already pretty much simple/common? | 14:36 |
| noonedeadpunk | as namespace is retrieved from galaxy.yml? | 14:36 |
| noonedeadpunk | so it's already pretty much true | 14:37 |
| sshnaidm | that's great | 14:37 |
| sshnaidm | sorry, didn't look at it in a while | 14:37 |
| sshnaidm | "it just works" (s) | 14:37 |
| sshnaidm | *(c) | 14:37 |
| noonedeadpunk | yeah, well, I pushed some edit to make it work in theory :) | 14:37 |
| sshnaidm | yeah, thanks :) | 14:38 |
| noonedeadpunk | so, we don't push secret at the end? | 14:38 |
| noonedeadpunk | and will store secret inside projects? | 14:38 |
| sshnaidm | yes | 14:38 |
| sshnaidm | with "pass to parent" | 14:38 |
| sshnaidm | I'll generate a secret for your project | 14:39 |
| noonedeadpunk | ok. I guess secret name and params should be quite static though? | 14:39 |
| sshnaidm | yeah | 14:39 |
| noonedeadpunk | Not sure that url should be in secret then? | 14:40 |
| fungi | a "secret" is usually considered as a bundle of closely-related bits of data, some of which can be encrypted but not all of which need to be | 14:41 |
| noonedeadpunk | looking at pypi example, you don't store pypi url there... | 14:42 |
| fungi | if the url is expected to vary by project then it should probably be a part of the secret. if it will be the same for every project then it can be a separate parameter/default | 14:43 |
| noonedeadpunk | yeah, fair | 14:44 |
| fungi | the classic example is a username and password. the password will be encrypted, the username may be in plaintext, but they're generally paired information and so would both be part of the "secret" | 14:45 |
| fungi | for some publication jobs we put a url or path in the secret because those are project-specific or at least namespace-specific | 14:46 |
| noonedeadpunk | also not sure how much iit sense would make to upload build collections to tarballs and publish from there... | 14:47 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/project-config master: Add job for publishing ansible collections https://review.opendev.org/c/openstack/project-config/+/850664 | 14:55 |
| noonedeadpunk | sshnaidm: was you thinking about smth like that ^ + per repo having secrets/jobs like https://review.opendev.org/c/openstack/ansible-config_template/+/850666 ? | 15:04 |
| noonedeadpunk | I decided to push to our repo change first as sample and then can adjust a-c-o | 15:04 |
| noonedeadpunk | if we're on the same page | 15:04 |
| sshnaidm | noonedeadpunk, looks good, jm1 can you take a look please? ^^ | 15:06 |
| noonedeadpunk | probably, we don't even need patch to project-config and just parent job from a-c-o, but likely it would be cleaner a bit if parent from project-config | 15:06 |
| clarkb | sahid isn't here anymore but we can do that. | 15:06 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/project-config master: Add job for publishing ansible collections https://review.opendev.org/c/openstack/project-config/+/850664 | 15:16 |
| sshnaidm | noonedeadpunk, yeah, having it in project-config will be more clear, so anyone can find it | 15:18 |
| clarkb | noonedeadpunk: sshnaidm did that job live in anothe repo originally? if so maybe credit it so that the code origin is known | 15:19 |
| noonedeadpunk | clarkb: yup, it's from https://opendev.org/openstack/ansible-collections-openstack/src/branch/master/.zuul.yaml#L355-L376 | 15:20 |
| clarkb | ok can you update the change to include that info? | 15:20 |
| noonedeadpunk | Should I put that to commit msg or description of the job? | 15:21 |
| clarkb | whenever you copy code from one place to another you should do this. Even if the code was appropriately licensed (that way we can all confirm that and if upstreams need to be updated to fix bugs that can happen too etc) | 15:21 |
| clarkb | I would put it in the content itself not the commit message | 15:21 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/project-config master: Add job for publishing ansible collections https://review.opendev.org/c/openstack/project-config/+/850664 | 15:25 |
| noonedeadpunk | clarkb: you meant smth like that? https://review.opendev.org/c/openstack/project-config/+/850664/3/playbooks/ansible-collections/run.yml | 15:25 |
| noonedeadpunk | as eventually there was no license header, so it's a bit confusing for me... | 15:26 |
| *** dviroel_ is now known as dviroel | 15:26 | |
| clarkb | noonedeadpunk: ya that works. Note the repo has a license file so I would assume that particular file falls under that license: https://opendev.org/openstack/ansible-collections-openstack/src/branch/master/COPYING | 15:27 |
| noonedeadpunk | yeah, fair | 15:27 |
| *** rlandy is now known as rlandy|afk | 15:28 | |
| noonedeadpunk | sshnaidm: one nasty thing is that secrets name should be unique between projects? | 15:29 |
| sshnaidm | noonedeadpunk, yeah, project name is part of the encryption | 15:29 |
| clarkb | noonedeadpunk: look at opendev's container image build jobs and zuuls | 15:29 |
| clarkb | sshnaidm: I think the secret blob itself usually contains key values for that sort of differentiation | 15:30 |
| clarkb | er sshnaidm noonedeadpunk ^ | 15:30 |
| *** ysandeep is now known as ysandeep|out | 15:30 | |
| noonedeadpunk | so yeah, basically secret name should be different but wen passing to job should be same, ie https://review.opendev.org/c/openstack/ansible-config_template/+/850666/4/zuul.d/jobs.yaml ? | 15:32 |
| clarkb | https://opendev.org/zuul/zuul/src/branch/master/.zuul.yaml#L248-L261 and https://opendev.org/opendev/system-config/src/branch/master/zuul.d/docker-images/base.yaml#L35-L38 illustate this | 15:34 |
| noonedeadpunk | yeah, exactly, thabks for confirming | 15:35 |
| fungi | if the secret is supplied from the projects via pass-to-parent they'll need to be encrypted separately with the keys for the projects they're inside. alternatively, secrets can be shared if you put them into the trusted config repo with the job definition (you could still have multiple secrets in there if different projects needed different logins to different publication namespaces, | 15:45 |
| fungi | for example) | 15:45 |
| clarkb | ya the examples I showed are for separately encrypted secrets using pass to parent. I Think that would work here | 15:45 |
| fungi | if you do share secrets centrally, you'll want to keep a list of allowed projects for those jobs though | 15:46 |
| fungi | since otherwise any project could run them and publish to the same namespace | 15:46 |
| lajoskatona | clarkb, fungi: Hi, we decided finally to fix the unit tests for the timeout (after with your help we found the suspicious groups): https://review.opendev.org/c/openstack/neutron/+/850670 | 18:08 |
| lajoskatona | clarkb, fungi: sorry if I burned your time, my intention was only to check if you have seen any changes around infra which can be interesting to check first after no recent code changes happened around Neutron | 18:09 |
| fungi | no need to apologize, i'm chair of the openstack testing and collaboration tools sig precisely because i care that testing works well for openstack projects! i wouldn't have helped you look into it if it wasn't something i had a vested interest in getting to the bottom of | 18:10 |
| fungi | i'm especially happy to discover that narrowing it down to a kernel update was a worthwhile use of our time | 18:11 |
| fungi | to be honest, it felt like a shot in the dark. it's not that often a random kernel security backport impacts performance like that | 18:13 |
| fungi | we definitely did our best to rule out more likely suspects first | 18:14 |
| fungi | to quote sherlock holmes: "Once you eliminate the impossible, whatever remains, no matter how improbable, must be the truth." | 18:16 |
| lajoskatona | fungi: :-) | 18:46 |
| lajoskatona | fungi: the kernel backport was a good tip, though we have no clear idea what was the root cause, and strange that with tempest things are working well, so that's why we said the best is to fix unit tests to avoid them to touch sysctl | 18:49 |
| fungi | lajoskatona: one of the cve's fixed in that patch involved the kernel's pf interface, so it's a good bet that's related | 19:08 |
| JayF | Performance regression in a kernel security fix is very surprising :-O. Nice find. | 19:28 |
| clarkb | lajoskatona it may still be a good idea to look at adding global test timeouts too. I added them to a number of openstack projects way way back specifically to help make these sorts of problems easier to debug | 19:53 |
| *** dasm|ruck is now known as dasm|off | 21:04 | |
| *** dviroel is now known as dviroel|out | 21:07 | |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!