| -@gerrit:opendev.org- Dong Zhang proposed: [zuul/zuul] 836616: WIP: Debug log for unexpected skip https://review.opendev.org/c/zuul/zuul/+/836616 | 07:33 | |
| -@gerrit:opendev.org- Andy Ladjadj proposed: [zuul/zuul-jobs] 836744: fix(packer): prevent task failure when packer_variables is not defined https://review.opendev.org/c/zuul/zuul-jobs/+/836744 | 08:48 | |
| -@gerrit:opendev.org- Zuul merged on behalf of Clark Boylan: [zuul/zuul] 836108: Handle greedy project name regexes https://review.opendev.org/c/zuul/zuul/+/836108 | 09:24 | |
| -@gerrit:opendev.org- Andy Ladjadj proposed: [zuul/zuul-jobs] 836744: fix(packer): prevent task failure when packer_variables is not defined https://review.opendev.org/c/zuul/zuul-jobs/+/836744 | 09:27 | |
| -@gerrit:opendev.org- Dong Zhang proposed: [zuul/zuul] 836616: WIP: Debug log for unexpected skip https://review.opendev.org/c/zuul/zuul/+/836616 | 09:33 | |
| -@gerrit:opendev.org- Albin Vass proposed: [zuul/zuul] 833001: Allow using 'unique' workspace scheme in jobs https://review.opendev.org/c/zuul/zuul/+/833001 | 11:11 | |
| -@gerrit:opendev.org- Albin Vass proposed: [zuul/zuul] 833001: Allow using 'unique' workspace scheme in jobs https://review.opendev.org/c/zuul/zuul/+/833001 | 13:43 | |
| -@gerrit:opendev.org- Albin Vass marked as active: [zuul/zuul] 833001: Allow using 'unique' workspace scheme in jobs https://review.opendev.org/c/zuul/zuul/+/833001 | 13:45 | |
| @hanson76:matrix.org | Hi, | 14:44 |
|---|---|---|
| I just reported a bugreport (https://storyboard.openstack.org/#!/story/2009974) on Nodepool. | ||
| The "bug" is that the Kubernetes driver in Nodepool is using an api that has been removed in K8s 1.22 | ||
| The affected api is rbac.authorization.k8s.io/v1beta1. | ||
| It was deprecated in 1.8 and replaced by rbac.authorization.k8s.io/v1. | ||
| We noticed this when we upgraded our test k8s cluster today. | ||
| @clarkb:matrix.org | Looks like nodepool testing pins minikube to version 1.22.0 as 1.23.0 broke create_namespaced_role? Do minikube versions align with k8s versions? If so I wonder why testing doesn't catch that | 15:09 |
| @clarkb:matrix.org | Does anyone know why the graphs at https://grafana.opendev.org/d/94891e7b01/resource-usage-by-tenants-and-projects?orgId=1 stopped on March 25? I suspect that the paths for stats that zuul publishes may have changed? | 15:09 |
| @jim:acmegating.com | > <@clarkb:matrix.org> Does anyone know why the graphs at https://grafana.opendev.org/d/94891e7b01/resource-usage-by-tenants-and-projects?orgId=1 stopped on March 25? I suspect that the paths for stats that zuul publishes may have changed? | 15:16 |
| possibly this change: https://zuul-ci.org/docs/zuul/latest/releasenotes.html#relnotes-5-2-0-upgrade-notes | ||
| @clarkb:matrix.org | I should learn to read the release notes. Deploying from master ish has meant they aren't always useful, but a good first check anyway. Thank you | 15:21 |
| @clarkb:matrix.org | corvus: is_use is the snapshot value for current usage? | 15:24 |
| @jim:acmegating.com | Clark: yeah, the other includes ready nodes allocated to the tenant. depending on exactly what you're looking for, the new stats may be of interest. | 15:25 |
| @clarkb:matrix.org | I guess the values we were using before are now the in_use values so using the in_use values would preserve behavior | 15:26 |
| @jim:acmegating.com | yep | 15:26 |
| @clarkb:matrix.org | > <@clarkb:matrix.org> Looks like nodepool testing pins minikube to version 1.22.0 as 1.23.0 broke create_namespaced_role? Do minikube versions align with k8s versions? If so I wonder why testing doesn't catch that | 16:27 |
| Reading the minikube logs it appears to install k8s 1.21.2 which explains that | ||
| @clarkb:matrix.org | I guess the work that would need to be done is updating the minikube version and then fixing the fallout. I can push up a change that bumps the minikube version, but I doubt I'm the best person to fix the problems that show up | 16:27 |
| -@gerrit:opendev.org- Clark Boylan proposed: [zuul/nodepool] 836860: Unpin minikube in testing https://review.opendev.org/c/zuul/nodepool/+/836860 | 16:31 | |
| @clarkb:matrix.org | tristanC: ^ you may find that interesting | 16:31 |
| @clarkb:matrix.org | 836860 installs k8s 1.23.3 and as expected the test fails due to waiting for ready nodes | 16:54 |
| -@gerrit:opendev.org- Clark Boylan proposed: [zuul/nodepool] 836860: Support K8s 1.22.0 and newer. https://review.opendev.org/c/zuul/nodepool/+/836860 | 17:01 | |
| @clarkb:matrix.org | Hanson: ^ I'm not sure if that is a complete fix. Feel free to update the change as necessary if you like. I don't have a kubernetes to test with beyond the functional test job that zuul will run for us | 17:03 |
| -@gerrit:opendev.org- Zuul merged on behalf of Ian Wienand: [zuul/nodepool] 836263: Docker image: use unstable debootstrap https://review.opendev.org/c/zuul/nodepool/+/836263 | 17:21 | |
| @clarkb:matrix.org | oh thats neat. Even when you set the specific version the client is using the beta version. I think this means we actually need to update the openshift client? | 17:22 |
| -@gerrit:opendev.org- Clark Boylan proposed: [zuul/nodepool] 836860: Support K8s 1.22.0 and newer. https://review.opendev.org/c/zuul/nodepool/+/836860 | 17:27 | |
| @clarkb:matrix.org | based on how things have historically gone with updating the openshift and kubernetes client libraries I expect ^ to explode spectacularly | 17:27 |
| -@gerrit:opendev.org- Clark Boylan proposed: [zuul/nodepool] 836860: Support K8s 1.22.0 and newer. https://review.opendev.org/c/zuul/nodepool/+/836860 | 17:46 | |
| @clarkb:matrix.org | its interesting that we set the version in the manifest and also the version of the client is part of the class name | 17:46 |
| @clarkb:matrix.org | oh wow that passed the k8s test. It might just be that simple? | 18:00 |
| -@gerrit:opendev.org- Clark Boylan proposed: [zuul/nodepool] 836875: Update openshift dependency https://review.opendev.org/c/zuul/nodepool/+/836875 | 18:08 | |
| -@gerrit:opendev.org- Mohammed Naser proposed: | 18:54 | |
| - [zuul/zuul-registry] 835772: tests: Add regression test for manifests https://review.opendev.org/c/zuul/zuul-registry/+/835772 | ||
| - [zuul/zuul-registry] 835773: Fix "docker manifest create" issue https://review.opendev.org/c/zuul/zuul-registry/+/835773 | ||
| @caiquemello:matrix.org | Hi guys! | 19:05 |
| I'm having some issues trying to upload logs to a S3 Bucket using the role upload-logs-s3[0]. I've passed access_key_id, secret_access_key, plublic-endpoint and bucket_name to the role. | ||
| The errors messages are: | ||
| WARNING:urllib3.connectionpool:Connection pool is full, discarding connection: s3.us-east-1.amazonaws.com | ||
| botocore.exceptions.ClientError: An error occurred (AccessDenied) when calling the PutObject operation: Access Denied | ||
| I've tested manually the credentials and the others parameters uploading a file to the same bucket using aws CLI, and it works just fine. Witch I believe shows that my bucket policies are fine. | ||
| [0] https://opendev.org/zuul/zuul-jobs/src/branch/master/roles/upload-logs-s3/tasks/main.yaml#L23 | ||
| boto3: https://opendev.org/zuul/zuul-jobs/src/branch/master/roles/upload-logs-base/library/zuul_s3_upload.py | ||
| @caiquemello:matrix.org | Do you guys have any ideia to help me see a little further? | 19:08 |
| @clarkb:matrix.org | caiquemello: one semi common thing is a stray newline at the end of your secret that gets there during the encoding step. Might want to double check that isn't a problem | 19:08 |
| @caiquemello:matrix.org | > <@clarkb:matrix.org> caiquemello: one semi common thing is a stray newline at the end of your secret that gets there during the encoding step. Might want to double check that isn't a problem | 19:14 |
| Thank you Clark, but I dont know if i got your point. By secret, do you mean using secrets to enconding the credentials? if so, I'm not using it right now, cause I'm runnig it in a test env. | ||
| @clarkb:matrix.org | yes the encoding of the credentials. I guess we can rule that out hten | 19:16 |
| @caiquemello:matrix.org | yeap | 19:16 |
| @clarkb:matrix.org | I don't have any other ideas. We don't use s3 so I don't have much exposure to it. | 19:22 |
| @jim:acmegating.com | caiquemello: here's an idea for more debugging: you can invoke the log upload script as a cli: https://opendev.org/zuul/zuul-jobs/src/branch/master/roles/upload-logs-base/library/zuul_s3_upload.py | 19:23 |
| @jim:acmegating.com | there are no instructions for setting up a test env and executing that, but if you can work through the pre-requisites, that may help you discover the problem | 19:24 |
| @jim:acmegating.com | (alternatively, you can write a simple ansible playbook that invokes the role and run that locally) | 19:24 |
| @caiquemello:matrix.org | thx corvus I'm going to do some tests here | 19:30 |
| @jpew:matrix.org | I;m running nodepool-builder in kubernetes, and I get DIB images stuck in the `building` state when the pod restarts while building. Is there a way to clean these up, or will they timeout somehow eventually? | 19:34 |
| @mnaser:matrix.org | jpew: are you running /var/lib/nodepool as a persistent volume? | 19:34 |
| @jpew:matrix.org | mnaser: No | 19:34 |
| @mnaser:matrix.org | i _think_ that might be part of the problem me thinks | 19:35 |
| -@gerrit:opendev.org- Mohammed Naser proposed: | 19:37 | |
| - [zuul/zuul-registry] 835772: tests: Add regression test for manifests https://review.opendev.org/c/zuul/zuul-registry/+/835772 | ||
| - [zuul/zuul-registry] 835773: Fix "docker manifest create" issue https://review.opendev.org/c/zuul/zuul-registry/+/835773 | ||
| @clarkb:matrix.org | hrm our builders are restarted whenever new images a built and they don't leak building images in zookeeper | 19:38 |
| @clarkb:matrix.org | It is possible that they are looking for state t ocleanup like any intermediate artifacts though | 19:38 |
| @jpew:matrix.org | @Clark possibly because I'm using a deployment instead of a statefulset (so the hostname changes each time)? | 19:40 |
| @clarkb:matrix.org | oh yes I think that is it | 19:40 |
| @mnaser:matrix.org | ah yes, that will also be an issue | 19:40 |
| @clarkb:matrix.org | or something along those lines. I think it may use a hash now insead of th ehostname directly and that may be stored on disk | 19:40 |
| @clarkb:matrix.org | its possible one or the other or both is necessary to address it | 19:40 |
| @mnaser:matrix.org | Does anyone know if zuul-registry is an OCI-complaint registry? i'm thinking of trying to upload Helm charts into zuul-registry but wondering if anyone has heard of any experience around that? https://helm.sh/docs/topics/registries/ | 19:41 |
| @clarkb:matrix.org | but basically a builder will only clean up builds it knows belong to it that are not running. Other builders will ignore them | 19:41 |
| @jpew:matrix.org | Ah, makes sense | 19:41 |
| @jpew:matrix.org | So, how do I manually clean up the stuck ones then? | 19:41 |
| @clarkb:matrix.org | jpew: you can use zk-shell to delete the entres in the zk db tree | 19:41 |
| @jpew:matrix.org | Clark: Just to check before I break anything, does `delete /nodepool/images/ubuntu-bionic/builds/0000000001` look correct? | 19:57 |
| @clarkb:matrix.org | jpew: yes I think so. If you get the contents of that there may be some info in json about paths to stuff | 20:01 |
| @clarkb:matrix.org | Though it may just construct that from the image name and build number that is encoded in the zk path | 20:01 |
| @jpew:matrix.org | Ya, there's no files or other state to clean up (the pods are gone), it's just the zk stuff that's left over | 20:02 |
| @clarkb:matrix.org | Oh but the state of the build should be recorded | 20:02 |
| @jpew:matrix.org | Yep, I can see that | 20:02 |
| @clarkb:matrix.org | But ya I think if you get that entry it should say building in a json blob which is good indication it isn't a completed build | 20:02 |
| -@gerrit:opendev.org- Mohammed Naser proposed: [zuul/zuul-registry] 836885: Add support for Helm charts https://review.opendev.org/c/zuul/zuul-registry/+/836885 | 20:48 | |
| @mnaser:matrix.org | turns out it mostly works (tested locally) other than a small OCI-manifest related thing :) | 20:49 |
| @mnaser:matrix.org | ps Clark i took care of making this more dynamic here -- https://review.opendev.org/c/zuul/zuul-registry/+/835772/6 | 20:50 |
| @clarkb:matrix.org | ah needed a .stdout update. +2 again | 20:52 |
| @mnaser:matrix.org | yes I made a little oopsie :) | 20:54 |
| @andyladjadj:matrix.org | Clark: | 20:58 |
| -@gerrit:opendev.org- Mohammed Naser proposed: [zuul/zuul-registry] 836885: Add support for Helm charts https://review.opendev.org/c/zuul/zuul-registry/+/836885 | 21:00 | |
| -@gerrit:opendev.org- Mohammed Naser proposed: [zuul/zuul-registry] 836885: Add support for Helm charts https://review.opendev.org/c/zuul/zuul-registry/+/836885 | 21:19 | |
| @mnaser:matrix.org | I'd appreciate feedback on that Helm chart support patch, "localhost" == "not ssl" for some reason according to Helm | 21:31 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!