| rpittau | fungi: saw your message only now, I see there's a new version of the package available 2024.02-2ubuntu0.7 I will give that a try to see if it fixes the issue | 07:33 |
|---|---|---|
| rpittau | same for jammy 2022.02-3ubuntu0.22.04.5 | 07:36 |
| *** haleyb|out is now known as haleyb | 15:54 | |
| sean-k-mooney | fungi: frickler so going back to a released verion did no help but what did was removing project form the tenant list | 18:44 |
| sean-k-mooney | https://github.com/SeanMooney/zuul-homelab/commit/cb59b031b0bd74cf5f3ddb4d32b087175a389650 | 18:44 |
| sean-k-mooney | specificly i belive it was openstack/sunbeam-charms | 18:44 |
| sean-k-mooney | i use - include: [] | 18:44 |
| sean-k-mooney | to well include no zuul config form those repo | 18:45 |
| sean-k-mooney | but with openstack/sunbeam-charms in the list | 18:45 |
| sean-k-mooney | zuul was trying call github to get info for changes for a repo that is not in the tenant and it was hitting gitbub ratelimites | 18:46 |
| sean-k-mooney | so i think there is a bug with haveing repos form diffent connections escpially when you are not including anything from the repo other then making the souce aviable | 18:47 |
| sean-k-mooney | the net result was somehwo this was causeing zuul to not properly process events | 18:48 |
| sean-k-mooney | it apprently took about 9 days for me to run out of api quota on github which is hwy this initaly appread to not be a problem | 18:49 |
| clarkb | sean-k-mooney: the project was in the tenant list which means zuul needs to cache the repo and handle changes from it if depends-on is used etc | 18:49 |
| clarkb | that is separate of whwether or not you load config from it | 18:49 |
| sean-k-mooney | the project it was trying to download was not in the teanant config | 18:50 |
| sean-k-mooney | openstack/sunbeam-charms was not the problem | 18:50 |
| clarkb | it is possible that there is an inefficiency in how zuul handles the inability ot cache a repo (like maybe that can be non fatal if no config is loaded?) | 18:50 |
| sean-k-mooney | the problem was canoncia/<somehting>-charms | 18:50 |
| clarkb | where does that come from? | 18:50 |
| sean-k-mooney | no idea it was not in my config at all | 18:50 |
| sean-k-mooney | i assume the sunbeam repo because that was the only charm related repo | 18:51 |
| clarkb | re :latest we deploy that in opendev | 18:53 |
| clarkb | it works quite well and when it breaks we tend to fix it quickly. That said we use a subset of features in opendev | 18:53 |
| sean-k-mooney | ya i didn have any issue with using latest before and this does not seam to be realted to that persay | 18:53 |
| sean-k-mooney | even with a release build one of the repos i removed seam to be causign zuul to try an load a git repo that was not in my tenant config | 18:54 |
| sean-k-mooney | i was just removing it to make the debugging eiser and say that it fixed it | 18:55 |
| clarkb | I wonder if the issue is having some depends-on in changes in gerrit to github | 18:55 |
| sean-k-mooney | that is definelty possible | 18:56 |
| clarkb | basically event comes in for say openstack/sunbeam-charms new change/patchset with a depends on to github | 18:56 |
| sean-k-mooney | i dont know if i still have logs bug i can check | 18:56 |
| clarkb | zuul then needs to prepare the git state for those jobs and will go and query things I think. | 18:56 |
| sean-k-mooney | right but its ment to check if they are in teh teant config before doing that | 18:56 |
| clarkb | I'm not positive of that | 18:56 |
| clarkb | I think for the basic "is this thing merged" it may check directly | 18:57 |
| clarkb | but also that shouldn't generaet a ton of api requests if it does work that way | 18:57 |
| sean-k-mooney | i woudl expect ti to alwasy filter to be honest for security reasons | 18:59 |
| sean-k-mooney | i.e. if i dont list a repo in the github conneciton it shoudl never be used by zuul IMO | 18:59 |
| sean-k-mooney | in my case it took down the ci once the ratelimit on github was reached. | 19:00 |
| clarkb | yes I think there is an argument down that path but I am not positive it works that way today. I seem to recall depends-on being a ltitle special. If logs imply this was the source you could bring it up with the zuul room | 19:00 |
| sean-k-mooney | ack | 19:00 |
| clarkb | but also the anonymous rate limit for github is like 5k requests per hour or something so wouldn't expect that to hit rate limits either | 19:00 |
| sean-k-mooney | well i dont need ot list all the opendev repo for my usecase | 19:00 |
| sean-k-mooney | clarkb: it may have been this https://review.opendev.org/c/openstack/sunbeam-charms/+/964390 | 19:08 |
| sean-k-mooney | it has a depend on againg a github repo that is not in the tenat config | 19:09 |
| sean-k-mooney | https://github.com/openstack/project-config/blob/master/zuul/main.yaml | 19:09 |
| clarkb | ya that is my best guess at what might trigger the problem but I'm still surprised that github rate limis would be hit by a depends on | 19:10 |
| clarkb | maybe the rate limits are much lower than I think they are | 19:11 |
| sean-k-mooney | there pretty high but i think this kept gettin evaulatied or soemthing like htat | 19:11 |
| sean-k-mooney | i ended up wiht almost 2000 pending eveent before i removed the proejct and prunged the db/zookeeper state | 19:12 |
| sean-k-mooney | looking at https://zuul.opendev.org/t/openstack/build/1518705a15564ddfab86a14dfcabd5f2/log/zuul-info/inventory.yaml#340 | 19:13 |
| sean-k-mooney | zuul is no including tha tin the proejct that its preparing | 19:14 |
| sean-k-mooney | but like you said it might be just checking if its merged | 19:14 |
| *** agalica_ is now known as agalica | 22:43 | |
Generated by irclog2html.py 4.0.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!