Sunday, 2023-12-17

eanderssonThanks for fixing this and sorry that y'all had to work over the weekend!00:45
tonybeandersson: all good.01:36
Clark[m] this is the issue02:40
Clark[m]Or wait we don't do http though. I suspect the problems are related though 02:40
tonybwe can also spin up a test node next week to play with 2.9.102:43
fricklerinfra-root: it seems a number of jobs in the periodic-weekly pipeline were being run multiple times yesterday (hourly even?) even when finishing successfully, i.e. even after the haproxy issue should have been solved08:46
fricklerfor noticed this when there were multiple updates for but see also e.g.
fricklerthis may be an issue if people rely on actually jobs running not more than once a week in that pipeline08:47
fricklercorvus: ^^ this commit in zuul may be related
fricklerthis also seems to affect the periodic pipeline, like this job running something like hourly now, which may put significant increased load onto our infra
frickleroh, grafana seems to confirm that suspicion, showing we're running continously at full load with periodic + periodic-stable retriggering every hour
frickleryes, parsing of the hour and dow crontab entries seems to be broken
fricklerregarding haproxy, I would think matches our issue more. or did we actually not see the issue with 2.9.0?09:14
frickleras I've threatened earlier I've added a couple of topics I mentioned earlier to the meeting agenda, including some thoughts regarding yesterday's issues. feel free to amend/reorder/prune/handle outside of meeting10:44
fungipip 23.3.2 was released earlier today:

Generated by 2.17.3 by Marius Gedminas - find it at!