| eandersson | Thanks for fixing this and sorry that y'all had to work over the weekend! | 00:45 |
|---|---|---|
| tonyb | eandersson: all good. | 01:36 |
| Clark[m] | https://github.com/haproxy/haproxy/issues/2393 this is the issue | 02:40 |
| Clark[m] | Or wait we don't do http though. I suspect the problems are related though | 02:40 |
| tonyb | we can also spin up a test node next week to play with 2.9.1 | 02:43 |
| Clark[m] | ++ | 02:51 |
| frickler | infra-root: it seems a number of jobs in the periodic-weekly pipeline were being run multiple times yesterday (hourly even?) even when finishing successfully, i.e. even after the haproxy issue should have been solved | 08:46 |
| frickler | for noticed this when there were multiple updates for https://review.opendev.org/c/openstack/requirements/+/903802 but see also e.g. https://zuul.opendev.org/t/openstack/builds?job_name=kolla-publish-ubuntu-dockerhub&project=openstack%2Fkolla&branch=master&skip=0 | 08:46 |
| frickler | this may be an issue if people rely on actually jobs running not more than once a week in that pipeline | 08:47 |
| frickler | corvus: ^^ this commit in zuul may be related https://review.opendev.org/c/zuul/zuul/+/894839 | 08:51 |
| frickler | this also seems to affect the periodic pipeline, like this job running something like hourly now, which may put significant increased load onto our infra https://zuul.opendev.org/t/openstack/builds?job_name=octavia-amphora-image-build&project=openstack%2Foctavia&branch=stable%2Ftrain&skip=0 | 08:56 |
| frickler | oh, grafana seems to confirm that suspicion, showing we're running continously at full load with periodic + periodic-stable retriggering every hour https://grafana.opendev.org/d/21a6e53ea4/zuul-status?orgId=1&from=now-2d&to=now | 08:59 |
| frickler | yes, parsing of the hour and dow crontab entries seems to be broken https://review.opendev.org/c/zuul/zuul/+/894839/comment/ed3ba086_9d8c0413/ | 09:09 |
| frickler | regarding haproxy, I would think https://github.com/haproxy/haproxy/issues/2393 matches our issue more. or did we actually not see the issue with 2.9.0? | 09:14 |
| frickler | as I've threatened earlier I've added a couple of topics I mentioned earlier to the meeting agenda, including some thoughts regarding yesterday's issues. feel free to amend/reorder/prune/handle outside of meeting | 10:44 |
| fungi | pip 23.3.2 was released earlier today: https://pip.pypa.io/en/stable/news/#v23-3-2 | 15:44 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!