| mnasiadka | tonyb, frickler: willing to +w https://review.opendev.org/c/opendev/zuul-providers/+/966200 ? It seems osuosl got better | 06:44 |
|---|---|---|
| tonyb | mnasiadka: I'm fine with the change, but I don't have the cycles this evening to monitor it for any breakage. | 06:55 |
| tonyb | mnasiadka: Not that I expect any | 06:55 |
| frickler | done, I should be around if something goes wrong | 07:26 |
| mnasiadka | thanks | 08:26 |
| mnasiadka | If it goes wrong, it will just not gate ;-) | 08:26 |
| opendevreview | Michal Nasiadka proposed opendev/zuul-providers master: Limit arm64 image builds to producing raw images https://review.opendev.org/c/opendev/zuul-providers/+/968029 | 10:27 |
| mnasiadka | Post failures again - I guess Clark[m]’s approach at 968029 should merge first | 10:27 |
| opendevreview | Michal Nasiadka proposed opendev/zuul-providers master: Limit arm64 image builds to producing raw images https://review.opendev.org/c/opendev/zuul-providers/+/968029 | 10:34 |
| frickler | rebased off the parent with W-1, I don't think that relation was needed? | 10:35 |
| tonyb | frickler: I agree | 10:35 |
| Clark[m] | Note we increased the timeout rather than force them all to raw since it should be a one time issue with the first build | 14:17 |
| Clark[m] | This was why I didn't remove my wip vote | 14:20 |
| *** mrunge_ is now known as mrunge | 15:26 | |
| mnasiadka | Clark[m]: well, I think the problem is that all arm64 jobs run at the same time in a patch and we’re hammering osuosl storage? | 15:30 |
| opendevreview | Michal Nasiadka proposed opendev/zuul-providers master: arm64: Introduce semaphore for arm64 build jobs https://review.opendev.org/c/opendev/zuul-providers/+/968724 | 15:38 |
| mnasiadka | Clark[m]: because raising the timeout to 3h doesn’t help - I’ll see if ^^ makes it better | 15:39 |
| fungi | mnasiadka: though the other half of the problem is that we seem to keep getting node_failure results there even after i increased boot-timeout to 20 minutes (at least as of yesterday) | 16:13 |
| fungi | though i guess i should rebase 968386 on top of 966200 since it the last recheck only failed with timeouts | 16:15 |
| opendevreview | Jeremy Stanley proposed opendev/zuul-providers master: Instrument dstat in image build jobs https://review.opendev.org/c/opendev/zuul-providers/+/968386 | 16:16 |
| corvus | if we need to slow things down more, we could make an arm64 image build semaphore with, say 50% of our quota, so we never use more than half the cloud for image builds. or something like that. | 16:28 |
| corvus | hopefully more data collection will help us decide whether that's necessary, and if so, how to tune it. | 16:29 |
| opendevreview | Michal Nasiadka proposed opendev/zuul-providers master: arm64: Introduce semaphore for arm64 build jobs https://review.opendev.org/c/opendev/zuul-providers/+/968724 | 16:31 |
| mnasiadka | corvus: I added semaphore in ^^, but it seems it’s only one job at a time - any idea what I did wrong? | 16:57 |
| corvus | mnasiadka: i think the semaphore addition must merge first; without that i think it's using an implicitly created semaphore with a max of 1. so just make a standalone change to add the semaphore without using it. we can merge that asap, then you can play around with using it. | 17:09 |
| corvus | (to be clear: i think it's okay to merge the semaphore creation right now, and i think it's okay to propose changes to use it to experiment, but i wouldn't want to merge the changes to use it until next week when everyone's back and we've thought/discussed it more) | 17:10 |
| opendevreview | Michal Nasiadka proposed opendev/zuul-providers master: arm64: Introduce semaphore for arm64 build jobs https://review.opendev.org/c/opendev/zuul-providers/+/968724 | 18:07 |
| opendevreview | Michal Nasiadka proposed opendev/zuul-providers master: arm64: Introduce semaphore for arm64 build jobs https://review.opendev.org/c/opendev/zuul-providers/+/968724 | 18:07 |
| mnasiadka | corvus: ^^ that only adds a semaphore with max: 5 | 18:13 |
| opendevreview | Merged opendev/zuul-providers master: arm64: Introduce semaphore for arm64 build jobs https://review.opendev.org/c/opendev/zuul-providers/+/968724 | 18:15 |
| corvus | mnasiadka: i single-core approved that to keep things moving -- we can decide if we really want to use it next week | 18:15 |
| mnasiadka | corvus: thanks | 18:16 |
| opendevreview | Michal Nasiadka proposed opendev/zuul-providers master: Use opendev-build-arm64 semaphore https://review.opendev.org/c/opendev/zuul-providers/+/968737 | 18:19 |
Generated by irclog2html.py 4.0.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!