| *** jamesmcarthur has quit IRC | 00:14 | |
| *** cdearborn has quit IRC | 01:06 | |
| *** swest has quit IRC | 01:55 | |
| erbarr | soo, if I added a new job and I see it on the queue but it just stays as queued and gets skipped, what am I missing? | 02:05 |
|---|---|---|
| *** rlandy has quit IRC | 02:08 | |
| clarkb | erbarr: queued means waiting for a test node iirc | 02:09 |
| *** swest has joined #zuul | 02:09 | |
| clarkb | you might want to check nodepool logs for that label type? | 02:09 |
| *** rlandy has joined #zuul | 02:10 | |
| erbarr | clarkb: ok, i'll check on that, i'm pretty sure they're all the same label | 02:11 |
| *** rlandy has quit IRC | 02:12 | |
| *** bhavikdbavishi has joined #zuul | 02:43 | |
| *** bhavikdbavishi1 has joined #zuul | 02:45 | |
| *** bhavikdbavishi has quit IRC | 02:47 | |
| *** bhavikdbavishi1 is now known as bhavikdbavishi | 02:47 | |
| *** zxiiro has quit IRC | 03:03 | |
| *** jamesmcarthur has joined #zuul | 03:06 | |
| openstackgerrit | Ian Wienand proposed zuul/nodepool master: Test buildx https://review.opendev.org/725456 | 04:28 |
| *** jamesmcarthur has quit IRC | 04:29 | |
| *** evrardjp has quit IRC | 04:35 | |
| *** evrardjp has joined #zuul | 04:36 | |
| *** jamesmcarthur has joined #zuul | 04:47 | |
| *** sanjayu_ has joined #zuul | 04:48 | |
| *** jamesmcarthur has quit IRC | 04:52 | |
| *** sgw has quit IRC | 04:54 | |
| *** jamesmcarthur has joined #zuul | 04:57 | |
| *** jamesmcarthur has quit IRC | 05:25 | |
| *** jamesmcarthur has joined #zuul | 05:25 | |
| *** jamesmcarthur has quit IRC | 05:30 | |
| *** ysandeep|away is now known as ysandeep | 05:49 | |
| *** dpawlik has joined #zuul | 05:59 | |
| openstackgerrit | Merged zuul/zuul-jobs master: Fix fetch-sphinx-tarball fails https://review.opendev.org/725210 | 06:05 |
| *** jamesmcarthur has joined #zuul | 06:05 | |
| *** jamesmcarthur has quit IRC | 06:15 | |
| openstackgerrit | Andreas Jaeger proposed zuul/zuul-jobs master: DNM: TEst sphinx failures 1 https://review.opendev.org/725472 | 06:33 |
| openstackgerrit | Andreas Jaeger proposed zuul/zuul-jobs master: DNM: Test sphinx failure 2 https://review.opendev.org/725473 | 06:33 |
| *** bhavikdbavishi has quit IRC | 06:42 | |
| *** jamesmcarthur has joined #zuul | 06:48 | |
| openstackgerrit | Andreas Jaeger proposed zuul/zuul-jobs master: DNM: Test sphinx 3 https://review.opendev.org/725479 | 06:51 |
| *** jamesmcarthur has quit IRC | 06:52 | |
| *** sanjayu_ has quit IRC | 07:09 | |
| *** lseki has quit IRC | 07:09 | |
| *** ianw has quit IRC | 07:09 | |
| *** toabctl has quit IRC | 07:09 | |
| *** iurygregory has quit IRC | 07:09 | |
| *** pots has quit IRC | 07:09 | |
| *** reiterative has quit IRC | 07:09 | |
| *** threestrands has quit IRC | 07:09 | |
| *** tobias-urdin has quit IRC | 07:09 | |
| *** noonedeadpunk has quit IRC | 07:09 | |
| *** msuszko has quit IRC | 07:09 | |
| *** AJaeger has quit IRC | 07:09 | |
| *** PrinzElvis has quit IRC | 07:10 | |
| *** tflink has quit IRC | 07:10 | |
| *** zenkuro has quit IRC | 07:10 | |
| *** logan- has quit IRC | 07:10 | |
| *** johanssone has quit IRC | 07:10 | |
| *** ianychoi_ has quit IRC | 07:10 | |
| *** sugaar has quit IRC | 07:10 | |
| *** Tahvok has quit IRC | 07:10 | |
| *** jlk has quit IRC | 07:10 | |
| *** yoctozepto has quit IRC | 07:10 | |
| *** nhicher has quit IRC | 07:10 | |
| *** dpawlik has quit IRC | 07:10 | |
| *** panda|pto has quit IRC | 07:10 | |
| *** ChanServ has quit IRC | 07:10 | |
| *** mnaser has quit IRC | 07:10 | |
| *** evgenyl has quit IRC | 07:10 | |
| *** stevthedev has quit IRC | 07:10 | |
| *** mugsie has quit IRC | 07:10 | |
| *** gundalow has quit IRC | 07:10 | |
| *** mnasiadka has quit IRC | 07:10 | |
| *** adam_g has quit IRC | 07:10 | |
| *** evrardjp has quit IRC | 07:10 | |
| *** jtanner has quit IRC | 07:10 | |
| *** andreaf has quit IRC | 07:10 | |
| *** fbo|off has quit IRC | 07:10 | |
| *** ttx has quit IRC | 07:10 | |
| *** tdasilva has quit IRC | 07:10 | |
| *** donnyd has quit IRC | 07:10 | |
| *** rfolco has quit IRC | 07:10 | |
| *** jkt has quit IRC | 07:10 | |
| *** cloudnull has quit IRC | 07:10 | |
| *** sshnaidm|afk has quit IRC | 07:10 | |
| *** dmsimard has quit IRC | 07:10 | |
| *** fdegir has quit IRC | 07:10 | |
| *** jhesketh has quit IRC | 07:10 | |
| *** corvus has quit IRC | 07:10 | |
| *** pleia2 has quit IRC | 07:10 | |
| *** tristanC has quit IRC | 07:10 | |
| *** irclogbot_0 has quit IRC | 07:10 | |
| *** EmilienM has quit IRC | 07:10 | |
| *** Miouge has quit IRC | 07:10 | |
| *** arxcruz has quit IRC | 07:10 | |
| *** SpamapS has quit IRC | 07:10 | |
| *** decimuscorvinus has quit IRC | 07:10 | |
| *** masterpe has quit IRC | 07:10 | |
| *** Open10K8S has quit IRC | 07:10 | |
| *** kklimonda has quit IRC | 07:10 | |
| *** smyers has quit IRC | 07:10 | |
| *** mwhahaha has quit IRC | 07:10 | |
| *** tributarian has quit IRC | 07:10 | |
| *** persia has quit IRC | 07:10 | |
| *** dcastellani has quit IRC | 07:10 | |
| *** samccann has quit IRC | 07:10 | |
| *** bstinson has quit IRC | 07:10 | |
| *** mgagne has quit IRC | 07:10 | |
| *** openstackgerrit has quit IRC | 07:10 | |
| *** mgoddard has quit IRC | 07:10 | |
| *** frickler has quit IRC | 07:10 | |
| *** bolg has quit IRC | 07:10 | |
| *** ysandeep has quit IRC | 07:10 | |
| *** swest has quit IRC | 07:10 | |
| *** zbr has quit IRC | 07:10 | |
| *** mordred has quit IRC | 07:10 | |
| *** maxamillion has quit IRC | 07:10 | |
| *** vblando has quit IRC | 07:10 | |
| *** rpittau|afk has quit IRC | 07:10 | |
| *** jpena|off has quit IRC | 07:10 | |
| *** ironfoot has quit IRC | 07:10 | |
| *** aspiers has quit IRC | 07:10 | |
| *** erbarr has quit IRC | 07:10 | |
| *** johnsom has quit IRC | 07:10 | |
| *** SotK has quit IRC | 07:10 | |
| *** iamweswilson has quit IRC | 07:10 | |
| *** webknjaz has quit IRC | 07:10 | |
| *** ChrisShort has quit IRC | 07:10 | |
| *** ofosos has quit IRC | 07:10 | |
| *** gmann has quit IRC | 07:10 | |
| *** fungi has quit IRC | 07:10 | |
| *** dmellado has quit IRC | 07:10 | |
| *** shanemcd has quit IRC | 07:10 | |
| *** etp has quit IRC | 07:10 | |
| *** clarkb has quit IRC | 07:10 | |
| *** aluria has quit IRC | 07:10 | |
| *** calebb has quit IRC | 07:10 | |
| *** portdirect has quit IRC | 07:10 | |
| *** Pilou has quit IRC | 07:10 | |
| *** jbryce has quit IRC | 07:10 | |
| *** tobberydberg has quit IRC | 07:10 | |
| *** marvs has quit IRC | 07:10 | |
| *** andreykurilin has quit IRC | 07:10 | |
| *** amotoki has quit IRC | 07:10 | |
| *** freefood has quit IRC | 07:10 | |
| *** mmedvede has quit IRC | 07:10 | |
| *** gothicmindfood has quit IRC | 07:10 | |
| *** jcapitao has joined #zuul | 07:16 | |
| *** avass has joined #zuul | 07:16 | |
| *** dpawlik has joined #zuul | 07:16 | |
| *** sanjayu_ has joined #zuul | 07:16 | |
| *** evrardjp has joined #zuul | 07:16 | |
| *** swest has joined #zuul | 07:16 | |
| *** rfolco has joined #zuul | 07:16 | |
| *** threestrands has joined #zuul | 07:16 | |
| *** lseki has joined #zuul | 07:16 | |
| *** zenkuro has joined #zuul | 07:16 | |
| *** panda|pto has joined #zuul | 07:16 | |
| *** gundalow has joined #zuul | 07:16 | |
| *** dmellado has joined #zuul | 07:16 | |
| *** jkt has joined #zuul | 07:16 | |
| *** ianw has joined #zuul | 07:16 | |
| *** irclogbot_0 has joined #zuul | 07:16 | |
| *** iurygregory has joined #zuul | 07:16 | |
| *** toabctl has joined #zuul | 07:16 | |
| *** zbr has joined #zuul | 07:16 | |
| *** pots has joined #zuul | 07:16 | |
| *** openstackgerrit has joined #zuul | 07:16 | |
| *** johanssone has joined #zuul | 07:16 | |
| *** cloudnull has joined #zuul | 07:16 | |
| *** sugaar has joined #zuul | 07:16 | |
| *** masterpe has joined #zuul | 07:16 | |
| *** PrinzElvis has joined #zuul | 07:16 | |
| *** reiterative has joined #zuul | 07:16 | |
| *** yoctozepto has joined #zuul | 07:16 | |
| *** tflink has joined #zuul | 07:16 | |
| *** ianychoi_ has joined #zuul | 07:16 | |
| *** Tahvok has joined #zuul | 07:16 | |
| *** jlk has joined #zuul | 07:16 | |
| *** ChanServ has joined #zuul | 07:16 | |
| *** AJaeger has joined #zuul | 07:16 | |
| *** msuszko has joined #zuul | 07:16 | |
| *** noonedeadpunk has joined #zuul | 07:16 | |
| *** tobias-urdin has joined #zuul | 07:16 | |
| *** logan- has joined #zuul | 07:16 | |
| *** mgagne has joined #zuul | 07:16 | |
| *** sshnaidm|afk has joined #zuul | 07:16 | |
| *** mordred has joined #zuul | 07:16 | |
| *** mnaser has joined #zuul | 07:16 | |
| *** maxamillion has joined #zuul | 07:16 | |
| *** vblando has joined #zuul | 07:16 | |
| *** tepper.freenode.net sets mode: +o ChanServ | 07:16 | |
| *** Open10K8S has joined #zuul | 07:16 | |
| *** kklimonda has joined #zuul | 07:16 | |
| *** dmsimard has joined #zuul | 07:16 | |
| *** jtanner has joined #zuul | 07:16 | |
| *** portdirect has joined #zuul | 07:16 | |
| *** fdegir has joined #zuul | 07:16 | |
| *** jhesketh has joined #zuul | 07:16 | |
| *** mugsie has joined #zuul | 07:16 | |
| *** stevthedev has joined #zuul | 07:16 | |
| *** evgenyl has joined #zuul | 07:16 | |
| *** andreaf has joined #zuul | 07:16 | |
| *** adam_g has joined #zuul | 07:16 | |
| *** mnasiadka has joined #zuul | 07:16 | |
| *** Pilou has joined #zuul | 07:16 | |
| *** ttx has joined #zuul | 07:16 | |
| *** tobberydberg has joined #zuul | 07:16 | |
| *** rpittau|afk has joined #zuul | 07:16 | |
| *** smyers has joined #zuul | 07:16 | |
| *** nhicher has joined #zuul | 07:16 | |
| *** EmilienM has joined #zuul | 07:16 | |
| *** mgoddard has joined #zuul | 07:16 | |
| *** corvus has joined #zuul | 07:16 | |
| *** pleia2 has joined #zuul | 07:16 | |
| *** Miouge has joined #zuul | 07:16 | |
| *** fbo|off has joined #zuul | 07:16 | |
| *** jpena|off has joined #zuul | 07:16 | |
| *** mwhahaha has joined #zuul | 07:16 | |
| *** tributarian has joined #zuul | 07:16 | |
| *** ironfoot has joined #zuul | 07:16 | |
| *** aspiers has joined #zuul | 07:16 | |
| *** frickler has joined #zuul | 07:16 | |
| *** ofosos has joined #zuul | 07:16 | |
| *** bolg has joined #zuul | 07:16 | |
| *** shanemcd has joined #zuul | 07:16 | |
| *** marvs has joined #zuul | 07:16 | |
| *** andreykurilin has joined #zuul | 07:16 | |
| *** donnyd has joined #zuul | 07:16 | |
| *** tdasilva has joined #zuul | 07:16 | |
| *** arxcruz has joined #zuul | 07:16 | |
| *** SpamapS has joined #zuul | 07:16 | |
| *** ysandeep has joined #zuul | 07:16 | |
| *** decimuscorvinus has joined #zuul | 07:16 | |
| *** persia has joined #zuul | 07:16 | |
| *** tristanC has joined #zuul | 07:16 | |
| *** erbarr has joined #zuul | 07:16 | |
| *** freefood has joined #zuul | 07:16 | |
| *** johnsom has joined #zuul | 07:16 | |
| *** calebb has joined #zuul | 07:16 | |
| *** SotK has joined #zuul | 07:16 | |
| *** etp has joined #zuul | 07:16 | |
| *** iamweswilson has joined #zuul | 07:16 | |
| *** webknjaz has joined #zuul | 07:16 | |
| *** dcastellani has joined #zuul | 07:16 | |
| *** samccann has joined #zuul | 07:16 | |
| *** ChrisShort has joined #zuul | 07:16 | |
| *** gmann has joined #zuul | 07:16 | |
| *** jbryce has joined #zuul | 07:16 | |
| *** amotoki has joined #zuul | 07:16 | |
| *** clarkb has joined #zuul | 07:16 | |
| *** mmedvede has joined #zuul | 07:16 | |
| *** gothicmindfood has joined #zuul | 07:16 | |
| *** fungi has joined #zuul | 07:16 | |
| *** bstinson has joined #zuul | 07:16 | |
| *** aluria has joined #zuul | 07:16 | |
| *** yolanda has joined #zuul | 07:29 | |
| *** tosky has joined #zuul | 07:32 | |
| *** rpittau|afk is now known as rpittau | 07:34 | |
| *** bhavikdbavishi has joined #zuul | 07:43 | |
| *** jpena|off is now known as jpena | 07:52 | |
| *** guillaumec has joined #zuul | 07:53 | |
| *** threestrands has quit IRC | 08:01 | |
| *** ysandeep is now known as ysandeep|lunch | 08:17 | |
| zbr | AJaeger: https://review.opendev.org/#/c/725091/ if you can, is the kind of review that can easily need rebase if it gets outdated | 08:19 |
| openstackgerrit | Merged zuul/zuul master: Add a link to the user survey to the end of the quickstart. https://review.opendev.org/725415 | 08:21 |
| *** jamesmcarthur has joined #zuul | 08:26 | |
| *** decimuscorvinus has quit IRC | 08:34 | |
| *** decimuscorvinus has joined #zuul | 08:34 | |
| *** jamesmcarthur has quit IRC | 08:35 | |
| *** bhavikdbavishi has quit IRC | 08:45 | |
| *** bhavikdbavishi has joined #zuul | 08:46 | |
| openstackgerrit | Merged zuul/zuul-jobs master: Enable yamllint https://review.opendev.org/725091 | 08:46 |
| openstackgerrit | Jan Kubovy proposed zuul/zuul master: Connect fingergw to Zookeeper https://review.opendev.org/716875 | 08:55 |
| *** hashar has joined #zuul | 09:08 | |
| zbr | AJaeger: thanks! | 09:09 |
| openstackgerrit | Sorin Sbarnea (zbr) proposed zuul/zuul-jobs master: yamlint: EOF newlines and comments indent https://review.opendev.org/725516 | 09:16 |
| *** sshnaidm|afk is now known as sshnaidm | 09:22 | |
| *** bhavikdbavishi has quit IRC | 10:02 | |
| *** tobias-urdin has quit IRC | 10:30 | |
| *** AJaeger has quit IRC | 10:30 | |
| *** hashar has quit IRC | 10:30 | |
| *** noonedeadpunk has quit IRC | 10:30 | |
| *** AJaeger has joined #zuul | 10:30 | |
| *** noonedeadpunk has joined #zuul | 10:31 | |
| *** jamesmcarthur has joined #zuul | 10:31 | |
| *** ysandeep|lunch is now known as ysandeep | 10:38 | |
| *** jamesmcarthur has quit IRC | 10:46 | |
| *** rpittau is now known as rpittau|bbl | 10:52 | |
| *** jcapitao is now known as jcapitao_lunch | 11:00 | |
| *** armstrongs has joined #zuul | 11:04 | |
| *** jpena is now known as jpena|lunch | 11:40 | |
| *** bhavikdbavishi has joined #zuul | 11:42 | |
| *** hashar has joined #zuul | 11:51 | |
| *** smyers has quit IRC | 12:03 | |
| *** smyers has joined #zuul | 12:04 | |
| *** rpittau|bbl is now known as rpittau | 12:06 | |
| *** bhavikdbavishi has quit IRC | 12:07 | |
| *** jcapitao_lunch is now known as jcapitao | 12:22 | |
| *** rlandy has joined #zuul | 12:31 | |
| openstackgerrit | Sorin Sbarnea (zbr) proposed zuul/zuul-jobs master: Made sequence indent consistent https://review.opendev.org/725538 | 12:33 |
| *** Goneri has joined #zuul | 12:42 | |
| *** jpena|lunch is now known as jpena | 12:43 | |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Support per branch change queues https://review.opendev.org/718531 | 12:44 |
| *** bhavikdbavishi has joined #zuul | 12:46 | |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: WIP: Move queue from pipeline to project https://review.opendev.org/720182 | 12:47 |
| *** ysandeep is now known as ysandeep|brb | 12:47 | |
| AJaeger | mnaser: can we merge https://review.opendev.org/724132 now, please? | 12:50 |
| mnaser | AJaeger: +2 but tbh not feeling great that we cant really test it :\ | 12:51 |
| * mordred is landing the buildx patch - it has 3x +2 | 12:58 | |
| *** sgw has joined #zuul | 13:03 | |
| AJaeger | mnaser: understood - do you have a quick way to test it once it merges? | 13:05 |
| mnaser | AJaeger: short of merging a change that promotes an image (i'll probably do that at some point today in our operator work).. nope | 13:07 |
| AJaeger | mnaser: I see - ok. | 13:09 |
| avass | mnser: yeah I don't like it either | 13:10 |
| avass | mnaser, AJaeger: it would be good if we could come up with a way to do that | 13:11 |
| mordred | ++ | 13:18 |
| openstackgerrit | Merged zuul/zuul-jobs master: Support multi-arch image builds with docker buildx https://review.opendev.org/722339 | 13:22 |
| *** tobiash has joined #zuul | 13:23 | |
| *** ysandeep|brb is now known as ysandeep | 13:24 | |
| tristanC | tobiash: hello, welcome back! when you have a moment, could you please check some operator refactor, starting from https://review.opendev.org/719965 . Otherwise if that's ok with you, i'd like to move forward with those | 13:31 |
| tobiash | tristanC: sure, I'll have a look later today | 13:32 |
| *** bhavikdbavishi has quit IRC | 13:36 | |
| tristanC | mnaser: corvus: mordred: following up on the cert-manager discussion, the zuul-operator can now uses Issuer and Certificate ressources by default and setting the `withCertManager` attribute to false enables backward compatibility using the legacy cert creation tasks. Is that ok for you? (implementation is https://review.opendev.org/718840 , https://review.opendev.org/719110 and | 13:40 |
| tristanC | https://review.opendev.org/719185) | 13:40 |
| *** jamesmcarthur has joined #zuul | 13:59 | |
| *** michael-beaver has joined #zuul | 14:04 | |
| openstackgerrit | Monty Taylor proposed zuul/zuul-jobs master: Write buildkitd.toml in use-buildset-registry https://review.opendev.org/724837 | 14:06 |
| mordred | tobiash: https://review.opendev.org/#/c/725431/ <-- clarkb found that jemalloc with py3.7 and py3.8 seems to be related to our memory leak issue. I believe you pinned to 3.6 for leak issues - thought you might be interested. winning combo for us seems like it might be 3.8 with no jemalloc | 14:14 |
| tobiash | mordred: interesting, it's correct that we pinned to 3.6 due to leak issues | 14:15 |
| tobiash | our zuul-web still has leak issues btw, but we don't care atm due to openshift | 14:15 |
| *** armstrongs has quit IRC | 14:16 | |
| openstackgerrit | Merged zuul/zuul-operator master: Refactor file creation to function files https://review.opendev.org/719965 | 14:17 |
| openstackgerrit | Merged zuul/zuul-operator master: Refactor kubernetes resources constructor to the function file https://review.opendev.org/719991 | 14:18 |
| mordred | tobiash: nod | 14:18 |
| mordred | tobiash: well - we'll let you know how zuul-web does for us with 3.8 | 14:19 |
| tobiash | mordred: cool, thanks | 14:19 |
| openstackgerrit | Monty Taylor proposed zuul/nodepool master: Build multi-arch images for x86 and arm https://review.opendev.org/722483 | 14:23 |
| openstackgerrit | Monty Taylor proposed zuul/nodepool master: Build nodepool with python3.8 https://review.opendev.org/725611 | 14:23 |
| openstackgerrit | Merged zuul/zuul-operator master: Refactor deployment and statefulset functions https://review.opendev.org/720007 | 14:23 |
| openstackgerrit | Merged zuul/zuul-operator master: Refactor backend services https://review.opendev.org/720019 | 14:23 |
| openstackgerrit | Merged zuul/zuul-operator master: Refactor Zuul services https://review.opendev.org/720024 | 14:23 |
| openstackgerrit | Monty Taylor proposed zuul/zuul-registry master: Bump to python 3.8 base images https://review.opendev.org/725612 | 14:24 |
| jkt | what is the status of that merger failure? (false messages "change failed to merge", and/or a job testing something else than supposed to test) | 14:27 |
| jkt | I think it was fixed by https://opendev.org/zuul/zuul/commit/9b300bc8dfc59584e3046337e7c001fc927d670f, but then I saw https://review.opendev.org/#/c/723800/ | 14:28 |
| mordred | corvus, avass, clarkb, tristanC: when you get a sec, https://review.opendev.org/#/c/724837/ is green, and so is the tempfile cleanup: https://review.opendev.org/#/c/725387/ - also, https://review.opendev.org/#/c/725380/ is another buildx cleanup that clarkb noticed | 14:34 |
| openstackgerrit | Merged zuul/zuul-jobs master: Revert "Revert "Do not set buildset_fact if it's not present in results.json"" https://review.opendev.org/724132 | 14:35 |
| *** jamesmcarthur has quit IRC | 14:53 | |
| *** jamesmcarthur_ has joined #zuul | 14:53 | |
| *** zxiiro has joined #zuul | 15:08 | |
| openstackgerrit | Merged zuul/zuul-jobs master: Don't pull and retag in buildx workflow https://review.opendev.org/725380 | 15:14 |
| *** ysandeep is now known as ysandeep|afk | 15:15 | |
| AJaeger | zuul-jobs maintainers, two reviews for buildset-registry, please: https://review.opendev.org/724841 and https://review.opendev.org/724717 | 15:17 |
| AJaeger | and a third one: https://review.opendev.org/724840 | 15:19 |
| AJaeger | mordred: did you see the comment on https://review.opendev.org/#/c/725387/1/roles/build-docker-image/tasks/setup-buildx.yaml | 15:25 |
| clarkb | mordred: tristanC has a good comment on the tempfile change | 15:35 |
| clarkb | mordred: and it applies to the next change in the series | 15:37 |
| *** ysandeep|afk is now known as ysandeep | 15:38 | |
| clarkb | mordred: I left a couple of things on https://review.opendev.org/#/c/724837/6 where I don't feel I know enough about ansible or buildkit to know if those are issues | 15:44 |
| clarkb | butI think they should eb considered and ruled out at least | 15:44 |
| *** panda|pto has quit IRC | 15:46 | |
| *** jamesmcarthur_ has quit IRC | 15:49 | |
| *** ysandeep is now known as ysandeep|away | 16:02 | |
| mordred | clarkb, tristanC : oh - hah. for some reason I thought they'd get cleaned up - but no, we do need to rm them ourselves. I'll update that | 16:05 |
| tobiash | corvus: the repl just saved my ass, we had an endless loop in processQueue of one pipeline and a huge queue of running items and a blocking zuul dequeue command | 16:08 |
| tobiash | the repl server allowed me to dequeue out of band the items of that pipeline :) | 16:08 |
| corvus | tobiash: wow, that's great it was there and terrible there was a bug like that -- do you know the endless loop happened? | 16:09 |
| tobiash | the scenario was one project had a queue in the gate and force merged a config change that removed a project-template with some jobs because they were broken. This lead to a tenant reconfig removing those jobs from the live items. Then the processQueue always returned changed=True despite doing nothing in that pipeline | 16:10 |
| tobiash | I suspect something in processOneItem marked the item as changed but I didn't find anything yet, and there was also no useful logging about why it thought it changed an item | 16:11 |
| AJaeger | speaking about tempfile, do we need cleanup these tempfile usages as well: https://review.opendev.org/#/c/718694/10/roles/ensure-dhall/tasks/main.yaml and https://review.opendev.org/#/c/717507/13/roles/ensure-package-repositories/tasks/RedHat.yaml ? | 16:12 |
| *** rpittau is now known as rpittau|afk | 16:14 | |
| *** michael-beaver has quit IRC | 16:14 | |
| corvus | tristanC: looks great, thanks! | 16:18 |
| openstackgerrit | Monty Taylor proposed zuul/zuul-jobs master: Use tempfile in buildx build https://review.opendev.org/725387 | 16:19 |
| openstackgerrit | Monty Taylor proposed zuul/zuul-jobs master: Write buildkitd.toml in use-buildset-registry https://review.opendev.org/724837 | 16:19 |
| *** sgw has quit IRC | 16:20 | |
| mordred | clarkb, tristanC, corvus: ^^I cleaned up the tempfiles. in the first patch I did not remove the buildkitd.toml tempfile, because it gets bind-mounted in. we get rid of that tempfile in the next patch, so I think it's ok | 16:20 |
| corvus | jkt: i don't think we understand the error surfaced by https://opendev.org/zuul/zuul/commit/9b300bc8dfc59584e3046337e7c001fc927d670f yet to know if https://review.opendev.org/723800 is a good idea -- our best guess is that maybe we need to put a slightly longer delay than "now" on the gc, but i don't think anyone has gotten a reproducer yet to verify | 16:20 |
| clarkb | mordred: re mirror locations if the job was run on a static host and then run again (granted not cleaning up after all that seems like it brings its own set of problems) | 16:22 |
| avass | AJaeger: We probably should but I don't think it's urgent | 16:24 |
| avass | AJaeger: oh, but those actually use the tempfile module? | 16:25 |
| clarkb | mordred: ya looks like tristanC has pointed out that same mirror duplicates issue. I think we should address that since it is simple to do so | 16:26 |
| mordred | clarkb: ok. let's do it in a followup - because we should update the other file that this was copied from too | 16:27 |
| AJaeger | avass: does tempfile module clean up? Checking the docs, it does not. Let me propose a change... | 16:27 |
| clarkb | mordred: ya tristanc has already done that | 16:27 |
| clarkb | mordred: its linked to in the change | 16:27 |
| clarkb | fwiw I think my suggestion is a simpler fix than the one tristanC has proposed in the other python module | 16:28 |
| clarkb | but maybe I don't understand the data structurethere | 16:28 |
| avass | AJaeger: it doesn't clean it up as far as I know but it (tries) to make sure it's put in a place where it can put the file | 16:28 |
| *** panda has joined #zuul | 16:29 | |
| AJaeger | avass: yes - and uses restrictive access. | 16:30 |
| openstackgerrit | Andreas Jaeger proposed zuul/zuul-jobs master: Remove some temporary files https://review.opendev.org/725650 | 16:30 |
| AJaeger | avass, ^ | 16:30 |
| avass | AJaeger: where is the ensure-dhall tempfile created? | 16:31 |
| *** sgw has joined #zuul | 16:31 | |
| AJaeger | avass: top of the file | 16:32 |
| openstackgerrit | Monty Taylor proposed zuul/zuul-jobs master: Use tempfile in buildx build https://review.opendev.org/725387 | 16:33 |
| avass | AJaeger: ah, wait. I feel stupid, I missed that it folded the beginning of the file | 16:33 |
| openstackgerrit | Monty Taylor proposed zuul/zuul-jobs master: Write buildkitd.toml in use-buildset-registry https://review.opendev.org/724837 | 16:33 |
| openstackgerrit | Monty Taylor proposed zuul/zuul-jobs master: use-buildset-registry: fix modify_registries_conf library idempotency https://review.opendev.org/725651 | 16:33 |
| mordred | clarkb, tristanC, corvus : ^^ I rebased tristanC's change into the stack then applied the same fix to the buildkitd change | 16:33 |
| openstackgerrit | Merged zuul/zuul-operator master: Add initial withCertManager input toggle https://review.opendev.org/718840 | 16:35 |
| *** evrardjp has quit IRC | 16:35 | |
| *** evrardjp has joined #zuul | 16:36 | |
| openstackgerrit | Merged zuul/zuul-operator master: Add gearman tls secret provided by cert-manager https://review.opendev.org/719110 | 16:38 |
| *** dpawlik has quit IRC | 16:40 | |
| openstackgerrit | Merged zuul/zuul-operator master: Add registry tls secret provided by cert-manager https://review.opendev.org/719185 | 16:40 |
| openstackgerrit | Merged zuul/zuul-operator master: Add zuul-preview service https://review.opendev.org/718419 | 16:41 |
| *** sgw has quit IRC | 16:47 | |
| *** sgw has joined #zuul | 16:47 | |
| openstackgerrit | Merged zuul/zuul-operator master: test: refactor run tasks to include file https://review.opendev.org/718158 | 16:51 |
| avass | mordred: ++, idempotency is a bit more important if you're not using single use nodes :) | 16:52 |
| AJaeger | mordred, tristanC, did you duplicate https://review.opendev.org/724840 with https://review.opendev.org/725651 ? | 16:53 |
| AJaeger | so, want to abandon 724840? | 16:54 |
| AJaeger | mordred: https://review.opendev.org/#/c/725387/3/roles/build-docker-image/tasks/setup-buildx.yaml does not remove the toml tempfile, or does it? | 16:55 |
| clarkb | mordred: tristanC can you review my comments on https://review.opendev.org/#/c/724837/ and https://review.opendev.org/#/c/725651/1 | 16:57 |
| tristanC | clarkb: done | 16:58 |
| clarkb | tristanC: right but there could be 5 buildset registries one for each different upstream registry, dockerhub, quay, google cloud, etc | 16:59 |
| clarkb | tristanC: you can't assume there is only one | 16:59 |
| clarkb | at least that was my understanding of it | 16:59 |
| openstackgerrit | Monty Taylor proposed zuul/zuul-jobs master: use-buildset-registry: fix modify_registries_conf library idempotency https://review.opendev.org/725651 | 16:59 |
| openstackgerrit | Monty Taylor proposed zuul/zuul-jobs master: Write buildkitd.toml in use-buildset-registry https://review.opendev.org/724837 | 16:59 |
| mordred | clarkb: thanks! good catches | 17:00 |
| tristanC | clarkb: then checking if the location is an element of the mirror list is not enough, we also need to ensure it is before any non buildset registry | 17:00 |
| openstackgerrit | Merged zuul/zuul-operator master: Add support for custom container image name https://review.opendev.org/720762 | 17:00 |
| openstackgerrit | Merged zuul/zuul-operator master: Add support for global image prefix https://review.opendev.org/720780 | 17:00 |
| openstackgerrit | Merged zuul/zuul-operator master: Refactor ZooKeeper service configuration https://review.opendev.org/720822 | 17:00 |
| mordred | clarkb, tristanC: hrm. | 17:01 |
| corvus | clarkb: there's only ever one buildset registry | 17:01 |
| clarkb | corvus: weren't we proxying with it to the backends too? | 17:01 |
| mordred | so maybe checking mirrors[0] is the better thing? | 17:01 |
| tristanC | and if there is only one buildset registry, as it was in my case, then checking for the head was enough | 17:01 |
| mordred | yeah. I think I should revert to checking for [0] | 17:02 |
| corvus | clarkb: i don't think we got around to that yet -- but that wouldn't be another buildset registry, that would be a second mirror, right? | 17:02 |
| mordred | it's possible that, on a static node, perhaps someone later added another registry mirror in front of the buildset registry | 17:02 |
| *** ysandeep|away is now known as ysandeep | 17:02 | |
| clarkb | corvus: maybe? I'm looking at the get_location code and it seems to assume its mapping there | 17:03 |
| mordred | so maybe we should check to see if it's first - and also check to see if it's later in the list and if so remove it from there and move it to the front | 17:03 |
| mordred | so that we don't wind up in a cycle of adding it over and over again fighting some other misbehaving system | 17:03 |
| corvus | clarkb: this is reasonable: quay.io mirrors=['buildset_registry:/quay.io', 'mirror.rax/quay.io'] | 17:03 |
| mordred | yeah | 17:03 |
| clarkb | dict(location=get_location(reg['prefix'], location)) specifically is saying map quay to this thing and google clodu to that thing | 17:03 |
| corvus | i don't think we do that, but that's something we can/should do | 17:03 |
| mordred | and I think, given sequencing, we'd expect the "set mirrors" code to run before the "use-buildset-registry" code | 17:03 |
| mordred | so it would go from [] to ['mirror.rax/quay.io'] to ['buildset_registry:/quay.io', 'mirror.rax/quay.io'] | 17:04 |
| clarkb | corvus: that data structure doesn't seem to match what we've got | 17:04 |
| mordred | clarkb: it's different for buildkitd.toml and registries.conf | 17:04 |
| mordred | because of course it is | 17:04 |
| clarkb | corvus: what we have in the code is mirrors=[ {quay, buildset/quay} ] | 17:05 |
| corvus | clarkb: well, i haven't looked at these changes, i'm just responding to your suggestion that we would have more than one buildset registry | 17:05 |
| clarkb | mordred: ya I'm talking about tristanC's change | 17:05 |
| corvus | clarkb: "corvus: what we have in the code is mirrors=[ {quay, buildset/quay} ]" <-- where do we have that? | 17:06 |
| mordred | clarkb: yeah. that's registries. it's [{'location': 'buildset_registry:/quay.io'}] | 17:06 |
| clarkb | (I'm inferring the datastructure from the code here so could be wrong, but it seems we already support doing multiple different locations in the mirror dict list, but tristanc's change assumes one entry only) | 17:06 |
| mordred | corvus: nowhere | 17:06 |
| mordred | clarkb: the dict structure is mirrors= | 17:06 |
| clarkb | mordred: mirrors is a list | 17:07 |
| mordred | yes. of dicts with the key location | 17:07 |
| mordred | so for registries.conf it's mirrors=[{'location': 'buildset_registry:/quay.io'}, {'location': 'mirror.rax/quay.io'}] and for buildkitd.toml it's mirrors=['buildset_registry:/quay.io', 'mirror.rax/quay.io'] | 17:08 |
| clarkb | mordred: and they are both already pre prefixed? | 17:10 |
| clarkb | (the toml one was pretty clearly already namespaced) | 17:10 |
| corvus | buildkitd.toml and registries.conf are both toml | 17:11 |
| clarkb | ah ok I see we modify the existing data structure in place | 17:11 |
| clarkb | (how did they end up with different data structures using the same serialization format to encode the same data...) | 17:12 |
| mordred | clarkb: right? | 17:13 |
| clarkb | ok I'm up to speed now. I think both versions are correct beacuse we insert at the front. So unless someone changed the order out of band we'd still be enforcing that list order | 17:13 |
| openstackgerrit | Monty Taylor proposed zuul/zuul-jobs master: use-buildset-registry: fix modify_registries_conf library idempotency https://review.opendev.org/725651 | 17:13 |
| openstackgerrit | Monty Taylor proposed zuul/zuul-jobs master: Write buildkitd.toml in use-buildset-registry https://review.opendev.org/724837 | 17:13 |
| clarkb | tristanC's version will override any out of band changes which may be desireable | 17:13 |
| mordred | clarkb: yeah - but I think tristanC's original is more correct- yeah | 17:13 |
| clarkb | since now that you are back into the zuul context we want to enforce zuul's expectations | 17:13 |
| mordred | I think the result of use-buildset-registry is for the buildset registry to be correctly configured | 17:13 |
| mordred | yup | 17:13 |
| tobiash | tristanC: it seems like your operator stack has merged already, is anything left for review? | 17:14 |
| corvus | i think if we want to include mirrors, we should do it by standardizing mirror site variables and using it directly | 17:14 |
| mordred | corvus: ++ | 17:14 |
| tristanC | clarkb: mordred: for it worth i also don't really understand the toml serialized data structure, i just reverse engineered the function to avoid further modification after the first usage | 17:15 |
| tristanC | tobiash: nop, it seems like mordred and corvus approved all the important changes, though i'd be happy to answer any question you might have about the refactor | 17:16 |
| avass | mordred: weren't we supposed to check if the mirror already was in the list and move it if it was, insert it at index 0 if it wasn't? or did i get lost? | 17:20 |
| mordred | avass: I decided to just stick with inserting at 0 for now | 17:21 |
| avass | mordred: sure | 17:21 |
| *** bhavikdbavishi has joined #zuul | 17:21 | |
| openstackgerrit | Monty Taylor proposed zuul/zuul-jobs master: Use tempfile in buildx build https://review.opendev.org/725387 | 17:24 |
| openstackgerrit | Monty Taylor proposed zuul/zuul-jobs master: use-buildset-registry: fix modify_registries_conf library idempotency https://review.opendev.org/725651 | 17:24 |
| openstackgerrit | Monty Taylor proposed zuul/zuul-jobs master: Write buildkitd.toml in use-buildset-registry https://review.opendev.org/724837 | 17:24 |
| *** y2kenny41 has joined #zuul | 17:24 | |
| mordred | corvus, clarkb, avass : ^^ sorry for the churn - the tests caught a sequencing error - yay for tests! | 17:24 |
| *** y2kenny41 has left #zuul | 17:24 | |
| mordred | corvus, clarkb : 725387 is the one that had to get updated | 17:24 |
| *** bhavikdbavishi1 has joined #zuul | 17:24 | |
| *** bhavikdbavishi has quit IRC | 17:26 | |
| *** bhavikdbavishi1 is now known as bhavikdbavishi | 17:26 | |
| jkt | corvus: ok. It was not on "mine" Zuul, so I cannot really debug this one, it was at mnaser's at vexxhost | 17:31 |
| * jkt is on a release, not on git | 17:31 | |
| mnaser | the one thing to note on that, is that jkt doesnt use gating but submits using gerrit themselves | 17:31 |
| mnaser | so i've been pondering if that has been a potential root cause | 17:32 |
| avass | mnaser, jkt: I think we've experienced problems like that | 17:32 |
| jkt | that's right, we're only using check | 17:32 |
| *** jcapitao has quit IRC | 17:33 | |
| avass | I don't remember exactly what the problem was, but I do remember having some kind of problem when submitting manually | 17:37 |
| *** jpena is now known as jpena|off | 17:44 | |
| *** iurygregory has quit IRC | 17:49 | |
| openstackgerrit | Andreas Jaeger proposed zuul/zuul-jobs master: Remove some temporary files https://review.opendev.org/725650 | 17:51 |
| AJaeger | mordred: could you check https://review.opendev.org/725298, please? | 17:55 |
| *** ysandeep is now known as ysandeep|away | 17:57 | |
| openstackgerrit | Monty Taylor proposed zuul/zuul-registry master: Run unittests on python 3.8 https://review.opendev.org/725662 | 17:59 |
| *** yolanda has quit IRC | 17:59 | |
| openstackgerrit | Andreas Jaeger proposed zuul/zuul-jobs master: Add py38 testing https://review.opendev.org/725663 | 18:00 |
| openstackgerrit | Andreas Jaeger proposed zuul/zuul-jobs master: Add py38 testing https://review.opendev.org/725663 | 18:03 |
| corvus | clarkb: fyi i spent yesterday afternoon working on a test case for the loading_errors bug; haven't quite gotten there yet. | 18:16 |
| clarkb | corvus: thanks! | 18:17 |
| openstackgerrit | Merged zuul/zuul-jobs master: Use tempfile in buildx build https://review.opendev.org/725387 | 18:22 |
| openstackgerrit | Andreas Jaeger proposed zuul/zuul-jobs master: Fix bare 'item' in build-container-image https://review.opendev.org/725298 | 18:22 |
| AJaeger | avass, corvus, mordred, I think that is the right fix ^ | 18:23 |
| AJaeger | avass: sorry for taking over - that was quicker than explaining. You found a genuine bug - thanks | 18:23 |
| AJaeger | mordred: thanks for your question mark when reviewing - that let me triple check ;) | 18:24 |
| mordred | AJaeger: ooh neat. we should maybe put a loop_control: loop_var on that include | 18:26 |
| AJaeger | mordred: yes, in https://review.opendev.org/#/c/724910/22/roles/build-container-image/tasks/main.yaml | 18:27 |
| AJaeger | (which needs updating due to merge conflict) | 18:27 |
| *** y2kenny has joined #zuul | 18:27 | |
| *** hashar has quit IRC | 18:27 | |
| mordred | AJaeger: yup - so I think we should rebase one on the other - because the loop var isn't going to be right when those two merge | 18:27 |
| avass | mordred, AJaeger: I'll update that :) | 18:27 |
| avass | AJaeger: so the set_fact was in the wrong task file? | 18:28 |
| AJaeger | avass: yes, read roles/build-container-image/common.rst on the semantics | 18:28 |
| AJaeger | avass: if you want to +2A https://review.opendev.org/#/c/725298/3, you can rebase 724910 on top of it | 18:30 |
| mordred | ++ | 18:30 |
| AJaeger | https://zuul.opendev.org/t/zuul/build/8b65dd48638446399e4f0d7bb6d933ca hsa "Error: Error copying image to the remote destination: Error writing blob: Error uploading layer to https://zuul-jobs.buildset-registry:5000/v2/downstream/image/blobs/uploads/27436eb77a2447b4a05b14c2790b986a?digest=sha256%3A1dbcab28ce46b65c0174e5e82658492107396fead31e9144c343e6bc96e471c7: received unexpected HTTP status: 500 | 18:31 |
| AJaeger | Internal Server Error" in the podman job | 18:31 |
| AJaeger | same error in https://zuul.opendev.org/t/zuul/build/f9862a50edf44b64ba891ac94babe143 | 18:32 |
| *** bhavikdbavishi has quit IRC | 18:32 | |
| *** jamesmcarthur has joined #zuul | 18:33 | |
| *** bhavikdbavishi has joined #zuul | 18:34 | |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Fix bare 'item' in build-container-image https://review.opendev.org/725298 | 18:34 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: ansible-lint: use matchplay instead of matchtask https://review.opendev.org/724910 | 18:34 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: use zj_image instead of image as loopvar https://review.opendev.org/725012 | 18:34 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: use zj_log_file instead of item as loop_var https://review.opendev.org/725013 | 18:34 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Check blocks recursively for loops https://review.opendev.org/724967 | 18:34 |
| avass | oh, oops, rebased the first one as well | 18:34 |
| avass | ah, a rebase doesn't reset the votes, good :) | 18:34 |
| AJaeger | ok, I'll add my +3 to it ;) | 18:35 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: ansible-lint: use matchplay instead of matchtask https://review.opendev.org/724910 | 18:37 |
| fungi | avass: yeah, a rebase with no substantive changes shouldn't reset votes, but if you alter the commit message or have to solve a merge conflict then it clears them | 18:39 |
| fungi | code review votes that is | 18:39 |
| fungi | verify votes still get cleared the way we've got gerrit configured | 18:39 |
| AJaeger | avass: you missed to change the item that I moved, see my inline comment | 18:40 |
| avass | AJaeger: yeah working on that ;) | 18:40 |
| AJaeger | no stress ;) | 18:41 |
| avass | some tests like eating up all my ram so my laptop freezes so I usually just push to get the testresults instead | 18:41 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: ansible-lint: use matchplay instead of matchtask https://review.opendev.org/724910 | 18:42 |
| avass | especially flake8 in zuul for some reason | 18:43 |
| y2kenny | I understand that if I push a patch series, jobs will get triggered on all of them speculatively. Is there a way to trigger jobs on a specific range of commits outside of this use case? The use case I am thinking of is git piset (https://hoelz.ro/blog/git-pisect) but there could be other use cases where I want to leverage the CI infrastructure to | 18:43 |
| y2kenny | find some optimum. | 18:43 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: use zj_image instead of image as loopvar https://review.opendev.org/725012 | 18:43 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: use zj_log_file instead of item as loop_var https://review.opendev.org/725013 | 18:43 |
| AJaeger | avass: thanks, LGTM | 18:43 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Check blocks recursively for loops https://review.opendev.org/724967 | 18:43 |
| fungi | y2kenny: you can set a queue "window" size in a dependent pipeline | 18:45 |
| AJaeger | mordred: do you have time to review 724910 ? | 18:46 |
| y2kenny | fungi: how would I specify the range? or is this a case where using the cli enqueue would be better? | 18:47 |
| avass | AJaeger: I figured out what that first field of the return value from matchplay does by the way. It's the message shown when you don't set parseable output from ansible-lint :) | 18:47 |
| *** bhavikdbavishi has quit IRC | 18:47 | |
| y2kenny | fungi: it's possible that I am not describing my use case clearly. Please let me know if that is the case. | 18:47 |
| fungi | y2kenny: https://zuul-ci.org/docs/zuul/reference/pipeline_def.html#attr-pipeline.window | 18:48 |
| *** panda is now known as panda|pto | 18:48 | |
| fungi | it's the n newest changes to enter the pipeline for that queue at any point in time | 18:48 |
| fungi | er, i mean n oldest changes, sorry | 18:48 |
| AJaeger | avass: interesting | 18:48 |
| y2kenny | fungi: ok. | 18:50 |
| mordred | AJaeger: lgtm - I think it might be good to have mnaser or tristanC look too - just so we've got some eyes from folks running other zuuls too | 18:50 |
| avass | mordred: sure | 18:51 |
| fungi | y2kenny: the idea with the window is that you can set an urgent precedence for a dependent pipeline but prevent it from monopolizing all your available resources and starving lower precedence pipelines when there's a lot of activity | 18:51 |
| AJaeger | tristanC, mnaser, could you review 724910 , please? | 18:51 |
| y2kenny | fungi: I think I haven't wrap my head around dependent pipeline just yet. | 18:52 |
| fungi | dependent pipelines just construct ordered queues for changes entering them, so that each change is tested with the changes ahead of it in the queue | 18:53 |
| *** iurygregory has joined #zuul | 18:53 | |
| y2kenny | fungi: also I am not sure how it fit with the pisect use case. For dependent pipeline, changes are still queued by the source repo event right? The only other way to queue an arbitary change is via the enqueue command? | 18:54 |
| fungi | i'm looking for pisect now. is it some raspberry pi based test platform? | 18:55 |
| fungi | oh, a distributed git bisect engine | 18:55 |
| fungi | so you're wanting to do git bisect on the branch history? | 18:56 |
| y2kenny | I want to do multiple build on the branch history | 18:56 |
| fungi | and run each bisection point as a separate buildset i guess? | 18:56 |
| avass | interesting | 18:56 |
| y2kenny | with bisect essentially you pretty much do things serially one pivot at a time right? | 18:56 |
| mordred | y2kenny: https://docs.openstack.org/infra/publications/zuul/#(18) has an animation | 18:56 |
| y2kenny | what pisect does is try more than 1 pivot at the same time | 18:57 |
| mordred | y2kenny: click in the browser window to advance each animation point | 18:57 |
| mordred | but it shows what the dependent pipeline does | 18:57 |
| y2kenny | mordred: oh pretty :) | 18:57 |
| fungi | y2kenny: got it, i think it's a very cool use case, i don't know that zuul has anything to map to that design (yet) since it's primarily code review event driven | 18:57 |
| fungi | i'm trying to imagine how the reporting would work, for one thing | 18:58 |
| clarkb | a dependent pipeline is basically that for predicted states | 18:58 |
| mordred | y2kenny: it's not 100% as understandable without our narration - but it might be helpful. there are youtube videos of both corvus and I narrating versions of that animation , but I don't know where they are off the top of my head | 18:58 |
| fungi | zuul's main focus is on testing things which haven't merged, while git bisect is focused on locating the point at which a regression was introduced (possibly long) after it merged | 18:58 |
| corvus | mordred, y2kenny: there's a video on the zuul homepage: https://zuul-ci.org/ | 18:59 |
| y2kenny | corvus, mordred: I will look at the video... the narration is important :) | 18:59 |
| y2kenny | fungi: I think the reporting as it is right now should be fine. Just knowing the pass/fail is useful. The idea is to let people leverage the existing CI farm and job definitions that already exist. | 19:01 |
| fungi | what exactly passes or fails though, and what state are you reporting on? | 19:02 |
| openstackgerrit | Merged zuul/zuul-jobs master: use-buildset-registry: add Fedora certifacts vars https://review.opendev.org/724717 | 19:02 |
| openstackgerrit | Merged zuul/zuul-jobs master: Fix bare 'item' in build-container-image https://review.opendev.org/725298 | 19:02 |
| avass | fungi: couldn't that work similar to tags though? report the state of the commit? | 19:03 |
| y2kenny | fungi: pass/fails of the job. Perhaps you are thinking that all the jobs should be passing on any of the commit because they are all gated if fail. | 19:03 |
| fungi | avass: sure, which commit? | 19:03 |
| avass | fungi: for each commit that is tested? | 19:04 |
| avass | fungi: not sure if I'm missing something | 19:04 |
| y2kenny | but there could be new job introduced after the commit has pass the gate | 19:04 |
| fungi | avass: yeah, i'm not sure how to consolidate those and what triggers them (a timer?) | 19:04 |
| avass | fungi: probably a timer yeah | 19:04 |
| y2kenny | fungi: I think that's probably what's missing... a trigger that's like a timer | 19:05 |
| fungi | y2kenny: oh, so you would want to run regression testing immediately after each change merges rather than before it merges> | 19:05 |
| fungi | oh, if a timer trigger is okay, zuul already has that | 19:05 |
| fungi | https://zuul-ci.org/docs/zuul/reference/drivers/timer.html | 19:06 |
| y2kenny | fungi: but not quite a timer trigger though. Time trigger essentially enqueue a commit by a specific time. What this crazy thing needs is a trigger that enqueue a range of commit. | 19:06 |
| avass | fungi, y2kenny: maybe something like, trigger on a timer or each x number of merges, start a buildset for y number of those commits and keep iterating until you find the most recent stable commit? | 19:06 |
| y2kenny | fungi: anyway... I am probably abusing zuul use case a bit here :) | 19:06 |
| mordred | y2kenny: you wou;dn't be able to add a new job | 19:07 |
| mordred | that didn't pass | 19:07 |
| fungi | so once a day take the branch tip state and run pisect on that but tell it to only consider the n most recent commits in the branch history and prove it with an n-node nodeset? | 19:07 |
| mordred | y2kenny: because adding new jobs tests running the jobs against the repos and is also done in the dependent pipeline | 19:07 |
| avass | fungi: something like that | 19:08 |
| y2kenny | mordred: oh yea, that's true | 19:08 |
| fungi | avass: that should already be doable with what zuul has today | 19:08 |
| avass | yeah I guess it would | 19:08 |
| y2kenny | mordred: but do you understand what I am trying to abuse zuul to do? ;) | 19:09 |
| fungi | avass: but it would treat the pisect jobs for the current branch state as a single buildset rather than separate buildsets for each n commits in the branch history | 19:09 |
| avass | fungi: yeah | 19:09 |
| fungi | no idea if that's acceptable for the desired case or not | 19:09 |
| fungi | and what the real difference is between treating them as individual buildsets vs one buildsets where the jobs are multinode | 19:10 |
| avass | fungi: individual buildsets, vs one build | 19:10 |
| fungi | avass: well, unless you had multuple pisect jobs you wanted to run which tested different things | 19:11 |
| avass | yeah :) | 19:11 |
| fungi | but regardless, wondering why one job couldn't perform a distributed pisect across a multinode nodeset and report the results instead of needing it to be run multiple times with individual commits as separate builds | 19:12 |
| avass | I think it could be useful for tests that are too long to run in the gate or requires resources that are very limited | 19:12 |
| fungi | if you want to restrict it to max 5-node nodesets, have a n-5..n job, a n-10..n-6 job, a n-15..n-11 job and so on | 19:13 |
| y2kenny | I think the motivation is similar to the timer trigger that trigger jobs that are not run at gate | 19:14 |
| y2kenny | another use case is outside of regression actually. | 19:14 |
| avass | y2kenny: thanks for the tip :) | 19:14 |
| y2kenny | let say the test result is not some pass/fail but some performance values (like completion time for example) | 19:15 |
| openstackgerrit | Merged zuul/zuul-jobs master: use-buildset-registry: fix modify_registries_conf library idempotency https://review.opendev.org/725651 | 19:15 |
| fungi | y2kenny: so how about if the timer trigger could trigger on multiple states in the branch history besides just the branch tips? | 19:17 |
| y2kenny | fungi: yes, something like that | 19:17 |
| fungi | also yeah, we've seen similar post-merge testing interest for things like tracking test coverage | 19:17 |
| openstackgerrit | Monty Taylor proposed zuul/zuul-jobs master: Write buildkitd.toml in use-buildset-registry https://review.opendev.org/724837 | 19:22 |
| openstackgerrit | Monty Taylor proposed zuul/zuul-jobs master: Include distro-specific vars https://review.opendev.org/725668 | 19:22 |
| mordred | clarkb, avass, tristanC: ^^ the final "write buildkitd.toml in use-buildset-registry" patch needs further debuggin, so if you don't mind, that should unstick the nodepool multi-arch job | 19:23 |
| mordred | corvus: ^^ | 19:24 |
| avass | mordred: done | 19:27 |
| mordred | avass: thx! | 19:27 |
| y2kenny | oh a separate note, I started to introduce my zuul deployment to a teammate and I got another "better than Jenkins" testimony | 19:27 |
| y2kenny | (just one more data point) | 19:28 |
| mordred | \o/ | 19:29 |
| *** michael-beaver has joined #zuul | 19:30 | |
| mordred | corvus: https://zuul.opendev.org/t/zuul/build/02cbe4f061114b629734022c0ff084c8 multi-arch failed with the latest version of the use-buildset-registry patch | 19:44 |
| mordred | corvus: I'm hoping maybe you can see where it went poorly | 19:45 |
| mordred | corvus: nm. I see the logic error | 19:46 |
| fungi | y2kenny: that's awesome, thanks for the feedback!!! | 19:46 |
| clarkb | zuulians, opendev updated its python-base image for python3.7 and 3.8 to remove jemalloc. Based on process of elimination jemalloc seemed to be the source of our leaks with zuul-web. | 19:46 |
| clarkb | tobiash: ^ I know you've seen leaks on newer pyhton too so wanted to point that out | 19:47 |
| clarkb | this also means the next zuul commit to land will rebuild on top of that and not use jemalloc | 19:47 |
| corvus | mordred: i don't see access logs on the buildset registry from that build; looks like buildkit isn't talking to it at all | 19:47 |
| openstackgerrit | Monty Taylor proposed zuul/zuul-jobs master: Write buildkitd.toml in use-buildset-registry https://review.opendev.org/724837 | 19:47 |
| mordred | corvus: it isn't | 19:47 |
| mordred | corvus: look at the inter-diff between 12 and 13 there ^^ | 19:47 |
| corvus | oh, heh, i just now saw your "nm" :) | 19:47 |
| corvus | mordred: can confirm test output matches logic bug :) | 19:48 |
| mordred | clarkb: ^^ please see above - I think our idempotency patch had a bug in it that the gate didn't gatch | 19:48 |
| mordred | or catch | 19:48 |
| clarkb | mordred: will do | 19:48 |
| mordred | corvus, clarkb : should we lift out the fix for modify_registries_conf.py into its own patch so we can fast-track it? | 19:49 |
| clarkb | mordred: I'm ok with the change there if we want to roll forward +2'd | 19:50 |
| *** hashar has joined #zuul | 19:50 | |
| AJaeger | mordred: really "if not mirrors - and then mirrors.insert"? | 19:50 |
| openstackgerrit | Merged zuul/zuul-jobs master: Include distro-specific vars https://review.opendev.org/725668 | 19:51 |
| clarkb | AJaeger: its either [] or [stuff] | 19:51 |
| clarkb | AJaeger: so the not catches [] case | 19:51 |
| AJaeger | clarkb: ah! | 19:51 |
| mordred | yeah | 19:53 |
| mordred | so for the "normal" case if empty file, it was happily not doing anything :) | 19:53 |
| AJaeger | zuul-jobs reviewers, two small changes: tempfile cleanup and py38 jobs: https://review.opendev.org/725650 and https://review.opendev.org/725663 - reviews welcome, please | 19:57 |
| AJaeger | and two small renames https://review.opendev.org/#/c/725012/4 and https://review.opendev.org/#/c/725013/4 | 19:58 |
| AJaeger | mordred: https://review.opendev.org/#/c/724841/ is another registry change you might want to review | 20:00 |
| *** jamesmcarthur has quit IRC | 20:06 | |
| openstackgerrit | Merged zuul/zuul-jobs master: Write buildkitd.toml in use-buildset-registry https://review.opendev.org/724837 | 20:12 |
| openstackgerrit | Merged zuul/zuul-jobs master: ensure-pip: use python2-pip when running under Python 2 https://review.opendev.org/724777 | 20:12 |
| openstackgerrit | Merged zuul/zuul-jobs master: Add py38 testing https://review.opendev.org/725663 | 20:12 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Add upload-artifactory role https://review.opendev.org/725678 | 20:17 |
| avass | just realized i forgot to push that | 20:18 |
| openstackgerrit | Merged zuul/zuul-jobs master: ansible-lint: use matchplay instead of matchtask https://review.opendev.org/724910 | 20:18 |
| openstackgerrit | Merged zuul/zuul-jobs master: use-buildset-registry: do not update ca when not necessary https://review.opendev.org/724841 | 20:22 |
| *** jamesmcarthur has joined #zuul | 20:23 | |
| *** jamesmcarthur has quit IRC | 20:26 | |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Add upload-artifactory role https://review.opendev.org/725678 | 20:26 |
| *** jamesmcarthur has joined #zuul | 20:29 | |
| *** jamesmcarthur has quit IRC | 20:35 | |
| *** rlandy is now known as rlandy|brb | 20:35 | |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: ensure-pip: Install backported pip for Xenial https://review.opendev.org/724788 | 20:42 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: Add plain nodes to testing https://review.opendev.org/724776 | 20:42 |
| openstackgerrit | Merged zuul/zuul-jobs master: use zj_image instead of image as loopvar https://review.opendev.org/725012 | 20:45 |
| *** jamesmcarthur has joined #zuul | 20:49 | |
| *** Goneri has quit IRC | 20:56 | |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Check blocks recursively for loops https://review.opendev.org/724967 | 20:57 |
| *** hashar has quit IRC | 21:00 | |
| *** rlandy|brb is now known as rlandy | 21:03 | |
| *** jamesmcarthur has quit IRC | 21:06 | |
| clarkb | zuulians https://review.opendev.org/#/c/725363/1 looks like a straightforward logging update that will help us get new imgaes built without jemalloc | 21:07 |
| clarkb | if anyone has a moment for that it would be great as I'd like to be able to redeploy our zuul with that at some point soon | 21:07 |
| *** jamesmcarthur has joined #zuul | 21:08 | |
| corvus | clarkb: done | 21:10 |
| clarkb | thank you | 21:10 |
| *** y2kenny has left #zuul | 21:11 | |
| *** jamesmcarthur has quit IRC | 21:11 | |
| *** jamesmcarthur has joined #zuul | 21:13 | |
| openstackgerrit | Monty Taylor proposed zuul/zuul-jobs master: Set up siblings dirs for each build in the loop https://review.opendev.org/725683 | 21:13 |
| mordred | clarkb, corvus: ^^ | 21:14 |
| mordred | https://zuul.opendev.org/t/zuul/build/dfe1c2cd37b643879e889cfd67813b07/console | 21:14 |
| mordred | is the error | 21:14 |
| mordred | we only notice this when images is a list and siblings are in use | 21:14 |
| corvus | mordred: and i guess our test only does one image | 21:14 |
| *** jamesmcarthur has quit IRC | 21:15 | |
| *** jamesmcarthur has joined #zuul | 21:17 | |
| *** jamesmcarthur has quit IRC | 21:17 | |
| *** jamesmcarthur has joined #zuul | 21:17 | |
| mordred | yeah - updating to change that | 21:18 |
| openstackgerrit | Monty Taylor proposed zuul/zuul-jobs master: Set up siblings dirs for each build in the loop https://review.opendev.org/725683 | 21:18 |
| mordred | corvus: there we go - that should exercise the code path | 21:18 |
| mordred | corvus: fascinating new error on the normal nodepool build: https://zuul.opendev.org/t/zuul/build/f4cfb40d86f746a985c72d1473f6469f | 21:19 |
| mordred | corvus: #22 ERROR: failed commit on ref "layer-sha256:c2ce0712fbc150db6136e121fa38328976919c059529d19ce9ccc73b4ba4d337": unexpected size 0, expected 1999588 | 21:20 |
| openstackgerrit | Merged zuul/zuul-jobs master: use zj_log_file instead of item as loop_var https://review.opendev.org/725013 | 21:20 |
| ianw | mordred: hrm, how come https://review.opendev.org/#/c/725456/ didn't pick that up? | 21:21 |
| mordred | ianw: oh - haha. BECAUSE | 21:24 |
| mordred | ianw: we use build-docker-imag ein a config project- so it's not depends-on-able | 21:24 |
| mordred | ianw: that's not the first time I've forgotten that | 21:24 |
| ianw | oh ... well i tried :) | 21:26 |
| mordred | clarkb: I transferred your +A from patchset 1 - ps2 just added tests: https://review.opendev.org/#/c/725683/ | 21:30 |
| ianw | yeah, opendev /base-jobs getting in the way | 21:30 |
| *** jamesmcarthur has quit IRC | 21:39 | |
| mordred | corvus: in good news - the new tests exercise the code path! | 21:43 |
| *** jamesmcarthur has joined #zuul | 21:44 | |
| openstackgerrit | Monty Taylor proposed zuul/zuul-jobs master: Set up siblings dirs for each build in the loop https://review.opendev.org/725683 | 21:45 |
| mordred | corvus, clarkb : in bad news, the tests found a flaw - so plz to review one more time | 21:46 |
| mordred | corvus: also - I'm already leaning towards "just use buildx" instead of this dual path | 21:46 |
| corvus | mordred: yeah, that's sounding like a win | 21:46 |
| clarkb | does buildx run reasonably quickly on its native platform? | 21:47 |
| corvus | clarkb: yes. it even runs reasonably quickly non-native. | 21:47 |
| clarkb | nice | 21:47 |
| corvus | clarkb: i think we decided there was ~no cost native | 21:47 |
| corvus | (mostly a little extra setup cost; actual build isn't doing any emulation) | 21:47 |
| mordred | yeah - for native single-arch is basically the same | 21:47 |
| mordred | in fact, it might build quicker - because buildkit is better at running non-dependent tasks in parallel | 21:48 |
| corvus | true, we were already looking at that after the denver ptg | 21:48 |
| mordred | yeah - I think "use the buildx path" may be the easiest "use buildkit builder" | 21:49 |
| mordred | although - apparently if we *actually* just used buildkit without docker involved, there's even more stuff we can do - but my brain can't really deal with that today | 21:50 |
| mordred | (there's a buildkitd and a buildctl) | 21:50 |
| clarkb | is there any concern that a dockerfile that is valid for docker proper wouldn't be valid for buildx somehow? | 21:50 |
| clarkb | if not then simplifying seems like a good idea | 21:51 |
| mordred | clarkb: not really no | 21:51 |
| mordred | clarkb: the risk is the other way - there are dockerfile constructs buildkit allows that do not work in normal docker | 21:51 |
| clarkb | ya | 21:52 |
| clarkb | is there a "strict" mode? | 21:52 |
| mordred | clarkb: although these days if you just enable buildkit with normal docker those are available | 21:52 |
| mordred | i don't think so | 21:52 |
| mordred | to be fair - all of the things have obscure syntax and are hard to use | 21:52 |
| mordred | so it's really unlikely anyone will accidentally use them | 21:53 |
| clarkb | ah so not like perl :) | 21:53 |
| mordred | haha | 21:54 |
| mordred | clarkb: RUN --mount=type=secret,id=top-secret-passwd my_command | 21:54 |
| mordred | clarkb: stuff like that | 21:54 |
| mordred | clarkb: but all you have to do if you do let some buildkit stuff in is do "DOCKER_BUILDKIT=1 docker build" | 21:55 |
| *** jamesmcarthur has quit IRC | 21:55 | |
| *** jamesmcarthur has joined #zuul | 21:57 | |
| *** jamesmcarthur has quit IRC | 22:01 | |
| corvus | clarkb: looks like a transient image build failure (kubic repo failure) on 725363; i revoked my +w so it won't merge | 22:01 |
| corvus | clarkb: and re-+3 after abandon/restore; it should try again | 22:02 |
| corvus | https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_a90/725363/1/gate/zuul-upload-image/a9014fe/job-output.txt is the log | 22:03 |
| mordred | GAH | 22:03 |
| mordred | same issue with build-container-image | 22:03 |
| corvus | is kubic unreliable enough we're going to need to get the mirrors working inside the build? | 22:03 |
| corvus | we've just been skating by so far without mirrors because zuul-scale isn't openstack-scale :/ | 22:04 |
| mordred | corvus: golly I hope not | 22:04 |
| mordred | corvus: cause also - we'll have to figure out plumbing mirrors into container builds | 22:05 |
| mordred | corvus: which we should also do but haven't done yet | 22:05 |
| corvus | mordred: that's what i meant | 22:06 |
| clarkb | in theory it has all the same infrastructure as bionic and xenial | 22:06 |
| mordred | corvus: ah - nod | 22:06 |
| clarkb | though its probably under a lot more post release churn? | 22:06 |
| mordred | clarkb: kubic does? | 22:06 |
| mordred | clarkb: this isn't PPA - this is opensuse build | 22:06 |
| clarkb | oh kubic sorry in my head I had it as focal | 22:06 |
| mordred | nod | 22:06 |
| clarkb | ya thats the pull from obs right? | 22:07 |
| clarkb | so its not even debian specific infrastructure | 22:07 |
| mordred | yeah | 22:07 |
| clarkb | we should maybe consider mirroring the skopeo, podman, etc builds from obs into our mirrors | 22:07 |
| openstackgerrit | Monty Taylor proposed zuul/zuul-jobs master: Set up siblings dirs for each build in the loop https://review.opendev.org/725683 | 22:08 |
| mordred | I had to rebase - so that's a blank rebase patch - next patch coming | 22:08 |
| corvus | right, but before we bother with that, we should figure out how to plumb them into the builder; that's the thing we were dodging earlier -- i think the right way to do that is to flesh out the zuul site mirror variable to describe docker mirroring then update the buildkitd.toml file (and similar) to add mirrors. | 22:09 |
| openstackgerrit | Monty Taylor proposed zuul/zuul-jobs master: Set up siblings dirs for each build in the loop https://review.opendev.org/725683 | 22:09 |
| clarkb | corvus: does buildkit know how to set up distro mirrors within the build? | 22:09 |
| corvus | oh, but it's going to be even more complicated to get the debian mirrors in there | 22:09 |
| clarkb | otherwise we probably need to bind mount in a apt repos config | 22:09 |
| corvus | i'm not sure how we'll do that | 22:09 |
| mordred | corvus, clarkb: there was actually a latent bug in siblings that the new siblings test uncovered - in addition to the bug I added | 22:09 |
| corvus | we basically have no mirroring in there | 22:10 |
| mordred | corvus, clarkb : I believe we can do things with COPY --mount | 22:10 |
| corvus | tobiash has a start on that here: https://review.opendev.org/649448 | 22:10 |
| mordred | which will allow us to bind-mount some stuff in without having it wind up in the final image | 22:10 |
| clarkb | mordred: ++ | 22:10 |
| corvus | mordred: so maybe translate tobiash's change into doing that with a bindmount? | 22:11 |
| mordred | corvus: yeah - maybe so | 22:11 |
| mordred | corvus: also - we might be able to abstract that into something we can put into builder/base | 22:11 |
| corvus | (his was basically a "write it out; install, then cleanup" change | 22:11 |
| mordred | so that it's not quite so much boilerplate | 22:11 |
| mordred | yeah | 22:11 |
| clarkb | I do think addressing the mirror config dict in zuul makes this cleaner in the job consumption side | 22:11 |
| corvus | clarkb: yeah | 22:11 |
| mordred | agree | 22:11 |
| mordred | but I thnk that's half the story - and the other half is making something that's reusable not in the gate and also usable in the gate - which apparently nobody cares because CDN yolo | 22:12 |
| corvus | basicially, if we have a good mirror dict, we can have the buildx setup and build write and use the files to bind mount | 22:12 |
| mordred | so we'll be ... having fun again | 22:12 |
| mordred | yeag | 22:12 |
| mordred | at least- I hope we can | 22:12 |
| corvus | mordred: why does it have to be useable outside the gate? | 22:12 |
| mordred | corvus: I mean - I'd like "docker build ." to continue to work | 22:13 |
| corvus | mordred: yeah, i don't see the problem | 22:13 |
| corvus | i'm imagining we just plop some apt configs in the image when it's building | 22:14 |
| corvus | mordred: you do 'docker build .'; zuul does 'docker buildx --bindstuff' | 22:14 |
| mordred | ah - sorry -that's where we're missing each other | 22:14 |
| mordred | the buildkit extention for bind mounting is an extension to the dockerfile syntax | 22:14 |
| corvus | ooooooooooooh | 22:14 |
| mordred | now - with _buildah_ you can totally do the mount on the command line | 22:15 |
| mordred | but we don't have a multi-arch story with buildah yet | 22:15 |
| corvus | headdesk | 22:15 |
| mordred | they are working on it - and it's possible we could have something scriptable before they have a full buildx command | 22:15 |
| mordred | lemme find the issue I was reading earlier | 22:16 |
| mordred | corvus: https://github.com/containers/buildah/issues/1590 | 22:16 |
| clarkb | mordred: you mean the COPY --mount thing above is buildkit specific? | 22:17 |
| mordred | https://github.com/grooverdan/buildah/tree/args-target-platform | 22:17 |
| mordred | clarkb: yes | 22:17 |
| mordred | although I think buildah has implemented at least some of it | 22:17 |
| mordred | honestly - the buildah story here is much better for zuul | 22:17 |
| mordred | (assuming multi-arch gets sorted) | 22:18 |
| mordred | because you can pass -v commands to the build to do the bind-mount - which means it can be a zuul action not a dockerfile action | 22:18 |
| mordred | docker explicitly disallows that because of concerns about people being confused about contents of mounts not winding up in final image contents | 22:18 |
| clarkb | wow thats exactly what we want though | 22:19 |
| mordred | yeah | 22:19 |
| mordred | like - it is EXACTLY what we want | 22:19 |
| corvus | good thing buildah didn't maintain bug-compatibility with docker on that one :) | 22:20 |
| mordred | which makes me want to figure out how to get multi-arch to work with buildah so that we can just switch to "screw it - we always use buildah" | 22:20 |
| mordred | if someone REALLY wants docker then, they can always just use-buildset-registry and roll the dice | 22:20 |
| corvus | mordred: i'm having trouble figuring out if grooverdan is still working on that | 22:24 |
| mordred | me too | 22:25 |
| mordred | corvus: so - worst case - buildkit isn't doing magic - I'm pretty sure we could do a binfmt thing for each arch outside of an invocation - build each layer and then build and push a manifest | 22:26 |
| mordred | corvus: like - we could almost certainly do it all just in ansible if we dive in | 22:26 |
| corvus | mordred: yeah, i was having the same hunch, though i don't quite know the mechanics of that. | 22:26 |
| mordred | corvus: me either | 22:27 |
| mordred | but I think we can figure it out - and mirrors might be the compelling reason to | 22:27 |
| mordred | now that buildx is there, we're unblocked on multiarch for nodepool | 22:27 |
| corvus | yeah. and once we do that we know we can build the manifest | 22:27 |
| mordred | yup | 22:28 |
| mordred | and then we've got one story for build/upload that works for all targets | 22:28 |
| mordred | basically - I think this is what porting over to build-container-image looks like | 22:28 |
| mordred | and then we just start using build-container-image | 22:28 |
| clarkb | how is that different than just using buildx? | 22:28 |
| clarkb | (seems like we should just use buildx in that case?) | 22:29 |
| mordred | clarkb: no bind-mounts in buildx | 22:29 |
| mordred | and recommend everyone use that unless they REALLY need to explicitly test something specific about building in docker proper | 22:29 |
| clarkb | mordred: maybe it would be easier to change that in builx than rebuilding buildx from scratch in ansible? | 22:30 |
| mordred | clarkb: but basically - we can learn what we need from what buildx is doing on the backend for multi-arch - which actually isn't that exciting it's just already done and easy | 22:30 |
| mordred | clarkb: nope. buildx is fundamentally philosophically opposed to the idea | 22:30 |
| *** michael-beaver has quit IRC | 22:30 | |
| mordred | we could fork buildx | 22:30 |
| clarkb | I guess I just see what you described as "make buildx in ansible" | 22:30 |
| mordred | no - sorry | 22:30 |
| clarkb | and so my immediate thought is why not use buildx | 22:31 |
| mordred | because buildx is missing the fundamental feature that makes this interesting and is fundamentally opposed to it | 22:31 |
| clarkb | which is the mount that doesn't end up in final image | 22:31 |
| mordred | yeah | 22:31 |
| mordred | the thing that buildx does that we need is not hard and we can totally script around it until such a time as buildah supports it natively | 22:31 |
| mordred | which they _will_ do becaues they'll have to | 22:32 |
| clarkb | which is run a qemu vm with the correct arch | 22:32 |
| mordred | yeah | 22:32 |
| corvus | (well, heh, they're fundamentally opposed to it, except for when they aren't by adding --mount=) | 22:32 |
| mordred | corvus: sssh | 22:32 |
| corvus | i'm sure the rules make sense to them | 22:32 |
| clarkb | dealing with mounts through qemu might also be ugly | 22:33 |
| clarkb | probably worth checking how painful that is in their existing codebase | 22:33 |
| clarkb | I know its something that kata perpetually struggles with | 22:33 |
| corvus | clarkb: on friday when we hit the loading_errors bug -- do you happen to remember if zuul-jobs had some kind of a configuration error? | 22:36 |
| corvus | (like, a persistent one that was already merged in the repo?) | 22:36 |
| clarkb | corvus: I don't recall it having one, but I couldn't say for sure | 22:36 |
| clarkb | corvus: I think the first time we hit it (or observed it atl east) the openstack tenant did have persistent errors | 22:37 |
| corvus | i can't find evidence in the logs, but i think we only log those on full reconfigurations | 22:37 |
| corvus | i know the zuul tenant has and had errors, but none in the zuul-jobs repo | 22:37 |
| openstackgerrit | Monty Taylor proposed zuul/zuul-jobs master: Set up siblings dirs for each build in the loop https://review.opendev.org/725683 | 22:38 |
| mordred | clarkb: ^^ headdesk | 22:38 |
| mordred | clarkb, corvus: that's going to do it though, based on the last error | 22:39 |
| corvus | i think one of the other requirements for the bug to trigger is for there to be a config error of some kind; either one in the change itself, or one in a dependent change | 22:39 |
| corvus | hrm, now that i think about it, i think it may have to be a config error in a different repo in a dependent change, which definitely isn't what happened on friday (the series was entirely zuul-jobs changes) | 22:40 |
| corvus | so i'm still missing something | 22:40 |
| clarkb | mordred: whoops | 22:40 |
| mordred | clarkb: also - I don't think they're runnign qemu vms - I think they're doing binfmt_misc equiv of cross-compiling | 22:42 |
| clarkb | mordred: I think binfmt maps the execution of the processes onto qemu emulation | 22:44 |
| clarkb | that may not be a full VM in the sense that we normally think of it with a kernel and all that hough | 22:44 |
| clarkb | https://github.com/qemu/qemu/blob/master/scripts/qemu-binfmt-conf.sh | 22:47 |
| clarkb | https://wiki.debian.org/QemuUserEmulation also useful | 22:48 |
| clarkb | its emulated more like an interpreter than a virtual machine (if that makes sense) | 22:49 |
| clarkb | which likely simplifies the mounting thingsI was worred about as there is a single kernel | 22:49 |
| mordred | yeah - there's the stuff in the debian article about using it for cross-compiling | 22:50 |
| clarkb | and I guess buildkit comes with everything you need to build an arm64 image if you can run as arm64? | 22:51 |
| ianw | (does it crossbuild? because we don't have a good story for mixing arm64 and x86 nodes in jobs) | 22:52 |
| corvus | we may be able to use some of the buildkit container images to get the necessary deps, even to use with buildah | 22:52 |
| openstackgerrit | Monty Taylor proposed zuul/zuul-jobs master: Set up siblings dirs for each build in the loop https://review.opendev.org/725683 | 22:52 |
| clarkb | ianw: ya the current buildx stuff is crossbuilding for that reason aiui | 22:53 |
| mordred | corvus, clarkb you're REALLY goig to liek the latest patchset | 22:53 |
| ianw | right i meant buildkit sorry | 22:53 |
| corvus | ianw: buildkit==buildx for this | 22:53 |
| mordred | yeah | 22:53 |
| mordred | corvus: and yes - I think we can almost certainly use the buildkit container images | 22:53 |
| corvus | (the buildx command invokes a buildkit based builder) | 22:53 |
| mordred | specifically the docker/binfmt image sets up the right binaries in binfmt_misc | 22:53 |
| clarkb | mordred: today is just not my day to review simple changes :) | 22:54 |
| mordred | clarkb: nor mine to write them | 22:54 |
| corvus | mordred: so a hand-wavey sketch of this might look like "run buildah in a container based on a binfmt_misc image" | 22:54 |
| mordred | corvus: well - yes - except we don't even have to do that so much ... | 22:55 |
| mordred | corvus: the binfmt image installs stuff into /proc/sys/fs/binfmt_misc - which is why it has to be run privileged | 22:55 |
| clarkb | ya those are the mappings from https://github.com/qemu/qemu/blob/master/scripts/qemu-binfmt-conf.sh basically | 22:55 |
| mordred | at that point, I believe we could even run buildah on the host | 22:55 |
| clarkb | it tells the kernel if you see these magic numbers use this emulator | 22:55 |
| clarkb | (roughly) | 22:55 |
| mordred | yeah | 22:55 |
| mordred | so - it might still be about using that container - or that might just e a setup container | 22:56 |
| mordred | but - hand wave hand wave - it's ultimately jst code | 22:56 |
| ianw | ahh, thanks, i think i'm beginning to see ... it talks about "ucp" compatibility which i first read as "uucp" and said no ... surely they're not ... but half believed they would :) | 22:57 |
| mordred | ianw: uucp is entirely too old school | 22:57 |
| mordred | ;) | 22:57 |
| clarkb | ipv6 too | 22:57 |
| clarkb | :) | 22:57 |
| corvus | the time between uucp and ipv6 is comparable to the time between ipv6 and now. | 22:58 |
| mordred | that's just depressing | 22:58 |
| *** tumble has joined #zuul | 23:04 | |
| openstackgerrit | Merged zuul/zuul master: merger: warn about invalid object type https://review.opendev.org/725363 | 23:04 |
| clarkb | promote for ^ appears to have completed already so we should be good to go without jemalloc on python3.8 now | 23:06 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Add some debug lines to help with the loading_errors bug https://review.opendev.org/725732 | 23:09 |
| tumble | hi guys, I'm new to the zuul ecosystem and still trying to figure out how to fit it into my toolchain. Did I understand it correctly that the git driver only supports public repositories, i.e. with no authentication? | 23:10 |
| corvus | clarkb, tobiash: ^ i think i could fix what i suspect to be the immediate cause of the loading_errors bug, but i think there may still be some secondary issue that's triggering it. i'd like to understand that first since it's easier to track down now. | 23:10 |
| corvus | tumble: i believe that's the case; the driving use case for that is so that other zuuls can share the zuul-jobs repo hosted by opendev (or similar). the main workflow happens with the gerrit, github, or pagure drivers (which do support authn) | 23:12 |
| tumble | I see, so I guess I would need to write my own driver for our internal gitea then. Thanks for the confirmation. :) | 23:14 |
| corvus | tumble: oh you're using gitea, nice :) | 23:14 |
| corvus | tumble: we'd love a gitea driver; and i bet you could trick some of us into collaborating on it :) | 23:15 |
| tumble | yeah and I hoped since opendev is gitea-based as well, I'd be lucky to find a driver for it, but unfortunately not the case :D | 23:15 |
| tumble | I looked at the github and gitlab drivers and while the github one looks quite huge, the gitlab one ain't so bad so I at least consider it doable | 23:15 |
| corvus | it's also super easy for us to test too, which is nice | 23:16 |
| corvus | tumble: also check out the pagure driver | 23:17 |
| tumble | will do, tbh I didn't hear of pagure before, should look at it anyway. You never stop learning :D | 23:17 |
| corvus | tumble: it might be a good combination of "simpler than github, but more finished than bitbucket" | 23:17 |
| openstackgerrit | Merged zuul/zuul-jobs master: Set up siblings dirs for each build in the loop https://review.opendev.org/725683 | 23:17 |
| corvus | tumble: we'll want to make sure we have good webhook support in the gitea driver, and the bitbucket one is largely based on polling which won't scale as well | 23:18 |
| openstackgerrit | Merged zuul/zuul-jobs master: ensure-pip: Install backported pip for Xenial https://review.opendev.org/724788 | 23:18 |
| tumble | corvus, yeah good point. I'm used to on-commit hooks from our current jenkins workflow and I don't wanna go back to polling | 23:19 |
| corvus | tumble: if you have some time for this, i can probably help out starting later this week | 23:22 |
| corvus | probably the easiest way to start would be to set up a local zuul dev environment (one where you can easily update and interactively run the zuul-scheduler process) with a gitea, then start on the driver. the first thing the driver will want to do is to query gitea for a list of branches for each of the projects specified in the config. that's a good checkpoint. :) | 23:25 |
| corvus | then it's implementing webhooks to respond to events, some more queries, etc. | 23:25 |
| openstackgerrit | Merged zuul/zuul-jobs master: Add plain nodes to testing https://review.opendev.org/724776 | 23:26 |
| corvus | once that's all working with a real gitea, we'll add some unit tests that fake out part of it so we can regression test drivers without running the whole thing | 23:26 |
| tumble | corvus, I'll be hanging out here from now (if I don't forget to launch my hexchat) and sure, let's see what we can do. Friday is a bank holiday here and I'll probably find some time to look into this over the weekend :) | 23:26 |
| corvus | tumble: sounds great :) | 23:27 |
| *** tosky has quit IRC | 23:27 | |
| tumble | since I'm basically working on our zuul setup at the moment and can easily reproduce it (all written in ansible and using the official docker containers) that's gonna be my playground for a start. Can even host it on some public VM for playing. A gitea playbook I also have somewhere. | 23:28 |
| tumble | oh well, the nodepool part might be difficult in public, but I guess that's not so relevant for this scenario | 23:29 |
| corvus | tumble: yeah, just running "noop" jobs is sufficient for testing | 23:30 |
| *** tumble has quit IRC | 23:44 | |
| *** guillaumec has quit IRC | 23:59 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!