| clarkb | corvus: looks like zuul is on its way to +1'ing the nodepool change | 00:01 |
|---|---|---|
| corvus | clarkb: now's probably a good time for https://review.opendev.org/776290 then | 00:03 |
| clarkb | approved | 00:06 |
| *** jamesmcarthur has joined #zuul | 00:07 | |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Update upload-logs-swift and upload-logs-gcs https://review.opendev.org/c/zuul/zuul-jobs/+/774650 | 00:09 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Update upload-logs roles to support endpoint override https://review.opendev.org/c/zuul/zuul-jobs/+/774650 | 00:10 |
| corvus | Open10K8S: I made a 1 character change on PS 15, and updated the commit message in PS16. also, in the future, if you can avoid rebasing your change when making new patchsets, i'd appreciate it. it makes it hard to tell what you changed from one version to the next. To review that, I had to download PS13 and rebase it on the parent of PS12 to figure out what you did. | 00:11 |
| corvus | Open10K8S: anyway, with those changes it lgtm. +2 | 00:12 |
| corvus | mnaser, tobiash: ^ | 00:12 |
| *** sduthil has quit IRC | 00:15 | |
| openstackgerrit | Merged zuul/zuul-jobs master: ensure-openshift: remove unused role var https://review.opendev.org/c/zuul/zuul-jobs/+/775888 | 00:22 |
| openstackgerrit | Merged zuul/zuul-jobs master: ensure-zookeeper: add use_tls role var https://review.opendev.org/c/zuul/zuul-jobs/+/776290 | 00:22 |
| *** jamesmcarthur has quit IRC | 00:35 | |
| *** jamesmcarthur has joined #zuul | 00:40 | |
| openstackgerrit | James E. Blair proposed zuul/zuul-storage-proxy master: Add zuul user to container https://review.opendev.org/c/zuul/zuul-storage-proxy/+/776566 | 00:42 |
| *** sduthil has joined #zuul | 01:09 | |
| *** jamesmcarthur has quit IRC | 02:22 | |
| *** jamesmcarthur has joined #zuul | 02:27 | |
| *** jamesmcarthur has quit IRC | 02:33 | |
| *** jamesmcarthur has joined #zuul | 02:34 | |
| *** jamesmcarthur has quit IRC | 02:35 | |
| openstackgerrit | James E. Blair proposed zuul/nodepool master: Require TLS https://review.opendev.org/c/zuul/nodepool/+/776286 | 02:46 |
| *** jamesmcarthur has joined #zuul | 02:54 | |
| *** jamesmcarthur has quit IRC | 03:07 | |
| *** jamesmcarthur has joined #zuul | 03:08 | |
| *** rlandy has quit IRC | 03:38 | |
| *** maxamillion has quit IRC | 04:01 | |
| *** maxamillion has joined #zuul | 04:03 | |
| *** ikhan has quit IRC | 04:21 | |
| *** ykarel has joined #zuul | 04:53 | |
| *** jamesmcarthur has quit IRC | 04:59 | |
| *** jamesmcarthur has joined #zuul | 05:01 | |
| *** evrardjp has quit IRC | 05:33 | |
| *** evrardjp has joined #zuul | 05:33 | |
| *** jfoufas1 has joined #zuul | 06:02 | |
| *** iurygregory_ has joined #zuul | 06:26 | |
| *** iurygregory has quit IRC | 06:27 | |
| *** zbr has quit IRC | 06:44 | |
| *** zbr has joined #zuul | 06:51 | |
| *** jamesmcarthur has quit IRC | 07:18 | |
| *** jamesmcarthur has joined #zuul | 07:19 | |
| *** jamesmcarthur has quit IRC | 07:25 | |
| *** piotrowskim has joined #zuul | 07:33 | |
| *** jamesmcarthur has joined #zuul | 07:49 | |
| *** jcapitao has joined #zuul | 08:01 | |
| *** rpittau|afk is now known as rpittau | 08:06 | |
| *** harrymichal has joined #zuul | 08:09 | |
| *** hashar has joined #zuul | 08:11 | |
| *** jpena|off is now known as jpena | 08:24 | |
| *** mordred has quit IRC | 08:25 | |
| *** Eighth_Doctor has quit IRC | 08:25 | |
| *** jamesmcarthur has quit IRC | 08:30 | |
| *** tosky has joined #zuul | 08:40 | |
| *** jamesmcarthur has joined #zuul | 08:44 | |
| *** zbr8 has joined #zuul | 08:48 | |
| *** mgoddard has quit IRC | 08:48 | |
| *** zbr has quit IRC | 08:49 | |
| *** zbr8 is now known as zbr | 08:49 | |
| *** jamesmcarthur has quit IRC | 08:52 | |
| *** hashar has quit IRC | 09:01 | |
| *** nils has joined #zuul | 09:02 | |
| *** hashar has joined #zuul | 09:03 | |
| *** jfoufas1 has quit IRC | 09:04 | |
| *** hashar has quit IRC | 09:06 | |
| *** jamesmcarthur has joined #zuul | 09:07 | |
| *** zbr has quit IRC | 09:10 | |
| *** zbr has joined #zuul | 09:11 | |
| *** jamesmcarthur has quit IRC | 09:12 | |
| *** Eighth_Doctor has joined #zuul | 09:22 | |
| *** jamesmcarthur has joined #zuul | 09:25 | |
| *** jamesmcarthur has quit IRC | 09:30 | |
| *** masterpe has joined #zuul | 09:40 | |
| *** mordred has joined #zuul | 09:40 | |
| *** jamesmcarthur has joined #zuul | 09:44 | |
| *** zbr has quit IRC | 09:46 | |
| *** jamesmcarthur has quit IRC | 09:50 | |
| *** ssbarnea_ has joined #zuul | 09:52 | |
| *** zbr has joined #zuul | 09:53 | |
| *** ssbarnea_ has quit IRC | 09:54 | |
| *** jamesmcarthur has joined #zuul | 10:02 | |
| *** jamesmcarthur has quit IRC | 10:07 | |
| *** jamesmcarthur has joined #zuul | 10:21 | |
| *** msuszko has joined #zuul | 10:25 | |
| *** jamesmcarthur has quit IRC | 10:27 | |
| *** hashar has joined #zuul | 10:30 | |
| *** harrymichal has quit IRC | 10:35 | |
| *** harrymichal has joined #zuul | 10:35 | |
| *** jamesmcarthur has joined #zuul | 10:40 | |
| *** jamesmcarthur has quit IRC | 10:46 | |
| *** jamesmcarthur has joined #zuul | 10:59 | |
| *** jcapitao has quit IRC | 10:59 | |
| *** bhavikdbavishi has joined #zuul | 11:01 | |
| *** jamesmcarthur has quit IRC | 11:04 | |
| *** jamesmcarthur has joined #zuul | 11:17 | |
| *** jamesmcarthur has quit IRC | 11:23 | |
| zbr | something is happening: https://gerrit-review.googlesource.com/c/gerrit/+/296851 :D | 11:26 |
| *** bhavikdbavishi has quit IRC | 11:34 | |
| avass | zbr: what's that? | 11:36 |
| zbr | i think they will finally improve the wrapping of comments in the UI, switching from 80 to 120ch. | 11:37 |
| *** jamesmcarthur has joined #zuul | 11:38 | |
| zbr | we also have a change to disable wrapping for ourselves, but is good to see the default gerrit improved. | 11:38 |
| *** harrymichal has quit IRC | 11:38 | |
| *** harrymichal has joined #zuul | 11:38 | |
| *** jamesmcarthur has quit IRC | 11:44 | |
| *** zbr0 has joined #zuul | 11:49 | |
| avass | tristanC: I think nix could work really nice in zuul with cross repo dependencies. they could use a fetchGit with branch/revision as input and push the result to a nix cache so not everything has to be rebuilt for every change. | 11:50 |
| avass | especcially where there aren't already well established package managers | 11:50 |
| *** zbr has quit IRC | 11:52 | |
| *** zbr0 is now known as zbr | 11:52 | |
| *** jamesmcarthur has joined #zuul | 11:58 | |
| *** jamesmcarthur has quit IRC | 12:03 | |
| *** maxamillion has quit IRC | 12:08 | |
| *** rpittau has quit IRC | 12:08 | |
| *** johnsom has quit IRC | 12:08 | |
| *** ericsysmin has quit IRC | 12:08 | |
| *** piotrowskim has quit IRC | 12:08 | |
| *** jamesmcarthur has joined #zuul | 12:16 | |
| *** maxamillion has joined #zuul | 12:20 | |
| *** ikhan has joined #zuul | 12:20 | |
| *** ericsysmin has joined #zuul | 12:20 | |
| *** rpittau has joined #zuul | 12:21 | |
| *** johnsom has joined #zuul | 12:23 | |
| *** jamesmcarthur has quit IRC | 12:24 | |
| *** piotrowskim has joined #zuul | 12:25 | |
| *** rlandy has joined #zuul | 12:28 | |
| *** mgoddard has joined #zuul | 12:30 | |
| *** jpena is now known as jpena|lunch | 12:31 | |
| *** sshnaidm is now known as sshnaidm|off | 12:36 | |
| *** harrymichal has quit IRC | 12:36 | |
| *** harrymichal has joined #zuul | 12:37 | |
| *** jamesmcarthur has joined #zuul | 12:37 | |
| *** harrymichal has quit IRC | 12:38 | |
| *** jamesmcarthur has quit IRC | 12:44 | |
| *** iurygregory_ is now known as iurygregory | 12:49 | |
| *** jamesmcarthur has joined #zuul | 12:58 | |
| *** jamesmcarthur has quit IRC | 13:04 | |
| tristanC | avass: that would nicely solve the caching challenge, it should be safe to share the derivations built in job, so that subsequent job could re-use any intermediate results | 13:15 |
| avass | tristanC: exactly | 13:17 |
| *** jamesmcarthur has joined #zuul | 13:17 | |
| *** ssbarnea_ has joined #zuul | 13:20 | |
| avass | it would be perfect if the default revision could be bumped on every change as well | 13:20 |
| *** zbr7 has joined #zuul | 13:21 | |
| avass | I guess it could just be left out for local development since anything merged should be stable | 13:22 |
| *** zbr has quit IRC | 13:22 | |
| *** zbr7 is now known as zbr | 13:22 | |
| *** jamesmcarthur has quit IRC | 13:24 | |
| tristanC | avass: here is an example ci powered by hydra: https://hydra.dhall-lang.org/build/81817#tabs-summary | 13:24 |
| avass | tristanC: oooh interesting | 13:25 |
| tristanC | i'm not familiar with this setup, but perhaps we can use zuul to trigger hydra build? | 13:25 |
| avass | what's hydra? | 13:26 |
| avass | oh nix ci | 13:27 |
| avass | what would be the benefit of triggering a hydra job vs running nix natively in zuul somehow? | 13:28 |
| *** zbr1 has joined #zuul | 13:31 | |
| tristanC | because it will act as a cache and it can serve individual build log | 13:32 |
| *** zbr has quit IRC | 13:33 | |
| *** zbr1 is now known as zbr | 13:33 | |
| *** jpena|lunch is now known as jpena | 13:34 | |
| *** ssbarnea_ has quit IRC | 13:34 | |
| *** jamesmcarthur has joined #zuul | 13:35 | |
| openstackgerrit | Felix Edel proposed zuul/zuul master: Simplify ZooKeeper client initialization https://review.opendev.org/c/zuul/zuul/+/754360 | 13:37 |
| openstackgerrit | Felix Edel proposed zuul/zuul master: Improve typings in context of builds via ZooKeeper https://review.opendev.org/c/zuul/zuul/+/753578 | 13:37 |
| openstackgerrit | Felix Edel proposed zuul/zuul master: Make ZooKeeper mandatory for Scheduler https://review.opendev.org/c/zuul/zuul/+/756716 | 13:38 |
| openstackgerrit | Felix Edel proposed zuul/zuul master: Make ConnectionRegistry mandatory for Scheduler https://review.opendev.org/c/zuul/zuul/+/757095 | 13:38 |
| openstackgerrit | Felix Edel proposed zuul/zuul master: Improve typings in context of 756716 and 757095 https://review.opendev.org/c/zuul/zuul/+/757148 | 13:38 |
| openstackgerrit | Felix Edel proposed zuul/zuul master: Instantiate executor client, merger, nodepool and app within Scheduler https://review.opendev.org/c/zuul/zuul/+/757149 | 13:38 |
| openstackgerrit | Felix Edel proposed zuul/zuul master: Improve typings in context of lock nodes on executor https://review.opendev.org/c/zuul/zuul/+/757097 | 13:38 |
| openstackgerrit | Felix Edel proposed zuul/zuul master: DNM: Reduce number of jobs for SOS development https://review.opendev.org/c/zuul/zuul/+/775081 | 13:38 |
| openstackgerrit | Felix Edel proposed zuul/zuul master: Component Registry in ZooKeeper https://review.opendev.org/c/zuul/zuul/+/759187 | 13:38 |
| tristanC | avass: for example, the zuul jobs would prepare and submit the hydra job, and once completed, it can pull the now cached derivation and run test on the nodepool node. | 13:38 |
| openstackgerrit | Felix Edel proposed zuul/zuul master: Move management and result events to model https://review.opendev.org/c/zuul/zuul/+/761163 | 13:39 |
| openstackgerrit | Felix Edel proposed zuul/zuul master: Allow (de-)serialization of management events https://review.opendev.org/c/zuul/zuul/+/761164 | 13:39 |
| openstackgerrit | Felix Edel proposed zuul/zuul master: Allow (de-)serialization of result events https://review.opendev.org/c/zuul/zuul/+/761165 | 13:39 |
| openstackgerrit | Felix Edel proposed zuul/zuul master: Add and fix fields in driver trigger event models https://review.opendev.org/c/zuul/zuul/+/761166 | 13:39 |
| avass | tristanC: what I don't like about that is that it requires a static build service. Imo it would be better if the cache was separate from the build system | 13:41 |
| avass | so you could still build on a dynamic node with a static cache | 13:42 |
| *** jamesmcarthur has quit IRC | 13:42 | |
| tristanC | avass: i think there is service for that, but it's not free: https://cachix.org/ | 13:43 |
| tristanC | avass: or it should be possible to setup your own cache, check the `Populating a binary cache` section of https://nixos.wiki/wiki/Binary_Cache | 13:45 |
| avass | tristanC: I thought this was it: https://nixos.wiki/wiki/Binary_Cache | 13:45 |
| avass | tristanC: yeah that's what I was thinking about | 13:45 |
| tristanC | avass: i guess the benefit of hydra is that similar job may build faster as the cache is local | 13:46 |
| avass | I think at least if I can get something set up at volvo it would probably require a cache. | 13:46 |
| avass | yeah | 13:47 |
| avass | you could also have image builds with nix caches heh :) | 13:47 |
| tristanC | i think we need an `ensure-nix` job to setup the toolchain and configure custom substitute | 13:47 |
| tristanC | role* | 13:47 |
| felixedel | corvus: I've move the initialization of the ZK connection to the server classes https://review.opendev.org/c/zuul/zuul/+/776640 and rebased our stack to the latest master. I did the change not directly on top of the connection refactoring but after https://review.opendev.org/c/zuul/zuul/+/756716/24. | 13:49 |
| felixedel | *moved | 13:49 |
| avass | oh and now that I'm thinking about it, bumping revisions would be better than following master since then it would be possible to build from every commit | 13:49 |
| avass | I guess it depends on what the requirements are | 13:50 |
| *** zbr9 has joined #zuul | 13:53 | |
| *** jfoufas1 has joined #zuul | 13:54 | |
| *** jamesmcarthur has joined #zuul | 13:55 | |
| *** zbr has quit IRC | 13:56 | |
| *** zbr9 is now known as zbr | 13:56 | |
| *** jamesmcarthur has quit IRC | 14:02 | |
| *** ykarel_ has joined #zuul | 14:15 | |
| *** jamesmcarthur has joined #zuul | 14:15 | |
| *** ykarel has quit IRC | 14:17 | |
| mordred | avass: I agree - having a separate hydra farm/service is gross when you have nodepool already. I'd think that some setup similar to how docker registries work in zuul would be really neat. that way you could also make use of speculative cache elements in dependency chains. obviously would take some engineering - but the end result would likely be really neat for zuul+nix | 14:18 |
| mordred | (we've speculated a bit about a similar need for a setup to interact with bazel cache for the folks using bazel) | 14:19 |
| *** jamesmcarthur has quit IRC | 14:22 | |
| *** zbr7 has joined #zuul | 14:22 | |
| *** zbr has quit IRC | 14:25 | |
| *** zbr7 is now known as zbr | 14:25 | |
| *** hashar has quit IRC | 14:29 | |
| *** jamesmcarthur has joined #zuul | 14:34 | |
| *** jamesmcarthur has quit IRC | 14:40 | |
| *** zbr3 has joined #zuul | 14:47 | |
| *** zbr has quit IRC | 14:49 | |
| *** zbr3 is now known as zbr | 14:49 | |
| *** ykarel_ is now known as ykarel | 14:53 | |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: multi-node debian: Update package lists before installing https://review.opendev.org/c/zuul/zuul-jobs/+/775866 | 14:53 |
| *** jamesmcarthur has joined #zuul | 14:55 | |
| openstackgerrit | James E. Blair proposed zuul/nodepool master: Require TLS https://review.opendev.org/c/zuul/nodepool/+/776286 | 14:58 |
| mordred | avass: ooh - so two things - a) the nix cache and the bazel cache seem to be at least _structurally_ quite similar - running your own both cases is "run an nginx" | 15:06 |
| tristanC | mordred: the folks at tweag wrote some rules to integrate nix in bazel: https://www.tweag.io/blog/2018-03-15-bazel-nix/ | 15:07 |
| *** rpittau is now known as rpittau|afk | 15:07 | |
| mordred | this: https://nixos.wiki/wiki/Binary_Cache#Populating_a_binary_cache - potentially provides a nice pattern - which is to have check jobs use a binary cache but not push new content in to it - and to have gate jobs actually push new cache entries when they are successful | 15:09 |
| corvus | mordred: ++ i think that integrates well with the check/gate concepts | 15:10 |
| mordred | I think it would be a really neat system | 15:10 |
| mordred | and a nice improvement to the zuul ecosystem | 15:10 |
| *** hashar has joined #zuul | 15:10 | |
| avass | yep something like that was what I was thinking :) | 15:11 |
| avass | it would work the same as the s3 cache but more specific to nix | 15:11 |
| mordred | yah | 15:11 |
| corvus | the gerrit folks would like to use more zuul for gerrit; maybe we can try out the idea with bazel there | 15:12 |
| mordred | ++ | 15:12 |
| corvus | avass: by s3 cache you mean the work you've been doing in zuul jobs? | 15:12 |
| corvus | https://review.opendev.org/764808 ? | 15:12 |
| avass | corvus: yep | 15:12 |
| mordred | and if you wanted to get fancy, you could have a buildset nix-cache/bazel-cache to allow sharing of built nix packages between multiple jobs in a buildset even in check | 15:12 |
| mordred | because spinning up a node with an nginx is ... easy | 15:13 |
| tristanC | mordred: it should be safe to share derivation built in check, the hash should be unique | 15:13 |
| corvus | tristanC: i think the worry in check is intentional poisoning | 15:13 |
| mordred | well - and also just cache size saturation | 15:14 |
| mordred | if you cache all of the derivations in check, you're potentially caching many thigns that are not useful over a given period | 15:14 |
| corvus | [if a check job has enough access to the buildset cache that it could write poisoned data to a hash that would be known to be used later, then it's a vector for producing compromised binaries] | 15:15 |
| fungi | i assume the way you'd address cache poisoning risks would be the same as for the zuul cache we'd been discussing previously | 15:15 |
| corvus | fungi: i think isolation is mostly what we discussed | 15:16 |
| fungi | yeah, not entrusting the cache write key to untrusted playbooks, only writing to the cache in post-review pipelines, et cetera | 15:17 |
| fungi | and making it per-project | 15:17 |
| fungi | were some of the options | 15:17 |
| corvus | ++ | 15:17 |
| fungi | but more generally, i think if we add or promote some cache solution, we need to make sure security concerns are first and foremost in mind | 15:18 |
| mordred | "As nix-serve is capable of serving only on IPv4, redirecting is also useful to make the binary cache available on IPv6." ... *sigh* | 15:18 |
| mordred | I swear - people writing "modern" softare that only speaks ipv4 is one of the lamest things out there | 15:19 |
| corvus | ipv6 is 25 years old | 15:19 |
| fungi | ipv6 is only for people who haven't yet embraced the ipv4-only world of containers | 15:19 |
| mordred | yeah | 15:19 |
| mordred | fungi: even containers have caught up with the 25-year-old future finally | 15:19 |
| fungi | woah! dockerhub added aaaa records? | 15:19 |
| mordred | oh - that I don't know | 15:20 |
| corvus | i feel like the container world is moving on from dockerhub | 15:20 |
| mordred | I just mean containers themselves finally learned how to get an ipv6 address | 15:20 |
| mordred | corvus: ++ | 15:20 |
| fungi | yeah | 15:20 |
| fungi | amusingly, quay.io also only publishes ipv4 addresses | 15:20 |
| mordred | well - both of them run in AWS | 15:21 |
| fungi | oh, has aws not figured out ipv6 yet? | 15:21 |
| mordred | which is not exactly a bastino of ipv6 glory | 15:21 |
| mordred | I think you can do ipv6 there - but you have to want it | 15:21 |
| fungi | having never tried aws, i did't realize it was a problem there | 15:21 |
| mordred | and it's not real ipv6 like you'd get from vexxhost - it's more like ipv6 from comcast from 6 years ago | 15:22 |
| avass | aws owns so many ipv4 cidrs that they're not really in a position to promote ipv6 either | 15:22 |
| *** icey has quit IRC | 15:32 | |
| *** icey has joined #zuul | 15:33 | |
| corvus | hrm, looks like gerrit's zuul is hitting a bunch of post_failures; i'm watching a console log now to try to catch it | 15:40 |
| *** zbr9 has joined #zuul | 15:41 | |
| corvus | i pushed up a change to see if the gerrit master job has bitrotted -- https://gerrit-review.googlesource.com/c/zuul/jobs/+/297362/ we could try iterating on a bazel cache with that change | 15:42 |
| *** zbr has quit IRC | 15:43 | |
| *** zbr9 is now known as zbr | 15:43 | |
| corvus | it looks like the checks 'reboot' work is happening in the checks plugin; i see ongoing work there | 15:44 |
| *** zbr6 has joined #zuul | 16:01 | |
| *** jamesmcarthur has quit IRC | 16:02 | |
| *** zbr6 has quit IRC | 16:02 | |
| *** zbr9 has joined #zuul | 16:03 | |
| *** zbr has quit IRC | 16:04 | |
| *** zbr9 is now known as zbr | 16:04 | |
| *** mordred has quit IRC | 16:12 | |
| openstackgerrit | Merged zuul/zuul-jobs master: Update upload-logs roles to support endpoint override https://review.opendev.org/c/zuul/zuul-jobs/+/774650 | 16:12 |
| *** mordred has joined #zuul | 16:13 | |
| *** jfoufas1 has quit IRC | 16:15 | |
| *** harrymichal has joined #zuul | 16:24 | |
| *** hashar has quit IRC | 16:41 | |
| *** ikhan has quit IRC | 16:41 | |
| *** ikhan has joined #zuul | 16:43 | |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Update GCS log upload to new google auth api https://review.opendev.org/c/zuul/zuul-jobs/+/776671 | 16:44 |
| fungi | i guess that was the post_failure cause? | 16:44 |
| corvus | yeah | 16:45 |
| corvus | http://paste.openstack.org/show/802837/ | 16:46 |
| fungi | non-backward-compatible changes in the api? | 16:46 |
| corvus | we override the method :( | 16:46 |
| corvus | i'd love to figure out a better way to do that, but i'm not seeing it right now | 16:46 |
| clarkb | they added the new param with a default so if you were calling it directly you'd be fine, but I assume something in the lib is calling it expecting to set a scope and that breaks the override | 16:46 |
| clarkb | ya the traceback seems to say ^ | 16:47 |
| fungi | yep, okay makes sense | 16:47 |
| corvus | oh, i think i may see another way of doing this | 16:49 |
| corvus | i'm going to +w that change to stop the bleeding | 16:49 |
| corvus | but then i'll see if i can do a better fix | 16:49 |
| mnaser | fiou | 16:49 |
| * mnaser though we broke opendev with upload-logs change for endpoint overrides | 16:49 | |
| tosky | yes, please | 16:49 |
| *** harrymichal has quit IRC | 16:50 | |
| *** harrymichal has joined #zuul | 16:50 | |
| corvus | nope, just google is broken; unrelated | 16:50 |
| fungi | mnaser: are you seeing a separate log upload issue in opendev? | 16:53 |
| mnaser | fungi: oh no, i just saw you mention post_failure after that merged and yeah :p | 16:53 |
| fungi | oh, sorry, the post_failure in gerrit's zuul we were discussing earlier | 16:53 |
| fungi | around 15:40 utc | 16:54 |
| fungi | so nothing related to opendev | 16:54 |
| *** jamesmcarthur has joined #zuul | 17:00 | |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: GCS logs: more robust Credential class https://review.opendev.org/c/zuul/zuul-jobs/+/776675 | 17:02 |
| corvus | fungi, clarkb: ^ i think that's slightly better pythoning. | 17:02 |
| tosky | is it expected for https://review.opendev.org/c/zuul/zuul-jobs/+/776671 to be failing in the gate queue with post_failure? | 17:03 |
| clarkb | there are a bunch of post failures | 17:06 |
| clarkb | its possible that mnaser's fear is valid but for different reasons | 17:06 |
| clarkb | https://review.opendev.org/c/zuul/zuul-jobs/+/774650 was the change in question /me looks at it | 17:06 |
| corvus | i have an error | 17:06 |
| *** jamesmcarthur has quit IRC | 17:06 | |
| corvus | http://paste.openstack.org/show/802839/ | 17:06 |
| corvus | the test i did was on a system that was not using the "quick-download" role | 17:06 |
| corvus | we should revert 774650 | 17:07 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Revert "Update upload-logs roles to support endpoint override" https://review.opendev.org/c/zuul/zuul-jobs/+/776676 | 17:08 |
| fungi | docs promotion, maybe we have an afs issue | 17:08 |
| fungi | looking | 17:08 |
| clarkb | fungi: if you look at zuul status its all the things | 17:08 |
| clarkb | I bet it is 774650 | 17:08 |
| fungi | 'dict object' has no attribute 'url' | 17:11 |
| clarkb | fungi: does it give a location for that url attribute? | 17:11 |
| clarkb | 774650 switched from using foo.url to foo.endpoint and foo.path | 17:11 |
| fungi | trying to find an actual traceback | 17:11 |
| clarkb | I'm guessing a foo.url was left behind somewhere | 17:11 |
| openstackgerrit | Merged zuul/zuul-jobs master: Revert "Update upload-logs roles to support endpoint override" https://review.opendev.org/c/zuul/zuul-jobs/+/776676 | 17:12 |
| *** jonass_ has joined #zuul | 17:13 | |
| corvus | there's no traceback, it's an ansible jinja error | 17:13 |
| fungi | The error appears to be in '/var/lib/zuul/builds/ca6d75bd1f90426aa3b28803cacd7cb5/trusted/project_0/opendev.org/opendev/base-jobs/playbooks/base/post-logs.yaml': line 30, column 7 | 17:13 |
| fungi | so maybe our base jobs are incompatible with that change? | 17:13 |
| clarkb | I see | 17:13 |
| clarkb | ya | 17:13 |
| clarkb | https://opendev.org/opendev/base-jobs/src/branch/master/playbooks/base/post-logs.yaml#L38 is the problem | 17:13 |
| *** ykarel has quit IRC | 17:14 | |
| fungi | https://opendev.org/opendev/base-jobs/src/branch/master/playbooks/base/post-logs.yaml#L38 | 17:15 |
| fungi | beat me to the punch | 17:15 |
| clarkb | mnaser: corvus ^ do we need to rewrite that using the large ansibel jinja filter at https://review.opendev.org/c/zuul/zuul-jobs/+/774650/16/roles/upload-logs-swift/tasks/main.yaml ? | 17:15 |
| fungi | we'll also have to bypas gating to fix it | 17:15 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Revert "Revert "Update upload-logs roles to support endpoint override"" https://review.opendev.org/c/zuul/zuul-jobs/+/776677 | 17:16 |
| *** aluria_ has joined #zuul | 17:16 | |
| corvus | fungi: the revert change to fix it merged at 17:12 | 17:17 |
| fungi | corvus: yep | 17:17 |
| fungi | i didn't see any of that in here until 17:16 | 17:18 |
| fungi | i have a feeling there's some massive lag between irc servers | 17:18 |
| corvus | i think there's big irc lag | 17:18 |
| clarkb | oh ok so multiple issues | 17:18 |
| clarkb | and ya I didn't see it either | 17:18 |
| corvus | clarkb, fungi, mnaser, Open10K8S: see my comment on https://review.opendev.org/776677 | 17:18 |
| *** aluria has quit IRC | 17:18 | |
| *** jonass has quit IRC | 17:18 | |
| *** mmedvede has quit IRC | 17:18 | |
| clarkb | looking | 17:18 |
| -openstackstatus- NOTICE: All jobs are failing with POST_FAILURE due to a backward incompatible change made in the swift log upload libarary role. Working on a fix now. | 17:19 | |
| *** mmedvede has joined #zuul | 17:19 | |
| clarkb | corvus: ++ to keeping the api | 17:19 |
| fungi | yes, not changing the api avoids breaking weird people like opendev who might have been relying on the old api ;) | 17:19 |
| corvus | plus, it's a good api :) | 17:20 |
| corvus | like, "emit the url the logs are at" is pretty boss. | 17:20 |
| *** aluria has joined #zuul | 17:21 | |
| *** jonass has joined #zuul | 17:21 | |
| *** 07IAANVQ9 has joined #zuul | 17:21 | |
| clarkb | it would simplify the ansible in the change too if it can avoid that large amount of jinja2 | 17:21 |
| clarkb | and keep referring to the url | 17:21 |
| corvus | it might be a bit tricky to override a deep var in a dict like that, but should be worth doing. | 17:22 |
| clarkb | corvus: can't the python just define it when it defines the endpoint and the path? | 17:22 |
| *** aluria has quit IRC | 17:22 | |
| *** jonass has quit IRC | 17:22 | |
| *** 07IAANVQ9 has quit IRC | 17:22 | |
| clarkb | set it at line 146 in https://review.opendev.org/c/zuul/zuul-jobs/+/774650/16/roles/upload-logs-base/library/zuul_swift_upload.py | 17:22 |
| avass | fungi: you guys don't upgrade to the latest version as soon as it's out? pff ;) | 17:23 |
| corvus | clarkb: yes. i suggested we *don't* do that though on ps4 of the original change. | 17:23 |
| clarkb | avass: we did thats why it broke :P | 17:23 |
| corvus | clarkb: i don't think the python code should care about endpoint overrides | 17:23 |
| clarkb | corvus: oh | 17:23 |
| corvus | it's job should be to upload logs and tell ansible where it put them | 17:24 |
| corvus | then if people want to go mucking with the url it gave them, fine, do that in ansible | 17:24 |
| corvus | i think the thing we missed really is that apparently this role has a return value api | 17:25 |
| *** mordred has quit IRC | 17:26 | |
| *** Eighth_Doctor has quit IRC | 17:26 | |
| clarkb | ya I think being able to set artifacts like that is generally useful and not entirely crazy. | 17:27 |
| clarkb | Basically jobs can provide hints to devs/users | 17:27 |
| corvus | clarkb: back to the question of whether to do it in python or ansible -- part of my motivation for asking for ansible is that there's a lot of plumbing needed to do it in python -- you have to add an extra input param to all the python modules, and it's a param they ultimately don't use except for a string replacement) | 17:27 |
| *** jpena is now known as jpena|off | 17:27 | |
| clarkb | corvus: isn't the info already known via the if cdn_url: do on thing else: do another ? | 17:28 |
| clarkb | but even then can't you just say url = join(endpoint, path) ? | 17:29 |
| corvus | clarkb: no, this is for the user to supply a replacement endpoint that isn't actually used except it's shown in the final url | 17:29 |
| clarkb | oh I see that is why the ansible uses some_var | default(other_var, true) | 17:29 |
| clarkb | its the some_var that isn't know to the python | 17:29 |
| clarkb | cdn there is the whole rax cdn thing | 17:29 |
| corvus | clarkb: yeah. essentially, it's to allow the site admin to tell the upload log role to lie about where the logs are stored so users download them through a proxy instead of the actual endpoint | 17:30 |
| clarkb | (I had conflated the cdn_url with the proxy_url and they are distinct) | 17:30 |
| avass | corvus: if you're using that as an output I think it should be set as a fact and documented. I'd argue that it isn't a 'true' output currently since it wouldn't persist across playbooks | 17:31 |
| avass | +cacheable in set_fact | 17:31 |
| corvus | avass: i agree; it is not documented as an output parameter in the readme | 17:32 |
| clarkb | But the utility is useful | 17:32 |
| clarkb | it allows jobs to provide helpful hints to people for finding useful log paths | 17:32 |
| corvus | as a referree, i would have to give opendev the yellow card here for using an undocumented output :) | 17:32 |
| corvus | an alternative would be for opendev to read the log url from the zuul json return file | 17:33 |
| fungi | the "download logs script" we have was admittedly a bit of a bolt-on | 17:34 |
| fungi | maybe an actual proxy which knows how to bundle logs on demand would be an improvement | 17:34 |
| clarkb | corvus: the zuul json return file is written when the playbook ends though? | 17:34 |
| fungi | (like, not proxy our logs normally, but have a special proxy for people who want to download a log bundle, since swift doesn't do that automagically) | 17:35 |
| clarkb | fungi: well in general I think the use case of "job points users directly at log urls that are useful" is something to figure out. And maybe a proxy can help with that too, but the job saying "where are my logs so I can do things with them" seems useful too | 17:35 |
| clarkb | I think tripleo may actually do this somewhere too | 17:35 |
| clarkb | but maybe it was just all relative links in an index file (which would also work) | 17:36 |
| corvus | clarkb: it's written when zuul_return is called | 17:36 |
| fungi | i still miss the old ftpd implementation where you could ask for a directory but tack on .tar.gz and get a compressed recursive archive of the tree from that point | 17:36 |
| corvus | fungi: we did a lot of work so that opendev doesn't have to run a log proxy that does magic things. | 17:37 |
| corvus | fungi: you may be itching for the good old days of logs.openstack.org+osla; i do not miss them. | 17:37 |
| fungi | corvus: yep, like i said, not proxy normal log urls, just add a url to artifacts which points at a bundling proxy for people who want bundles | 17:37 |
| corvus | i think the download script is great and it should be trivial to support | 17:38 |
| fungi | rather than continue endorsing people to wget|bash | 17:38 |
| clarkb | corvus: that would look like run the upload role (whcih calls zuul_return?), load the vars off disk with a json file loader taks, run a subsequent zuul_return which uses the var loaded from disk? | 17:39 |
| clarkb | having the role define it as an output and set as a fact seems more natural but ^ is probably workable too | 17:39 |
| fungi | but yes, ultimately adding a proxy service off to the side just to support the rare downloads of tree archives seems like a lot of maintenance for minimal benefit | 17:39 |
| *** mordred has joined #zuul | 17:40 | |
| *** zbr1 has joined #zuul | 17:40 | |
| corvus | clarkb: yes. it's an option. i agree a documented return value would be good. | 17:40 |
| corvus | if we're going to support it as an api, we should probably rename it to something other than "upload_results" | 17:41 |
| corvus | that actually would make the overwrite easier | 17:41 |
| corvus | keep upload_results as the temporary registered fact, then do the rewrite to "upload_logs_results_ | 17:42 |
| corvus | grr | 17:42 |
| *** zbr has quit IRC | 17:42 | |
| *** zbr1 is now known as zbr | 17:42 | |
| corvus | keep upload_results as the temporary registered fact, then do the rewrite to "zuul_upload_logs_results.url" and return that as a cacheable fact | 17:42 |
| corvus | add the new api now, switch opendev to use it, assounce the change to zuul-announce, then do the log proxy change | 17:43 |
| fungi | sgtm | 17:43 |
| corvus | (or if we want to do it immediately, do a little extra work to support .url as an attribute of "upload_results" temporarily, then remove it in a couple weeks) | 17:44 |
| clarkb | ya that sounds like a good way to consume the data in playbooks | 17:44 |
| *** Eighth_Doctor has joined #zuul | 17:44 | |
| corvus | clarkb, fungi, avass, mnaser, Open10K8S: next steps outlined in https://review.opendev.org/776677 | 17:47 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: GCS logs: more robust Credential class https://review.opendev.org/c/zuul/zuul-jobs/+/776675 | 17:50 |
| mnaser | corvus: thanks for that plan — well try and work on it, it’s always scary to work on these roles :( | 17:51 |
| avass | corvus: ++ lgtm | 17:51 |
| fungi | mnaser: beware the overheating spacebar! | 17:51 |
| fungi | https://xkcd.com/1172/ | 17:52 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: GCS logs: more robust Credential class https://review.opendev.org/c/zuul/zuul-jobs/+/776675 | 17:53 |
| corvus | clarkb, fungi: ^ can you review that real quick pls? we still need a solution for gerrit's zuul | 17:53 |
| fungi | yeop | 17:53 |
| corvus | since the first fix didn't merge, i figure let's just jump straight to the second one | 17:54 |
| clarkb | yup I guess we should do that instead of https://review.opendev.org/c/zuul/zuul-jobs/+/776671 ? | 17:54 |
| clarkb | corvus: your change creates an intereting python typing question in my head. When you call super(Credentials,.....).etc() are you getting back a Credentials or gce_cred.Credentials object? becuse _set_path is only valid on Credentials and not gce_cred.Credentials | 17:58 |
| clarkb | it depends on what self.__class__'s value is inside that call I think | 17:58 |
| clarkb | I think your code is right just puzzling through it in my head fwiw | 17:59 |
| -openstackstatus- NOTICE: The change to the upload role has been reverted. Jobs started since the revert appear to be functioning normally. You can recheck changes that failed for builds started between 16:12 and 17:13UTC reporting POST_FAILURE safely now. | 17:59 | |
| corvus | clarkb: it's the subclass | 17:59 |
| clarkb | because the self object doesn't change we're just calling the parent method on it | 17:59 |
| clarkb | if the gce code didn't use self.__class__ then it could be a problem | 18:00 |
| corvus | clarkb: yeah if it did google.auth.Credential() | 18:00 |
| clarkb | ok approved | 18:00 |
| fungi | interesting, yeah i didn't even think about needing self.__class__ indirection in the original class | 18:01 |
| *** jamesmcarthur has joined #zuul | 18:03 | |
| *** zbr7 has joined #zuul | 18:05 | |
| *** zbr has quit IRC | 18:07 | |
| *** zbr7 is now known as zbr | 18:07 | |
| *** wuchunyang has joined #zuul | 18:10 | |
| *** hashar has joined #zuul | 18:11 | |
| openstackgerrit | Merged zuul/zuul-jobs master: GCS logs: more robust Credential class https://review.opendev.org/c/zuul/zuul-jobs/+/776675 | 18:12 |
| *** wuchunyang has quit IRC | 18:15 | |
| *** jamesmcarthur has quit IRC | 18:16 | |
| corvus | okay this is really weird. the 2 nodepool-functional-container-openstack-* jobs timed out because the nodepool procesesses periodically lose their zk connection | 18:20 |
| corvus | i'm not sure why that's happening | 18:20 |
| corvus | i held a node for one of them, and it's still happenning, even though the node is under light load | 18:20 |
| clarkb | corvus: do the containers use host networking? (just to rule out any weirdness with container networking) | 18:20 |
| corvus | lets see | 18:21 |
| clarkb | I think the zk processes are on the host network and not container networks | 18:21 |
| corvus | clarkb: yes host networking | 18:21 |
| corvus | and to be clear, they do connect to zk, they just lose their connections fairly frequently | 18:21 |
| corvus | https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_82d/776286/14/check/nodepool-functional-container-openstack-release/82d81e0/docker/nodepool_nodepool-builder_1.txt | 18:22 |
| fungi | could the zk process be crashing and restarting? or split-brain/fighting over who's primary? | 18:22 |
| corvus | only one server there, and it's up continually | 18:23 |
| fungi | bizarre | 18:23 |
| clarkb | corvus: does this happen in the openstackfunctional job without containers | 18:23 |
| clarkb | (just trying to think about ways to isolate the behavior) | 18:23 |
| corvus | clarkb: nope, that passed this round | 18:24 |
| corvus | the zoo.cfg file is different than it was before since we're using ensure-zookeeper instead of test-setup.sh (which would have installed zk from ubuntu) | 18:25 |
| clarkb | could it be a python version issue? the other jobs just run on base distro python iirc. | 18:26 |
| clarkb | corvus: oh maybe a tick rate connection reset from the server side if the config is using lower values? | 18:27 |
| corvus | clarkb: something like "the python in the container images interacts poorly with kazoo and tls" ? | 18:27 |
| clarkb | ya, though that is just an idea based on difference between those two jobs | 18:27 |
| clarkb | corvus: looks like the container is running python3.7 | 18:27 |
| clarkb | but the other jobs would all be 3.6 or 3.8? | 18:27 |
| corvus | clarkb: we are running these containers in prod with tls though.... | 18:28 |
| clarkb | good point and it should be the same container with python3.7 | 18:29 |
| clarkb | (I guess we updated zuul to 3.8 but not nodepool) | 18:29 |
| corvus | i'm trying to dig up a stock ubuntu bionic zoo.cfg | 18:29 |
| corvus | http://paste.openstack.org/show/802846/ | 18:30 |
| corvus | oh | 18:30 |
| corvus | i was about to say: the only difference between that stock config and what this job was running before was that we mounted the datadir on a tmpfs. | 18:31 |
| corvus | then i realized the implications of that. :) | 18:31 |
| corvus | so, yes, there are some other differences, but that's probably the one to address first | 18:31 |
| clarkb | corvus: meaning when it worked it was on a tmpfs? | 18:31 |
| corvus | mind you, that's not a difference between this job and the non-container jobs, but maybe we're right on the edge of what these nodes can support | 18:32 |
| corvus | yeah | 18:32 |
| fungi | yeah, i suppose tmpfs writes will be a lot faster or less i/o intensive as long as there's free memory | 18:33 |
| fungi | reads will likely be about the same due to the fs cache | 18:33 |
| fungi | unless we end up under mild memory pressure and the cache is getting evacuated | 18:34 |
| *** rlandy is now known as rlandy|biab | 18:37 | |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: ensure-zookeeper: use a tmpfs https://review.opendev.org/c/zuul/zuul-jobs/+/776696 | 18:40 |
| *** nils has quit IRC | 18:42 | |
| openstackgerrit | James E. Blair proposed zuul/nodepool master: Require TLS https://review.opendev.org/c/zuul/nodepool/+/776286 | 18:42 |
| corvus | okay lets give that a shot | 18:42 |
| *** zbr2 has joined #zuul | 18:43 | |
| corvus | meanwhile, i'm rechecking a change in gerrit to see if the log situation is resolved | 18:44 |
| *** zbr has quit IRC | 18:46 | |
| *** zbr2 is now known as zbr | 18:46 | |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: ensure-zookeeper: use a tmpfs https://review.opendev.org/c/zuul/zuul-jobs/+/776696 | 18:52 |
| clarkb | corvus: should we make the use of tmpfs optional? | 18:56 |
| *** ikhan has quit IRC | 19:04 | |
| *** ikhan has joined #zuul | 19:06 | |
| *** jamesmcarthur has joined #zuul | 19:15 | |
| *** gmann is now known as gmann_afk | 19:19 | |
| *** jamesmcarthur has quit IRC | 19:20 | |
| corvus | clarkb: i can't think of a reason to do so in the zuul context | 19:25 |
| corvus | no objection to that, it just didn't seem worth all the typing right now :) | 19:25 |
| *** Eighth_Doctor has quit IRC | 19:36 | |
| *** aluria_ has quit IRC | 19:36 | |
| *** mordred has quit IRC | 19:36 | |
| *** ttx has quit IRC | 19:36 | |
| *** asettle has quit IRC | 19:36 | |
| *** johanssone has quit IRC | 19:36 | |
| *** tobberydberg has quit IRC | 19:36 | |
| *** jhesketh has quit IRC | 19:36 | |
| *** yoctozepto has quit IRC | 19:36 | |
| *** reiterative has quit IRC | 19:36 | |
| *** dmsimard has quit IRC | 19:36 | |
| *** cloudnull has quit IRC | 19:36 | |
| *** saneax has quit IRC | 19:36 | |
| *** arxcruz has quit IRC | 19:36 | |
| *** holser has quit IRC | 19:36 | |
| *** avass has quit IRC | 19:36 | |
| *** EmilienM has quit IRC | 19:36 | |
| *** irclogbot_0 has quit IRC | 19:36 | |
| *** sduthil has quit IRC | 19:36 | |
| *** gouthamr has quit IRC | 19:36 | |
| *** sshnaidm|off has quit IRC | 19:36 | |
| *** openstackgerrit has quit IRC | 19:36 | |
| *** gmann_afk has quit IRC | 19:36 | |
| *** samccann has quit IRC | 19:36 | |
| *** Open10K8S has quit IRC | 19:36 | |
| *** mgoddard has quit IRC | 19:36 | |
| *** rlandy|biab has quit IRC | 19:36 | |
| *** parallax has quit IRC | 19:36 | |
| *** stevthedev has quit IRC | 19:36 | |
| *** aprice has quit IRC | 19:36 | |
| *** corvus has quit IRC | 19:36 | |
| *** fdegir has quit IRC | 19:36 | |
| *** mnasiadka has quit IRC | 19:36 | |
| *** Tahvok has quit IRC | 19:36 | |
| *** mvadkert has quit IRC | 19:36 | |
| *** guillaumec has quit IRC | 19:36 | |
| *** ikhan has quit IRC | 19:36 | |
| *** mmedvede has quit IRC | 19:36 | |
| *** jkt has quit IRC | 19:36 | |
| *** smyers has quit IRC | 19:36 | |
| *** mhu has quit IRC | 19:36 | |
| *** noonedeadpunk has quit IRC | 19:36 | |
| *** ianychoi_ has quit IRC | 19:36 | |
| *** freefood_ has quit IRC | 19:36 | |
| *** dpawlik has quit IRC | 19:36 | |
| *** fbo has quit IRC | 19:36 | |
| *** SotK has quit IRC | 19:36 | |
| *** ericsysmin has quit IRC | 19:36 | |
| *** maxamillion has quit IRC | 19:36 | |
| *** mnaser has quit IRC | 19:36 | |
| *** mugsie has quit IRC | 19:36 | |
| *** paladox has quit IRC | 19:36 | |
| *** PrinzElvis has quit IRC | 19:36 | |
| *** erbarr has quit IRC | 19:36 | |
| *** persia has quit IRC | 19:36 | |
| *** melwitt has quit IRC | 19:36 | |
| *** kgz has quit IRC | 19:36 | |
| *** johnsom has quit IRC | 19:36 | |
| *** ofosos has quit IRC | 19:36 | |
| *** iamweswilson has quit IRC | 19:36 | |
| *** leoluk has quit IRC | 19:36 | |
| *** harrymichal has quit IRC | 19:36 | |
| *** masterpe has quit IRC | 19:36 | |
| *** logan- has quit IRC | 19:36 | |
| *** ChanServ has quit IRC | 19:36 | |
| *** jonass_ has quit IRC | 19:36 | |
| *** icey has quit IRC | 19:36 | |
| *** msuszko has quit IRC | 19:36 | |
| *** hashar has quit IRC | 19:36 | |
| *** piotrowskim has quit IRC | 19:36 | |
| *** rpittau|afk has quit IRC | 19:36 | |
| *** iurygregory has quit IRC | 19:36 | |
| *** tflink has quit IRC | 19:36 | |
| *** clarkb has quit IRC | 19:36 | |
| *** jpena|off has quit IRC | 19:36 | |
| *** ironfoot has quit IRC | 19:36 | |
| *** swest has quit IRC | 19:36 | |
| *** fungi has quit IRC | 19:36 | |
| *** zbr has quit IRC | 19:36 | |
| *** ianw has quit IRC | 19:36 | |
| *** SpamapS has quit IRC | 19:36 | |
| *** Pilou has quit IRC | 19:36 | |
| *** lyr has quit IRC | 19:36 | |
| *** guilhermesp has quit IRC | 19:36 | |
| *** bodgix has quit IRC | 19:36 | |
| *** tristanC has quit IRC | 19:36 | |
| *** amotoki has quit IRC | 19:36 | |
| *** pabelanger has quit IRC | 19:36 | |
| *** odyssey4me has quit IRC | 19:36 | |
| *** bschanzel has quit IRC | 19:36 | |
| *** etp has quit IRC | 19:36 | |
| *** evrardjp has quit IRC | 19:36 | |
| *** pots has quit IRC | 19:36 | |
| *** dcastellani has quit IRC | 19:36 | |
| *** mwhahaha has quit IRC | 19:36 | |
| *** webknjaz has quit IRC | 19:36 | |
| *** ChrisShort has quit IRC | 19:36 | |
| *** donnyd has quit IRC | 19:36 | |
| *** frickler has quit IRC | 19:36 | |
| *** felixedel has quit IRC | 19:36 | |
| *** systemd has quit IRC | 19:36 | |
| *** tosky has quit IRC | 19:36 | |
| *** gundalow has quit IRC | 19:36 | |
| *** jbryce has quit IRC | 19:36 | |
| *** Shrews has quit IRC | 19:36 | |
| *** decimuscorvinus has quit IRC | 19:36 | |
| *** jamesmcarthur has joined #zuul | 19:40 | |
| *** ikhan has joined #zuul | 19:40 | |
| *** zbr has joined #zuul | 19:40 | |
| *** hashar has joined #zuul | 19:40 | |
| *** Eighth_Doctor has joined #zuul | 19:40 | |
| *** mordred has joined #zuul | 19:40 | |
| *** mmedvede has joined #zuul | 19:40 | |
| *** aluria_ has joined #zuul | 19:40 | |
| *** jonass_ has joined #zuul | 19:40 | |
| *** harrymichal has joined #zuul | 19:40 | |
| *** icey has joined #zuul | 19:40 | |
| *** rlandy|biab has joined #zuul | 19:40 | |
| *** piotrowskim has joined #zuul | 19:40 | |
| *** johnsom has joined #zuul | 19:40 | |
| *** rpittau|afk has joined #zuul | 19:40 | |
| *** ericsysmin has joined #zuul | 19:40 | |
| *** maxamillion has joined #zuul | 19:40 | |
| *** msuszko has joined #zuul | 19:40 | |
| *** masterpe has joined #zuul | 19:40 | |
| *** tosky has joined #zuul | 19:40 | |
| *** iurygregory has joined #zuul | 19:40 | |
| *** evrardjp has joined #zuul | 19:40 | |
| *** sduthil has joined #zuul | 19:40 | |
| *** mvadkert has joined #zuul | 19:40 | |
| *** reiterative has joined #zuul | 19:40 | |
| *** gouthamr has joined #zuul | 19:40 | |
| *** ianw has joined #zuul | 19:40 | |
| *** SpamapS has joined #zuul | 19:40 | |
| *** tflink has joined #zuul | 19:40 | |
| *** dmsimard has joined #zuul | 19:40 | |
| *** jkt has joined #zuul | 19:40 | |
| *** jhesketh has joined #zuul | 19:40 | |
| *** cloudnull has joined #zuul | 19:40 | |
| *** saneax has joined #zuul | 19:40 | |
| *** sshnaidm|off has joined #zuul | 19:40 | |
| *** mnaser has joined #zuul | 19:40 | |
| *** arxcruz has joined #zuul | 19:40 | |
| *** smyers has joined #zuul | 19:40 | |
| *** mugsie has joined #zuul | 19:40 | |
| *** mhu has joined #zuul | 19:40 | |
| *** noonedeadpunk has joined #zuul | 19:40 | |
| *** clarkb has joined #zuul | 19:40 | |
| *** openstackgerrit has joined #zuul | 19:40 | |
| *** holser has joined #zuul | 19:40 | |
| *** ianychoi_ has joined #zuul | 19:40 | |
| *** ofosos has joined #zuul | 19:40 | |
| *** avass has joined #zuul | 19:40 | |
| *** pots has joined #zuul | 19:40 | |
| *** ttx has joined #zuul | 19:40 | |
| *** asettle has joined #zuul | 19:40 | |
| *** logan- has joined #zuul | 19:40 | |
| *** gundalow has joined #zuul | 19:40 | |
| *** tobberydberg has joined #zuul | 19:40 | |
| *** johanssone has joined #zuul | 19:40 | |
| *** irclogbot_0 has joined #zuul | 19:40 | |
| *** EmilienM has joined #zuul | 19:40 | |
| *** Open10K8S has joined #zuul | 19:40 | |
| *** samccann has joined #zuul | 19:40 | |
| *** gmann_afk has joined #zuul | 19:40 | |
| *** paladox has joined #zuul | 19:40 | |
| *** Pilou has joined #zuul | 19:40 | |
| *** PrinzElvis has joined #zuul | 19:40 | |
| *** iamweswilson has joined #zuul | 19:40 | |
| *** lyr has joined #zuul | 19:40 | |
| *** bodgix has joined #zuul | 19:40 | |
| *** guilhermesp has joined #zuul | 19:40 | |
| *** parallax has joined #zuul | 19:40 | |
| *** dcastellani has joined #zuul | 19:40 | |
| *** mwhahaha has joined #zuul | 19:40 | |
| *** webknjaz has joined #zuul | 19:40 | |
| *** stevthedev has joined #zuul | 19:40 | |
| *** aprice has joined #zuul | 19:40 | |
| *** corvus has joined #zuul | 19:40 | |
| *** jpena|off has joined #zuul | 19:40 | |
| *** freefood_ has joined #zuul | 19:40 | |
| *** dpawlik has joined #zuul | 19:40 | |
| *** fdegir has joined #zuul | 19:40 | |
| *** ironfoot has joined #zuul | 19:40 | |
| *** yoctozepto has joined #zuul | 19:40 | |
| *** mnasiadka has joined #zuul | 19:40 | |
| *** ChrisShort has joined #zuul | 19:40 | |
| *** donnyd has joined #zuul | 19:40 | |
| *** erbarr has joined #zuul | 19:40 | |
| *** jbryce has joined #zuul | 19:40 | |
| *** Shrews has joined #zuul | 19:40 | |
| *** Tahvok has joined #zuul | 19:40 | |
| *** fbo has joined #zuul | 19:40 | |
| *** tristanC has joined #zuul | 19:40 | |
| *** guillaumec has joined #zuul | 19:40 | |
| *** pabelanger has joined #zuul | 19:40 | |
| *** decimuscorvinus has joined #zuul | 19:40 | |
| *** systemd has joined #zuul | 19:40 | |
| *** bschanzel has joined #zuul | 19:40 | |
| *** etp has joined #zuul | 19:40 | |
| *** fungi has joined #zuul | 19:40 | |
| *** felixedel has joined #zuul | 19:40 | |
| *** ChanServ has joined #zuul | 19:40 | |
| *** odyssey4me has joined #zuul | 19:40 | |
| *** frickler has joined #zuul | 19:40 | |
| *** leoluk has joined #zuul | 19:40 | |
| *** swest has joined #zuul | 19:40 | |
| *** kgz has joined #zuul | 19:40 | |
| *** amotoki has joined #zuul | 19:40 | |
| *** melwitt has joined #zuul | 19:40 | |
| *** persia has joined #zuul | 19:40 | |
| *** SotK has joined #zuul | 19:40 | |
| *** tepper.freenode.net sets mode: +o ChanServ | 19:40 | |
| *** rlandy|biab is now known as rlandy | 19:41 | |
| *** mgoddard has joined #zuul | 19:41 | |
| *** zbr7 has joined #zuul | 19:44 | |
| *** zbr has quit IRC | 19:46 | |
| *** zbr7 is now known as zbr | 19:46 | |
| *** jamesmcarthur has quit IRC | 19:48 | |
| *** jamesmcarthur has joined #zuul | 19:48 | |
| *** jamesmcarthur has quit IRC | 19:54 | |
| *** hamalq has joined #zuul | 19:58 | |
| *** zbr0 has joined #zuul | 20:08 | |
| *** zbr has quit IRC | 20:09 | |
| *** zbr0 is now known as zbr | 20:09 | |
| clarkb | corvus: looks like one of the containers jobs just passed | 20:12 |
| corvus | be still my heart | 20:15 |
| *** jamesmcarthur has joined #zuul | 20:19 | |
| *** ikhan has quit IRC | 20:22 | |
| *** ikhan has joined #zuul | 20:22 | |
| *** zbr7 has joined #zuul | 20:28 | |
| *** jamesmcarthur has quit IRC | 20:28 | |
| *** jamesmcarthur has joined #zuul | 20:28 | |
| *** zbr has quit IRC | 20:30 | |
| *** zbr7 is now known as zbr | 20:30 | |
| *** zbr9 has joined #zuul | 20:41 | |
| *** zbr has quit IRC | 20:43 | |
| *** zbr9 is now known as zbr | 20:43 | |
| *** jamesmcarthur has quit IRC | 20:50 | |
| *** jamesmcarthur has joined #zuul | 20:50 | |
| clarkb | corvus: zuul has +1'd the require tls change in nodepool | 20:53 |
| *** jamesmcarthur has quit IRC | 20:55 | |
| clarkb | I've +2d the change with a note about a file that can be removed | 20:57 |
| *** jamesmcarthur has joined #zuul | 21:03 | |
| corvus | clarkb: thanks; i think we should keep that file (responded inline). | 21:04 |
| corvus | zuul-maint: anyone want to +3 https://review.opendev.org/776286 ? | 21:04 |
| corvus | when that merges, we can restart opendev again, then i think tag 4.0 on monday | 21:05 |
| mordred | corvus: what happens to tox runs without those new ZK vars? | 21:06 |
| corvus | mordred: they loop until timeout trying to connect to zk | 21:07 |
| mordred | nod | 21:09 |
| openstackgerrit | Clark Boylan proposed zuul/zuul master: Noop change to Dockerfile to trigger image builds https://review.opendev.org/c/zuul/zuul/+/776710 | 21:09 |
| clarkb | corvus: ^ fyi | 21:09 |
| corvus | +3 | 21:10 |
| *** jamesmcarthur has quit IRC | 21:11 | |
| clarkb | thanks | 21:11 |
| *** jamesmcarthur has joined #zuul | 21:12 | |
| openstackgerrit | Merged zuul/zuul-jobs master: ensure-zookeeper: use a tmpfs https://review.opendev.org/c/zuul/zuul-jobs/+/776696 | 21:15 |
| *** jamesmcarthur has quit IRC | 21:17 | |
| clarkb | I'm going to pop out for a bike ride while we wait on image builds and such | 21:22 |
| *** zbr3 has joined #zuul | 21:27 | |
| *** zbr has quit IRC | 21:29 | |
| *** zbr3 is now known as zbr | 21:29 | |
| fungi | i approved the require tls change for nodepool | 21:32 |
| fungi | dinner time, but will be around in a while for an opendev service restart | 21:32 |
| *** zbr6 has joined #zuul | 21:35 | |
| *** zbr has quit IRC | 21:38 | |
| *** zbr6 is now known as zbr | 21:38 | |
| *** jamesmcarthur has joined #zuul | 21:41 | |
| *** jamesmcarthur has quit IRC | 21:54 | |
| *** hashar has quit IRC | 22:05 | |
| *** jamesmcarthur has joined #zuul | 22:09 | |
| *** zbr7 has joined #zuul | 22:10 | |
| *** zbr has quit IRC | 22:13 | |
| *** zbr7 is now known as zbr | 22:13 | |
| *** rlandy has quit IRC | 22:13 | |
| *** jamesmcarthur has quit IRC | 22:15 | |
| corvus | mordred: i ran a gerrit build and told it to use a local (empty) bazel cache, then ran a find on that and here's what it produced: https://ci.gerritcodereview.com/t/gerrit/build/bf79fb37cb784824ab93d3e72415e537/log/job-output.txt#3292 | 22:17 |
| corvus | we can do that for "good" builds, then upload that to gcs and use gcs as a read-only cache for untrusted builds | 22:19 |
| mordred | yeah | 22:20 |
| corvus | what would be really cool is if we could then use a read-only gcs cache for the good builds too, then only get a smaller delta we'd need to upload | 22:20 |
| corvus | (why not just use gcs as a read-write cache for good builds? because the google folks don't want to issue service account tokens, so we have to do weird stuff to write to gcs -- that log upload stuff from earlier being an example) | 22:21 |
| mordred | corvus: there's a part of me that thinks that the zuul-registry architecture of having upstream cache sources is *almost* applicable | 22:21 |
| mordred | but that might just be the "large pile of content addressible storage blobs" talking to me | 22:22 |
| openstackgerrit | Merged zuul/zuul master: Noop change to Dockerfile to trigger image builds https://review.opendev.org/c/zuul/zuul/+/776710 | 22:22 |
| corvus | mordred: yeah, well if bazel happens to allow both a read-only and read-write cache, that'd be great. i think we need a tiered cache. so read from remote if present, write to local if not. that can *probably* be implemented with apache/nginx if necessary. | 22:23 |
| mordred | yeah | 22:24 |
| *** zbr1 has joined #zuul | 22:24 | |
| *** zbr has quit IRC | 22:27 | |
| *** zbr1 is now known as zbr | 22:27 | |
| *** jamesmcarthur has joined #zuul | 22:27 | |
| fungi | overlayfs? | 22:29 |
| fungi | no idea if the "cache" is expected to be a posix fs or not | 22:29 |
| fungi | oh, the apache/nginx mention makes me think it has some sort of network cache protocol | 22:30 |
| fungi | so nevermind | 22:30 |
| corvus | fungi: yeah, the network protocol is http get/put, so overlayfs on a webdav mount might actually work :) | 22:31 |
| fungi | d'oh | 22:31 |
| * fungi had somehow repressed all memories of webdav | 22:31 | |
| *** jamesmcarthur has quit IRC | 22:34 | |
| openstackgerrit | Merged zuul/nodepool master: Require TLS https://review.opendev.org/c/zuul/nodepool/+/776286 | 22:35 |
| corvus | \o/ ^ | 22:37 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Support credentials supplied by nodepool https://review.opendev.org/c/zuul/zuul/+/774362 | 22:45 |
| *** jamesmcarthur has joined #zuul | 22:47 | |
| *** zbr7 has joined #zuul | 22:49 | |
| *** zbr has quit IRC | 22:51 | |
| *** zbr7 is now known as zbr | 22:51 | |
| *** zbr1 has joined #zuul | 22:54 | |
| corvus | mordred: i think several (~4) minutes of the gerrit job may be the git repo prep (*after* zuul is done with it). i think we might be able to improve that by making that role just a single giant shell script. | 22:55 |
| corvus | i suspect the ansible task overhead is getting us there | 22:55 |
| mordred | ++ | 22:55 |
| *** zbr has quit IRC | 22:55 | |
| *** zbr1 is now known as zbr | 22:55 | |
| mordred | I've become disillusioned with the concept of a lot of the ansible modules vs just using it to run shell scripts | 22:56 |
| mordred | the ansible task overhead is horrible - and honestly a lot of the time it's easier to read the shell. NOW - there are times, like your ansible REST stuff - where the ansible is super clear and that's awesome | 22:56 |
| corvus | yeah, i'm thinking we should aim for a smarter use of ansible. where we write *highly tuned* ansible that has as few tasks as possible for boilerplate pre/post playbooks that run all the time | 22:57 |
| mordred | corvus: by git repo prep, you mean the all the submodule stuff in this case right? | 22:58 |
| corvus | mordred: yep | 22:58 |
| mordred | corvus: ++ | 22:58 |
| corvus | mordred: in this case there's about 10 tasks for each repo | 22:58 |
| mordred | yeah. that's too many | 22:58 |
| mordred | it's not generalizable into "zuul supports submodules in the following manner..." yet is it? | 22:59 |
| corvus | (and that's *after* the standard zuul git repo prep, which could use the same kind of haircut) | 22:59 |
| corvus | mordred: i think it's pretty close.... | 23:00 |
| mordred | I wish the debug cycle on python modules was nicer - I'd say "we should encode these important pieces of functionality into some nice python" - but I'd still vote for shell in this case | 23:00 |
| corvus | mordred: the entire job takes 14m, the gerrit build with no bazel cache takes 6m | 23:02 |
| corvus | honestly, it's like 50/50 as far is which one is a better win to optimize | 23:02 |
| *** ikhan has quit IRC | 23:03 | |
| mordred | how long is the gerrit build with cache? | 23:04 |
| fungi | yeah, i looked at the timing profile of the git prep role recently and there seems to be a ton of overhead, particularly from the looping | 23:04 |
| fungi | it's not super noticeable for one or a handful of repos, but we've got some jobs which have >100 required-projects entries and it does stack up | 23:04 |
| corvus | mordred: i don't have a cache yet | 23:04 |
| mordred | if we make the git prep role more efficient, that's a win for opendev and gerrit and bmw and softwarefactory | 23:04 |
| mordred | corvus: good point | 23:04 |
| fungi | even if we compressed it all into a single loop it would probably be a significant improvement. right now it re-loops over the required-projects list multiple times for different activities | 23:05 |
| corvus | fungi, mordred: on my list is taking a look at https://review.opendev.org/740005 and maybe taking that opportunity to expand the use of shell scripts (it's already starting to head in that direction) | 23:06 |
| corvus | then we'd have the grand-unified-shell-script-for-git-prep-on-all-platforms | 23:06 |
| fungi | i'll just call that gussfgpoap | 23:07 |
| *** jamesmcarthur has quit IRC | 23:07 | |
| *** jamesmcarthur has joined #zuul | 23:07 | |
| mordred | obviously | 23:07 |
| mordred | looking at roles/prepare-gerrit-repos real quick ... | 23:09 |
| mordred | do we have any thoughts on how to sanely deal with stuff like loop: "{{ zuul.projects.values() | list }}" in shell? | 23:10 |
| corvus | mordred: i bet we could do some jq fanciness. but also, i think a good start might be to keep that loop but then replace everything in the file it includes with a shell | 23:11 |
| fungi | stuff zuul.projects into a structure in an envvar and parse in shell? good question though | 23:11 |
| fungi | i agree serialization will be key for some of this | 23:11 |
| fungi | and yeah, jq is pretty awesome | 23:11 |
| clarkb | corvus: that sounds like something out of hitchikers guide | 23:12 |
| mordred | yeah. that would be a good first start - but it also might be the case with some of these that, even with the bad debug cycle, python module would be a better choice. | 23:12 |
| clarkb | I know the overhead to run ansibel tasks has come up before. Is this something we should maybe talk to upstream ansible about too? | 23:12 |
| mordred | (mostly just scanning the inner file and there's a bunch of processing of the zuul dict | 23:12 |
| fungi | passing a structure to a python module which is invoked once is probably still reasonable | 23:12 |
| fungi | re-invoking python over and over in a tight loop, less so | 23:12 |
| mordred | clarkb: we have - they have not been as receptive as one might otherwise hope | 23:13 |
| clarkb | as a "hey this is noticeably painful in our CI environments is this fixable in ansible itself" type of engagement | 23:13 |
| corvus | mordred: point. the debug cycle isn't terrible if you set it up like i did the log upload stuff -- make it invokable from the cli. you can get 99% code coverage that way. | 23:13 |
| clarkb | :( | 23:13 |
| clarkb | corvus: ++ invocable from command line makes a huge difference | 23:13 |
| fungi | what we really need is "compiled ansible" | 23:13 |
| mordred | clarkb: the biggest issue was the move from threading to multiprocessing | 23:13 |
| * fungi calls "not it" for writing a jit ansible compiler | 23:14 | |
| *** zbr5 has joined #zuul | 23:19 | |
| *** gmann_afk is now known as gmann | 23:21 | |
| *** zbr has quit IRC | 23:21 | |
| *** zbr5 is now known as zbr | 23:21 | |
| clarkb | is the plan to try and restart zuul and nodepool today or monday? | 23:22 |
| * clarkb is around to help both days but didn't want to weekend early if that was still on the cards for today | 23:22 | |
| *** zbr2 has joined #zuul | 23:23 | |
| *** zbr has quit IRC | 23:25 | |
| *** zbr2 is now known as zbr | 23:25 | |
| corvus | i'd like to restart today then release on monday | 23:29 |
| clarkb | wfm, I'll be around to help out too then | 23:29 |
| corvus | looks like it's a good time; lemme make sure all the images are updated | 23:29 |
| corvus | the promote jobs work for both the zuul and nodepool changes, so i assume we're good there | 23:30 |
| corvus | -> #opendev | 23:30 |
| *** zbr1 has joined #zuul | 23:43 | |
| mordred | corvus: remote: https://gerrit-review.googlesource.com/c/zuul/jobs/+/297442 WIP Replace ansible loops and tasks with a python module [NEW] | 23:44 |
| mordred | VERY WIP - just starting taking a stab at setting something up | 23:44 |
| *** zbr has quit IRC | 23:45 | |
| *** zbr1 is now known as zbr | 23:45 | |
| corvus | mordred: okay whew! i'm guessing "implement process_repo()" is the next task? :) | 23:47 |
| mordred | corvus: yeah ... # TODO: implement | 23:48 |
| *** hamalq has quit IRC | 23:50 | |
| *** zbr8 has joined #zuul | 23:52 | |
| corvus | mordred: you have performed a DOS on gerrit's zuul with that change... we only have 4 nodes available... maybe wait till it's working to push up the next rev? :) | 23:52 |
| *** zbr has quit IRC | 23:54 | |
| *** zbr8 is now known as zbr | 23:54 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!