| openstackgerrit | Jamie Lennox proposed openstack-infra/zuul feature/zuulv3: Expose buildset to the executor and url formatter https://review.openstack.org/463457 | 01:06 |
|---|---|---|
| openstackgerrit | Jamie Lennox proposed openstack-infra/zuul feature/zuulv3: Add tenant to url formatting. https://review.openstack.org/463628 | 01:06 |
| *** jamielennox is now known as jamielennox|away | 02:04 | |
| *** jamielennox|away is now known as jamielennox | 02:15 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Allow per-repo selection of configuration classes to load https://review.openstack.org/472009 | 02:36 |
| jeblair | SpamapS, mordred, pabelanger: ^ that is the last thing on my list of things that must be done before we start to scale out use. :) | 02:38 |
| jeblair | (turns out it wasn't a breaking change after all) | 02:38 |
| mordred | jeblair: yay! | 02:47 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Allow per-repo selection of configuration classes to load https://review.openstack.org/472009 | 02:49 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Allow per-repo selection of configuration classes to load https://review.openstack.org/472009 | 03:32 |
| openstackgerrit | Jamie Lennox proposed openstack-infra/zuul feature/zuulv3: Expose buildset to the executor and url formatter https://review.openstack.org/463457 | 03:53 |
| openstackgerrit | Jamie Lennox proposed openstack-infra/zuul feature/zuulv3: Add tenant to url formatting. https://review.openstack.org/463628 | 03:53 |
| *** smyers has quit IRC | 04:18 | |
| *** smyers has joined #zuul | 04:52 | |
| openstackgerrit | Jamie Lennox proposed openstack-infra/zuul feature/zuulv3: Strip not split the untrusted_wrapper https://review.openstack.org/472066 | 05:04 |
| *** jamielennox is now known as jamielennox|away | 05:12 | |
| *** jamielennox|away is now known as jamielennox | 05:19 | |
| *** jamielennox is now known as jamielennox|away | 05:56 | |
| *** jamielennox|away is now known as jamielennox | 06:03 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool feature/zuulv3: Add webapp port and listen_address configuration https://review.openstack.org/472128 | 07:54 |
| *** hashar has joined #zuul | 08:11 | |
| *** isaacb has joined #zuul | 08:34 | |
| *** jkilpatr has joined #zuul | 10:59 | |
| *** jkilpatr has quit IRC | 11:00 | |
| *** jkilpatr has joined #zuul | 11:00 | |
| *** dougbtv_ has quit IRC | 11:08 | |
| mordred | jeblair: yah - I shoud have mentioned - the 5th patch doesn't work | 11:25 |
| mordred | jeblair: I do not know WHY the 5th patch doesn't wor | 11:29 |
| mordred | work | 11:29 |
| mordred | jlk: thought for the gerrit refUpdated event question ... I wonder if we could ask the mergers. | 11:32 |
| mordred | jamielennox: I swear - in https://review.openstack.org/#/c/463457/5/zuul/executor/client.py the dict has tenant= and project= ... and it NEVER gets unfunny to me in my head | 11:36 |
| mordred | although I suppose it's https://review.openstack.org/#/c/463628 that's the actual funny one | 11:37 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Extend in-repo config update support to github https://review.openstack.org/471477 | 11:41 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Re-enable test_worker_update_metadata https://review.openstack.org/468667 | 11:42 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Re-enable test_merge_failure_reporters https://review.openstack.org/468668 | 11:42 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Fix race in test_crd_gate_unknown https://review.openstack.org/471833 | 11:43 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Move log streaming to share with zuul_console https://review.openstack.org/471921 | 11:45 |
| Shrews | mordred: yeah, 1 copy of the streaming code seems logical | 12:24 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool feature/zuulv3: Add webapp port and listen_address configuration https://review.openstack.org/472128 | 12:36 |
| *** jroll has left #zuul | 12:52 | |
| mordred | Shrews: if only it worked :) | 13:00 |
| *** yolanda has quit IRC | 14:24 | |
| jlk | mordred: right, it'd take some git activity to figure out what changed, since gerrit doesn't expose it in the data. | 14:24 |
| *** yolanda has joined #zuul | 14:28 | |
| *** GheRivero has joined #zuul | 14:36 | |
| jeblair | SpamapS, pabelanger: can you take a look at https://review.openstack.org/472009 ? | 14:41 |
| *** dkranz has joined #zuul | 14:41 | |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Add a test which exercises a speculative role checkout https://review.openstack.org/462677 | 14:57 |
| *** hashar has quit IRC | 15:17 | |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Add tenant to url formatting. https://review.openstack.org/463628 | 15:25 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Expose buildset to the executor and url formatter https://review.openstack.org/463457 | 15:26 |
| jeblair | i'm heading out to an appointment, biab. | 15:49 |
| SpamapS | jeblair: reviewing 472009 .. I don't see how the tenant-parser fixture ever gets used. | 16:00 |
| SpamapS | pabelanger: tox-cover doesn't actually copy the coverage report to the logs server btw. | 16:04 |
| *** isaacb has quit IRC | 16:07 | |
| *** jkilpatr_ has joined #zuul | 16:33 | |
| *** jkilpatr has quit IRC | 16:36 | |
| mordred | SpamapS, pabelanger: we have several artifact copying things I think we can do better on :) | 16:50 |
| SpamapS | mordred: that reminds me that we need to spec out the "drop a json file that zuul reads or feeds to dependent jobs" thing | 16:57 |
| SpamapS | I think things like golang will need it | 16:57 |
| SpamapS | so you can build an artifact, drop it somewhere, and then feed that location to the dependent jobs | 16:58 |
| SpamapS | CD will want that too | 16:58 |
| jlk | speaking of future looking | 16:58 |
| mordred | SpamapS: yes - we do need to do that | 16:59 |
| SpamapS | I want to also get it written down so I can be present-focused and get this stuff done. :-P | 16:59 |
| * jlk needs to find that presentation he saw regarding azure | 16:59 | |
| mordred | SpamapS: speaking of - this: https://review.openstack.org/#/c/471891 is in that general area and is ready for review ;) | 16:59 |
| mordred | (well, and the three ones it depends on) | 17:01 |
| mordred | SpamapS: I mean, it's not actually related - but it's in the area of executing ansible - so it's MILDLY related ;) | 17:01 |
| jlk | https://www.linuxfestnorthwest.org/2017/sessions/continuous-delivery-microsoft-azure-docker-through-codeship was the session, it kind of explained a bit of how codeship worked, and how resources could be created in one step and used in later steps, somewhat visually. It was an interesting watch. | 17:04 |
| jlk | but I can't find easy docs that explain it in a quick search. | 17:04 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Strip not split the untrusted_wrapper https://review.openstack.org/472066 | 17:07 |
| * jlk peeks at monty's stack | 17:07 | |
| mordred | also - if anybody can spot why patch 5 in that stack doesn't work, I will give you a fatted calf | 17:07 |
| jlk | I feel like "daemon-stamp" is the Unix version of a "tramp-stamp" | 17:10 |
| mordred | jlk: ++ | 17:10 |
| openstackgerrit | Clint 'SpamapS' Byrum proposed openstack-infra/zuul feature/zuulv3: Add coverage artifacts to .gitignore https://review.openstack.org/472353 | 17:16 |
| clarkb | fwiw I really really wish we had a better story for shared storage in $clouds | 17:16 |
| clarkb | a manilla filesystem with a bunch of RO clients or even a cinder volume with a bunch of RO attachers would solve these problems fairly neatly if we constraints a "flow" of jobs to a region | 17:17 |
| SpamapS | clarkb: what's your issue with swift? | 17:17 |
| clarkb | SpamapS: it requires a lot more overhead to use and is deployed in clunky ways (like at rackspace) | 17:17 |
| clarkb | SpamapS: all of a sudden to use swift properly you have to figure out dns for every container for example | 17:17 |
| SpamapS | That overhead is client side, for sure, but it should result in extremely low costs overall. | 17:17 |
| SpamapS | dns for containers wut? | 17:18 |
| clarkb | its also not in every compute cloud | 17:18 |
| * SpamapS is a little naive on this | 17:18 | |
| clarkb | SpamapS: yes that is how rackspace has deployed their swift "public" access requires CDN which requires DNS records | 17:18 |
| SpamapS | I've used rackspace's and softlayers swift primarily.. they both worked about the same. | 17:18 |
| SpamapS | Not in every compute cloud isn't a problem though. | 17:18 |
| SpamapS | I mean it's faster if it's closer. | 17:18 |
| clarkb | SpamapS: it is if someone wants to deploy zuul and they don't have a swift | 17:19 |
| clarkb | granted manialla and cinder aren't there either | 17:19 |
| jlk | swift is a HUGE cost to implement | 17:19 |
| SpamapS | There are a few big swifts | 17:19 |
| SpamapS | all with nice fat pipes | 17:19 |
| SpamapS | and low prices | 17:19 |
| SpamapS | also you can s3 each cheaper | 17:19 |
| SpamapS | (which isn't a terrible idea) | 17:19 |
| clarkb | SpamapS: s3 each cheaper? | 17:19 |
| SpamapS | s/each/even/ | 17:19 |
| clarkb | ovh is cheaper than s3 fwiw | 17:19 |
| clarkb | like significantly | 17:20 |
| SpamapS | Oh that's great to hear | 17:20 |
| SpamapS | I have not looked at their swift | 17:20 |
| SpamapS | but I imagine their region mix may not be as good as s3 | 17:20 |
| clarkb | they are cheaper for compute too :) | 17:20 |
| SpamapS | I knew their compute was cheaper. Which is \o/ btw. :) | 17:20 |
| clarkb | this message comes to you from a $3.50/month compute instance :) | 17:20 |
| SpamapS | Do they offer native v6? | 17:21 |
| jlk | I should consider OVH when I am forced off of the blue box cloud I have things parked on | 17:21 |
| clarkb | SpamapS: they do | 17:21 |
| SpamapS | because I want to v6 my stuff but my current very cheap bro-priced VPC that I run all my stuff on is dragging their feet on v6 | 17:21 |
| clarkb | SpamapS: its a little weird to setup as its through their legacy dashboard that you get the info | 17:21 |
| clarkb | but it works | 17:21 |
| SpamapS | and I'm too lazy to make a tunnel :-P | 17:21 |
| clarkb | and you have to statically config it on the instance after looking at their dashboard and docs | 17:22 |
| SpamapS | clarkb: that's always a problem for network subnet allocation though, since Neutron doesn't have a "buy more addresses" feature that makes sense. | 17:22 |
| SpamapS | oh | 17:22 |
| clarkb | so clunky but functioanl. I poked them about it in boston as something i'd like to be easier | 17:22 |
| SpamapS | That weird. | 17:22 |
| clarkb | SpamapS: uh | 17:22 |
| SpamapS | I was hoping it would just get you a v6 subnet to use in neutron. | 17:22 |
| clarkb | neutron absolutely has a featuer that makes sense for that | 17:22 |
| clarkb | its quite awesome | 17:22 |
| clarkb | subnetpools | 17:22 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/zuul feature/zuulv3: Correct location of ansible log file https://review.openstack.org/472355 | 17:22 |
| Shrews | mordred: not sure if that ^^^ affects any work you are doing | 17:23 |
| clarkb | was just playing with it in citycloud, its such a nice way to set this up | 17:23 |
| SpamapS | in 10 minutes, clarkb has taught me more about the state of OpenStack's APIs than I have learned in 1 year of skimming the ML and docs. ;-) | 17:23 |
| mordred | Shrews: it'll conflict with patch 5, but that doesn't work anyway | 17:23 |
| SpamapS | clarkb: that's pretty awesome | 17:23 |
| clarkb | basically you say "give me a subnet and allocate it from the global ipam" | 17:23 |
| SpamapS | somebody should shake their fist at OVH and tell them to do that. | 17:23 |
| mordred | yup. it's amazing. we need more people to enable it in their deployments | 17:23 |
| clarkb | and neutron hands you back a /24 or /64 or whatever its configured to give out | 17:23 |
| clarkb | then you add it to a network and a router and bam | 17:24 |
| mordred | Shrews: but it shouldn't impact the others | 17:24 |
| jlk | I'd get Blue Box to do that, but WELP. | 17:25 |
| mordred | oh - well, it'll conflict with one of the othres - but in a minor way | 17:25 |
| mordred | super easy to fix/align whichever lands first | 17:25 |
| clarkb | anyways using swift is significantly more overhead to use than a filesystem or even a block device imo. | 17:25 |
| Shrews | mordred: well, your changes conflict with what I have locally, so I'll call it even | 17:25 |
| clarkb | I think its great if you are able to build an entire application about it and need globally distributed redundant files | 17:25 |
| mordred | it's possible that once we get into artifact transfer between jobs that it'll be more useful for us (since it's a targetted use case we totally control) than it was for log publication | 17:27 |
| SpamapS | clarkb: from a node perspective, yes, much simpler to do shared filesystem. From a cloud perspective, much cheaper to support simple object storage like s3/swift/etc. | 17:27 |
| mordred | although we _also_ need to have support for simple shared fs so that local zuul deployments don't grow an object store hard requirement | 17:27 |
| SpamapS | Less shared infrastructure, mountains of fault tolerance, and very few guarantees to support. | 17:27 |
| SpamapS | And for CI needs, I actually think it's the perfect fit. | 17:28 |
| clarkb | mordred: right thats what I was saying | 17:28 |
| clarkb | basically can't rely on object storage existing for $user | 17:28 |
| mordred | yup | 17:28 |
| SpamapS | But if swift providers are standing in their own way... :( | 17:28 |
| clarkb | SpamapS: just double checked ovh swift is half the price of s3 per GB and about the same on network transfer $.011/GB ovh vs $.01/GB s3 | 17:28 |
| clarkb | I know they have been active with upstream swift dev too so they likely understand the system very well | 17:29 |
| SpamapS | and I do agree that if you have like, one exector and 1 cloud and you just want to scp files between jobs, that should work w/o a swift. | 17:29 |
| SpamapS | executor | 17:29 |
| SpamapS | which is basically how the jenkins artifacts plugin works | 17:29 |
| mordred | SpamapS: yah. I think the MOST important thing is that we have a good interface for expressing artifacts to base jobs | 17:30 |
| clarkb | another difficult thing with swift (and maybe I just haven't spent enough time writing tooling to do it :/) is quickly auditing your utilization of the resource | 17:30 |
| mordred | then if the transfer mechanism is swift or rsync or whatever, its' all good | 17:30 |
| clarkb | you basically have to maintain your own index for everything production like | 17:30 |
| SpamapS | mordred: yeah, it's more important that we abstract this so you just write a job and put files somewhere and then zuul and the playbooks make artifacts appear. :) | 17:30 |
| mordred | clarkb: utilization in terms of "how much do I have stored?" | 17:30 |
| mordred | SpamapS: EXACTLY | 17:31 |
| clarkb | mordred: how do I have stored, who/what stored it, when will it be expired, etc | 17:31 |
| clarkb | mordred: at a container level you can get really high level aggregate info | 17:31 |
| mordred | clarkb: yup. but for moar deets you need to walk the tree | 17:31 |
| clarkb | but otherwise you have to basically list every single object, collect the metadata from each, and then do things with it | 17:31 |
| mordred | clarkb: to be fair - youhave to do that with filesystems too | 17:31 |
| clarkb | vs say `find` | 17:31 |
| clarkb | not really | 17:32 |
| clarkb | at a syscall level you might do the equivalent, but its basically write a one liner in find and done | 17:32 |
| mordred | I mean, we could write a find equiv for swift in short order if we wanted to | 17:32 |
| clarkb | the hard work is already solved for you | 17:32 |
| jlk | What if we just stuff objects as base64 encoded "strings" in zookeeper, so that they can be grabbed by any zoo client? (I'm not serious) | 17:32 |
| mordred | right. I meant at the syscall level | 17:32 |
| clarkb | mordred: right I'm saying I shouldn't have to :P | 17:32 |
| SpamapS | Travis has a nice mechanism for utilizing S3 for artifacts btw. You just config an environment variable on your github repo and they use it to push to S3. And then you setup an expire policy in S3 for those creds + that bucket and viola! you have a rolling bucket of the last few months of artifacts. | 17:32 |
| mordred | clarkb: sure :) | 17:32 |
| SpamapS | Of course, Travis doesn't have pipelines | 17:33 |
| SpamapS | so those artifacts are for "I wonder why that job failed in that way, I want to look at the binaries / test logs / etc" | 17:33 |
| mordred | SpamapS: I imagine we could set up that exact thing | 17:33 |
| mordred | should we desire | 17:33 |
| jlk | we could, that's useful for end user storage and retrieval of artifacts | 17:33 |
| jlk | but not for system proliferation of artifacts during builds | 17:33 |
| mordred | agree | 17:33 |
| SpamapS | so unfortunate that the words 'swift' and 'container' are not unique to OpenStack's swift :-P | 17:34 |
| SpamapS | https://docs.openstack.org/developer/swift/overview_expiring_objects.html btw | 17:34 |
| clarkb | SpamapS: how do you browse that though? | 17:34 |
| clarkb | SpamapS: ^ this was the problem we ran into first when we tried swifting all the build artificats | 17:34 |
| SpamapS | clarkb: the bucket is setup for public read access | 17:34 |
| clarkb | its 12TB compressed for ~6 weeks now | 17:35 |
| clarkb | so its basically unbrowsable on its own | 17:35 |
| mordred | SpamapS: browsing/nagivating is the bigger problem - although the auto-expire is cool | 17:35 |
| clarkb | (and we did use auto expire and it was a nice feature indeed) | 17:35 |
| pabelanger | SpamapS: ya, likely need to add the copy report part still. I can look at that shortly | 17:37 |
| mordred | SpamapS: browse is made harder for us because given job uploads its logs independently | 17:37 |
| mordred | SpamapS: so while a job can produce an index for the files it's uploading - there is nothing in a position to do so for browsing all the jobs for a given change | 17:38 |
| mordred | SpamapS: (this is one of those things where notmyname would say "oh, you can just..." and then we'd work on it with him for a bit then he'd be like "oh, gotcha... yah, you're going to have to implement a layer on top" | 17:38 |
| jlk | mordred: I have one immediate theory as to why patch 5 fails, we may be totally unprepared for use of Ansible 2.3 instead of 2.0.0. There's a fair amount of delta between the two. Our module replacements and plugins may be failing. | 17:40 |
| jlk | (I haven't looked at the logs) | 17:40 |
| SpamapS | mordred: interesting | 17:41 |
| mordred | jlk: unfortunately it fails local testing with simple playbooks - so it's more fundamental | 17:41 |
| jlk | k | 17:41 |
| mordred | jlk: the BEST part is that a previous iteration of that patch did work ... | 17:41 |
| jlk | eww | 17:41 |
| mordred | jlk: but when I changed it from using if statements in the class to a class with 2 subclasses to contain the logic differences, it just magically broke | 17:41 |
| mordred | BUT - if I run the remote streamer by hand in the foreground, the code works | 17:42 |
| mordred | maybe if I come back to it monday it'll magically make sense | 17:42 |
| jeblair | back and catching up on scrollback | 17:44 |
| SpamapS | mordred: I suppose zuul could append each job index to a separate index container as it finds job indexes.. then you'd be able to just list all the indexes relatively easily and cheaply..? | 17:44 |
| mordred | SpamapS: but then zuul would have to scan the containers for content - and without a secondary index that's actually quite expensive - because there is a limit on the number of top-level containers you can use (this doesn't wind up being one-container-per-job or anything) | 17:45 |
| mordred | SpamapS: also, it's not zuul doing any of this - but jobs themselves - so there's still no way for a given job to know "I'm the last job that was executing, I should take on the job of updating the generated index" | 17:46 |
| SpamapS | err no, not scan. I'm saying, every time it uploads to 'artifacts-container' it just dumps a uniquely named reference to that upload in 'artifacts-index'. | 17:47 |
| mordred | SpamapS: sorry - I was talking about logs | 17:48 |
| mordred | SpamapS: for artifacts I think this is MUCH more straight forward | 17:48 |
| SpamapS | what's different about logs that I'm missing? | 17:48 |
| mordred | SpamapS: hyumans need to be able to browse and discover logs | 17:48 |
| mordred | jobs know which artifacts they need | 17:48 |
| SpamapS | they do? I thought they found logs by reporters? | 17:48 |
| mordred | SpamapS: that is one of the ways - there are other logs that are found by browsing/navigating | 17:49 |
| mordred | and also in the current system you can work your way back up a level and look at related builds | 17:49 |
| SpamapS | also.. btw.. this discussion led me to find out that gearmand had been spewing the upload-only S3 keys into every travis job log... so I was hesitant to point out how it worked until now... rotated keys and encrypted in travis now... so I can paste this link and show how artifact uploads work: | 17:50 |
| jeblair | mordred, clarkb: SpamapS had the idea of a "cleanup" job which i think we can implement fairly easily in zuul. that lets us build swift container pipelines in zuul pretty easily (create a container -> fan out to many jobs -> coalesce to cleanup job and remove container (or make index, or whatever) | 17:50 |
| SpamapS | https://travis-ci.org/gearman/gearmand/jobs/240043868 | 17:50 |
| mordred | SpamapS: for instance: http://logs.openstack.org/91/471891/ shows you both revisions of 471891 | 17:50 |
| mordred | SpamapS: and http://logs.openstack.org/91/471891/2/check/ shows you all of the jobs for patchset 2 | 17:50 |
| SpamapS | mordred: Neat. I've never used that or had occasion to. | 17:51 |
| SpamapS | Not to say I won't now that I know it exists :) | 17:51 |
| mordred | SpamapS: :) | 17:51 |
| SpamapS | I've always found logs by looking in gerrit comments. | 17:51 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Allow per-repo selection of configuration classes to load https://review.openstack.org/472009 | 17:51 |
| mordred | jeblair: ++ | 17:52 |
| jeblair | SpamapS: you were right, i missed 'git add' on the file that added all the new tests ^ :) | 17:52 |
| mordred | SpamapS: but I think th emodel they're doing for artifacts there would also work for us and not be terribly hard | 17:52 |
| SpamapS | jeblair: oh good, I'm not crazy :) | 17:52 |
| jeblair | not what i said ;) | 17:53 |
| SpamapS | <sigh> so true | 17:53 |
| SpamapS | Oh and I just discovered that the artifact repository is not open to global reads.. and I think I decided to do that so that somebody couldn't make a PR that uploads porn. | 17:53 |
| SpamapS | so if somebody wants gearmand artifacts, they have to ask me | 17:54 |
| * jlk quietly closes his PR | 17:54 | |
| clarkb | SpamapS: thats the situation that is hard without additional tooling to handle in swift | 17:54 |
| clarkb | SpamapS: good actors will upload logs and build artifacts and set them to expire appopriately. Bad ones could hide random crap fairly easily and use you as cheap hosting | 17:55 |
| clarkb | basically oyu need `find` and are stuck writing it yourself :) | 17:55 |
| SpamapS | there are some ways we could prevent evil artifacting | 17:56 |
| SpamapS | simple things | 17:56 |
| SpamapS | file magic checks | 17:56 |
| SpamapS | this is where the trusted post jobs come in handy | 17:56 |
| SpamapS | an untrusted job can just drop files in the node itself.. but the trusted job is the thing that can push them up to swift. In so doing, we can pre-qualify files. | 17:57 |
| mordred | SpamapS: yup | 17:57 |
| SpamapS | whitelist and blacklist mimetypes and that would get you pretty far | 17:57 |
| jeblair | mordred: jlk had questions on https://review.openstack.org/471799 | 17:57 |
| * SpamapS snaps back to present day problems | 17:58 | |
| SpamapS | mordred: reviewing your stuff. | 17:58 |
| jeblair | pabelanger: i'm ready to dive into getting the server running and then working on jobs with you. | 17:59 |
| pabelanger | jeblair: sure, I think we just need 472360 | 17:59 |
| pabelanger | both zuulv3.o.o and ze01.o.o are running right now | 18:00 |
| jeblair | i'm hoping 472009 will land soon and we'll be able to share with bonny | 18:00 |
| mordred | jeblair, SpamapS, jlk: I'll do a followup real quick to get those taken care of | 18:00 |
| pabelanger | but ze01.o.o is missing the connection info | 18:00 |
| mordred | jeblair: I'd also love to get logging patches in and live so that iterating on job content can actually see all the job output ;) | 18:00 |
| SpamapS | jeblair: FYI our current task over in Bonny deployment land is shutting down 2.5 and moving all the things to v3 :) | 18:00 |
| jeblair | mordred: yep | 18:01 |
| jeblair | mordred, jlk: interesting question about whether the host is unique; i don't think zuul has any requirement that hosts in a nodeset are uniquely named... does ansible have such a requirement in an inventory? | 18:02 |
| jlk | kinda yea | 18:03 |
| jlk | I don't know what happens if you have two hosts with the same name | 18:03 |
| jlk | undefined behavior | 18:03 |
| mordred | I believe latest wins | 18:03 |
| jeblair | mordred, jlk: maybe we should enforce uniqueness in zuul? | 18:04 |
| jlk | I may have misread the intent of that patch though, I was thinking that it was starting multiple streams per-host, so the logging of just host wasn't sufficiently unique enough to trace what's happening | 18:04 |
| jeblair | then the host would be a sufficient identifier in the logs, yeah? | 18:04 |
| mordred | yes | 18:04 |
| jeblair | jlk: ah, i believe we guarantee they are serialized within a host | 18:04 |
| jeblair | (we wait for logging of one task to complete before we start the next) | 18:04 |
| jlk | ah okay | 18:04 |
| mordred | yes. at this point we should be serialized. however, since that's debug logging, I think adding the info is nice | 18:04 |
| pabelanger | Shrews: is there a way to list builds using finger? | 18:05 |
| pabelanger | Shrews: it looks like you need to know the build ID today | 18:05 |
| jeblair | mordred: ah, that's for -vvv only. sounds good then. | 18:05 |
| Shrews | pabelanger: no, not yet | 18:06 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Log a little extra debug info on log streaming https://review.openstack.org/472367 | 18:06 |
| pabelanger | Shrews: okay, that's what I suspected. Thanks | 18:06 |
| mordred | jeblair: should we expose the ability for someone to request their jobs be run with -vvv ? | 18:06 |
| * SpamapS drops 4 +3's into the gate | 18:06 | |
| SpamapS | now to play with that broken one | 18:07 |
| SpamapS | btw, I am glad we're pinning to Ansible 2.3. I think we're going to have to start making Zuul capable of using multiple ansible versions at the job level at some point | 18:08 |
| SpamapS | especially whe Ansible 3 comes out | 18:08 |
| *** jkilpatr_ has quit IRC | 18:08 | |
| mordred | SpamapS: I at least want to get to the point where we're running latest release so that we can run jobs on prs to ansible/ansible and have them not be too wildly divergent/informative | 18:09 |
| SpamapS | mordred: definitely. The problem is going to be when there's just a divergent path and playbooks have to be updated to use newer ansibles. | 18:09 |
| mordred | yup | 18:10 |
| jeblair | mordred: let's not expose -vvv for now. | 18:10 |
| mordred | hopefully we can learn some things there | 18:10 |
| mordred | jeblair: kk | 18:10 |
| jeblair | mordred: in v2.5, an admin can switch that on and off for the executor; we should definitely add that back :) | 18:10 |
| mordred | jeblair: fo sho | 18:10 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Bail with an error on a non-existent jobroot_dir https://review.openstack.org/469524 | 18:15 |
| *** jkilpatr has joined #zuul | 18:15 | |
| jeblair | mordred, jlk: oh, derp, zuul does have duplicate node detection, just not where i thought it was :) | 18:16 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Make log streaming to executor per-task https://review.openstack.org/471799 | 18:16 |
| mordred | jeblair: oh good | 18:16 |
| jeblair | i'm cleaning it up to make a better error message | 18:16 |
| SpamapS | mordred: ehhh... so, your test fail | 18:16 |
| SpamapS | mordred: seems like you have a v6 socket, and you're trying to bind it to 0.0.0.0 | 18:17 |
| mordred | SpamapS: patch 5? yah - it also doesn't work with live execution with a simple playbook | 18:17 |
| mordred | gah. did I miss an 0.0.0.0 somewhere? | 18:17 |
| SpamapS | yerp | 18:17 |
| mordred | blast | 18:17 |
| mordred | thank you | 18:17 |
| jlk | I'm a bit wary of overexposing the underlying ansible to folks | 18:17 |
| SpamapS | mordred: also, why not using super(CustomForkingTCPServer)? | 18:17 |
| jlk | or ever exposing a choice of ansibles | 18:17 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Strip double-timestamps from output log https://review.openstack.org/471890 | 18:18 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Update log to include expected data on task results https://review.openstack.org/471891 | 18:19 |
| SpamapS | mordred: (in zuul/ansible/module_utils/log_streamer.py line 235) | 18:19 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Move log streaming to share with zuul_console https://review.openstack.org/471921 | 18:19 |
| SpamapS | q | 18:19 |
| SpamapS | mordred: oh neat, you found it quick :) | 18:20 |
| mordred | SpamapS: yah - although that still doesn't fix it to actually work - it's definitely an issue | 18:20 |
| SpamapS | mordred: aw derp, wtf? | 18:22 |
| * SpamapS is now truly confused | 18:23 | |
| mordred | SpamapS: http://paste.openstack.org/show/612042/ is how I'm poking at it, fwiw | 18:23 |
| SpamapS | wait no, it's failing elsewhere now | 18:23 |
| SpamapS | mordred: heh, the new fail is the opposite problem | 18:24 |
| SpamapS | you have '::' for an AF_INET | 18:24 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Add friendly error messages and tests for nodeset dupes https://review.openstack.org/472370 | 18:24 |
| clarkb | I thought 0.0.0.0 was supposed to magically work for ipv6 too | 18:24 |
| jeblair | mordred, jlk: ^ | 18:25 |
| mordred | does it? kk. we had '::' in one place and '0.0.0.0' in another- so I figured just having it be the same would be good | 18:26 |
| SpamapS | mordred: change the AF_INET's to AF_INET6 in that test, and it works | 18:26 |
| mordred | one does at least need AF_INET6 on the socket though | 18:26 |
| SpamapS | though | 18:27 |
| SpamapS | I get a detached process that spits some stuff on stderr later | 18:27 |
| SpamapS | http://paste.openstack.org/show/612043/ | 18:27 |
| mordred | oh. hrm | 18:29 |
| SpamapS | mordred: in case I wasn't clear, this got the test to pass for me: http://paste.openstack.org/show/612044/ | 18:30 |
| SpamapS | but seems like we may need to be careful about leaving old processes sitting around. | 18:31 |
| mordred | so - I think the issue I'm having with the synthetic test is somewhere related to forking/process management - if I run the remote zuul_console process by hand, the playbook works and everything logs | 18:31 |
| mordred | but if I have the playbook run the zuul_console task - it hangs waiting for log streaming of the first task | 18:31 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Move log streaming to share with zuul_console https://review.openstack.org/471921 | 18:32 |
| mordred | SpamapS: agree re: old processes | 18:32 |
| mordred | SpamapS: and that would anecdotaly concur with suppositions about there being wonkiness in the forking/process management | 18:32 |
| SpamapS | yeah, the test is clearly also leaving a streamer waiting for log streaming of something | 18:33 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Change node 'image' to 'label' https://review.openstack.org/472372 | 18:33 |
| jeblair | mordred, SpamapS, pabelanger, Shrews: ^ i almost forgot, we should land that change before we go hog wild on jobs since it changes the config syntax | 18:35 |
| jlk | Okay I'm going ot pick up the task of creating a cache of github change objects and updating them when necessary. | 18:36 |
| jeblair | jlk: yay! | 18:37 |
| jlk | I think our delta to bonnyci is down to one patch, that jamielennox is working on | 18:38 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Allow per-repo selection of configuration classes to load https://review.openstack.org/472009 | 18:38 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Change node 'image' to 'label' https://review.openstack.org/472372 | 18:38 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Add friendly error messages and tests for nodeset dupes https://review.openstack.org/472370 | 18:38 |
| jlk | mordred: in your spec for multiple zuul inputs, that still uses a single scheduler right? it's not going to tackle trying to scale out schedulers and dealing with shared caches across multiple? | 18:39 |
| jlk | (see also years of nova scheduler scale out issues) | 18:40 |
| mordred | jlk: that is correct | 18:40 |
| jeblair | jlk: multiple schedulers is zuulv4 | 18:40 |
| jlk | good deal | 18:40 |
| jeblair | mordred's spec is forward compatible with that :) | 18:40 |
| jlk | at least we'll have already finished the bikeshed over zookeeper | 18:40 |
| mordred | heh | 18:40 |
| mordred | you're so optimistic | 18:41 |
| mordred | I do not assume that to be true ;) | 18:41 |
| jeblair | we can absolutely revisit the zookeeper decision for zuul v5. :) | 18:41 |
| SpamapS | The problem is that the bikeshed, even painted red, can ALWAYS be repainted. | 18:42 |
| mordred | jeblair: oh - I thought jlk meant the bikeshed about zk v mqtt for ingestors | 18:42 |
| jlk | I know, lets powdercoat the damn thing. | 18:42 |
| mordred | jlk: YES | 18:42 |
| mordred | SpamapS: no matter how many years it has been, I still try to type "git shelve" | 18:43 |
| jeblair | mordred: i have a 'git update' alias | 18:43 |
| mordred | jeblair: :) | 18:44 |
| *** hashar has joined #zuul | 18:44 | |
| clarkb | shelve is the bzr thing like stash? | 18:45 |
| SpamapS | yep | 18:45 |
| clarkb | when I'm not lazy I've actually started committing instead of stashing because git stash pop with a conflict is the worst thing ever | 18:46 |
| mordred | clarkb: yah - I tend to do that too | 18:46 |
| mordred | same how I tend to use reset --hard instead of git checkout | 18:46 |
| mordred | jlk, SpamapS, jeblair, Shrews: I marked https://review.openstack.org/#/c/471921 as WIP beause although it now passes tests, it doesn't actually work live | 19:24 |
| mordred | which needs to be fixed - and then preferrably a test that shows the real work/no-work status needs to be added | 19:24 |
| mordred | and if it's not related, we need to figure out the orphaned process issue | 19:25 |
| jeblair | mordred: ok | 19:26 |
| jeblair | mordred: i haven't really dug into it; let me know if you think it's a problem you'd like me to help poke at. | 19:26 |
| mordred | jeblair: I do not think it is an essential patch to land | 19:26 |
| mordred | jeblair: it's a "refactor to share code between finger streamer and node streamer" | 19:26 |
| mordred | so I think we shold do it - but the current landed code has the things we need for actually getting log content | 19:27 |
| mordred | which is a long-winded way of saying that I will almost certainly need your help, but there are other much more important things to do currently | 19:27 |
| jeblair | SpamapS: can you re-review 472009 and take a look at 472372? | 19:43 |
| jeblair | SpamapS: bonus change 472370, but i'm okay approving that one as it stands if you're busy | 19:44 |
| Shrews | how on earth is my 1 line change in merge conflict already? *sigh* | 19:52 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/zuul feature/zuulv3: Correct location of ansible log file https://review.openstack.org/472355 | 19:56 |
| *** jkilpatr has quit IRC | 19:56 | |
| mordred | Shrews: sorry bout that | 19:57 |
| Shrews | no worries | 20:01 |
| * SpamapS is just back from lunch now | 20:04 | |
| *** harlowja has quit IRC | 20:08 | |
| jlk | SpamapS: so I updated the z8s repo with a few things to help. You can set where to clone zuul from and which branch using shell vars at docker-compose build time, and you can tell it to just volume mount a zuul checkout into the container at docker-compose run time too, and I re-did the pip install so the volume mount over the clone from build just works. | 20:09 |
| jlk | so now it's really fast to spin up a container set running code out of your local zuul clone to test in-flight code | 20:10 |
| jlk | docker-compose -f docker-compose.yaml -f devel.yaml up zuul-zookeeper zuul-scheduler zuul-executor | 20:10 |
| *** jkilpatr has joined #zuul | 20:14 | |
| SpamapS | jlk: so... now the bad news.. kompose can only convert docker-compose v2 files | 20:14 |
| SpamapS | it does not work with the 3.1 format you're using | 20:14 |
| SpamapS | which means we get to hand-convert to k8s | 20:14 |
| SpamapS | (or fix kompose) | 20:14 |
| Shrews | z8s? | 20:14 |
| *** jkilpatr has quit IRC | 20:14 | |
| SpamapS | Zulernetes ;) | 20:15 |
| SpamapS | Zuulernetes I should say | 20:15 |
| jlk | a repo I'm putting up to track work to get Zuul into kubernetes | 20:15 |
| jlk | a journey of sorts. | 20:15 |
| jlk | starting with docker-compose, then moving onto k8s | 20:15 |
| SpamapS | btw, the name collision thing is stalled... Netflix Badger don't care. Netflix Badger DGAF https://github.com/Netflix/zuul/issues/315 | 20:16 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Correct location of ansible log file https://review.openstack.org/472355 | 20:16 |
| SpamapS | jeblair, mordred: Random community thing, but at some point we should talk about possible solutions we could do without the Netflix Zuul team's cooperation, since they seem not to care. | 20:16 |
| Shrews | so netflix zuul _is_ older | 20:17 |
| SpamapS | ehhhhhh | 20:17 |
| SpamapS | the REPO is older | 20:17 |
| SpamapS | It may not have been released publicly until after that git commit. | 20:18 |
| SpamapS | https://medium.com/netflix-techblog/announcing-zuul-edge-service-in-the-cloud-ab3af5be08ee <-- June, 2013 | 20:18 |
| SpamapS | I've actually been trying _not_ to throw that in their face, to engender cooperation and let THEM point it out. Seems like they DGAF though. | 20:19 |
| *** hashar has quit IRC | 20:19 | |
| clarkb | yes this zuul is older as a public name | 20:20 |
| clarkb | but I'm also not sure it matters too much? | 20:20 |
| SpamapS | clarkb: it's been mistaken a number of times. | 20:21 |
| SpamapS | They're both pretty abstract things. | 20:21 |
| SpamapS | so it's easy for a high level journalist review to find the wrong one and kind of go "well ok I ... guess?" | 20:22 |
| SpamapS | why wouldn't I use an HTTP routing service to test CNCF's apps? | 20:22 |
| SpamapS | clarkb: In particular, I worry that it will hamper efforts to grow Zuul's community. | 20:23 |
| Shrews | let's just redirect all of our bug reports to netflix's GH repo, then maybe they'll care after it gets annoying enough :) | 20:24 |
| SpamapS | honestly | 20:25 |
| * SpamapS refrains from grumbling about storyboard UI | 20:26 | |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Allow per-repo selection of configuration classes to load https://review.openstack.org/472009 | 20:29 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Add friendly error messages and tests for nodeset dupes https://review.openstack.org/472370 | 20:32 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Change node 'image' to 'label' https://review.openstack.org/472372 | 20:34 |
| clarkb | SpamapS: this is a problem that exists elsewhere (as you mentioned re swift and containers earlier) I'm not sure there is a good way around it other than being as clear as possible when communicating about the tools? You can always rename on your end until someone else decides to conflict with you and create confusion | 20:34 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Log a little extra debug info on log streaming https://review.openstack.org/472367 | 20:35 |
| jlk | geez this is super convenient to just restart the container stack, which picks up the new code in my checkout, and stuff an event through it | 20:43 |
| SpamapS | clarkb: There is that weird phenomenon when HUGE players forget to do due dilligence, or ignore it because they know they can. Golang was the _SECOND_ open source programming language named Go. :P | 20:49 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-jobs master: Add initial license, docs, and other config https://review.openstack.org/472410 | 21:01 |
| * SpamapS adds another gertty subscription | 21:10 | |
| jeblair | File "/usr/local/lib/python3.5/dist-packages/zuul/cmd/executor.py", line 74, in send_command | 21:28 |
| jeblair | s.sendall('%s\n' % cmd) | 21:28 |
| jeblair | TypeError: a bytes-like object is required, not 'str' | 21:28 |
| jeblair | pabelanger: ^ on ze01, when trying to stop | 21:28 |
| pabelanger | I missed that | 21:29 |
| jeblair | i'll patch | 21:29 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: UTF8 encode commands sent to executor socket https://review.openstack.org/472419 | 21:32 |
| jeblair | pabelanger: ^ | 21:32 |
| pabelanger | +3 | 21:33 |
| SpamapS | Thou shalt utf8 encode thine strings when thou dost send them in thy wire. | 21:33 |
| jeblair | verily and it was good | 21:38 |
| jeblair | pabelanger: i will just kill the running proc and restart it to pick up new connection info | 21:38 |
| pabelanger | jeblair: ok | 21:38 |
| jeblair | The authenticity of host '[review.openstack.org]:29418 ([2001:4800:7819:103:be76:4eff:fe05:8525]:29418)' can't be established. | 21:40 |
| jeblair | RSA key fingerprint is SHA256:RXNl/GKyDaKiIQ93BoDvrNSKUPFvA1PNeAO9QiirYZU. | 21:40 |
| jeblair | Are you sure you want to continue connecting (yes/no)? no | 21:40 |
| jeblair | pabelanger: looks like we're missing ssh known hosts | 21:41 |
| pabelanger | oops, yes, that is possible | 21:41 |
| pabelanger | I can fix quickly | 21:41 |
| pabelanger | jeblair: I am okay if you want to manually accept for now | 21:41 |
| pabelanger | we have the puppet code today, just copy pasta for executor | 21:42 |
| *** adam_g has quit IRC | 21:42 | |
| jeblair | cool, will do to keep moving | 21:42 |
| jeblair | pabelanger: /var/lib/zuul/ssh/id_rsa is empty | 21:43 |
| jeblair | er, maybe we should switch this back to -infra, sorry :) | 21:43 |
| jeblair | we're out of the zuul bugs woods and into the zuul puppet bugs woods | 21:43 |
| pabelanger | k | 21:43 |
| *** adam_g has joined #zuul | 21:43 | |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: UTF8 encode commands sent to executor socket https://review.openstack.org/472419 | 21:50 |
| jlk | That face when you get all excited because tests pass, and then you realize you checked out the wrong branch. | 22:19 |
| SpamapS | hrm | 22:19 |
| SpamapS | translator is proving a bigger thing than I'd thought | 22:19 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-jobs master: Add revoke-sudo role https://review.openstack.org/472461 | 22:19 |
| SpamapS | why is it always the day I have afternoon kid stuff planned that I get into an actual fun coding problem? :-P | 22:21 |
| jlk | lol, I know that feel | 22:21 |
| jlk | woo only one failure. | 22:23 |
| jlk | Uh oh, brain storm time | 22:30 |
| jlk | What things that might change in a github PR should cause a dequeue of something already in the pipeline? | 22:31 |
| jlk | for gerrit, seems that it only dequeues if the patchset changes. | 22:31 |
| jlk | maybe that's right for github too, don't worry about votes or statuses or labels or comments | 22:31 |
| jlk | just the patch set | 22:31 |
| mordred | yah. if someone leaves a blocking vote after a job is already running, we just let the job finish and report but the blocking vote blocks the final merge | 22:32 |
| mordred | since, at the time it goes to merge, that is no longer satisfyable | 22:32 |
| jlk | that's the one thing I managed to break with cached changes | 22:32 |
| mordred | oh neat | 22:32 |
| jlk | oh hrm. I think I broke what the "patchset" is... | 22:32 |
| jlk | or this test is really bad. hrm. | 22:33 |
| mordred | SpamapS: were you able to get as far as getting the auto-translation from jjb to playbooks like we're producing in 2.5? | 22:33 |
| mordred | SpamapS: (I'd be happy to share thoughts that have been in my brainhole on that topic - but also happy to just leave you to it :) ) | 22:34 |
| jlk | ahhhh, the cache is by patchset in gerrit land, I need to do the same in github land. | 22:35 |
| jlk | jeblair: so uh, with the change cache, it doesn't look like anything ever prunes it. So if I were to keep scheduler running for ... months, and it gets lots of changes, wouldn't that memory just grow without bound (until the host runs out of memory)? | 22:46 |
| jeblair | jlk: yeah. there are some pruning routines in there, but i think we disabled them due to $problems? (turns out, it's not that much memory). i'd like to clean that up again though. | 22:48 |
| jlk | yeah that can be tricky. | 22:49 |
| openstackgerrit | Jesse Keating proposed openstack-infra/zuul feature/zuulv3: Implement a cache of github change objects https://review.openstack.org/472468 | 22:51 |
| jlk | Caching!! ^^ | 22:52 |
| jlk | jamielennox: that should be of interest to you. | 22:52 |
| mordred | \o/ | 22:58 |
| pabelanger | do we use dogpile.cache for that? | 22:58 |
| jeblair | no. that may be an option; or we may just want to leave this until zuulv4 where, presumably, the cache would be in zk anyway. | 22:59 |
| pabelanger | ++ | 23:00 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-jobs master: Add extra-test-setup role https://review.openstack.org/472472 | 23:09 |
| jeblair | pabelanger, mordred, SpamapS: i'm working on moving over the roles we already have in openstack-zuul-roles into zuul-jobs. i'm also setting up some a basic documentation framework so we have that out of the gate. | 23:10 |
| jlk | is zuul-base-jobs ? | 23:10 |
| jlk | er is zuul-base-jobs used any more? | 23:11 |
| jeblair | an early look at 472472 and parents would be nice. | 23:11 |
| jeblair | jlk: the idea with that repo is for it to hold only one (or a small number) of literal 'base' jobs. that way sites can use zuul-jobs + zuul-base-jobs if that works for them, or if they have complex base job requirements, they can supply their own 'base' job instead of zuul-base-jobs, and use that with zuul-jobs | 23:12 |
| jeblair | jlk: i don't think we've populated it yet, but we should soon | 23:12 |
| jeblair | jlk: (but all the jobs in zuul-jobs will inherit from 'base', and zuul-base-jobs will define 'base') | 23:13 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-jobs master: Add extra-test-setup role https://review.openstack.org/472472 | 23:14 |
| pabelanger | jeblair: sure, I'll start reviewing in the morning | 23:18 |
| clarkb | ya they all just took longer | 23:22 |
| clarkb | er ww | 23:22 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Add 'description' field to jobs https://review.openstack.org/472483 | 23:43 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-jobs master: Add python unit test jobs https://review.openstack.org/472484 | 23:48 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-jobs master: Add documentation to jobs https://review.openstack.org/472485 | 23:48 |
| *** harlowja has joined #zuul | 23:57 | |
| *** jamielennox is now known as jamielennox|away | 23:57 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!