| *** rcarrillocruz has joined #zuul | 00:05 | |
| jamielennox | mordred: i'm not sure what channel this belongs in any more but in os_server_volume device: may not be respected? | 00:30 |
|---|---|---|
| mordred | jamielennox: well that doesn't seem great :) | 00:32 |
| mordred | jamielennox: #openstack-shade is as good a place as any for stuff like that -- but we should dig in and figure out why not | 00:33 |
| openstackgerrit | Paul Belanger proposed openstack-infra/nodepool: Make diskimage-builder command configurable for testing https://review.openstack.org/404976 | 00:40 |
| openstackgerrit | Paul Belanger proposed openstack-infra/nodepool: Properly cleanup failed diskimage builds https://review.openstack.org/409327 | 00:40 |
| *** Shuo has quit IRC | 01:18 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: WIP triggers https://review.openstack.org/408848 | 01:31 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: WIP organize connections into drivers https://review.openstack.org/408849 | 01:31 |
| *** harlowja has quit IRC | 03:43 | |
| openstackgerrit | Jerry Zhao proposed openstack-infra/zuul: add check if an event matches the connection which triggered it. https://review.openstack.org/409397 | 04:03 |
| *** bhavik1 has joined #zuul | 04:18 | |
| *** harlowja has joined #zuul | 04:21 | |
| *** bhavik1 has quit IRC | 04:55 | |
| *** harlowja has quit IRC | 05:08 | |
| *** harlowja has joined #zuul | 05:56 | |
| *** bhavik1 has joined #zuul | 05:58 | |
| *** bhavik1 has joined #zuul | 06:32 | |
| *** abregman has joined #zuul | 06:52 | |
| *** saneax-_-|AFK is now known as saneax | 07:02 | |
| *** harlowja has quit IRC | 07:25 | |
| *** bhavik1 has quit IRC | 07:56 | |
| *** hashar has joined #zuul | 08:30 | |
| *** hashar has quit IRC | 09:20 | |
| *** hashar has joined #zuul | 09:24 | |
| *** willthames has quit IRC | 09:31 | |
| *** hashar is now known as hasharLunch | 11:21 | |
| *** hasharLunch is now known as hashar | 12:39 | |
| *** dmsimard has joined #zuul | 14:18 | |
| dmsimard | hey zuulers o/ unless mistaken, you can merge this if you want, it's not a POC/WIP or anything: https://review.openstack.org/#/c/405613/ | 14:18 |
| dmsimard | rcarrillocruz, jeblair ^ | 14:18 |
| dmsimard | clarkb: ^ | 14:18 |
| *** dmsimard has quit IRC | 14:21 | |
| *** dmsimard has joined #zuul | 14:21 | |
| *** dmsimard has quit IRC | 14:28 | |
| *** saneax is now known as saneax-_-|AFK | 14:29 | |
| *** dmsimard has joined #zuul | 14:30 | |
| mordred | dmsimard: neat! | 14:47 |
| dmsimard | if you have any feedback regarding ara, do let me know .. most of the stuff that has gone in is driven directly out of user feedback | 15:01 |
| dmsimard | doesn't mean I'll be able to do it (quickly or at all) but writing it down in a story that can eventually be picked up (by me or someone else) is always useful | 15:01 |
| mordred | ++ | 15:05 |
| mordred | dmsimard: I love how simple it is | 15:05 |
| mordred | (like, the d-g patch is tiny :) ) | 15:05 |
| dmsimard | simplicity is one of the core concepts of it, it was designed from the ground up to be that simple | 15:06 |
| dmsimard | i.e, Tower gets in the way of your workflow and wants to control everything | 15:06 |
| dmsimard | this is the extreme opposite | 15:06 |
| pabelanger | mordred: Shrews: jeblair: mind reviewing https://review.openstack.org/#/c/404976 need to land our failed dib cleanup patch | 15:17 |
| * Shrews reviewed 4 days ago :-P | 15:19 | |
| pabelanger | Shrews: Ah, yes. lack of coffee, danke | 15:19 |
| mordred | jeblair: ^^ the patch linked there is good by me - but you had concerns originally, so I'd prefer if you pulled the trigger | 15:22 |
| pabelanger | think today I'm going to poke at statsd stuff for builder.py, see how we can better trace diskimage success and failures. I know we have (had?) something | 15:25 |
| phschwartz | mordred: I should have called you when I was in dallas Friday night but was short on time. Next time I am out there we will have to get together for drinks | 15:26 |
| mordred | phschwartz: ++ | 15:26 |
| phschwartz | I flew in around 10pm to help a friend clean out his apartment and drive to south FL. So it was leaving early the next morning for the 23 hour drive | 15:27 |
| mordred | oh lovely | 15:28 |
| mordred | phschwartz: you got to enjoy the lovely cold spell! | 15:28 |
| phschwartz | It was 38 when I landed and 43 when we left in the morning. Crazy thing was it was 29 when we stopped for the night in pensacola and then 25 when we woke up and got on the road at 7am | 15:29 |
| *** saneax-_-|AFK is now known as saneax | 15:48 | |
| *** abregman has quit IRC | 16:04 | |
| openstackgerrit | Merged openstack-infra/nodepool: Make diskimage-builder command configurable for testing https://review.openstack.org/404976 | 16:48 |
| openstackgerrit | Merged openstack-infra/nodepool: Properly cleanup failed diskimage builds https://review.openstack.org/409327 | 16:49 |
| *** hashar has quit IRC | 16:54 | |
| pabelanger | mordred: jeblair: https://review.openstack.org/#/c/408324/ should fix a race in our nodepool testing too | 17:21 |
| openstackgerrit | Merged openstack-infra/nodepool: Update waitForBuildDeletion() to protect against delete race https://review.openstack.org/408324 | 17:30 |
| rcarrillocruz | back | 18:31 |
| rcarrillocruz | glad it got merged the ara change | 18:31 |
| rcarrillocruz | clarkb , mordred : https://review.openstack.org/#/c/401975/ , are we good now? jlk also +1'd now | 18:33 |
| jeblair | Shrews: did you get anywhere with the 'only use one zk client' in nodepool-builder idea? | 18:46 |
| Shrews | jeblair: i experimented with it a bit (thinking it might be easy to just create singletons), but run into the "everyone shares the same chroot so breaks tests" issue, then didn't think too much more about it | 18:48 |
| Shrews | we'd have to change it to pass connection info in, then hash to a singleton based on that info | 18:49 |
| jeblair | Shrews: how about an approach where the builder holds a single client and the workers share it? (the the builder for each test gets its own) | 18:49 |
| openstackgerrit | James E. Blair proposed openstack-infra/nodepool: Merge feature/zuulv3 into master https://review.openstack.org/410357 | 18:50 |
| Shrews | jeblair: maybe? but thinking about it, if we really only have 5 ZK connections now (1 build worker, 4 upload workers), then we could really only reduce it to 2. Are 3 less connections going to really buy us a lot here? (maybe it will, just thinking out loud) | 18:52 |
| Shrews | well, i guess we *could* reduce to 1 if both builders and uploaders share, too | 18:53 |
| clarkb | rcarrillocruz: ya I will try taking a look at that stack today | 18:53 |
| clarkb | probably after meeting and lunch | 18:53 |
| Shrews | i'm very surprised 5 connections cause so much cpu churn | 18:54 |
| jeblair | Shrews: yes, i was imagining they all share. each connection has 3 threads, so we currently have 18 threads dedicated to talking to zk. | 18:54 |
| jeblair | we should only need 1 connection/3 threads so i think it's worth looking into | 18:54 |
| Shrews | jeblair: i can take another stab at it | 18:56 |
| jeblair | Shrews: cool, thanks | 18:56 |
| openstackgerrit | Clint 'SpamapS' Byrum proposed openstack-infra/zuul: WIP: Enable test_swift_instructions https://review.openstack.org/410363 | 19:05 |
| Shrews | For the PTG, do we intend to have coordinated things beyond the Mon-Tue session? | 19:08 |
| Shrews | b/c 2 days of Atlanta would be about all I could stand | 19:08 |
| Shrews | :) | 19:08 |
| Shrews | jeblair: hrm, the config reload aspect of this makes things difficult. In order to prevent stepping on a thread's toes, we need a lock around the client calls so it can be re-established if the ZK config values change | 19:16 |
| SpamapS | Shrews: you just need moar bitchface | 19:16 |
| Shrews | SpamapS: ugh | 19:16 |
| Shrews | SpamapS: that's why i want to leave early... less chance of you corrupting me | 19:18 |
| SpamapS | the deed is done | 19:18 |
| SpamapS | I can feel it growing inside you | 19:18 |
| dmsimard | As someone who doesn't follow zuul/nodepool development too much | 19:18 |
| SpamapS | USE your bitchface, Luke | 19:18 |
| dmsimard | can someone give me a tl;dr of why zookeeper is now a thing ? | 19:18 |
| dmsimard | did we miss java because we took jenkins out of the picture? :) | 19:18 |
| SpamapS | dmsimard: There is a need to coordinate and order things. Gearman doesn't allow for that. | 19:19 |
| SpamapS | Zookeeper is the most well behaved java app I've ever worked with. | 19:19 |
| jeblair | dmsimard: see http://specs.openstack.org/openstack-infra/infra-specs/specs/nodepool-zookeeper-workers.html | 19:19 |
| Shrews | dmsimard: http://specs.openstack.org/openstack-infra/infra-specs/specs/nodepool-zookeeper-workers.html#problem-description (First section) | 19:19 |
| jeblair | Shrews: for the provider managers, we took an approach where if a thread used an "old" provider manager, we threw an error. so if a provider changed and was replaced, threads would error out at random places, but the whole system is designed for that anyway, so it's okay. | 19:20 |
| dmsimard | ok, thanks. Personal curiosity :) | 19:21 |
| Shrews | jeblair: I'm not comfortable with that. Not to mention that if a thread currently has a lock, and another thread resets that connection, then we lose that lock. I have no idea how the current code is going to react to that | 19:25 |
| jeblair | Shrews: well, regardless of whether you want to use that approach for solving this problem, we absolutely have to handle an error from any place at any time. because we get them. | 19:26 |
| jeblair | Shrews: if the locking is an issue, perhaps we should rework that a bit and not have the client object hold locks | 19:27 |
| jlk | jeblair: thanks for the comment on the etc_hosts PR. I was reading the old code, I just mentally lost the nuance that you only wanted to add a new line if the hostname didn't appear at all in the existing file. I kind of glossed over that. | 19:27 |
| jeblair | Shrews: we can also revisit how we handle configuration changes -- we don't have to keep the current approach | 19:29 |
| SpamapS | The one thing I've seen trip up ZK users in the two times I've used it, was that there has to be a thread that is responding to the server all the time and mutating state in the program for it to be really effective at "liveness". | 19:31 |
| SpamapS | ok hrm.. this is weird | 19:32 |
| SpamapS | clint@clint-ThinkPad-X250:/tmp/tmp4CzbkW/zuul-test/upstream/org/nonvoting-project$ git show-ref | 19:32 |
| SpamapS | 2d66fdbddcbc6fd5d5a103d893ae664577074739 refs/heads/master | 19:32 |
| SpamapS | clint@clint-ThinkPad-X250:/tmp/tmp4CzbkW/zuul-test/upstream/org/nonvoting-project$ | 19:32 |
| SpamapS | no refs/tags/init | 19:32 |
| SpamapS | clint@clint-ThinkPad-X250:/tmp/tmp4CzbkW/zuul-test/upstream/org/project1$ git show-ref | 19:33 |
| SpamapS | bb57c1731db46115a19d77e3aea36c28a652af9e refs/heads/master | 19:33 |
| SpamapS | bb57c1731db46115a19d77e3aea36c28a652af9e refs/tags/init | 19:33 |
| SpamapS | ah, only project1 has that. | 19:36 |
| jeblair | Shrews: (keep in mind, a zk connection change will almost never happen, which is why i'm okay with it being slightly disruptive (even to the point of aborting an image build) but i agree if there's another approach that isn't disruptive, that would be better) | 19:36 |
| SpamapS | hm no.. project1 and project2 have it. | 19:37 |
| * SpamapS is so confused :-P | 19:37 | |
| * SpamapS finds it | 19:40 | |
| openstackgerrit | Clint 'SpamapS' Byrum proposed openstack-infra/zuul: Re-enable test_nonvoting_job https://review.openstack.org/410376 | 19:44 |
| openstackgerrit | Clint 'SpamapS' Byrum proposed openstack-infra/zuul: Re-enable test_no_job_project https://review.openstack.org/410378 | 19:55 |
| SpamapS | I've been workin on the test.. mine... all the live long day... | 19:56 |
| * SpamapS swings pick axe to chip another test away | 19:56 | |
| openstackgerrit | Clint 'SpamapS' Byrum proposed openstack-infra/zuul: Re-enable test_no_job_project https://review.openstack.org/410378 | 20:08 |
| jlk | Hi all, I have a 2.5 question (yes, 2.5, I know). It seems to me that "get the source in place on the nodepool node" is not a baked in functionality, and implementers of zuul must write their own job/builder to get the source in place before trying to do anything. | 20:15 |
| jlk | Can somebody confirm this observation? | 20:15 |
| jlk | (I'm looking at things in infra like gerrit-git-prep) | 20:15 |
| pabelanger | right, we use things like gerrit-git-prep or zuul-cloner in 2.5 to get the git repo onto the node. | 20:17 |
| jlk | It looks like it's implemented as a job execution, like in the pipeline, rather than say a builder step, right? | 20:17 |
| pabelanger | let me double check | 20:18 |
| jlk | It seems that way from reading the jenkins job macros | 20:18 |
| pabelanger | for example: http://logs.openstack.org/18/406118/4/gate/gate-nova-python27-db-ubuntu-xenial/52b969d/_zuul_ansible/scripts/02-46ea63ccc8384e2d8de3141ed3d2f98a.sh | 20:20 |
| pabelanger | is the bash script used to clone | 20:20 |
| pabelanger | using: builder: http://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/jobs/python-jobs.yaml#n181 | 20:20 |
| pabelanger | defined: http://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/jobs/macros.yaml#n56 | 20:21 |
| jlk | sorry I was holding the wrong mental map of what a builder step was | 20:22 |
| pabelanger | zuul-git-prep-upper-constraints is more complex, but would suggest having zuul-git-prep for your setup. | 20:24 |
| jlk | yeah, something like that | 20:24 |
| jlk | Is that how v3 is set up as well, getting source onto the builder is up to the implementer? | 20:27 |
| mordred | nope | 20:27 |
| mordred | in v3, zuul pushes the prepared git repos directly onto the build node | 20:27 |
| jesusaur | oh, cool | 20:28 |
| jlk | oh good! | 20:28 |
| jlk | that feels much more reasonable :) | 20:28 |
| mordred | which also helps with a thing that was jesusaur's past hell | 20:28 |
| mordred | which is build nodes being somewhere that don't have a network path back to the central service | 20:28 |
| pabelanger | Interested to see how our zuul-launchers handle the extra IO and HDD requirements :) | 20:29 |
| mordred | so in v3 we require that the central service be able to have in-bound access to build nodes (which it needs to have anyway) - but we do not require a path from build node to central service | 20:29 |
| mordred | pabelanger: yah. otoh - they're a nice scale-out piece of the architecture - so "just add more nodes" :) | 20:29 |
| jesusaur | yeah, that means in v3 we'll no longer require routes from all the slaves to all the mergers | 20:29 |
| mordred | ++ | 20:29 |
| jlk | oh yeah that'll be nice | 20:30 |
| jlk | so, trigger comes in, merger merges the things and serves it up via http, launcher pops a node, clones from merger repo, pushes into node? | 20:31 |
| mordred | hand-wavey yes | 20:31 |
| jlk | I'm so good at the hand-wavey | 20:31 |
| mordred | it's also possible that leaving mergers and launchers separate turns out to be not worth it | 20:32 |
| mordred | and we end up with mergerlaunchers | 20:32 |
| jlk | yeah, that was a thought I had | 20:32 |
| mordred | but good arguments can be made in both directions | 20:32 |
| mordred | so we may just have to see how it goes one we start load testing | 20:32 |
| pabelanger | mordred: something I was thinking about today, anything stopping me in zuulv3 from randomly rebooting a node during a test run? assuming my playbook had the wait_for logic? | 20:32 |
| mordred | pabelanger: nope | 20:33 |
| jlk | seems /slightly/ inefficient to clone to push, but it also feels weird to put push to node capability on all the mergers | 20:33 |
| pabelanger | mordred: k, didn't think so | 20:33 |
| mordred | jlk: yah - especially when the launchers all have to be able to ansible to the nodes anyway | 20:33 |
| jlk | yeah. | 20:33 |
| jlk | when scaling out mergers, does each merger then get it's own http url, and it's unique to that merger, different from another merger? | 20:34 |
| pabelanger | mordred: maybe this week, I can bend your ear about the append hostname to git clones logic for zuul. So we don't have namespace issues if 2 gerrits have foobar | 20:35 |
| mordred | jlk: yes - and part of the payload to a job executor is the location of the merger to clone from | 20:35 |
| mordred | pabelanger: yes! | 20:35 |
| mordred | pabelanger: I just wrote a mention of that in an email list thing about go | 20:35 |
| mordred | pabelanger: I've been promising to write a spec about that for a while haven't I? | 20:36 |
| jlk | mordred: okay. That would mean if launcher/merger was all one thing, the launcher task would _have_ to go to the same host that did the merging | 20:36 |
| jlk | unless you collapse that into one functionality, merge+launch | 20:36 |
| pabelanger | mordred: neat, I was just reminded about it since RDO has the issue today. They're going to do a out-of-tree patch until we get it working properly | 20:36 |
| mordred | pabelanger: tl;dr from me : clone to $BASE/src/$git_host/$repo instead of $BASE/$repo | 20:37 |
| pabelanger | nods | 20:37 |
| mordred | pabelanger: and we need to make sure that, in a case like openstack where we have review.o.o and git.o.o we wind up with $BASE/src/git.openstack.org/openstack/nova | 20:37 |
| mordred | jlk: yah - or you could pass localhost to the launcher and a file:/// url perhaps and have it have short-circuit logic ... but yeah | 20:38 |
| jesusaur | jlk: the launcher would have to go to the same host that did the merging to push the refs to the slave anyway, since that constructed git tree ould only exist on the one merger that took the merge job | 20:38 |
| jlk | mordred: yeah. | 20:38 |
| jlk | jesusaur: unless the launcher did "clone from zuul merger + push to node" | 20:39 |
| jlk | instead of expecting it to exist somewhere on the local filesystem | 20:39 |
| jesusaur | oh, i see what you mean | 20:39 |
| jlk | Looking at zuul-git-prep, and thinking about what zuul merger does, don't I need a zuul ref to check out? I wouldn't guess that $ZUUL_PROJECT has the complete ref to clone, or does it? | 20:47 |
| jlk | oh maybe that gets set in the environment | 20:48 |
| jlk | and zuul-cloner pulls it from that | 20:48 |
| pabelanger | right | 20:49 |
| pabelanger | http://logs.openstack.org/18/406118/4/gate/gate-nova-python27-db-ubuntu-xenial/52b969d/_zuul_ansible/vars.yaml | 20:49 |
| pabelanger | would be the env vars we'd use for that job | 20:50 |
| jesusaur | jlk: in a typical check/gate pipeline there will also be a $ZUUL_REF var | 20:51 |
| jlk | alright | 20:51 |
| jesusaur | pabelanger: ah, that's super useful | 20:51 |
| jlk | pabelanger: do you have to configure all of those to be pushed into the env, or does that just automatically happen by way of ansiblelauncher? | 20:52 |
| jesusaur | but post and periodic pipelines look a little different | 20:52 |
| pabelanger | jlk: ya, in 2.5 we hardcode them in ansiblelauncher.py | 20:52 |
| pabelanger | jlk: http://git.openstack.org/cgit/openstack-infra/zuul/tree/zuul/launcher/ansiblelaunchserver.py#n1267 | 20:54 |
| jlk | Do any of you happen to know where you make zuul-cloner available in your dib images for nodepool? | 21:00 |
| jlk | (in infra) | 21:00 |
| jlk | is it in puppet somewhere? | 21:01 |
| jeblair | jesusaur, jlk: in v3 the merger that produces the code that gets used by a node is integrated into the launcher (it does not consult a remote launcher; standalone launchers are (if anything) only used for speculative merges for configuration changes, however, the merger-launchers can do that too) | 21:02 |
| jlk | jeblair: I see, so it's more of a scheduler -> $ONEBOXTHATMERGESANDLAUCNHES | 21:03 |
| jeblair | jlk: yep, for the easy case. | 21:03 |
| jeblair | (there is a scheduler -> merger -> scheduler -> merger-launcher path for live configuration changes) | 21:03 |
| pabelanger | jlk: we use puppet today (slave_common.pp) | 21:04 |
| jeblair | also, the nodepool 'pop' happens in the scheduler, so: scheduler [-> merger -> scheduler] -> nodepool -> scheduler -> merger-launcher | 21:05 |
| pabelanger | jlk: you could quick and dirty it with a in job builder, until you roll new images | 21:07 |
| openstackgerrit | Merged openstack-infra/nodepool: Merge feature/zuulv3 into master https://review.openstack.org/410357 | 21:39 |
| jlk | woo ^^ | 21:42 |
| jlk | Question on zuul-cloner | 21:46 |
| pabelanger | jeblair: clarkb: mordred: re: 410357 we still need to merged to master right? | 21:47 |
| jlk | It looks like it requires a git_base_url to clone from, which I would think would be the place zuul-merger makes available, since that's where the zuul-ref exists. (also why doesn't cloner get that from ENV vars?). | 21:47 |
| pabelanger | merge* | 21:47 |
| jlk | In infra's case, is git.openstack.org where zuul-merger puts it's merged refs? | 21:47 |
| jlk | and we'd need to use a url that's appropriate for our merger's published git repos? | 21:47 |
| mordred | jlk: each of the mergers runs their own git repo that they serve from direcly - so if you see the ZUUL_URL: http://zm07.openstack.org/p in the vars above | 21:48 |
| mordred | jlk: that's where zuul tells the job to clone from | 21:49 |
| jlk | hrm. | 21:49 |
| jlk | what does the base url passed to zuul-cloner come into play then? | 21:49 |
| mordred | one sec - lemme read something and make sure I'm not lying :) | 21:49 |
| jlk | reading http://docs.openstack.org/infra/zuul/cloner.html | 21:49 |
| clarkb | pabelanger: that is the merge to master | 21:50 |
| clarkb | pabelanger: we will need to tell puppet to go back to using master | 21:50 |
| pabelanger | clarkb: really? I read that as merge to feature/zuulv3 | 21:51 |
| mordred | jlk: oh. duh. sorry ECONTEXT_SWITCH ... git_base_url is where to clone from if it needs to clone from scratch. it can then do a git fetch from the ZUUL_URL provided to get the latest refs | 21:52 |
| clarkb | pabelanger: oh indeed | 21:52 |
| clarkb | pabelanger: the first parent is master though... | 21:52 |
| clarkb | this is what I get for reviewing locally rather than reading what gerrit is telling me | 21:53 |
| jlk | mordred: oh interesting. Well that kind of stinks, as we'll have to figure out a way to push through the base url for each of our github sites | 21:53 |
| pabelanger | clarkb: ya, I'm slightly confused :) | 21:53 |
| clarkb | pabelanger: its possible it was a mispush to the wrong dest branch | 21:54 |
| clarkb | and now that same change can be pushed to master | 21:54 |
| clarkb | jlk: why can't you use the merger? | 21:54 |
| clarkb | merger fetches from github or gerrit or wherever, makes ref, job fetches from merger | 21:54 |
| jlk | clarkb: set the base url to the merger? | 21:55 |
| jlk | isn't the future where you have multiple mergers, and eachmerger might have the ref and might not? | 21:55 |
| jlk | (or even the base repo) | 21:55 |
| clarkb | yes, maybe I misunderstand what the problem is | 21:56 |
| jlk | is the ZUUL_URL supposed to be the URL to the merger? | 21:56 |
| jlk | clarkb: zuul-cloner expects a git_base_url passed in as a cli argument. It does not fetch it out of env variables | 21:56 |
| clarkb | jlk: yes iirc ZUUL_URL will be url to one of the mergers | 21:56 |
| jlk | so in my builder, I have to specify one, or use a variable. | 21:56 |
| jlk | since we may be talking to multiple githubs, and we may have multiple builders, I need some way to pas sin one that we know the ref will be at | 21:57 |
| clarkb | you should only ever get the merger that knows what your refs are in zuul_url | 21:57 |
| clarkb | so you can do something like git fetch $ZUUL_URL/$ZUUL_PROJECT $ZUUL_REF && git checkout FETCH_HEAD | 21:58 |
| jlk | I shouldn't have to do that, isn't that what zuul-cloner does? | 21:58 |
| clarkb | yes under the hood | 21:58 |
| clarkb | its an illustration | 21:58 |
| jlk | like I want to maybe do zuul-cloner $ZUUL_URL $ZUUL_PROJECT | 21:58 |
| jlk | and I assume that $ZUUL_URL would be an appropriate place to construct the url | 21:59 |
| jlk | of $ZUUL_URL/$ZUUL_PROJECT | 21:59 |
| jlk | I see in one log from earlier on infra a setting of: ZUUL_URL: http://zm07.openstack.org/p | 21:59 |
| clarkb | as long as the project is part of zuul then I think that should work | 21:59 |
| SpamapS | My experience with zuul-cloner was that $ZUUL_URL was set to the merger and it worked fine. | 21:59 |
| jlk | http://logs.openstack.org/18/406118/4/gate/gate-nova-python27-db-ubuntu-xenial/52b969d/_zuul_ansible/vars.yaml | 21:59 |
| SpamapS | never had to pass in the url via CLI | 21:59 |
| jlk | So presumably http://zm07.openstack.org/p/openstack/nova is a valid URL | 22:00 |
| jlk | SpamapS: well the docs say it expects something | 22:00 |
| jlk | and upstream infra just toss git.openstack.org in there | 22:00 |
| clarkb | we don't just toss git.openstack.org in there | 22:02 |
| clarkb | its a complete mirror of all the repos we host | 22:02 |
| jlk | sorry, that's not what I meant. | 22:02 |
| jlk | "just toss" was poorly phrased. | 22:02 |
| jeblair | clarkb, pabelanger: yep, i neglected to account for the .gitreview, sorry. | 22:03 |
| jlk | infra uses git.openstack.org in there, because presumably everything zuul will be cloning is available from there. However we can't rely on a single base url to cover all our potential sources | 22:03 |
| pabelanger | jeblair: thanks, that explains it | 22:03 |
| pabelanger | jlk: should be able to it (git.o.o) an $UPSTREAM_GIT variable, then pass it the properly location. IIRC, that is what RDO did with 2 different gerrit servers | 22:05 |
| jeblair | pabelanger, clarkb: so i guess i'll push a version of that to master, but amend it to not touch .gitreview | 22:06 |
| clarkb | jeblair: sounds good | 22:06 |
| pabelanger | ++ | 22:06 |
| openstackgerrit | James E. Blair proposed openstack-infra/nodepool: Merge feature/zuulv3 into master https://review.openstack.org/410426 | 22:08 |
| jeblair | clarkb, pabelanger: ^ | 22:09 |
| clarkb | +2 | 22:09 |
| pabelanger | +2 | 22:09 |
| jeblair | note, it has lots of conflicts... that probably should have been a red flag for the other one :) | 22:09 |
| jeblair | but it's hard to see the absence of a red flag | 22:10 |
| clarkb | indeed | 22:11 |
| clarkb | fwiw I did submit a feature request against the new gerrit UI to better render the subway graph of proposed changes | 22:12 |
| pabelanger | 273 changes, nice | 22:12 |
| clarkb | which I think would make this much easier to understand | 22:12 |
| mordred | jeblair: I left if without a +A so you could do the honors | 22:19 |
| jeblair | mordred: thx, i'll let the devstack job run again and +3 | 22:20 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: WIP triggers https://review.openstack.org/408848 | 22:24 |
| clarkb | jlk: thinking about that situation more you may consider a central repo mirror like ours (reliabilty reasons if nothing else), another option may be to update the z-c map file contents to includeabsolute paths per repo and ignore the base in that case | 22:26 |
| jlk | clarkb: a central repo where every repo you care about gets mirrored to? | 22:27 |
| jlk | (if it doesn't originate from there) | 22:27 |
| clarkb | yes (tahts basically what git.openstack.org is for us) | 22:27 |
| jlk | that makes some sense, but if the "upstream" of a repo goes offline, we won't be able to accurately test anyway | 22:28 |
| jlk | the merger might not be able to reach the repo to create the zuul ref | 22:28 |
| clarkb | correct | 22:28 |
| jlk | We're going to toy with setting the base_url to ZUUL_URL | 22:28 |
| jlk | which should be the merger that made the ref | 22:29 |
| clarkb | at least for us I think we regularly push half a gigabit through git.o.o and so keeping the majority of that trffic off of gerrit means gerrit tiself is happier | 22:29 |
| jeblair | you will get *a* repo, you will not get a *correct* repo | 22:29 |
| clarkb | making it more reliable to serve the smaller set of refs that the zuul mergers need | 22:29 |
| jeblair | (it may not have all the branches/tags, etc needed) | 22:29 |
| jeblair | i should say you *may* not get a *correct* repo | 22:30 |
| jeblair | you might happen to get a correct one :) | 22:30 |
| jlk | yeah it seems the zuul mergers need to always be able to go out to the canonical upstream | 22:30 |
| jlk | but hopefully keep a cache. | 22:30 |
| jlk | so first time a merger works on a repo it may be a bunch of traffic, but the next time it'd be relatively little traffic | 22:30 |
| jlk | and the testers in turn fetch it from the merger | 22:30 |
| jeblair | a merger is only guaranteed to have a correct repo if it's involved in the change being tested | 22:34 |
| jeblair | and it's worth re-iterating none of this applies to v3 | 22:34 |
| jlk | yeah I know it's not in v3 | 22:34 |
| jlk | but what sets ZUUL_URL? Is it not the merger who merged the change? | 22:34 |
| jesusaur | also, the slaves have semi-recent clones baked into /opt/git iirc, so that they aren't pulling the full git tree from the mergers | 22:34 |
| jlk | e.g. ZUUL_URL: http://zm07.openstack.org/p | 22:34 |
| jlk | jesusaur: the slaves... | 22:35 |
| jlk | how does /opt/git get put in place though? That feels like an infra customization, not a zuul built in | 22:35 |
| jeblair | jlk: yeah, that's the merger that merged the repo(s) for that change | 22:35 |
| jesusaur | so that when the slave fetches the ZUUL_REF from ZUUL_URL it isn't pulling the entire git tree | 22:35 |
| mordred | jesusaur: jlk has an interal zuul - some of the infra optimizations don't apply to him directly | 22:35 |
| jesusaur | jlk: correct, that's going to be in the nodepool dib somewhere | 22:35 |
| jlk | jeblair: since the base url is per-job run defined, it seems safe to rely on $ZUUL_URL when the job runs | 22:35 |
| jeblair | jlk: we use nodepool and a diskimage builder element that caches repos (it's not actually a zuul-specific thing, but it tilts somewhat that way) | 22:36 |
| jeblair | jlk: unless you need a repo not being tested | 22:36 |
| jeblair | er, not in the change series being tested | 22:36 |
| mordred | jlk: problem is - if there are any git repos needed for the job and the zuul merger did not have to merge any changes for any of them, the merger may not have up to date copies of those repos | 22:36 |
| mordred | gah | 22:36 |
| mordred | I said it in more words than jeblair - but jeblair said it better | 22:36 |
| jlk | but that would be a different builder | 22:36 |
| jlk | and can use something else for that value | 22:37 |
| jeblair | mordred: i was about to say you said it better :) | 22:37 |
| mordred | jlk: imagine a devstack job - needs like 40 different repos to run - but there is only currently a proposed change to nova | 22:37 |
| jlk | I can see in that case, having a mirror could reduce traffic, at the expense of cache issues | 22:37 |
| mordred | jlk: the merger will only have nova | 22:37 |
| mordred | jlk: it will not have the other 39 repos | 22:37 |
| jlk | mordred: sure. | 22:37 |
| jlk | mordred: that feels like a different builder element though | 22:38 |
| jlk | it's not just "prep the source of this repo", it's "prep all the deps of this repo" | 22:38 |
| mordred | but you need to use zuul-cloner to clone all 40 repos because the job has no way of knowing if it will have a change from them | 22:38 |
| jlk | I don't see it being used that way in infra's zuul-git-prep | 22:39 |
| jlk | maybe that all still lives in gerrit-git-prep | 22:39 |
| jlk | and for v3 it'll be even weirder. What will launcher/merger push into the node? | 22:39 |
| jlk | just nova, or nova+deps? | 22:39 |
| mordred | jeblair: ^^ excellent question | 22:40 |
| jeblair | that macro only handles the single repo case; integration jobs use something else (either a different invocation of zuul-cloner, or its antecedent code in devstack-gate) | 22:40 |
| jeblair | mordred, jlk: it will push all the repos | 22:40 |
| jesusaur | oh, huh, yeah, if v3 is pushing the refs, do we lose the ability to configure custom clone paths with a clonemap? | 22:40 |
| jlk | jeblair: that was my point, we'll use a different macro for the dependent repos case | 22:41 |
| jlk | but first I need to solve the single repo case | 22:41 |
| mordred | GOTCHA - sorry - I was going too complex on you | 22:41 |
| jlk | we're crawling, and trying to walk | 22:41 |
| jlk | someday we'll run | 22:41 |
| mordred | for the single repo case, cloning from the merger should be fine - since the merger by definition would have merged the change you're testing to the single repo you are working with | 22:42 |
| mordred | jesusaur: I need to write up a spec - but I'm proposing that we follow go get semantics and clone everything to $BASE/src/${git_host}/${repo} - and if you need more complex layouts, you should be able to do symlinks or something | 22:44 |
| mordred | jesusaur: is there a use case I'm not thinking of that that would mess up? | 22:44 |
| * mordred thinks he should actually write up the spec so that people can tear it apart | 22:44 | |
| * jesusaur thinks that was for jeblair | 22:45 | |
| * jeblair thinks it was for jesusaur :) | 22:45 | |
| clarkb | mordred: one thing to be wary of doing that is the exec shebang limit in linux we hit with virtualenvs | 22:45 |
| jlk | that was an answer for the custom paths with the mapping | 22:45 |
| jesusaur | oh, so it was :) | 22:46 |
| mordred | jesusaur: it was for you ... in response to "do we lose the ability to configure custom clone paths with a clonemap?" | 22:46 |
| clarkb | mordred: adding morestuff topaths potentially makes exec unhappy | 22:46 |
| jlk | hrm, why would there be more stuff to path? | 22:46 |
| mordred | clarkb: it does- but for supporting multi-source (or supporting go at all) putting the host in the path winds up being essential | 22:46 |
| jlk | wouldn't they all get tossed into the same venv? | 22:46 |
| jlk | like, the source is in once place, but the venv you build is... combined? | 22:47 |
| jeblair | clarkb: we are dropping the job name, so i expect that path to be "/opt/workspace/git.openstack.org/openstack-infra/really-long-project-name" | 22:47 |
| mordred | ++ | 22:47 |
| jesusaur | mordred: oh i guess the symlinks work. the main case i was thinking of was cloning puppet modules into /etc/puppet/modules | 22:47 |
| mordred | yah - I think symlinks are going to be the friend there - might want something like clonemap that can declare a set of repos and symlink locations that can be added into the ansible of a job | 22:48 |
| jlk | Feels right to have a standard path that zuul exposes the sources, and then the jobs after that deal with pathing | 22:49 |
| jlk | instead of letting a job inform zuul how to expose the source on the node | 22:49 |
| SpamapS | so......... | 22:49 |
| SpamapS | forgive me if I missed backscroll about this | 22:49 |
| * mordred hands SpamapS a pie to distract him | 22:50 | |
| SpamapS | but the v3 future is a pushed-into-the-node future, not a zuul-cloner future... | 22:50 |
| mordred | yup | 22:50 |
| jlk | that's my understanding | 22:50 |
| SpamapS | ok now I"m backscrolled enough to see that was where we were already at | 22:50 |
| clarkb | jeblair: mordred ah ok no job name helps a ton | 22:50 |
| SpamapS | ok n/m I have nothing to add | 22:51 |
| * SpamapS should read forward instead of backwards | 22:51 | |
| jlk | clarkb: how does the path of the source checkout impact the shabang of a venv? | 22:51 |
| jeblair | remote: https://review.openstack.org/410442 Update zuulv3 spec to include job repo information | 22:51 |
| jeblair | i just realized that while the job-repos thing was in the pencil sketches, i could not find it in the spec itself; there's an update | 22:52 |
| clarkb | jlk: because tox for example is run relative to the repo | 22:52 |
| jlk | ooh right, to | 22:52 |
| jlk | x | 22:52 |
| clarkb | jlk so with tox you have $repopath/.tox/venv/bin/foo | 22:52 |
| clarkb | foo's shebang is what we are concerned about | 22:53 |
| jlk | yeah. | 22:53 |
| jesusaur | oh, and that can reach the max path length for a shebang | 22:53 |
| jlk | You can specify a path for tox to make the venv though right? .tox/venv/ is just the defualt? | 22:53 |
| clarkb | jlk: yes but that generally unfriendly to devs | 22:54 |
| clarkb | and assumes they want stuff in /tmp and know to look there for their install etc | 22:54 |
| jlk | fair enough | 22:54 |
| clarkb | jesusaur: yes 127 bytes is limit | 22:54 |
| *** willthames has joined #zuul | 23:10 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Re-enable zuultrigger test https://review.openstack.org/408848 | 23:25 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Reorganize connections into drivers https://review.openstack.org/408849 | 23:25 |
| jeblair | jhesketh: would you please take a look at 408849 and let me know what you think? i like the structure of it, but think that we will probably want to polish it up a bit before we declare it a public api. | 23:27 |
| jhesketh | jeblair: yes, I was excited by that change but haven't had a chance yet | 23:29 |
| jhesketh | will hopefully take a look today :-) | 23:29 |
| jeblair | jhesketh: cool, it's ready for actual review now :) | 23:29 |
| jhesketh | woo :-) | 23:30 |
| openstackgerrit | Merged openstack-infra/nodepool: Merge feature/zuulv3 into master https://review.openstack.org/410426 | 23:34 |
| mordred | \o/ | 23:34 |
| pabelanger | wee | 23:39 |
| jeblair | for realz this time | 23:40 |
| jhesketh | nice! | 23:41 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!