clarkb | independent of potentially solving this more betterer with ansible/zuul I feel like dns was built to solve a lot of these env specific problems but for raisins we can't actually sanely use dns this way | 00:00 |
---|---|---|
jeblair | hrm, it's not that elucidating anyway | 00:01 |
jamielennox | jeblair: i did, but i don't think the question was phrased the same way, the best answer seemed to be that there was a git connector, but whilst i could bend that to be usable it doesn't really fit the use case | 00:01 |
jeblair | jamielennox: okay, so you have a project-config repo -- why *can't* it be used in those other environments? | 00:02 |
openstackgerrit | Jamie Lennox proposed openstack-infra/zuul feature/zuulv3: Allow loading additional variables file for site config https://review.openstack.org/447734 | 00:02 |
jamielennox | jeblair: essentially because base job contains things like http://zuulv3-dev.openstack.org/logs/{build.uuid}/ | 00:03 |
jamielennox | review for comment | 00:03 |
jamielennox | jeblair: in production i can define that url, in vargrant i know what it is ahead of time | 00:03 |
jamielennox | in my dev environment it's hostvars['playpen-logs']['ansible_default_ipv4']['address'] | 00:04 |
jamielennox | and to make an end-to-end usable test i want to define that value differently for every person who spins up a dev environment | 00:04 |
jeblair | jamielennox: okay, so in true zuul fashion, we can throw more git repos at the problem and solve it that way. :) | 00:04 |
jamielennox | jeblair: that would mean me having to maintain a git repo with prior knowledge of every development environment | 00:05 |
jeblair | gimme a few minutes | 00:05 |
jeblair | i'm going to expand on this | 00:05 |
jamielennox | as opposed to something that the deployment simply templates out onto the file system | 00:05 |
SpamapS | jamielennox: a git repo could be templated out too :) | 00:06 |
SpamapS | but yeah, templating a file out for just one thing might be the better UX | 00:06 |
jamielennox | SpamapS: it certainly could | 00:06 |
* SpamapS has not thought enough about it | 00:06 | |
SpamapS | anyway, EOD | 00:06 |
jamielennox | i haven't looked yet as to whether the git connector tries to do things like pull updates | 00:06 |
SpamapS | BBL | 00:06 |
jeblair | jamielennox: basically, i think there are problems that we will encounter where we will have to remind ourselves that zuul (especially v3) works really well with lots of different repos, and quite often, the solution to a problem may involve rethinking the organization of repos. | 00:06 |
jeblair | (this may or may not be one of those cases, but we can explore what would be involved) | 00:07 |
jamielennox | jeblair: i am entirely open to reorganizing, and it's why i did this as a hack around a problem i had locally rather than go upstream first | 00:07 |
jeblair | so we could solve this problem by having the 'base' project defined in a repo which does little other than define this sort of site-specific configuration | 00:08 |
jeblair | er 'base' job rather | 00:08 |
jamielennox | basically i'm modelling these jobs off of what pabelanger has done for zuul which is i know designed to make things work fast | 00:08 |
jeblair | so each of the dev/prod/staging environments, as well as the vagrant stuff could have their own such repo | 00:08 |
jeblair | and with the 'git' driver, it doesn't have to be remote | 00:08 |
jeblair | that's essentially the way to do a 'local config file' | 00:09 |
jeblair | except it's a git repo instead of a file :) | 00:09 |
jeblair | my feeling is that this sort of thing may be really useful. generally speaking, it lets us modularize the configuration and reassemble the parts as fits different situations. | 00:10 |
jeblair | jamielennox: however, when i think about the idea of setting up a git repo solely so that "success-url" is something different, that seems like we've gone too far, and perhaps your site-local variables idea is the better way of dealing with that. :) | 00:11 |
jamielennox | i'm sure i could make that work, in practice though it'd end up being an entirely local git repo that i template out and depending on what was easier with ansible we'd either just wipe out the old repo or force commit whatever we actually wanted to be there | 00:12 |
jamielennox | i would need to look at what the git connector reponds to to see whether that would cause issues - but i don't konw if there would be a value to that history | 00:12 |
jamielennox | though i guess at least for production we would be able to publish it | 00:13 |
jeblair | jamielennox: yeah, so maybe it's good to have both of these things in our toolbox. if you want to change one variable, we have executor variables. if you want a slightly more substantial difference, you can define site-specific job configuration in a site-specific git repo. | 00:14 |
jamielennox | jeblair: so what's the rationale behind a local git repo connector? for most of this the advantage of git is having pull-requests and reviews etc but that wouldn't apply to a local repo | 00:14 |
jeblair | jamielennox: the third-party ci case. where you want to run a zuul pointing at someone else's github or gerrit, without running one of your own. | 00:15 |
jeblair | jamielennox: (and the reason it exists now is so that i can run zuul locally and point it at openstack's gerrit) | 00:15 |
jeblair | jamielennox: (since there is no way to supply zuulv3 configuration without having it in a git repo) | 00:16 |
jamielennox | ok, so practically it exists more for a testing or dependency case where you could have something in the background that could sync from upstream as part of a larger job? | 00:16 |
jeblair | jamielennox: well, i imagine a third-party ci would have their very site-specific configuration in a git repo without an upstream | 00:17 |
jeblair | jamielennox: they have configurations like "run test-my-network-switch on changes to openstack/neutron". that's never going to be an upstream thing, that's always local. | 00:18 |
jamielennox | but in which case something outside of zuul is responsible for doing that pull from upstream and merge my stuff | 00:19 |
jeblair | jamielennox: i'm probably not communicating this very well | 00:19 |
jeblair | jamielennox: i'm expecting the "git" driver in zuul to be used by third-party-ci systems to load a portion of their zuul configuration which has no correspondence with anything upstream. so there's no pull requests or proposed changes or merging to it at all. | 00:20 |
jeblair | just like today, third-party ci operators don't generally merge changes we make to zuulv2's layout.yaml into their own layout.yaml. they have a private shadow configuration. | 00:21 |
jeblair | they *could* set up their own gerrit to do that, but that's too much to ask of someone who just wants to run zuul. | 00:21 |
jamielennox | why would we not expect them to host that repository on their own gerrit/github | 00:22 |
jamielennox | right | 00:22 |
jeblair | because for many of them, it's a single small file that hardly ever changes | 00:22 |
jamielennox | ok, and it involves configuring via a baseurl - so i'm pretty sure i get the usecase, and we could make it work for changing urls but it does seem like overkill | 00:24 |
jeblair | (now, having said all that, third-party-ci operators will probably want to use *parts* of our configuration, which is why i think we need to update the tenant configuration syntax to allow folks to specify that they want to load *jobs* but not *projects* from certain repos, etc) | 00:24 |
jeblair | jamielennox: yep. i think if the extent of the problem is "i want a different success-url" i suspect your solution will be ideal. | 00:25 |
jamielennox | jeblair: well, it could be used/abused for injecting other deploy specific variables as well | 00:26 |
jamielennox | i'm not sure what they are yet | 00:26 |
jamielennox | and i'm not sure that's a good idea | 00:26 |
jeblair | jamielennox: if we include executor variables, we'll want to keep the other options in mind, and be careful of when we cross the threshold from things that are best done in variables to things that are best done with configuration topology. | 00:26 |
jamielennox | jeblair: agreed | 00:28 |
jeblair | jamielennox: also, with variables, we're going to want to think pretty carefully about safety. for instance, whether a job could influence where logs are copied because the logs post-playbook uses a variable. | 00:28 |
jamielennox | oh, hmm, whether a job can re-define logs_url over the top of what goes into executor variables | 00:28 |
jamielennox | is that fundamentally different though from a job just deciding to not inherit from base? | 00:29 |
jeblair | jamielennox: yes, because then they don't get the (presumably trusted) playbook which copies logs | 00:30 |
jeblair | so the outcome there is "no logs" vs "altered behavior of log copying playbook" | 00:31 |
jamielennox | so would it make sense to split executor variables to trusted and untrusted | 00:31 |
jamielennox | actually it shouldn't matter because in this case the log copying would be trusted wouldn't it | 00:31 |
jeblair | or perhaps say that they can not be overridden by jobs | 00:31 |
jamielennox | each playbook is executed in its own ansible process so if something in an untrusted scenario overrides a var it's not going to carry through to the trusted post anyway rigth? | 00:32 |
jeblair | jamielennox: right, but job variables are supplied to all of the playbook executions | 00:32 |
jamielennox | jeblair: oh, right, so they define it in the job variables, i was thinking like a set_fact | 00:33 |
jeblair | (fundamentally, this isn't very different from putting the same sort of thing in a base job 'vars' section, it's just more obvious (hopefully) that it can be overidden) | 00:34 |
jamielennox | hmm | 00:34 |
jeblair | jamielennox: as i think about this, i'm back to your example of 'success-url' and it occurs to me that isn't actually related to ansible at all; so executor variables couldn't (at the moment at least) be used to set that. | 00:35 |
jamielennox | so the only way i see around that is a trusted vars and untrusted vars file | 00:35 |
pabelanger | just catching up on backscoll. But what was the down side of having base->site-base->job for zuul.yaml? | 00:35 |
jamielennox | jeblair: what do you mean? it definitely works | 00:35 |
jamielennox | pabelanger: it means redefining job on top of site-base for every site including ones we spin up just for testing | 00:36 |
pabelanger | jamielennox: do you have an example today for your playbooks? | 00:37 |
jeblair | jamielennox: success-url is not used or set by ansible; that's internal to zuul. | 00:37 |
jamielennox | jeblair: oh? lol, ok i definitely used that as an example, sec | 00:37 |
pabelanger | ya, that is where I am getting confused | 00:38 |
jamielennox | https://github.com/jamielennox/project-config/blob/master/playbooks/roles/base/tasks/post.yaml#L16 | 00:38 |
jeblair | (log copying destination, however, would be something that could use a variable like this) | 00:38 |
jamielennox | same concept, i was just copying what was in -infra/project-config than the one out of my project-config | 00:38 |
jeblair | yeah, so that's the actual example :) | 00:39 |
jamielennox | pabelanger: so in my testing bonnyci.logs.dest needs to be: hostvars['playpen-logs']['ansible_default_ipv4']['address'] from the deploy scripts | 00:39 |
jeblair | (i think it would be nice if the log-copying playbook could feed information back to zuul, so that it could supply (part of?) the success-url) | 00:40 |
jeblair | (but that's a problem for a different day) | 00:41 |
jamielennox | jeblair: so in our case i'd happily just mandate a whole bunch of that stuff, like i don't need user's fiddling with workspace roots or logging urls or logging paths or anything like that | 00:41 |
jamielennox | but it's not a big concern for now | 00:41 |
jeblair | jamielennox: i think we might be saying the same things with different words; with you saying "trusted/untrusted variables files" and i saying "override" | 00:43 |
jeblair | jamielennox: i think we're in agreement that if executor_variables are to be a thing, we have to be able to rely on them. | 00:43 |
jeblair | regardless of how that actually gets implemented | 00:44 |
pabelanger | jamielennox: why don't you just use {{ hostvars['playpen-logs'].ansible_default_ipv4.address }} directly from your playbook? Or better yet, maybe use set_fact to do some condition checks | 00:44 |
jamielennox | jeblair: right, so i'm not sold on this solution it's just something i had come up - i think it will depend on how the trusted/untrusted split happens | 00:45 |
jamielennox | there's not much difference their atm | 00:45 |
jamielennox | realistically for now i don't care about feeding vars into the untrusted environment, but could maybe see that as something useful | 00:45 |
jamielennox | pabelanger: how? i have access to that variable at the time i deploy zuul, i need to save it out somewhere so i can have access to it at the time a job is running | 00:46 |
jamielennox | also that would involve encoding the term 'playpen-logs' into project-config right? that's not going to work on vagrant or production | 00:47 |
pabelanger | Hmm, so in our case. We are going to need logs.o.o in our job inventory file | 00:50 |
pabelanger | because, we'll need to delegate_host to logs.o.o, to rsync the file | 00:51 |
jeblair | pabelanger: we don't do that in v2.5. why would we do it in v3? | 00:51 |
pabelanger | well, we have site section in 2.5 that contains that info, doesn't it? | 00:52 |
pabelanger | cannot remember | 00:52 |
jamielennox | that information comes out of zuul.conf in 2.5 | 00:52 |
pabelanger | let me check | 00:52 |
jeblair | pabelanger: sure, but that's just emulating jenkins data structures. the actual implementation is just an rsync push from launcher to logs.o.o | 00:52 |
jeblair | pabelanger: the site defn does not get translated into an inventory entry. | 00:53 |
jamielennox | so realistically i don't know about rsync anyway, that would entail adding an ssh key to the executor capable of uploading files to our log server | 00:53 |
jeblair | yeah, that's the v2.5 approach and what i think we're likely to do for openstack in v3. to start with at least. | 00:54 |
jamielennox | for any sort of prod i think i'd want to put something a bit more secure there anyway - but that rules out adding anything to the ansible inventory | 00:54 |
pabelanger | right, I'm looking at it from the way of hosts: log_server for the playbook, which is an inventory look up | 00:55 |
jamielennox | i'm vaguely considering some kind of one off upload secret managed by (hashicorp) vault or something we can otherwise add to the job vars | 00:55 |
pabelanger | but, I know we do it today a little different with with rsync specific commands | 00:55 |
jeblair | jamielennox: it's worth noting that much of the design relies of the executor remaining secure. | 00:55 |
jamielennox | but we're really no where near that yet | 00:56 |
jeblair | pabelanger: only hosts that the job can manipulate should be in the inventory. the log server shouldn't be there. | 00:57 |
jeblair | (unless it's a job that runs on the log server) | 00:57 |
jamielennox | jeblair: so to my mind that depends a lot on what/how we let users define jobs and whether they can get code exec on the executor | 00:57 |
pabelanger | jeblair: not a job, but a task for example | 00:57 |
jeblair | jamielennox: right, that's an important consideration, which is why SpamapS has that spec | 00:58 |
pabelanger | let me quickly write a sudo playbook | 00:58 |
jamielennox | jeblair: well SpamapS's spec is how to stop a user breaking out of the executor, and in reality i think it is largely going to depend on us doing single use executors | 00:58 |
jeblair | jamielennox: what is a single-use executor? | 00:58 |
jamielennox | there's really no limit to what the user could do with code exec inside the executor | 00:59 |
jamielennox | jeblair: oh, create an executor, run one job, discard executor | 00:59 |
jeblair | jamielennox: sure there are limits -- we're contemplating 3 systems for enforcing those limits, all of which are detailed in the spec | 00:59 |
jamielennox | so anything the user does inside the executor is destroy | 01:00 |
jamielennox | ed | 01:00 |
SpamapS | The point of the spec is to make limits | 01:00 |
jeblair | jamielennox: and the spec is pretty clear that single-use executors is not the way we're going | 01:00 |
pabelanger | jeblair: http://paste.openstack.org/show/603544/ | 01:00 |
SpamapS | Too spensive and confusing | 01:01 |
pabelanger | that to me, would be a way of publish logs to logs.o.o | 01:01 |
pabelanger | but, logs.o.o is not in the inventory file | 01:01 |
jamielennox | jeblair: understood, but on last reading that wasn't ruling that out it was just about containment - but yes, the amount of work to launch and have a ready queue of executors is not worth it at this time | 01:02 |
jamielennox | pabelanger: so that's why i ended up having to do command: rsync - https://github.com/jamielennox/project-config/blob/master/playbooks/roles/base/tasks/post.yaml#L13 | 01:03 |
jamielennox | there was no way i could see to make ansible's synchronize push to an arbitrary host (which i'm guessing is intentional) | 01:03 |
jeblair | yeah, that's why v2.5 is constructed that way | 01:04 |
jeblair | (also, we should be careful about keys here, we may not want to use the zuul worker key to log into the log host) | 01:04 |
jeblair | (or maybe we do. we should think about it at least :) | 01:05 |
jeblair | it's past dinnertime for me, so i have to bow out now | 01:05 |
jamielennox | yea, i feel i've given people something to think about and i don't need a solution right now | 01:05 |
jamielennox | can discuss it more at another time | 01:05 |
pabelanger | add_host would be a work around to it not being in the inventory file | 01:06 |
pabelanger | but, keys need to be setup too | 01:06 |
pabelanger | for another day! | 01:06 |
jlk | Maybe we could teach Zuul how to contact an Object service! | 01:12 |
* jlk ducks | 01:12 | |
*** jamielennox is now known as jamielennox|away | 01:16 | |
*** pleia2_ is now known as pleia2 | 01:25 | |
*** jamielennox|away is now known as jamielennox | 01:28 | |
jamielennox | walked away from the computer for a while and realized that i super confused that conversation by using the word executor as both the server and the container environment that the executor will be creating | 02:15 |
jamielennox | then managed to confuse myself :) | 02:15 |
*** herlo has joined #zuul | 02:18 | |
*** herlo has quit IRC | 02:23 | |
*** herlo has joined #zuul | 02:25 | |
*** herlo has joined #zuul | 02:25 | |
*** MarkMielke has joined #zuul | 03:47 | |
*** jamielennox is now known as jamielennox|away | 04:12 | |
*** Guest45332 has joined #zuul | 04:49 | |
*** Guest82302 has joined #zuul | 04:59 | |
*** Guest45332 has quit IRC | 05:05 | |
*** Guest82302 has quit IRC | 05:09 | |
*** bhavik1 has joined #zuul | 06:29 | |
*** yolanda has quit IRC | 06:32 | |
*** yolanda has joined #zuul | 06:46 | |
*** hashar has joined #zuul | 08:45 | |
*** openstackgerrit has quit IRC | 09:03 | |
lennyb | Hi, I think that I've hit #link https://bugs.launchpad.net/zuul/+bug/1270029 . I am using zuul 2.5.1. Any ideas, comments, suggestions? | 09:28 |
openstack | Launchpad bug 1270029 in Zuul "zuul doesn't connect to gearman server" [Undecided,New] | 09:28 |
*** bhavik1 has quit IRC | 09:52 | |
*** jamielennox|away is now known as jamielennox | 10:30 | |
*** jamielennox is now known as jamielennox|away | 11:05 | |
tobiash_ | just curious, did someone experiment with somewhat unsafe filesystem tunings (like disable journaling) for single use nodes? | 11:09 |
tobiash_ | I'm thinking about reducing io footprint of the ci system in our environment | 11:10 |
*** jamielennox|away is now known as jamielennox | 11:12 | |
mordred | tobiash_: I don't think we have much - but now that you say it I want to | 11:26 |
pabelanger | Shrews: we ready to start nl01.o.o again this morning? Not sure if we are waiting on anything | 11:26 |
pabelanger | ah, maybe 447630 | 11:27 |
tobiash_ | mordred: I'm thinking about combining eatmydata with some mount options and vm.dirty* settings | 11:28 |
mordred | tobiash_: I think it's a great idea - if you wind up exploring it, I'd love to know what impact you find it to have | 11:37 |
tobiash_ | mordred: I think I'll try it out in our environment | 11:40 |
tobiash_ | and share knowledge if it had some possitive impact | 11:41 |
*** yolanda has quit IRC | 11:52 | |
*** openstackgerrit has joined #zuul | 11:53 | |
openstackgerrit | Paul Belanger proposed openstack-infra/zuul feature/zuulv3: Create zuul_workspace_root job variable https://review.openstack.org/441441 | 11:53 |
openstackgerrit | Paul Belanger proposed openstack-infra/zuul feature/zuulv3: Organize playbooks folder https://review.openstack.org/441547 | 11:53 |
openstackgerrit | Paul Belanger proposed openstack-infra/zuul feature/zuulv3: Add net-info to bootstrap role https://review.openstack.org/441617 | 11:53 |
openstackgerrit | Paul Belanger proposed openstack-infra/zuul feature/zuulv3: Add revoke-sudo role and update tox jobs https://review.openstack.org/441467 | 11:53 |
openstackgerrit | Paul Belanger proposed openstack-infra/zuul feature/zuulv3: Create tox-tarball job https://review.openstack.org/441609 | 11:53 |
*** yolanda has joined #zuul | 11:54 | |
pabelanger | jeblair: jlk: jamielennox: Shrews: so, in the effort to keep iterating forward on ^, jeblair suggestion of using the global zuul_workspace_root seems fine | 11:55 |
pabelanger | I do think we should establish a best practice guide if possible and try to avoid global vars whenever possible. | 11:55 |
pabelanger | But since these roles are still in flux, lets keep moving forward now | 11:56 |
mordred | pabelanger: BIKESHED .... | 11:57 |
mordred | pabelanger: is there any reason we're calling it zuul_workspace_root and not zuul.workspace_root that maybe I missed? | 11:57 |
pabelanger | mordred: zuul_workspace_root is in zuul.yaml, where our zuul. vars come from the vars.yaml file the executor creates | 11:58 |
pabelanger | so, even more namespacing of vars | 11:58 |
openstackgerrit | Paul Belanger proposed openstack-infra/zuul feature/zuulv3: Add ansible-lint to tox (pep8) https://review.openstack.org/438276 | 11:58 |
mordred | ah - gotcha. so in this case, that's actaully some infra job-content | 11:59 |
pabelanger | ya, we'll have to keep an eye on the variable prefix we use for job-content vars and zuul executor vars | 12:00 |
pabelanger | job_workspace_root might be better | 12:00 |
mordred | yah ... but yay for iteration :) | 12:01 |
pabelanger | 447089 is going to be the next one to bikeshed | 12:03 |
Shrews | pabelanger: yeah, i'd really like to see 447630 go in first | 12:12 |
Shrews | pabelanger: let me get some coffee and i'll fix it up for that -1 | 12:12 |
mordred | pabelanger: I think in my brain I'm thinking having the _role_ available is a good idea - but we don't necessarily need to create a centrally defined _job_ for that combo | 12:15 |
mordred | and maybe even the playbook | 12:16 |
mordred | pabelanger: so a user can say "job: my-local-py27 pre: setup-db" | 12:16 |
mordred | pabelanger: but I also agree, getting extra-test-setup will be nice | 12:16 |
openstackgerrit | Paul Belanger proposed openstack-infra/zuul feature/zuulv3: Create extra-test-setup role https://review.openstack.org/448042 | 12:20 |
pabelanger | mordred: ^ basically how we do it in JJB | 12:21 |
pabelanger | but, on the fence if we want extra-test-setup role added by default into our base tox-py27 job. Doing a file check doesn't feel like the right way to do it with ansible | 12:22 |
pabelanger | but ya, I think a pre-run: playbook, a project can opt into sets up a database is the right way | 12:22 |
pabelanger | for now, ^ should unblock zuul jobs from failing | 12:23 |
mordred | yup | 12:24 |
mordred | pabelanger: we may need to think about reusable playbooks that are not jobs or roles too - which should make jeblair aka "Mr. Multi-Repo Search Path" super happy to think about | 12:25 |
mordred | since needing to create central _jobs_ in all of the combinations seems like overkill, but just having a "setup-db" role means that to consume it a local user would need to make a setup-db playbook in their repo that just calls that one role - so we'll have that playbook cargo-culted in to a ton of repos | 12:26 |
mordred | and also - I'm with you on the extra-setup-role by default question. it both feels wrong to have the file search, but it also feels a bit overkill to do it the other way too maybe | 12:27 |
mordred | I knew it was going to get fun once we started actualy converting openstack's jobs | 12:28 |
mordred | turns out we have a LOT of edge cases available to us to consider just in zuul's unit tests :) | 12:28 |
pabelanger | indeed | 12:29 |
openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: Fix test_node_assignment_at_quota https://review.openstack.org/447630 | 12:32 |
Shrews | pabelanger: jeblair: clarkb: ^^^ | 12:33 |
mordred | Shrews: having a quota of 5 nodes seems to have really hepled us find bugs that happen at quota hasn't it? | 12:41 |
Shrews | mordred: definitely. the entire "pause processing" bit of code is slightly complex, and the limited quota has helped exercise the demons | 12:43 |
Shrews | but i feel like it's getting really solid now (after this bug fix) | 12:44 |
mordred | \o/ | 12:45 |
Shrews | though zuul is failing ALL the tests on that PS | 12:48 |
Shrews | http://logs.openstack.org/30/447630/4/check/gate-nodepool-pep8-ubuntu-xenial/f4565f8/console.html#_2017-03-21_12_38_50_844834 | 12:49 |
Shrews | neat | 12:49 |
Shrews | mordred: pabelanger: ^^^ setuptools issues today? | 12:49 |
Shrews | cryptography lib | 12:50 |
pabelanger | Hmm | 12:51 |
pabelanger | looking | 12:51 |
openstackgerrit | Paul Belanger proposed openstack-infra/nodepool feature/zuulv3: Add libffi development headers to bindep https://review.openstack.org/448076 | 12:53 |
pabelanger | should fix it | 12:54 |
openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: Fix test_node_assignment_at_quota https://review.openstack.org/447630 | 12:55 |
Shrews | cool. rebased | 12:56 |
openstackgerrit | Merged openstack-infra/nodepool feature/zuulv3: Add libffi development headers to bindep https://review.openstack.org/448076 | 13:56 |
*** hashar has quit IRC | 14:33 | |
*** hashar has joined #zuul | 14:35 | |
jeblair | mordred, pabelanger: i agree with most of what you've said, but i think if you consider what the tox-py27 job/playbook really is -- an implementation of openstack's common testing interface -- the fact that it encodes "policy" things like "run extra setup if it exists" doesn't seem very wrong. | 14:36 |
jeblair | mordred, pabelanger: i'm not sure i used the best words there, it's early, cut me some slack. :) | 14:37 |
mordred | jeblair: good morning! | 14:37 |
mordred | jeblair: however, if one considers a tox-py27 in the zuul standard lib that's not an openstack specific tox-py27 - this may be a fun rabbithole to consider | 14:38 |
jeblair | mordred: yeah -- i'm sure *roles* will end up in the standard lib; i'm not as certain that we're going to succeed in having jobs there. | 14:39 |
jeblair | so certainly the tox *role*, etc, need to be clean, modular, reusable. | 14:39 |
mordred | jeblair: yah - I guess I'm trying to suggest that if we _only_ have roles in our re-use bucket, there's going to be a gob of copy-pasta | 14:41 |
mordred | (this is a painpoint generally in ansible land ... roles are awesome, but people also need to share playbooks sometimes) | 14:41 |
jeblair | (for one thing -- in order to have a job in the stdlib, you'd have to have a convention on inhertance -- like they all ultimately inherit from 'base' -- but 'base' almost certainly needs site-local configuration -- so do we leave a hole in the inheritance chain for that? possible, but weird) | 14:41 |
jeblair | mordred: roles can (now) include other roles, so that is a way to achive some of what playbook sharing achieves | 14:42 |
pabelanger | The more I think about it, I do like the idea mordred had. If we added a new project to zuulv3, and tox-py27 was a trusted job, we could add a variant of tox-py27, which included a pre-run to a untrusted playbook called database_setup. Then use an upstream role on galaxy to do that step: https://github.com/geerlingguy/ansible-role-mysql | 14:42 |
jeblair | yep. that's still a possibility. and we did make 'pre-run' a list, so you can tack on "db-setup" "extra-setup" "bindep" "whateverelsewethinkup" all in a pre-run section of a job. | 14:44 |
pabelanger | I think it is more a question, do we want to provider a playbook in openstack-infra that will run tox and setup a database for you (pretty common) or expect the each project to write that logic them self | 14:46 |
jeblair | i think that the pre/post playbook option is important -- but i think that for something so common, it's okay for the 'openstack standard' tox job to do that for you automatically. | 14:48 |
pabelanger | ya, and there is so many ways to make that dynamic | 14:49 |
pabelanger | for now https://review.openstack.org/#/c/448042/ is that way | 14:49 |
pabelanger | basically a straight convert from JJB | 14:49 |
pabelanger | downside of that, more if / else logic in playbooks. Which, as it grows could get more problematic. Another reason I like the idea of a tox-db-py27 playbook specifically | 14:51 |
mordred | jeblair: yah - I thnk it's totally great for the openstack standard job to do it for folks automatically - since openstack after all | 14:52 |
mordred | jeblair: but if someone says "I want to run python27 jobs, but as part of setup I want to make sure that there is a mysql database set up" | 14:53 |
mordred | that's pretty easy for them to just do in their .zuul.yaml file - except as of right now it means they'll need to make a 3 line playbook that says "hosts: *\n roles:\n setup-mysql" and then reference that in their pre-run | 14:55 |
mordred | (ignoring the current openstack "always run a tools/test-setup.sh if it's in the repo" cantrip - using it more as an example of combining more than one logical thing) | 14:56 |
jeblair | mordred: ya. so maybe the way to do what you're suggesting is not to do a multi-repo search path, but rather to allow repo specification of playbooks | 14:57 |
jeblair | mordred: "pre-run: git.openstack.org/openstack-infra/project-config/playbooks/setup-mysql" or something | 14:58 |
jeblair | mordred: that way we don't have to deal with maintaining a file inventory of every project-branch combination, or worry about what happens when cloudkitty adds their own setup-mysql playbook | 14:59 |
mordred | jeblair: who doens't want the cloud-kitty playbook??? | 14:59 |
jeblair | ironic, that's who | 14:59 |
mordred | silly ironic | 15:00 |
pabelanger | pre-run: git.o.o/foo is a neat idea | 15:03 |
jroll | hey, ironic loves kitties, even cloudy ones | 15:03 |
mordred | especially if we can apply our syntactic sugar maybe - so that in openstack people can reference openstack-infra/project-config/playbooks/setup-mysql and that'll expand to have git.o.o on front of it maybe? (thinking outloud) | 15:05 |
mordred | jroll: :) | 15:05 |
* jeblair bows to our new feline overlords | 15:05 | |
jeblair | mordred: yeah, we may be able to make it a standard "project reference" whatever that ends up being. we'll just have to figure out how to tack a file path onto a project reference. (do we need a delimiter, or can we figure it out?) | 15:06 |
mordred | yah | 15:07 |
pabelanger | but, we can do pre-run: openstack-infra/project-config/playbooks/setup-mysql today, if we added a setup-mysql job in our trusted repo right? | 15:08 |
pabelanger | then a project adds a variant job, with pre-run: setup-mysql | 15:09 |
pabelanger | trying to wrap my brain around everything | 15:09 |
mordred | pabelanger: we'd just have to have a setup-mysql job that was a 'fake' job we weren't really expecting anyone to use as an actual job - which seems hacky to me | 15:11 |
jeblair | that doesn't sound right -- | 15:11 |
jeblair | i thought the issue we're talking about is that a playbook (whether pre, main, or post) is assumed to be in the repo where the job is defined, so you can't use a playbook in a different repo | 15:12 |
jeblair | even if you had a setup-mysql playbook in another repo, you couldn't tell zuul to run it | 15:13 |
mordred | yes. that is the problem | 15:13 |
mordred | it's entirely possible pabelanger are smoking something with our most recent thing | 15:13 |
mordred | s/pabelanger/pabelanger and I/ | 15:14 |
jeblair | you could have a setup-mysql job, and *inherit* from that, but we don't have multiple inheritance (yet?) so it's not something you can mix-in. so the inheritance model would be more like "tox-py27 -> tox-db -> tox -> base" | 15:15 |
mordred | right. and with that and single inheritance we'd wind up maybe with a big pile of jobs like "tox-py27-with-db" and "tox-py27-without-db" which would be silly | 15:16 |
jeblair | yup | 15:16 |
pabelanger | ya | 15:16 |
pabelanger | this is getting fun! | 15:17 |
pabelanger | :) | 15:17 |
mordred | since the thing pabelanger said is I think how the consumer mindset will be "I want a job that runs the python27 content that I didn't right, but a variant of that which ensures a database is installed that I also didn't write" | 15:17 |
mordred | so, I want a variant of the python27 job that does this other pre-canned thing in pre-run | 15:17 |
jeblair | anyway, i like the cross-project playbook reference -- though we do need to decide its effect on trustedness -- does a playbook in another project run with the trusted flag of the project it's in or the project that's referencing it? | 15:18 |
mordred | jeblair: that's an EXCELLENT question :) | 15:18 |
jeblair | i think referencing | 15:19 |
jeblair | but i'm not sure :) | 15:20 |
pabelanger | I like that, cross-project playbook reference | 15:20 |
mordred | I think you're right - since I per-repo opt-in actions like "I want you to publish my artifacts to this service I have" are most likely to want to be tenant credentials | 15:20 |
mordred | whereas base things that would need shared credentials are more likely to be from inheritance perhaps? | 15:20 |
mordred | or maybe I'm wrong abot that | 15:20 |
mordred | no - I think that's right - in openstack we don't really want joe-random-user to be able to write things to trusted AFS locations | 15:21 |
mordred | but - we _could_ maybe have a project that requests an AFS space that we give them along with a kerb principal and then they would put that into their tenant secrets and they could publish to _their_ AFS location with their security context | 15:22 |
*** dkranz has quit IRC | 15:23 | |
jeblair | mordred: yeah. i'm pretty sure it wants to be referencing, because otherwise, imagine the trusted playbook operated on variables. but within its defining (trusted) repo, the author knew that the variables were safe (only called from final jobs, etc -- like maybe a pypi-upload job). if you allowed an untrusted repo to invoke it in the trusted context, you've bypassed that. | 15:37 |
mordred | yup | 15:37 |
pabelanger | Oh, bikeshed! | 15:56 |
*** bhavik1 has joined #zuul | 15:56 | |
pabelanger | jeblair: I missed your comment in 441441 | 15:56 |
pabelanger | So, I did make zuul_workspace_root: /home/zuul, this was intentional. | 15:56 |
pabelanger | however, the question is, do we really need a /home/zuul/workspace? | 15:57 |
pabelanger | A /home/zuul is less characters, which helps tox.ini issue. | 15:58 |
pabelanger | but /home/zuul/workspace is more inline with existing things | 15:58 |
jeblair | pabelanger: as long as we aren't putting the git repos straight in /home/zuul. if we put them in /home/zuul/src/git.openstack.org/openstack/cloud-kitty that works for me. | 15:58 |
pabelanger | yes, that is where git repos live | 15:59 |
jeblair | ++ | 15:59 |
Shrews | oh, glad that's resolved. i was tempted to start a new bikeshed on the name "workspace" | 16:00 |
Shrews | only b/c i wanted in on the fun | 16:00 |
pabelanger | :) | 16:00 |
jeblair | pabelanger: jlk has some questions inline on ps8 | 16:00 |
pabelanger | jeblair: ya, pushed up some replied. Might have missed one | 16:01 |
Shrews | pabelanger: fyi, almost ready to start n-l on nl01. just waiting for the puppet to do things | 16:01 |
pabelanger | Shrews: yay | 16:02 |
pabelanger | jeblair: Oh, replies are drafts currently. | 16:03 |
pabelanger | let me fix | 16:03 |
jeblair | ah yeah, the old publish to wrong patchset issue :) | 16:04 |
*** isaacb has joined #zuul | 16:07 | |
*** isaacb_ has joined #zuul | 16:07 | |
*** isaacb_ has left #zuul | 16:08 | |
mordred | ++ to /home/zuul/src/git.openstack.org/openstack/cloud-kitty ... (and our go friends can then set GOPATH={{zuul_workspace_root}} and all will be joyous and wonderful | 16:16 |
Shrews | pabelanger: n-l started | 16:20 |
pabelanger | Shrews: cool | 16:20 |
Shrews | pabelanger: and very happy to see the ServerList output only 1/min now. :) | 16:22 |
pabelanger | Shrews: Yay, it works | 16:24 |
Shrews | will take a bit for it to catch up. over 100 requests to satisfy | 16:25 |
pabelanger | time to start using infracloud-vanilla? | 16:31 |
jlk | pabelanger: mordred: simple answer on zuul_workspace_root vs zuul.workspace_root: The latter is a hash, and by default, Ansible does not merge hashes together. So if at any level you wanted to redefine some sub-key of the zuul hash, you'd have to re-define the _entire_ hash. OR you have to turn on non-default behavior in Ansible, which is not the path of least surprise. | 16:34 |
*** isaacb has quit IRC | 16:38 | |
pabelanger | right, the zuul has (eg zuul.executor) is what we setup specifically from the executor: http://git.openstack.org/cgit/openstack-infra/zuul/tree/zuul/executor/server.py?h=feature/zuulv3#n820 | 16:38 |
jeblair | jlk, pabelanger, mordred: yeah -- maybe we should reserve "zuul." for zuul itself, and "zuul_" for site-local global variable things (which is what is happening here) | 16:38 |
pabelanger | s/has/hash* | 16:38 |
jeblair | i don't know if that's a useful distinction, or just confusing. :) | 16:38 |
jlk | yeah, I'm generally wary of hashes. | 16:38 |
jlk | particularly as role defaults | 16:38 |
pabelanger | ya, so may ways to do this | 16:40 |
pabelanger | zuul.vars (doesn't exist) could be our hash from the job vars section. then it is zuul.vars.zuul_workspace_root | 16:41 |
jeblair | fungi, clarkb, mordred, SpamapS: i've sent an email to openstack-infra and openstack-dev (tagged with infra and security to try to get lots of eyes) about encryption. i put a lot of work into it. hopefully it's accurate and a useful basis for making some decisions (or at least a starting point for discussion if it is way off). | 16:41 |
fungi | thanks jeblair!!! | 16:42 |
jeblair | please point it out to others who should see it and may otherwise miss it :) | 16:42 |
SpamapS | jeblair: will do, thanks for the heads up | 16:42 |
jeblair | i have to afk for a bit. bbl. | 16:42 |
*** bhavik1 has quit IRC | 16:48 | |
mordred | jeblair: it's worth noting that one of the options you list is also implemented by our friends in keystone land so there may be some experience with it in our community | 16:54 |
*** herlo has quit IRC | 17:03 | |
*** herlo has joined #zuul | 17:05 | |
pabelanger | mordred: do you mind looking at 446749, relates to your callback logic for zuul_stream | 17:09 |
openstackgerrit | Paul Belanger proposed openstack-infra/zuul feature/zuulv3: Remove ZooKeeperConnectionConfig class https://review.openstack.org/447683 | 17:18 |
openstackgerrit | Paul Belanger proposed openstack-infra/zuul feature/zuulv3: Remove ZooKeeperConnectionConfig class https://review.openstack.org/447683 | 17:27 |
*** hashar has quit IRC | 17:28 | |
mordred | pabelanger: looking | 17:37 |
mordred | jamielennox: ^^ nice +A | 17:39 |
openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Handle the 'all' group in callback. https://review.openstack.org/446749 | 17:47 |
openstackgerrit | Paul Belanger proposed openstack-infra/zuul feature/zuulv3: Create extra-test-setup role https://review.openstack.org/448042 | 18:27 |
pabelanger | jlk: thanks^. good news is our testing confirmed | 18:28 |
jlk | cool | 18:29 |
*** dkranz has joined #zuul | 18:31 | |
Shrews | nl01 all caught up now. pabelanger, jeblair: i say we think about introducing vanilla tomorrow-ish if no more problems crop up | 18:35 |
jlk | hahahah *sigh* | 18:39 |
jlk | wasted a fair amount of time because I didn't realize mysql stuff got added to bindeps | 18:39 |
jlk | for testing | 18:39 |
jlk | hrm, still not passing | 18:40 |
jlk | Do I need to do something special to get mysql tests to work? | 18:40 |
jlk | should the server be running? | 18:40 |
openstackgerrit | Paul Belanger proposed openstack-infra/zuul feature/zuulv3: Create extra-test-setup role https://review.openstack.org/448042 | 18:41 |
pabelanger | jlk: tools/test-setup.sh should setup the database for zuul | 18:41 |
jlk | oh huh | 18:41 |
jlk | I missed that file | 18:41 |
jlk | dammit that's asking for a sudo password, which I don't know. | 18:43 |
jlk | oh haha | 18:43 |
jlk | wrong box | 18:43 |
jlk | oh cool, so other people's tests are failing due to sql stuff too. | 18:47 |
jlk | Are the nodepool nodes in zuulv3-dev not set up right? | 18:47 |
pabelanger | jlk: that is what I am fixing with 448042 | 18:48 |
jlk | hahah okay | 18:48 |
pabelanger | we merged sql reporter support last week | 18:48 |
jlk | How was the addition of the need for this passed? | 18:48 |
jlk | did it never work on the zuulv3 nodes, and only wokred on the old system? | 18:49 |
pabelanger | no, we only readd database support | 18:49 |
pabelanger | so it only broke recently | 18:49 |
jlk | hrm. | 18:49 |
jlk | so it's broken in the "zuul" tests, which are voting | 18:49 |
pabelanger | and we've had nl01.o.o offline for a few days | 18:49 |
jlk | how did it merge? | 18:49 |
jlk | ooooh | 18:49 |
jlk | so it happened when zuul wasn't even testing | 18:50 |
pabelanger | because it works properly on nodepool.o.o | 18:50 |
pabelanger | right | 18:50 |
jlk | gotcha. Okay. | 18:50 |
pabelanger | zuul user doesn't gate yet either | 18:50 |
jlk | what's the difference between non-voting and gate? | 18:51 |
jlk | oh I get it | 18:51 |
jlk | it still "votes" a -1, but the gate pipeline doesn't care about votes from that user | 18:51 |
clarkb | non voting is any test that doesn't affect the +/- value. gating is any test that prevents a change from merging if it fails (not that voting is required for gating) | 18:52 |
pabelanger | right, zuul.o.o only care about jenkins right now | 18:52 |
clarkb | s/not/note/ | 18:52 |
pabelanger | we also only have a single pipeline on zuulv3-dev.o.o too, check | 18:52 |
jlk | alright this makes my world view a bit better. | 18:56 |
pabelanger | nice, https://review.openstack.org/#/c/448042/ all green again | 18:56 |
pabelanger | jeblair: mordred: Shrews: SpamapS: ^ mind a review, that makes our tests green again on zuulv3-dev | 18:57 |
Shrews | pabelanger: jeblair: i think it is time to think about making our zuul-np integration job voting. i would note, though, it seems to be broken now. | 18:58 |
Shrews | not sure where we broke it | 18:58 |
jhesketh | Morning | 18:58 |
jeblair | Shrews: i agree | 18:58 |
jeblair | also, all the more reason :) | 18:58 |
pabelanger | Shrews: ya, I've been trying to bring the job back online. 447683 was the first step, but not sure the hostname issue | 18:59 |
pabelanger | jhesketh: morning! do you think you could rebase master to feature/zuulv3 for nodepool again? | 19:01 |
jhesketh | pabelanger: yes. I was working on that but it was non trivial. I'll get it working today | 19:02 |
pabelanger | jhesketh: great! Ya, I suspect its some work | 19:03 |
jhesketh | It's more just repeating what I had done. There isn't an easy way to rebase a merge commit | 19:04 |
Shrews | pabelanger: what's up with this message in the ansible output? http://paste.openstack.org/show/603671/ | 19:05 |
Shrews | from http://zuulv3-dev.openstack.org/logs/f749706319924e38819d8c852b8eb36d/ansible_log.txt | 19:06 |
pabelanger | Shrews: https://review.openstack.org/#/c/446749/ fixes it | 19:08 |
pabelanger | we need to schedule an update of zuulv3-dev to pick it up | 19:08 |
Shrews | i love when there are already solutions to things | 19:08 |
openstackgerrit | Paul Belanger proposed openstack-infra/zuul feature/zuulv3: Fix hostname issue with nodepool integration job https://review.openstack.org/448239 | 19:08 |
pabelanger | I think that is our integration job fix too | 19:08 |
clarkb | pabelanger: re ^ why wouldn't you use the hostname of the test machine? | 19:09 |
clarkb | isn't that where it comes from in the real world? | 19:09 |
pabelanger | clarkb: I guess we could, that was just copypasta from another job | 19:10 |
clarkb | for integration I think its valuable to actually test those surfaces | 19:10 |
pabelanger | self.hostname = socket.gethostname() ? | 19:10 |
clarkb | or let whatever code reads the hostname already do it | 19:11 |
pabelanger | clarkb: we just default that for __init__ on the scheduler | 19:12 |
clarkb | ya I would let it do it for you then | 19:12 |
clarkb | rather than set it something different in the test | 19:12 |
clarkb | (I guess the risk there is it requires properly configured /etc/hosts) | 19:12 |
openstackgerrit | Paul Belanger proposed openstack-infra/zuul feature/zuulv3: Fix hostname issue with nodepool integration job https://review.openstack.org/448239 | 19:14 |
jlk | anybody know of an example in v3 where you have some jobs in a pipeline that only run if a higher level pipeline job finishes? | 19:38 |
jlk | aaah | 19:39 |
openstackgerrit | Jesse Keating proposed openstack-infra/zuul feature/zuulv3: Allow github trigger to match on branches/refs https://review.openstack.org/445625 | 19:48 |
openstackgerrit | Jesse Keating proposed openstack-infra/zuul feature/zuulv3: Better merge message for GitHub pull reqeusts https://review.openstack.org/445644 | 19:48 |
openstackgerrit | Jesse Keating proposed openstack-infra/zuul feature/zuulv3: Set filter according to PR/Change in URL https://review.openstack.org/446782 | 19:48 |
openstackgerrit | Jesse Keating proposed openstack-infra/zuul feature/zuulv3: Support for dependent pipelines with github https://review.openstack.org/445292 | 19:48 |
openstackgerrit | Jesse Keating proposed openstack-infra/zuul feature/zuulv3: GitHub file matching support https://review.openstack.org/446113 | 19:48 |
openstackgerrit | Jesse Keating proposed openstack-infra/zuul feature/zuulv3: Log GitHub API rate limit https://review.openstack.org/446150 | 19:48 |
openstackgerrit | Jesse Keating proposed openstack-infra/zuul feature/zuulv3: Query changes only via relevant sources https://review.openstack.org/448257 | 19:48 |
*** hashar has joined #zuul | 19:53 | |
openstackgerrit | Paul Belanger proposed openstack-infra/zuul feature/zuulv3: Create zuul_workspace_root job variable https://review.openstack.org/441441 | 20:06 |
openstackgerrit | Paul Belanger proposed openstack-infra/zuul feature/zuulv3: Organize playbooks folder https://review.openstack.org/441547 | 20:06 |
openstackgerrit | Paul Belanger proposed openstack-infra/zuul feature/zuulv3: Add net-info to bootstrap role https://review.openstack.org/441617 | 20:06 |
openstackgerrit | Paul Belanger proposed openstack-infra/zuul feature/zuulv3: Add ansible-lint to tox (pep8) https://review.openstack.org/438276 | 20:06 |
openstackgerrit | Paul Belanger proposed openstack-infra/zuul feature/zuulv3: Add revoke-sudo role and update tox jobs https://review.openstack.org/441467 | 20:06 |
openstackgerrit | Paul Belanger proposed openstack-infra/zuul feature/zuulv3: Create tox-tarball job https://review.openstack.org/441609 | 20:07 |
jeblair | jlk: http://git.openstack.org/cgit/openstack-infra/zuul/tree/tests/fixtures/config/dependency-graph/git/common-config/zuul.yaml?h=feature/zuulv3 | 20:10 |
jeblair | jlk: the rejiggering as a graph bit just recently landed | 20:10 |
jeblair | jlk: (it used to be a tree) | 20:10 |
jesusaur | jeblair: is there any context where it would make sense to create a merger:merge job for a Ref, or will they all be Changes? | 20:24 |
jesusaur | also, was someone working on refactoring the Changeish classes? | 20:25 |
jeblair | jesusaur: only changes (the changes we need to speculatively merge, but we can be certain that the executors will be able to check out the refs) | 20:28 |
jeblair | jesusaur: currently assigned to SpamapS: but i don't think he's started yet? https://storyboard.openstack.org/#!/story/2000781 | 20:29 |
SpamapS | jeblair: I have started analysis. Have not succeeded in making much progress. | 20:31 |
SpamapS | I would like to keep pushing forward on it. | 20:31 |
SpamapS | I just keep distracting myself with test re-enablement because it's easier to see immediate progress. | 20:32 |
jesusaur | jeblair: cool, thanks; I'm slowly making more progress | 20:35 |
* SpamapS will double back to it now actually | 20:36 | |
jeblair | cool; i have to afk again for a bit; back in a couple hours. | 20:37 |
openstackgerrit | K Jonathan Harker proposed openstack-infra/zuul feature/zuulv3: Perform pre-launch merge checks https://review.openstack.org/446275 | 20:41 |
pabelanger | Doh! | 20:42 |
pabelanger | another patch bomb, sorry | 20:43 |
openstackgerrit | Paul Belanger proposed openstack-infra/zuul feature/zuulv3: Create zuul_workspace_root job variable https://review.openstack.org/441441 | 20:45 |
openstackgerrit | Paul Belanger proposed openstack-infra/zuul feature/zuulv3: Organize playbooks folder https://review.openstack.org/441547 | 20:45 |
openstackgerrit | Paul Belanger proposed openstack-infra/zuul feature/zuulv3: Add net-info to bootstrap role https://review.openstack.org/441617 | 20:45 |
openstackgerrit | Paul Belanger proposed openstack-infra/zuul feature/zuulv3: Add ansible-lint to tox (pep8) https://review.openstack.org/438276 | 20:45 |
openstackgerrit | Paul Belanger proposed openstack-infra/zuul feature/zuulv3: Add revoke-sudo role and update tox jobs https://review.openstack.org/441467 | 20:45 |
openstackgerrit | Paul Belanger proposed openstack-infra/zuul feature/zuulv3: Create tox-tarball job https://review.openstack.org/441609 | 20:45 |
openstackgerrit | Paul Belanger proposed openstack-infra/zuul feature/zuulv3: Create extra-test-setup role https://review.openstack.org/448042 | 20:45 |
rbergeron | wow. that is da bomb o'patches | 20:45 |
pabelanger | okay, that should make the stack green | 20:45 |
jlk | jeblair: thanks, I found it and fixed up my set | 21:03 |
SpamapS | do I need to be running mysql for tests now too? | 21:37 |
pabelanger | SpamapS: yes | 21:40 |
SpamapS | oy | 21:40 |
SpamapS | getting heavy | 21:40 |
SpamapS | we should drop a Dockerfile or something in the root so devs can spin up fast | 21:41 |
jlk | there's a tools/ script | 21:41 |
jlk | tools/test-setup.sh | 21:41 |
jlk | but it doesn't do the bindep install | 21:41 |
SpamapS | I'm writing up a simple Dockerfile now | 21:41 |
jlk | yeah I honestly hadn't thought of doing that. Makes a fair amount of sense | 21:42 |
SpamapS | just from the standpoint of not having to build a big clean env and not polluting your laptop.. it's a win :-P | 21:43 |
jlk | yeah, I've been doing all my testing in a cloud VM | 21:44 |
jlk | gotta say though, the constant push / pull of code to/from my laptop to test is a pain | 21:44 |
SpamapS | trying not to have to | 21:44 |
SpamapS | yeah remote editting is t3h suck | 21:44 |
pabelanger | we could switch to sqlite I guess | 21:48 |
pabelanger | well | 21:48 |
pabelanger | one option is to switch to sqlite :) | 21:48 |
pabelanger | depending on if jeblair sees value in that or not | 21:48 |
clarkb | in the past we have explicitly switched away because raisins | 21:49 |
pabelanger | Dockerfile in tree might bitrot pretty quick, since we don't test against docker | 21:49 |
clarkb | *away from sqlite | 21:49 |
pabelanger | clarkb: right | 21:49 |
clarkb | if the dockerfile uses bindep and test-setup.sh it is probably fine | 21:49 |
clarkb | thats really similar to what we do on our test VMs | 21:49 |
SpamapS | pabelanger: not hard to make a docker test job though | 21:50 |
SpamapS | the only reason I say docker is that I want a lightweight local test env | 21:50 |
clarkb | I've had a docker that runs mysql just for nodepool tests for a long time. | 21:51 |
pabelanger | whatever happen to dox? | 21:51 |
pabelanger | wasn't that a docker for tox thing | 21:51 |
clarkb | but nothing fancy, I just grabbed upstream ubuntu image and apt-get installed mysql | 21:51 |
clarkb | pabelanger: I think it mostly died? mordred was deving on it. Not sure it really solved the problem people had? | 21:52 |
clarkb | I dunno I never used it beacuse I didn't like the idea of giving unittests root on my boxes | 21:52 |
pabelanger | ya, I never tried it | 21:52 |
pabelanger | just knew of it | 21:52 |
SpamapS | DNS is really slow from inside my docker containers for some reason. | 21:55 |
SpamapS | I think it's confused by my native ipv6 | 21:55 |
pabelanger | okay, dox cloned | 21:57 |
pabelanger | now to install docker | 21:57 |
clarkb | SpamapS: did docker stop hardcoding 8.8.8.8 and 8.8.4.4? | 21:59 |
SpamapS | clarkb: I dunno | 21:59 |
SpamapS | clarkb: the ubuntu docker image uses the host's resolv.conf | 21:59 |
SpamapS | which for me right now is the IBM vpn | 22:00 |
SpamapS | which is actually probably where much of the confusion comes from, not IPv6 | 22:00 |
SpamapS | With needing to install build-essential, zookeeperd, and mysql-server ... this is a big docker image. | 22:00 |
SpamapS | interesting with dox... | 22:01 |
SpamapS | reads .travis.yml | 22:01 |
SpamapS | Fetched 122 MB in 5min 41s (356 kB/s) | 22:04 |
SpamapS | slow ubuntu mirror.. argh | 22:05 |
pabelanger | clarkb: I see what you mean about root | 22:05 |
pabelanger | docker: Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock | 22:05 |
pabelanger | need to figure that out | 22:05 |
SpamapS | pabelanger: you just need to be in the docker group | 22:06 |
clarkb | pabelanger: you add yourself to the docker group but once you've done that the user effectively has root as long as the docker daemon is running | 22:06 |
clarkb | since docker can do all the things! | 22:06 |
pabelanger | ah | 22:06 |
pabelanger | ya | 22:06 |
SpamapS | clarkb: that's a bit of an oversimplification | 22:06 |
clarkb | I mean in as much as you can't just sudo foo | 22:06 |
clarkb | there are steps between here and there | 22:07 |
SpamapS | a lot | 22:07 |
clarkb | but they are straightforward and not difficult | 22:07 |
clarkb | you just make a container with root in the hosts user namespace aiui | 22:07 |
SpamapS | known steps.. defined steps.. but.. you're going to have to invite untrusted code to run in your containers first. | 22:07 |
SpamapS | yeah don't do that | 22:07 |
SpamapS | that's a privileged container | 22:07 |
SpamapS | don't do that :) | 22:07 |
clarkb | but I don't control what $randomtestsuite does | 22:08 |
SpamapS | just like don't have passwordless sudo | 22:08 |
clarkb | which is why I have found it safer to run in VM as unprivileged user | 22:08 |
SpamapS | clarkb: You trust yourself not to have any local root priv escalation too though, so... not that far off. | 22:08 |
clarkb | SpamapS: the problem isn't in the specific zuul case | 22:08 |
clarkb | SpamapS: its in the general use dox as test runner tool | 22:08 |
SpamapS | Like, you invite code to run on your box all the time. | 22:08 |
SpamapS | Same reason I don't use my dev box to access my PGP keys... I have so much garbage code in and out of this thing.. | 22:09 |
pabelanger | hey, that worked | 22:09 |
clarkb | SpamapS: right, but there is a difference between trusting my distro provided code and grabbing random openstack project and running `dox` | 22:09 |
clarkb | so what I do is run VM with kvm and run tox in there as ubuntu | 22:10 |
pabelanger | cool, dox doing thing | 22:10 |
SpamapS | same danger running 'tox' IMO | 22:10 |
pabelanger | we'll need to update this to use DIB | 22:10 |
SpamapS | clarkb: ah yeah you're sealing that off | 22:10 |
clarkb | SpamapS: its less because there isn't a direct line to root | 22:10 |
SpamapS | I'm not | 22:10 |
clarkb | my workstation stays pristine | 22:10 |
SpamapS | just accepting they might pwn me.. and this is a throw away box without secrets on it. | 22:10 |
clarkb | in as much as unprivilged user can escape kvm | 22:10 |
SpamapS | (well.. except my GH/Gerrit SSH keys.. but I can revoke those) | 22:11 |
SpamapS | clarkb: yeah, your way is way safer | 22:11 |
SpamapS | and WAY more of a PITA | 22:11 |
SpamapS | though.. I imagine you have a host->guest mount so you can edit local? | 22:11 |
SpamapS | that might make local vms more attractive | 22:11 |
clarkb | I've done that, but currently I just copy patches around | 22:12 |
SpamapS | yeah, f that! ;) | 22:12 |
clarkb | (either via git push or git apply whatever) | 22:12 |
SpamapS | push.. apply.. copy.. edit.. ggggggggggahhhh | 22:12 |
clarkb | eh | 22:12 |
SpamapS | no thanks :) | 22:12 |
SpamapS | but a host->guest would work too | 22:12 |
clarkb | its trivial to do the other thing | 22:12 |
clarkb | I found that further separating and not nuking my actual repos was desireable | 22:13 |
clarkb | just sshfs done | 22:13 |
jlk | I wind up using a github fork as a staging place to pass changes around | 22:13 |
jlk | git push -f and git fetch && git reset --hard | 22:13 |
SpamapS | jlk: yeah I've done that hell too | 22:13 |
SpamapS | damnit | 22:13 |
SpamapS | wasn't going to spend all afternoon dockering | 22:14 |
jlk | price one pays for using OS X | 22:14 |
SpamapS | but ubuntu's slow mirror is killin me | 22:14 |
clarkb | also I found virt install | 22:14 |
clarkb | which makes the boot VM process simple | 22:15 |
SpamapS | oh good work docker.. you just fell back to Readline but you can't read lines from me. | 22:15 |
pabelanger | okay, so dox isn't that bad! | 22:21 |
pabelanger | I have to rebuild base image, but it still works | 22:21 |
SpamapS | pabelanger: why not just write a Dockerfile? | 22:21 |
SpamapS | http://paste.openstack.org/show/603699 | 22:21 |
SpamapS | thats the one I'm testing right now | 22:22 |
mordred | pabelanger: I don't think any of us are working on dox anymore - but I'd be more than happy to give you core on it immediately if you wanted to | 22:22 |
mordred | (out of curiosity, what are we dockering?) | 22:22 |
SpamapS | mordred: zuul testing | 22:22 |
SpamapS | since now I have to run mysql-server and zookeeperd to run tests | 22:22 |
clarkb | SpamapS: oh I remember the other reason for not mounting git repos into VMs, it was devstack doing things and I realized I couldn't trust tests to not break me either | 22:23 |
SpamapS | which I think is cool | 22:23 |
SpamapS | not a bad thing, but a great thing, to test against the real things | 22:23 |
SpamapS | but my laptop doesn't need all those things most of the time | 22:23 |
clarkb | basically that was sufficient surface area for me to have a bad day so I removed it | 22:23 |
SpamapS | clarkb: yeah I'd make that mount RO | 22:23 |
mordred | SpamapS: cool | 22:23 |
clarkb | SpamapS: but then you can't commit your changes? it sort of defeats the sync back and forth use case | 22:24 |
clarkb | oh you do it on the other end derp | 22:24 |
clarkb | gotcha | 22:24 |
SpamapS | I'd be done with that Dockerfile but for some reason archive.ubuntu.com has always been really slow for me over ipv4. In fact, when it got fast was when I noticed I had native v6 from Spectrum. | 22:24 |
SpamapS | clarkb: I edit on my local box, with my vim, and my settings. | 22:24 |
* jlk tries to get his laptops docker to work again | 22:25 | |
SpamapS | clarkb: that's the main reason I want to mount things in the VM | 22:25 |
jlk | I"m sure this is all going to go sideways because OSX | 22:26 |
jlk | but worth a try | 22:26 |
SpamapS | another annoyance with both docker and vms is that I have to tell them how to find my local squid cache that has all these stupid debs in it already | 22:26 |
SpamapS | I remember there was a thing for a while.. squid-deb-proxy.. that would make your squid an avahi service | 22:27 |
SpamapS | until IIRC somebody MITM'd some debs at a UDS | 22:27 |
pabelanger | Hey it worked | 22:27 |
pabelanger | dox.tests.config.test_commands.TestCommands.test_commands(dox_yaml) 0.021 | 22:28 |
SpamapS | just to see if anyone would install anyway | 22:28 |
pabelanger | that was in my infra/trusty container | 22:28 |
SpamapS | pabelanger: it's 2017... time to xenial | 22:30 |
SpamapS | ;) | 22:30 |
pabelanger | ya, need to patch dox :) | 22:36 |
pabelanger | mordred: going to try working with it, see how it goes. So, expect some patches | 22:37 |
jlk | SpamapS: isn't that docker file missing the tools/test-setup.sh ? | 22:38 |
SpamapS | jlk: it is, because I'm still just trying to get bindep to work | 22:38 |
jlk | ah | 22:38 |
SpamapS | it's failing for some reason | 22:39 |
mordred | pabelanger: you are already core on it, since infra-core is core on it - but I added you explicitly anyway | 22:39 |
SpamapS | aha! | 22:40 |
SpamapS | bindep requires lsb_release | 22:40 |
SpamapS | a reasonable req | 22:41 |
pabelanger | mordred: WFM | 22:41 |
jlk | how else does it reliably determine which Linux you are? | 22:42 |
pabelanger | HAHA | 22:43 |
pabelanger | Trying again with ssh://pabelanger@review.openstack.org:29418/stackforge/dox.git | 22:43 |
SpamapS | jlk: yeah entirely reasonable | 22:44 |
jlk | image caching for the win, right? | 22:45 |
pabelanger | okay, will continue tomorrow | 22:45 |
jlk | oh oh what | 22:46 |
mordred | SpamapS: I love that distro cloud images don't always have lsb_release | 22:46 |
jlk | bindep only shows that you need to install things, it doesn't actually install things? | 22:46 |
mordred | it's the ultimate in "we stuck in a bunch of crazy vendor shit but left out the tiny utility that gives you metadata about your system" | 22:46 |
mordred | jlk: yes. | 22:46 |
clarkb | jlk: it will print the command though iirc | 22:46 |
clarkb | so you take its output and run it | 22:46 |
jlk | mordred: well, on Fedora to get lsb-release ou have to install a bunch of shit | 22:46 |
jeblair | mordred: i like that policy: if you can run dox you're core. | 22:46 |
jlk | # bindep -f /zuul/bindep.txt test | 22:47 |
jlk | Missing packages: | 22:47 |
jlk | libjpeg-dev mysql-client mysql-server | 22:47 |
jlk | that's all it says for me | 22:47 |
mordred | jlk: that's not true ... there is a minimal package you can install to just get lsb_release that isn't tied to all the X requirements | 22:47 |
clarkb | there is a switch | 22:47 |
SpamapS | crap | 22:47 |
clarkb | -b? | 22:47 |
SpamapS | I forgot docker's lovely "we don't really do pid 1" way of being | 22:47 |
jlk | that just spits out the packges one per line | 22:48 |
jlk | SpamapS: oh right, "running" mysql may be... diffiult | 22:48 |
SpamapS | so to get something behind /var/run/mysql.sock one has to somehow get mysqld running :-P | 22:48 |
mordred | jlk: https://github.com/ansible/ansible/pull/22159/files see line 60 | 22:48 |
jlk | you may have to run a mysql container linked to your tox container | 22:48 |
clarkb | jlk ya looks like you have to feed the -b output to your package manager | 22:48 |
SpamapS | if only we had shell scripts we could just run when we need something like that | 22:48 |
jlk | and a zookeeper container, and... | 22:48 |
mordred | SpamapS: you may want to poke at mysql-sandbox from guiseppe - I think it has docker support now :) | 22:48 |
SpamapS | jlk: except I need /var/run/mysql.sock, not 127.0.0.1:3306 | 22:48 |
jlk | SpamapS: you can do linked volumes! | 22:49 |
SpamapS | I'm just going to safe_mysqld | 22:49 |
mordred | oh - I never pushed up my fix for the 2.4 issue on the bindep module PR | 22:49 |
SpamapS | or not | 22:49 |
SpamapS | I'm spent | 22:49 |
clarkb | SpamapS: we that shell script is why I have never interacted with bindep directly | 22:50 |
clarkb | s/we// | 22:50 |
SpamapS | jlk: linked volumes sounds insane. So what, I can make the mysql container export its /var/run ? | 22:50 |
mordred | SpamapS: yes | 22:50 |
jlk | mordred: lsb_release comes from redhat-lsb-core-4.1-33.fc25.x86_64 | 22:51 |
jlk | which drags in a bunch of stuff on a stock fedora 25 docker image | 22:51 |
* SpamapS tries '/usr/sbin/mysqld_safe & ..' | 22:52 | |
jlk | SpamapS: yeah you can do volumes_from | 22:52 |
mordred | jlk: right. well, that's what I meant by 'it's the ultimate in "we stuck in a bunch of crazy vendor shit but left out the tiny utility that gives you metadata about your system"' | 22:54 |
mordred | since packaging lsb_release in a package that includes a giant ton of other useless crap is an insane choice | 22:54 |
mordred | redhat-lsb-core is the "minimal" version compared to redhat-lsb which pulls in a CRAZY amount of things | 22:55 |
jlk | ah. I see what you mean. Well, the whole world of LSB is full of useless crap and insane choices | 22:55 |
mordred | yah | 22:55 |
mordred | fo sho | 22:55 |
mordred | but the lsb_release utility is tiny and essential and it not being places is one of the most frustrating parts of being a person who consumes more than one distro | 22:55 |
jlk | why isn't zookeeperd in the bindep list? | 22:56 |
jlk | oh it's in platform:dpkg ? | 22:56 |
clarkb | because it doesn't exist on red hat distros | 22:56 |
mordred | yah | 22:56 |
mordred | someone is working on it | 22:56 |
jlk | weird seems there for Fedora 25 | 22:56 |
mordred | s/red hat distros/centos/ | 22:57 |
mordred | I think adding a bindep line for it for fedora would be great | 22:57 |
mordred | jlk: I think pabelanger was working with someone on getting in to a spin or something for centos/rhel land | 22:58 |
clarkb | oh cool looks like f24 has it too | 22:58 |
jlk | I'll look at the syntax | 22:58 |
SpamapS | ok, I figured out how to get a ghetto mysqld up | 22:58 |
mordred | jlk, pabelanger: btw - updated the bindep pr for ansible with docstring fixes - it should be green this time | 22:58 |
SpamapS | now just a ghetto zookeeperd | 22:58 |
jlk | kk | 22:59 |
jlk | SpamapS: what did you do in your docker file to ghetto mysql? | 22:59 |
jlk | safe_mysqld & ? as a RUN ? | 22:59 |
jlk | or maybe build a .sh that does it all | 22:59 |
SpamapS | http://paste.openstack.org/show/603703 | 22:59 |
SpamapS | yep | 22:59 |
SpamapS | yeah I think I'll put that in tools | 22:59 |
SpamapS | with the zookeeperd | 23:00 |
SpamapS | "daemons.sh" | 23:00 |
SpamapS | oops, forgot python-dev | 23:00 |
SpamapS | yay.. zookeeper kept its init script | 23:01 |
SpamapS | so /etc/init.d/zookeeper start _works_ | 23:01 |
* jlk eyeballs "/usr/bin/mysqld_pre_systemd" | 23:04 | |
* clarkb wonders if that included all the self daemonization code and now it lets systemd do it | 23:04 | |
SpamapS | jlk: lol! | 23:05 |
SpamapS | amazing | 23:05 |
SpamapS | clarkb: yeah probably | 23:06 |
jlk | it's a bash script | 23:06 |
*** harlowja has quit IRC | 23:06 | |
openstackgerrit | Jamie Lennox proposed openstack-infra/zuul feature/zuulv3: Re-add the ability to set username on zuul-executor https://review.openstack.org/446308 | 23:07 |
jlk | this is a pain | 23:10 |
SpamapS | aye, 'tis | 23:11 |
SpamapS | since we depend on cryptography, we should add python2.7-dev to our bindep | 23:11 |
SpamapS | or.. should we? | 23:11 |
jlk | ¯\_(ツ)_/¯ | 23:12 |
clarkb | SpamapS: ya a lot of projects have left it off because its implied on our test machines for other reasons (well zuul is installed on them in avirtualenv) | 23:13 |
clarkb | but I think it would be good to be explicit about the python versions required and whether or not you need the headers too | 23:13 |
SpamapS | is there a way to call tox where it runs the _setup_ for the venvs, but not the commands? | 23:14 |
clarkb | yes there is a setup only flag | 23:15 |
clarkb | I forget what it is though | 23:15 |
SpamapS | I'll look | 23:15 |
SpamapS | #lazyirc | 23:15 |
SpamapS | weird.. pep8 is trying to run with python3.5 .. but otherwise it seems to be workingish | 23:17 |
jesusaur | SpamapS: not that weird, since it's set in tox.ini | 23:18 |
SpamapS | oh! | 23:18 |
SpamapS | haha | 23:18 |
SpamapS | that seems.. odd? | 23:18 |
jesusaur | ya, I'm not sure why it's set there, but I noticed it a couple days ago | 23:18 |
jlk | hrm, struggling to get zookeeper up | 23:24 |
jlk | ah | 23:25 |
SpamapS | I almost have it now | 23:27 |
SpamapS | last step is to run as not-root :) | 23:28 |
SpamapS | and to figure out how to rebuild w/ new context | 23:29 |
jlk | heh | 23:30 |
*** jamielennox is now known as jamielennox|away | 23:35 | |
jlk | well you can docker run --no-cache | 23:38 |
jlk | er docker build --no-cache | 23:38 |
jlk | to rebuild the image without caching | 23:38 |
*** jamielennox|away is now known as jamielennox | 23:39 | |
SpamapS | jlk: but I want all the caching _except_ the context | 23:39 |
SpamapS | basically everything up to the ADD | 23:39 |
jlk | define "content" | 23:39 |
SpamapS | context | 23:39 |
SpamapS | the git repo | 23:39 |
SpamapS | and working dir | 23:39 |
jlk | oh I'm not putting that in the image at all | 23:39 |
jlk | that's just a volume mount | 23:39 |
SpamapS | I guess that's the better way | 23:40 |
jlk | so you literally change it on your host and it's changed in the container | 23:40 |
SpamapS | but I kind of liked the context thing | 23:40 |
jlk | what's context? | 23:40 |
SpamapS | by default everything in the dir with the Dockerfile | 23:41 |
SpamapS | and then you can ADD things from it | 23:41 |
SpamapS | but maybe that's borken | 23:41 |
jlk | I figured you'd want to have an image that's capable of running tox, and in our case, it has mysql and zookeeper running. You'd docker run with the volume of your zuul checkout | 23:46 |
jlk | and the default command would be just to run tox, or you can override it and do a docker run <image> <volume> tox <whatever> | 23:46 |
jlk | (a compose file may make this more easy) | 23:47 |
jesusaur | aha, I *finally* figured out why my test wasn't holding the builds, hold_jobs_in_queue != hold_jobs_in_build | 23:52 |
SpamapS | jlk: I want an image with the tox venvs already populated :) | 23:52 |
jlk | they will be | 23:52 |
jlk | because the tox build will be building in your zuul/ dir | 23:52 |
jlk | which is on your host | 23:52 |
SpamapS | Yeah true, but that only works by happenstance because I'm running Xenial. | 23:52 |
jlk | so every time you run, you already have .tox/ with populated content | 23:52 |
jlk | I mean, it'll get populated from within the container | 23:53 |
SpamapS | Yeah either way true | 23:53 |
jlk | it may not _work_ outside the container | 23:53 |
SpamapS | I still need the bindep file too | 23:53 |
SpamapS | but that's easy to add | 23:53 |
jlk | why? | 23:53 |
jlk | all it's good for is telling you what to install | 23:53 |
SpamapS | so I can install bindeps during docker build | 23:53 |
SpamapS | also I don't see how to control the uid of a volume | 23:54 |
SpamapS | but I can probably learn that by playing | 23:54 |
jlk | oh for non-root? | 23:55 |
jlk | yeah I dunno | 23:55 |
jlk | I haven't crossed that bridge yet | 23:55 |
SpamapS | I don't want anything running as root | 23:55 |
jlk | ERROR: py27: InvocationError: '/zuul/.tox/py27/bin/python /zuul/setup.py --name' | 23:56 |
jlk | huh, what does that mean. | 23:56 |
*** hashar has quit IRC | 23:56 | |
jlk | just running tox -e py27 | 23:56 |
jlk | I don't get it | 23:56 |
mordred | jlk: there may be logs in .tox | 23:57 |
mordred | jlk: it's possible it's a directory permissions issue - or that you don't have setuptools installed perhaps | 23:58 |
jlk | it created the venv, but no longs | 23:58 |
mordred | ah - awesome | 23:58 |
mordred | that's fantastic :) | 23:58 |
jlk | no setuptools in pip freeze tho | 23:59 |
jlk | either inside or outside the venv tox created | 23:59 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!