Tuesday, 2017-03-21

clarkbindependent of potentially solving this more betterer with ansible/zuul I feel like dns was built to solve a lot of these env specific problems but for raisins we can't actually sanely use dns this way00:00
jeblairhrm, it's not that elucidating anyway00:01
jamielennoxjeblair: i did, but i don't think the question was phrased the same way, the best answer seemed to be that there was a git connector, but whilst i could bend that to be usable it doesn't really fit the use case00:01
jeblairjamielennox: okay, so you have a project-config repo -- why *can't* it be used in those other environments?00:02
openstackgerritJamie Lennox proposed openstack-infra/zuul feature/zuulv3: Allow loading additional variables file for site config  https://review.openstack.org/44773400:02
jamielennoxjeblair: essentially because base job contains things like http://zuulv3-dev.openstack.org/logs/{build.uuid}/00:03
jamielennoxreview for comment00:03
jamielennoxjeblair: in production i can define that url, in vargrant i know what it is ahead of time00:03
jamielennoxin my dev environment it's hostvars['playpen-logs']['ansible_default_ipv4']['address']00:04
jamielennoxand to make an end-to-end usable test i want to define that value differently for every person who spins up a dev environment00:04
jeblairjamielennox: okay, so in true zuul fashion, we can throw more git repos at the problem and solve it that way.  :)00:04
jamielennoxjeblair: that would mean me having to maintain a git repo with prior knowledge of every development environment00:05
jeblairgimme a few minutes00:05
jeblairi'm going to expand on this00:05
jamielennoxas opposed to something that the deployment simply templates out onto the file system00:05
SpamapSjamielennox: a git repo could be templated out too :)00:06
SpamapSbut yeah, templating a file out for just one thing might be the better UX00:06
jamielennoxSpamapS: it certainly could00:06
* SpamapS has not thought enough about it00:06
SpamapSanyway, EOD00:06
jamielennoxi haven't looked yet as to whether the git connector tries to do things like pull updates00:06
SpamapSBBL00:06
jeblairjamielennox: basically, i think there are problems that we will encounter where we will have to remind ourselves that zuul (especially v3) works really well with lots of different repos, and quite often, the solution to a problem may involve rethinking the organization of repos.00:06
jeblair(this may or may not be one of those cases, but we can explore what would be involved)00:07
jamielennoxjeblair: i am entirely open to reorganizing, and it's why i did this as a hack around a problem i had locally rather than go upstream first00:07
jeblairso we could solve this problem by having the 'base' project defined in a repo which does little other than define this sort of site-specific configuration00:08
jeblairer 'base' job rather00:08
jamielennoxbasically i'm modelling these jobs off of what pabelanger has done for zuul which is i know designed to make things work fast00:08
jeblairso each of the dev/prod/staging environments, as well as the vagrant stuff could have their own such repo00:08
jeblairand with the 'git' driver, it doesn't have to be remote00:08
jeblairthat's essentially the way to do a 'local config file'00:09
jeblairexcept it's a git repo instead of a file :)00:09
jeblairmy feeling is that this sort of thing may be really useful.  generally speaking, it lets us modularize the configuration and reassemble the parts as fits different situations.00:10
jeblairjamielennox: however, when i think about the idea of setting up a git repo solely so that "success-url" is something different, that seems like we've gone too far, and perhaps your site-local variables idea is the better way of dealing with that.  :)00:11
jamielennoxi'm sure i could make that work, in practice though it'd end up being an entirely local git repo that i template out and depending on what was easier with ansible we'd either just wipe out the old repo or force commit whatever we actually wanted to be there00:12
jamielennoxi would need to look at what the git connector reponds to to see whether that would cause issues - but i don't konw if there would be a value to that history00:12
jamielennoxthough i guess at least for production we would be able to publish it00:13
jeblairjamielennox: yeah, so maybe it's good to have both of these things in our toolbox.  if you want to change one variable, we have executor variables.  if you want a slightly more substantial difference, you can define site-specific job configuration in a site-specific git repo.00:14
jamielennoxjeblair: so what's the rationale behind a local git repo connector? for most of this the advantage of git is having pull-requests and reviews etc but that wouldn't apply to a local repo00:14
jeblairjamielennox: the third-party ci case.  where you want to run a zuul pointing at someone else's github or gerrit, without running one of your own.00:15
jeblairjamielennox: (and the reason it exists now is so that i can run zuul locally and point it at openstack's gerrit)00:15
jeblairjamielennox: (since there is no way to supply zuulv3 configuration without having it in a git repo)00:16
jamielennoxok, so practically it exists more for a testing or dependency case where you could have something in the background that could sync from upstream as part of a larger job?00:16
jeblairjamielennox: well, i imagine a third-party ci would have their very site-specific configuration in a git repo without an upstream00:17
jeblairjamielennox: they have configurations like "run test-my-network-switch on changes to openstack/neutron".  that's never going to be an upstream thing, that's always local.00:18
jamielennoxbut in which case something outside of zuul is responsible for doing that pull from upstream and merge my stuff00:19
jeblairjamielennox: i'm probably not communicating this very well00:19
jeblairjamielennox: i'm expecting the "git" driver in zuul to be used by third-party-ci systems to load a portion of their zuul configuration which has no correspondence with anything upstream.  so there's no pull requests or proposed changes or merging to it at all.00:20
jeblairjust like today, third-party ci operators don't generally merge changes we make to zuulv2's layout.yaml into their own layout.yaml.  they have a private shadow configuration.00:21
jeblairthey *could* set up their own gerrit to do that, but that's too much to ask of someone who just wants to run zuul.00:21
jamielennoxwhy would we not expect them to host that repository on their own gerrit/github00:22
jamielennoxright00:22
jeblairbecause for many of them, it's a single small file that hardly ever changes00:22
jamielennoxok, and it involves configuring via a baseurl - so i'm pretty sure i get the usecase, and we could make it work for changing urls but it does seem like overkill00:24
jeblair(now, having said all that, third-party-ci operators will probably want to use *parts* of our configuration, which is why i think we need to update the tenant configuration syntax to allow folks to specify that they want to load *jobs* but not *projects* from certain repos, etc)00:24
jeblairjamielennox: yep.  i think if the extent of the problem is "i want a different success-url" i suspect your solution will be ideal.00:25
jamielennoxjeblair: well, it could be used/abused for injecting other deploy specific variables as well00:26
jamielennoxi'm not sure what they are yet00:26
jamielennoxand i'm not sure that's a good idea00:26
jeblairjamielennox: if we include executor variables, we'll want to keep the other options in mind, and be careful of when we cross the threshold from things that are best done in variables to things that are best done with configuration topology.00:26
jamielennoxjeblair: agreed00:28
jeblairjamielennox: also, with variables, we're going to want to think pretty carefully about safety.  for instance, whether a job could influence where logs are copied because the logs post-playbook uses a variable.00:28
jamielennoxoh, hmm, whether a job can re-define logs_url over the top of what goes into executor variables00:28
jamielennoxis that fundamentally different though from a job just deciding to not inherit from base?00:29
jeblairjamielennox: yes, because then they don't get the (presumably trusted) playbook which copies logs00:30
jeblairso the outcome there is "no logs" vs "altered behavior of log copying playbook"00:31
jamielennoxso would it make sense to split executor variables to trusted and untrusted00:31
jamielennoxactually it shouldn't matter because in this case the log copying would be trusted wouldn't it00:31
jeblairor perhaps say that they can not be overridden by jobs00:31
jamielennoxeach playbook is executed in its own ansible process so if something in an untrusted scenario overrides a var it's not going to carry through to the trusted post anyway rigth?00:32
jeblairjamielennox: right, but job variables are supplied to all of the playbook executions00:32
jamielennoxjeblair: oh, right, so they define it in the job variables, i was thinking like a set_fact00:33
jeblair(fundamentally, this isn't very different from putting the same sort of thing in a base job 'vars' section, it's just more obvious (hopefully) that it can be overidden)00:34
jamielennoxhmm00:34
jeblairjamielennox: as i think about this, i'm back to your example of 'success-url' and it occurs to me that isn't actually related to ansible at all; so executor variables couldn't (at the moment at least) be used to set that.00:35
jamielennoxso the only way i see around that is a trusted vars and untrusted vars file00:35
pabelangerjust catching up on backscoll.  But what was the down side of having base->site-base->job for zuul.yaml?00:35
jamielennoxjeblair: what do you mean? it definitely works00:35
jamielennoxpabelanger: it means redefining job on top of site-base for every site including ones we spin up just for testing00:36
pabelangerjamielennox: do you have an example today for your playbooks?00:37
jeblairjamielennox: success-url is not used or set by ansible; that's internal to zuul.00:37
jamielennoxjeblair: oh? lol, ok i definitely used that as an example, sec00:37
pabelangerya, that is where I am getting confused00:38
jamielennoxhttps://github.com/jamielennox/project-config/blob/master/playbooks/roles/base/tasks/post.yaml#L1600:38
jeblair(log copying destination, however, would be something that could use a variable like this)00:38
jamielennoxsame concept, i was just copying what was in -infra/project-config than the one out of my project-config00:38
jeblairyeah, so that's the actual example :)00:39
jamielennoxpabelanger: so in my testing bonnyci.logs.dest needs to be: hostvars['playpen-logs']['ansible_default_ipv4']['address'] from the deploy scripts00:39
jeblair(i think it would be nice if the log-copying playbook could feed information back to zuul, so that it could supply (part of?) the success-url)00:40
jeblair(but that's a problem for a different day)00:41
jamielennoxjeblair: so in our case i'd happily just mandate a whole bunch of that stuff, like i don't need user's fiddling with workspace roots or logging urls or logging paths or anything like that00:41
jamielennoxbut it's not a big concern for now00:41
jeblairjamielennox: i think we might be saying the same things with different words; with you saying "trusted/untrusted variables files" and i saying "override"00:43
jeblairjamielennox: i think we're in agreement that if executor_variables are to be a thing, we have to be able to rely on them.00:43
jeblairregardless of how that actually gets implemented00:44
pabelangerjamielennox: why don't you just use {{ hostvars['playpen-logs'].ansible_default_ipv4.address }} directly from your playbook? Or better yet, maybe use set_fact to do some condition checks00:44
jamielennoxjeblair: right, so i'm not sold on this solution it's just something i had come up - i think it will depend on how the trusted/untrusted split happens00:45
jamielennoxthere's not much difference their atm00:45
jamielennoxrealistically for now i don't care about feeding vars into the untrusted environment, but could maybe see that as something useful00:45
jamielennoxpabelanger: how? i have access to that variable at the time i deploy zuul, i need to save it out somewhere so i can have access to it at the time a job is running00:46
jamielennoxalso that would involve encoding the term 'playpen-logs' into project-config right? that's not going to work on vagrant or production00:47
pabelangerHmm, so in our case. We are going to need logs.o.o in our job inventory file00:50
pabelangerbecause, we'll need to delegate_host to logs.o.o, to rsync the file00:51
jeblairpabelanger: we don't do that in v2.5.  why would we do it in v3?00:51
pabelangerwell, we have site section in 2.5 that contains that info, doesn't it?00:52
pabelangercannot remember00:52
jamielennoxthat information comes out of zuul.conf in 2.500:52
pabelangerlet me check00:52
jeblairpabelanger: sure, but that's just emulating jenkins data structures.  the actual implementation is just an rsync push from launcher to logs.o.o00:52
jeblairpabelanger: the site defn does not get translated into an inventory entry.00:53
jamielennoxso realistically i don't know about rsync anyway, that would entail adding an ssh key to the executor capable of uploading files to our log server00:53
jeblairyeah, that's the v2.5 approach and what i think we're likely to do for openstack in v3.  to start with at least.00:54
jamielennoxfor any sort of prod i think i'd want to put something a bit more secure there anyway - but that rules out adding anything to the ansible inventory00:54
pabelangerright, I'm looking at it from the way of hosts: log_server for the playbook, which is an inventory look up00:55
jamielennoxi'm vaguely considering some kind of one off upload secret managed by (hashicorp) vault or something we can otherwise add to the job vars00:55
pabelangerbut, I know we do it today a little different with with rsync specific commands00:55
jeblairjamielennox: it's worth noting that much of the design relies of the executor remaining secure.00:55
jamielennoxbut we're really no where near that yet00:56
jeblairpabelanger: only hosts that the job can manipulate should be in the inventory.  the log server shouldn't be there.00:57
jeblair(unless it's a job that runs on the log server)00:57
jamielennoxjeblair: so to my mind that depends a lot on what/how we let users define jobs and whether they can get code exec on the executor00:57
pabelangerjeblair: not a job, but a task for example00:57
jeblairjamielennox: right, that's an important consideration, which is why SpamapS has that spec00:58
pabelangerlet me quickly write a sudo playbook00:58
jamielennoxjeblair: well SpamapS's spec is how to stop a user breaking out of the executor, and in reality i think it is largely going to depend on us doing single use executors00:58
jeblairjamielennox: what is a single-use executor?00:58
jamielennoxthere's really no limit to what the user could do with code exec inside the executor00:59
jamielennoxjeblair: oh, create an executor, run one job, discard executor00:59
jeblairjamielennox: sure there are limits -- we're contemplating 3 systems for enforcing those limits, all of which are detailed in the spec00:59
jamielennoxso anything the user does inside the executor is destroy01:00
jamielennoxed01:00
SpamapSThe point of the spec is to make limits01:00
jeblairjamielennox: and the spec is pretty clear that single-use executors is not the way we're going01:00
pabelangerjeblair: http://paste.openstack.org/show/603544/01:00
SpamapSToo spensive and confusing01:01
pabelangerthat to me, would be a way of publish logs to logs.o.o01:01
pabelangerbut, logs.o.o is not in the inventory file01:01
jamielennoxjeblair: understood, but on last reading that wasn't ruling that out it was just about containment - but yes, the amount of work to launch and have a ready queue of executors is not worth it at this time01:02
jamielennoxpabelanger: so that's why i ended up having to do command: rsync - https://github.com/jamielennox/project-config/blob/master/playbooks/roles/base/tasks/post.yaml#L1301:03
jamielennoxthere was no way i could see to make ansible's synchronize push to an arbitrary host (which i'm guessing is intentional)01:03
jeblairyeah, that's why v2.5 is constructed that way01:04
jeblair(also, we should be careful about keys here, we may not want to use the zuul worker key to log into the log host)01:04
jeblair(or maybe we do.  we should think about it at least :)01:05
jeblairit's past dinnertime for me, so i have to bow out now01:05
jamielennoxyea, i feel i've given people something to think about and i don't need a solution right now01:05
jamielennoxcan discuss it more at another time01:05
pabelangeradd_host would be a work around to it not being in the inventory file01:06
pabelangerbut, keys need to be setup too01:06
pabelangerfor another day!01:06
jlkMaybe we could teach Zuul how to contact an Object service!01:12
* jlk ducks01:12
*** jamielennox is now known as jamielennox|away01:16
*** pleia2_ is now known as pleia201:25
*** jamielennox|away is now known as jamielennox01:28
jamielennoxwalked away from the computer for a while and realized that i super confused that conversation by using the word executor as both the server and the container environment that the executor will be creating02:15
jamielennoxthen managed to confuse myself :)02:15
*** herlo has joined #zuul02:18
*** herlo has quit IRC02:23
*** herlo has joined #zuul02:25
*** herlo has joined #zuul02:25
*** MarkMielke has joined #zuul03:47
*** jamielennox is now known as jamielennox|away04:12
*** Guest45332 has joined #zuul04:49
*** Guest82302 has joined #zuul04:59
*** Guest45332 has quit IRC05:05
*** Guest82302 has quit IRC05:09
*** bhavik1 has joined #zuul06:29
*** yolanda has quit IRC06:32
*** yolanda has joined #zuul06:46
*** hashar has joined #zuul08:45
*** openstackgerrit has quit IRC09:03
lennybHi, I think that I've hit #link https://bugs.launchpad.net/zuul/+bug/1270029   . I am using zuul  2.5.1. Any ideas, comments, suggestions?09:28
openstackLaunchpad bug 1270029 in Zuul "zuul doesn't connect to gearman server" [Undecided,New]09:28
*** bhavik1 has quit IRC09:52
*** jamielennox|away is now known as jamielennox10:30
*** jamielennox is now known as jamielennox|away11:05
tobiash_just curious, did someone experiment with somewhat unsafe filesystem tunings (like disable journaling) for single use nodes?11:09
tobiash_I'm thinking about reducing io footprint of the ci system in our environment11:10
*** jamielennox|away is now known as jamielennox11:12
mordredtobiash_: I don't think we have much - but now that you say it I want to11:26
pabelangerShrews: we ready to start nl01.o.o again this morning? Not sure if we are waiting on anything11:26
pabelangerah, maybe 44763011:27
tobiash_mordred: I'm thinking about combining eatmydata with some mount options and vm.dirty* settings11:28
mordredtobiash_: I think it's a great idea - if you wind up exploring it, I'd love to know what impact you find it to have11:37
tobiash_mordred: I think I'll try it out in our environment11:40
tobiash_and share knowledge if it had some possitive impact11:41
*** yolanda has quit IRC11:52
*** openstackgerrit has joined #zuul11:53
openstackgerritPaul Belanger proposed openstack-infra/zuul feature/zuulv3: Create zuul_workspace_root job variable  https://review.openstack.org/44144111:53
openstackgerritPaul Belanger proposed openstack-infra/zuul feature/zuulv3: Organize playbooks folder  https://review.openstack.org/44154711:53
openstackgerritPaul Belanger proposed openstack-infra/zuul feature/zuulv3: Add net-info to bootstrap role  https://review.openstack.org/44161711:53
openstackgerritPaul Belanger proposed openstack-infra/zuul feature/zuulv3: Add revoke-sudo role and update tox jobs  https://review.openstack.org/44146711:53
openstackgerritPaul Belanger proposed openstack-infra/zuul feature/zuulv3: Create tox-tarball job  https://review.openstack.org/44160911:53
*** yolanda has joined #zuul11:54
pabelangerjeblair: jlk: jamielennox: Shrews: so, in the effort to keep iterating forward on ^, jeblair suggestion of using the global zuul_workspace_root seems fine11:55
pabelangerI do think we should establish a best practice guide if possible and try to avoid global vars whenever possible.11:55
pabelangerBut since these roles are still in flux, lets keep moving forward now11:56
mordredpabelanger: BIKESHED ....11:57
mordredpabelanger: is there any reason we're calling it zuul_workspace_root and not zuul.workspace_root that maybe I missed?11:57
pabelangermordred: zuul_workspace_root is in zuul.yaml, where our zuul. vars come from the vars.yaml file the executor creates11:58
pabelangerso, even more namespacing of vars11:58
openstackgerritPaul Belanger proposed openstack-infra/zuul feature/zuulv3: Add ansible-lint to tox (pep8)  https://review.openstack.org/43827611:58
mordredah - gotcha. so in this case, that's actaully some infra job-content11:59
pabelangerya, we'll have to keep an eye on the variable prefix we use for job-content vars and zuul executor vars12:00
pabelangerjob_workspace_root might be better12:00
mordredyah ... but yay for iteration :)12:01
pabelanger447089 is going to be the next one to bikeshed12:03
Shrewspabelanger: yeah, i'd really like to see 447630 go in first12:12
Shrewspabelanger: let me get some coffee and i'll fix it up for that -112:12
mordredpabelanger: I think in my brain I'm thinking having the _role_ available is a good idea - but we don't necessarily need to create a centrally defined _job_ for that combo12:15
mordredand maybe even the playbook12:16
mordredpabelanger: so a user can say "job: my-local-py27 pre: setup-db"12:16
mordredpabelanger: but I also agree, getting extra-test-setup will be nice12:16
openstackgerritPaul Belanger proposed openstack-infra/zuul feature/zuulv3: Create extra-test-setup role  https://review.openstack.org/44804212:20
pabelangermordred: ^ basically how we do it in JJB12:21
pabelangerbut, on the fence if we want extra-test-setup role added by default into our base tox-py27 job.  Doing a file check doesn't feel like the right way to do it with ansible12:22
pabelangerbut ya, I think a pre-run: playbook, a project can opt into sets up a database is the right way12:22
pabelangerfor now, ^ should unblock zuul jobs from failing12:23
mordredyup12:24
mordredpabelanger: we may need to think about reusable playbooks that are not jobs or roles too - which should make jeblair aka "Mr. Multi-Repo Search Path" super happy to think about12:25
mordredsince needing to create central _jobs_ in all of the combinations seems like overkill, but just having a "setup-db" role means that to consume it a local user would need to make a setup-db playbook in their repo that just calls that one role - so we'll have that playbook cargo-culted in to a ton of repos12:26
mordredand also - I'm with you on the extra-setup-role by default question. it both feels wrong to have the file search, but it also feels a bit overkill to do it the other way too maybe12:27
mordredI knew it was going to get fun once we started actualy converting openstack's jobs12:28
mordredturns out we have a LOT of edge cases available to us to consider just in zuul's unit tests :)12:28
pabelangerindeed12:29
openstackgerritDavid Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: Fix test_node_assignment_at_quota  https://review.openstack.org/44763012:32
Shrewspabelanger: jeblair: clarkb: ^^^12:33
mordredShrews: having a quota of 5 nodes seems to have really hepled us find bugs that happen at quota hasn't it?12:41
Shrewsmordred: definitely. the entire "pause processing" bit of code is slightly complex, and the limited quota has helped exercise the demons12:43
Shrewsbut i feel like it's getting really solid now (after this bug fix)12:44
mordred\o/12:45
Shrewsthough zuul is failing ALL the tests on that PS12:48
Shrewshttp://logs.openstack.org/30/447630/4/check/gate-nodepool-pep8-ubuntu-xenial/f4565f8/console.html#_2017-03-21_12_38_50_84483412:49
Shrewsneat12:49
Shrewsmordred: pabelanger: ^^^ setuptools issues today?12:49
Shrewscryptography lib12:50
pabelangerHmm12:51
pabelangerlooking12:51
openstackgerritPaul Belanger proposed openstack-infra/nodepool feature/zuulv3: Add libffi development headers to bindep  https://review.openstack.org/44807612:53
pabelangershould fix it12:54
openstackgerritDavid Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: Fix test_node_assignment_at_quota  https://review.openstack.org/44763012:55
Shrewscool. rebased12:56
openstackgerritMerged openstack-infra/nodepool feature/zuulv3: Add libffi development headers to bindep  https://review.openstack.org/44807613:56
*** hashar has quit IRC14:33
*** hashar has joined #zuul14:35
jeblairmordred, pabelanger: i agree with most of what you've said, but i think if you consider what the tox-py27 job/playbook really is -- an implementation of openstack's common testing interface -- the fact that it encodes "policy" things like "run extra setup if it exists" doesn't seem very wrong.14:36
jeblairmordred, pabelanger: i'm not sure i used the best words there, it's early, cut me some slack.  :)14:37
mordredjeblair: good morning!14:37
mordredjeblair: however, if one considers a tox-py27 in the zuul standard lib that's not an openstack specific tox-py27 - this may be a fun rabbithole to consider14:38
jeblairmordred: yeah -- i'm sure *roles* will end up in the standard lib; i'm not as certain that we're going to succeed in having jobs there.14:39
jeblairso certainly the tox *role*, etc, need to be clean, modular, reusable.14:39
mordredjeblair: yah - I guess I'm trying to suggest that if we _only_ have roles in our re-use bucket, there's going to be a gob of copy-pasta14:41
mordred(this is a painpoint generally in ansible land ... roles are awesome, but people also need to share playbooks sometimes)14:41
jeblair(for one thing -- in order to have a job in the stdlib, you'd have to have a convention on inhertance -- like they all ultimately inherit from 'base' -- but 'base' almost certainly needs site-local configuration -- so do we leave a hole in the inheritance chain for that?  possible, but weird)14:41
jeblairmordred: roles can (now) include other roles, so that is a way to achive some of what playbook sharing achieves14:42
pabelangerThe more I think about it, I do like the idea mordred had. If we added a new project to zuulv3, and tox-py27 was a trusted job, we could add a variant of tox-py27, which included a pre-run to a untrusted playbook called database_setup. Then use an upstream role on galaxy to do that step: https://github.com/geerlingguy/ansible-role-mysql14:42
jeblairyep.  that's still a possibility.  and we did make 'pre-run' a list, so you can tack on "db-setup" "extra-setup" "bindep" "whateverelsewethinkup" all in a pre-run section of a job.14:44
pabelangerI think it is more a question, do we want to provider a playbook in openstack-infra that will run tox and setup a database for you (pretty common) or expect the each project to write that logic them self14:46
jeblairi think that the pre/post playbook option is important -- but i think that for something so common, it's okay for the 'openstack standard' tox job to do that for you automatically.14:48
pabelangerya, and there is so many ways to make that dynamic14:49
pabelangerfor now https://review.openstack.org/#/c/448042/ is that way14:49
pabelangerbasically a straight convert from JJB14:49
pabelangerdownside of that, more if / else logic in playbooks. Which, as it grows could get more problematic.  Another reason I like the idea of a tox-db-py27 playbook specifically14:51
mordredjeblair: yah - I thnk it's totally great for the openstack standard job to do it for folks automatically - since openstack after all14:52
mordredjeblair: but if someone says "I want to run python27 jobs, but as part of setup I want to make sure that there is a mysql database set up"14:53
mordredthat's pretty easy for them to just do in their .zuul.yaml file - except as of right now it means they'll need to make a 3 line playbook that says "hosts: *\n  roles:\n  setup-mysql" and then reference that in their pre-run14:55
mordred(ignoring the current openstack "always run a tools/test-setup.sh if it's in the repo" cantrip - using it more as an example of combining more than one logical thing)14:56
jeblairmordred: ya.  so maybe the way to do what you're suggesting is not to do a multi-repo search path, but rather to allow repo specification of playbooks14:57
jeblairmordred: "pre-run: git.openstack.org/openstack-infra/project-config/playbooks/setup-mysql" or something14:58
jeblairmordred: that way we don't have to deal with maintaining a file inventory of every project-branch combination, or worry about what happens when cloudkitty adds their own setup-mysql playbook14:59
mordredjeblair: who doens't want the cloud-kitty playbook???14:59
jeblairironic, that's who14:59
mordredsilly ironic15:00
pabelangerpre-run: git.o.o/foo is a neat idea15:03
jrollhey, ironic loves kitties, even cloudy ones15:03
mordredespecially if we can apply our syntactic sugar maybe - so that in openstack people can reference openstack-infra/project-config/playbooks/setup-mysql and that'll expand to have git.o.o on front of it maybe? (thinking outloud)15:05
mordredjroll: :)15:05
* jeblair bows to our new feline overlords15:05
jeblairmordred: yeah, we may be able to make it a standard "project reference" whatever that ends up being.  we'll just have to figure out how to tack a file path onto a project reference.  (do we need a delimiter, or can we figure it out?)15:06
mordredyah15:07
pabelangerbut, we can do pre-run: openstack-infra/project-config/playbooks/setup-mysql today, if we added a setup-mysql job in our trusted repo right?15:08
pabelangerthen a project adds a variant job, with pre-run: setup-mysql15:09
pabelangertrying to wrap my brain around everything15:09
mordredpabelanger: we'd just have to have a setup-mysql job that was a 'fake' job we weren't really expecting anyone to use as an actual job - which seems hacky to me15:11
jeblairthat doesn't sound right --15:11
jeblairi thought the issue we're talking about is that a playbook (whether pre, main, or post) is assumed to be in the repo where the job is defined, so you can't use a playbook in a different repo15:12
jeblaireven if you had a setup-mysql playbook in another repo, you couldn't tell zuul to run it15:13
mordredyes. that is the problem15:13
mordredit's entirely possible pabelanger are smoking something with our most recent thing15:13
mordreds/pabelanger/pabelanger and I/15:14
jeblairyou could have a setup-mysql job, and *inherit* from that, but we don't have multiple inheritance (yet?) so it's not something you can mix-in.  so the inheritance model would be more like "tox-py27 -> tox-db -> tox -> base"15:15
mordredright. and with that and single inheritance we'd wind up maybe with a big pile of jobs like "tox-py27-with-db" and "tox-py27-without-db" which would be silly15:16
jeblairyup15:16
pabelangerya15:16
pabelangerthis is getting fun!15:17
pabelanger:)15:17
mordredsince the thing pabelanger said is I think how the consumer mindset will be "I want a job that runs the python27 content that I didn't right, but a variant of that which ensures a database is installed that I also didn't write"15:17
mordredso, I want a variant of the python27 job that does this other pre-canned thing in pre-run15:17
jeblairanyway, i like the cross-project playbook reference -- though we do need to decide its effect on trustedness -- does a playbook in another project run with the trusted flag of the project it's in or the project that's referencing it?15:18
mordredjeblair: that's an EXCELLENT question :)15:18
jeblairi think referencing15:19
jeblairbut i'm not sure :)15:20
pabelangerI like that, cross-project playbook reference15:20
mordredI think you're right - since I per-repo opt-in actions like "I want you to publish my artifacts to this service I have" are most likely to want to be tenant credentials15:20
mordredwhereas base things that would need shared credentials are more likely to be from inheritance perhaps?15:20
mordredor maybe I'm wrong abot that15:20
mordredno - I think that's right - in openstack we don't really want joe-random-user to be able to write things to trusted AFS locations15:21
mordredbut - we _could_ maybe have a project that requests an AFS space that we give them along with a kerb principal and then they would put that into their tenant secrets and they could publish to _their_ AFS location with their security context15:22
*** dkranz has quit IRC15:23
jeblairmordred: yeah.  i'm pretty sure it wants to be referencing, because otherwise, imagine the trusted playbook operated on variables.  but within its defining (trusted) repo, the author knew that the variables were safe (only called from final jobs, etc -- like maybe a pypi-upload job).  if you allowed an untrusted repo to invoke it in the trusted context, you've bypassed that.15:37
mordredyup15:37
pabelangerOh, bikeshed!15:56
*** bhavik1 has joined #zuul15:56
pabelangerjeblair: I missed your comment in 44144115:56
pabelangerSo, I did make zuul_workspace_root: /home/zuul, this was intentional.15:56
pabelangerhowever, the question is, do we really need a /home/zuul/workspace?15:57
pabelangerA /home/zuul is less characters, which helps tox.ini issue.15:58
pabelangerbut /home/zuul/workspace is more inline with existing things15:58
jeblairpabelanger: as long as we aren't putting the git repos straight in /home/zuul.  if we put them in /home/zuul/src/git.openstack.org/openstack/cloud-kitty that works for me.15:58
pabelangeryes, that is where git repos live15:59
jeblair++15:59
Shrewsoh, glad that's resolved. i was tempted to start a new bikeshed on the name "workspace"16:00
Shrewsonly b/c i wanted in on the fun16:00
pabelanger:)16:00
jeblairpabelanger: jlk has some questions inline on ps816:00
pabelangerjeblair: ya, pushed up some replied. Might have missed one16:01
Shrewspabelanger: fyi, almost ready to start n-l on nl01. just waiting for the puppet to do things16:01
pabelangerShrews: yay16:02
pabelangerjeblair: Oh, replies are drafts currently.16:03
pabelangerlet me fix16:03
jeblairah yeah, the old publish to wrong patchset issue :)16:04
*** isaacb has joined #zuul16:07
*** isaacb_ has joined #zuul16:07
*** isaacb_ has left #zuul16:08
mordred++ to /home/zuul/src/git.openstack.org/openstack/cloud-kitty ... (and our go friends can then set GOPATH={{zuul_workspace_root}} and all will be joyous and wonderful16:16
Shrewspabelanger: n-l started16:20
pabelangerShrews: cool16:20
Shrewspabelanger: and very happy to see the ServerList output only 1/min now.  :)16:22
pabelangerShrews: Yay, it works16:24
Shrewswill take a bit for it to catch up. over 100 requests to satisfy16:25
pabelangertime to start using infracloud-vanilla?16:31
jlkpabelanger: mordred: simple answer on zuul_workspace_root vs zuul.workspace_root: The latter is a hash, and by default, Ansible does not merge hashes together. So if at any level you wanted to redefine some sub-key of the zuul hash, you'd have to re-define the _entire_ hash. OR you have to turn on non-default behavior in Ansible, which is not the path of least surprise.16:34
*** isaacb has quit IRC16:38
pabelangerright, the zuul has (eg zuul.executor) is what we setup specifically from the executor: http://git.openstack.org/cgit/openstack-infra/zuul/tree/zuul/executor/server.py?h=feature/zuulv3#n82016:38
jeblairjlk, pabelanger, mordred: yeah -- maybe we should reserve "zuul." for zuul itself, and "zuul_" for site-local global variable things (which is what is happening here)16:38
pabelangers/has/hash*16:38
jeblairi don't know if that's a useful distinction, or just confusing.  :)16:38
jlkyeah, I'm generally wary of hashes.16:38
jlkparticularly as role defaults16:38
pabelangerya, so may ways to do this16:40
pabelangerzuul.vars (doesn't exist) could be our hash from the job vars section. then it is zuul.vars.zuul_workspace_root16:41
jeblairfungi, clarkb, mordred, SpamapS: i've sent an email to openstack-infra and openstack-dev (tagged with infra and security to try to get lots of eyes) about encryption.  i put a lot of work into it.  hopefully it's accurate and a useful basis for making some decisions (or at least a starting point for discussion if it is way off).16:41
fungithanks jeblair!!!16:42
jeblairplease point it out to others who should see it and may otherwise miss it :)16:42
SpamapSjeblair: will do, thanks for the heads up16:42
jeblairi have to afk for a bit.  bbl.16:42
*** bhavik1 has quit IRC16:48
mordredjeblair: it's worth noting that one of the options you list is also implemented by our friends in keystone land so there may be some experience with it in our community16:54
*** herlo has quit IRC17:03
*** herlo has joined #zuul17:05
pabelangermordred: do you mind looking at 446749, relates to your callback logic for zuul_stream17:09
openstackgerritPaul Belanger proposed openstack-infra/zuul feature/zuulv3: Remove ZooKeeperConnectionConfig class  https://review.openstack.org/44768317:18
openstackgerritPaul Belanger proposed openstack-infra/zuul feature/zuulv3: Remove ZooKeeperConnectionConfig class  https://review.openstack.org/44768317:27
*** hashar has quit IRC17:28
mordredpabelanger: looking17:37
mordredjamielennox: ^^ nice +A17:39
openstackgerritMerged openstack-infra/zuul feature/zuulv3: Handle the 'all' group in callback.  https://review.openstack.org/44674917:47
openstackgerritPaul Belanger proposed openstack-infra/zuul feature/zuulv3: Create extra-test-setup role  https://review.openstack.org/44804218:27
pabelangerjlk: thanks^. good news is our testing confirmed18:28
jlkcool18:29
*** dkranz has joined #zuul18:31
Shrewsnl01 all caught up now. pabelanger, jeblair: i say we think about introducing vanilla tomorrow-ish if no more problems crop up18:35
jlkhahahah *sigh*18:39
jlkwasted a fair amount of time because I didn't realize mysql stuff got added to bindeps18:39
jlkfor testing18:39
jlkhrm, still not passing18:40
jlkDo I need to do something special to get mysql tests to work?18:40
jlkshould the server be running?18:40
openstackgerritPaul Belanger proposed openstack-infra/zuul feature/zuulv3: Create extra-test-setup role  https://review.openstack.org/44804218:41
pabelangerjlk: tools/test-setup.sh should setup the database for zuul18:41
jlkoh huh18:41
jlkI missed that file18:41
jlkdammit that's asking for a sudo password, which I don't know.18:43
jlkoh haha18:43
jlkwrong box18:43
jlkoh cool, so other people's tests are failing due to sql stuff too.18:47
jlkAre the nodepool nodes in zuulv3-dev not set up right?18:47
pabelangerjlk: that is what I am fixing with 44804218:48
jlkhahah okay18:48
pabelangerwe merged sql reporter support last week18:48
jlkHow was the addition of the need for this passed?18:48
jlkdid it never work on the zuulv3 nodes, and only wokred on the old system?18:49
pabelangerno, we only readd database support18:49
pabelangerso it only broke recently18:49
jlkhrm.18:49
jlkso it's broken in the "zuul" tests, which are voting18:49
pabelangerand we've had nl01.o.o offline for a few days18:49
jlkhow did it merge?18:49
jlkooooh18:49
jlkso it happened when zuul wasn't even testing18:50
pabelangerbecause it works properly on nodepool.o.o18:50
pabelangerright18:50
jlkgotcha. Okay.18:50
pabelangerzuul user doesn't gate yet either18:50
jlkwhat's the difference between non-voting and gate?18:51
jlkoh I get it18:51
jlkit still "votes" a -1, but the gate pipeline doesn't care about votes from that user18:51
clarkbnon voting is any test that doesn't affect the +/- value. gating is any test that prevents a change from merging if it fails (not that voting is required for gating)18:52
pabelangerright, zuul.o.o only care about jenkins right now18:52
clarkbs/not/note/18:52
pabelangerwe also only have a single pipeline on zuulv3-dev.o.o too, check18:52
jlkalright this makes my world view a bit better.18:56
pabelangernice, https://review.openstack.org/#/c/448042/ all green again18:56
pabelangerjeblair: mordred: Shrews: SpamapS: ^ mind a review, that makes our tests green again on zuulv3-dev18:57
Shrewspabelanger: jeblair: i think it is time to think about making our zuul-np integration job voting. i would note, though, it seems to be broken now.18:58
Shrewsnot sure where we broke it18:58
jheskethMorning18:58
jeblairShrews: i agree18:58
jeblairalso, all the more reason :)18:58
pabelangerShrews: ya, I've been trying to bring the job back online. 447683 was the first step, but not sure the hostname issue18:59
pabelangerjhesketh: morning! do you think you could rebase master to feature/zuulv3 for nodepool again?19:01
jheskethpabelanger: yes. I was working on that but it was non trivial. I'll get it working today19:02
pabelangerjhesketh: great! Ya, I suspect its some work19:03
jheskethIt's more just repeating what I had done. There isn't an easy way to rebase a merge commit19:04
Shrewspabelanger: what's up with this message in the ansible output? http://paste.openstack.org/show/603671/19:05
Shrewsfrom http://zuulv3-dev.openstack.org/logs/f749706319924e38819d8c852b8eb36d/ansible_log.txt19:06
pabelangerShrews: https://review.openstack.org/#/c/446749/ fixes it19:08
pabelangerwe need to schedule an update of zuulv3-dev to pick it up19:08
Shrewsi love when there are already solutions to things19:08
openstackgerritPaul Belanger proposed openstack-infra/zuul feature/zuulv3: Fix hostname issue with nodepool integration job  https://review.openstack.org/44823919:08
pabelangerI think that is our integration job fix too19:08
clarkbpabelanger: re ^ why wouldn't you use the hostname of the test machine?19:09
clarkbisn't that where it comes from in the real world?19:09
pabelangerclarkb: I guess we could, that was just copypasta from another job19:10
clarkbfor integration I think its valuable to actually test those surfaces19:10
pabelangerself.hostname = socket.gethostname() ?19:10
clarkbor let whatever code reads the hostname already do it19:11
pabelangerclarkb: we just default that for __init__ on the scheduler19:12
clarkbya I would let it do it for you then19:12
clarkbrather than set it something different in the test19:12
clarkb(I guess the risk there is it requires properly configured /etc/hosts)19:12
openstackgerritPaul Belanger proposed openstack-infra/zuul feature/zuulv3: Fix hostname issue with nodepool integration job  https://review.openstack.org/44823919:14
jlkanybody know of an example in v3 where you have some jobs in a pipeline that only run if a higher level pipeline job finishes?19:38
jlkaaah19:39
openstackgerritJesse Keating proposed openstack-infra/zuul feature/zuulv3: Allow github trigger to match on branches/refs  https://review.openstack.org/44562519:48
openstackgerritJesse Keating proposed openstack-infra/zuul feature/zuulv3: Better merge message for GitHub pull reqeusts  https://review.openstack.org/44564419:48
openstackgerritJesse Keating proposed openstack-infra/zuul feature/zuulv3: Set filter according to PR/Change in URL  https://review.openstack.org/44678219:48
openstackgerritJesse Keating proposed openstack-infra/zuul feature/zuulv3: Support for dependent pipelines with github  https://review.openstack.org/44529219:48
openstackgerritJesse Keating proposed openstack-infra/zuul feature/zuulv3: GitHub file matching support  https://review.openstack.org/44611319:48
openstackgerritJesse Keating proposed openstack-infra/zuul feature/zuulv3: Log GitHub API rate limit  https://review.openstack.org/44615019:48
openstackgerritJesse Keating proposed openstack-infra/zuul feature/zuulv3: Query changes only via relevant sources  https://review.openstack.org/44825719:48
*** hashar has joined #zuul19:53
openstackgerritPaul Belanger proposed openstack-infra/zuul feature/zuulv3: Create zuul_workspace_root job variable  https://review.openstack.org/44144120:06
openstackgerritPaul Belanger proposed openstack-infra/zuul feature/zuulv3: Organize playbooks folder  https://review.openstack.org/44154720:06
openstackgerritPaul Belanger proposed openstack-infra/zuul feature/zuulv3: Add net-info to bootstrap role  https://review.openstack.org/44161720:06
openstackgerritPaul Belanger proposed openstack-infra/zuul feature/zuulv3: Add ansible-lint to tox (pep8)  https://review.openstack.org/43827620:06
openstackgerritPaul Belanger proposed openstack-infra/zuul feature/zuulv3: Add revoke-sudo role and update tox jobs  https://review.openstack.org/44146720:06
openstackgerritPaul Belanger proposed openstack-infra/zuul feature/zuulv3: Create tox-tarball job  https://review.openstack.org/44160920:07
jeblairjlk: http://git.openstack.org/cgit/openstack-infra/zuul/tree/tests/fixtures/config/dependency-graph/git/common-config/zuul.yaml?h=feature/zuulv320:10
jeblairjlk: the rejiggering as a graph bit just recently landed20:10
jeblairjlk: (it used to be a tree)20:10
jesusaurjeblair: is there any context where it would make sense to create a merger:merge job for a Ref, or will they all be Changes?20:24
jesusauralso, was someone working on refactoring the Changeish classes?20:25
jeblairjesusaur: only changes (the changes we need to speculatively merge, but we can be certain that the executors will be able to check out the refs)20:28
jeblairjesusaur: currently assigned to SpamapS: but i don't think he's started yet?  https://storyboard.openstack.org/#!/story/200078120:29
SpamapSjeblair: I have started analysis. Have not succeeded in making much progress.20:31
SpamapSI would like to keep pushing forward on it.20:31
SpamapSI just keep distracting myself with test re-enablement because it's easier to see immediate progress.20:32
jesusaurjeblair: cool, thanks; I'm slowly making more progress20:35
* SpamapS will double back to it now actually20:36
jeblaircool; i have to afk again for a bit; back in a couple hours.20:37
openstackgerritK Jonathan Harker proposed openstack-infra/zuul feature/zuulv3: Perform pre-launch merge checks  https://review.openstack.org/44627520:41
pabelangerDoh!20:42
pabelangeranother patch bomb, sorry20:43
openstackgerritPaul Belanger proposed openstack-infra/zuul feature/zuulv3: Create zuul_workspace_root job variable  https://review.openstack.org/44144120:45
openstackgerritPaul Belanger proposed openstack-infra/zuul feature/zuulv3: Organize playbooks folder  https://review.openstack.org/44154720:45
openstackgerritPaul Belanger proposed openstack-infra/zuul feature/zuulv3: Add net-info to bootstrap role  https://review.openstack.org/44161720:45
openstackgerritPaul Belanger proposed openstack-infra/zuul feature/zuulv3: Add ansible-lint to tox (pep8)  https://review.openstack.org/43827620:45
openstackgerritPaul Belanger proposed openstack-infra/zuul feature/zuulv3: Add revoke-sudo role and update tox jobs  https://review.openstack.org/44146720:45
openstackgerritPaul Belanger proposed openstack-infra/zuul feature/zuulv3: Create tox-tarball job  https://review.openstack.org/44160920:45
openstackgerritPaul Belanger proposed openstack-infra/zuul feature/zuulv3: Create extra-test-setup role  https://review.openstack.org/44804220:45
rbergeronwow. that is da bomb o'patches20:45
pabelangerokay, that should make the stack green20:45
jlkjeblair: thanks, I found it and fixed up my set21:03
SpamapSdo I need to be running mysql for tests now too?21:37
pabelangerSpamapS: yes21:40
SpamapSoy21:40
SpamapSgetting heavy21:40
SpamapSwe should drop a Dockerfile or something in the root so devs can spin up fast21:41
jlkthere's a tools/ script21:41
jlktools/test-setup.sh21:41
jlkbut it doesn't do the bindep install21:41
SpamapSI'm writing up a simple Dockerfile now21:41
jlkyeah I honestly hadn't thought of doing that. Makes a fair amount of sense21:42
SpamapSjust from the standpoint of not having to build a big clean env and not polluting your laptop.. it's a win :-P21:43
jlkyeah, I've been doing all my testing in a cloud VM21:44
jlkgotta say though, the constant push / pull of code to/from my laptop to test is a pain21:44
SpamapStrying not to have to21:44
SpamapSyeah remote editting is t3h suck21:44
pabelangerwe could switch to sqlite I guess21:48
pabelangerwell21:48
pabelangerone option is to switch to sqlite :)21:48
pabelangerdepending on if jeblair sees value in that or not21:48
clarkbin the past we have explicitly switched away because raisins21:49
pabelangerDockerfile in tree might bitrot pretty quick, since we don't test against docker21:49
clarkb*away from sqlite21:49
pabelangerclarkb: right21:49
clarkbif the dockerfile uses bindep and test-setup.sh it is probably fine21:49
clarkbthats really similar to what we do on our test VMs21:49
SpamapSpabelanger: not hard to make a docker test job though21:50
SpamapSthe only reason I say docker is that I want a lightweight local test env21:50
clarkbI've had a docker that runs mysql just for nodepool tests for a long time.21:51
pabelangerwhatever happen to dox?21:51
pabelangerwasn't that a docker for tox thing21:51
clarkbbut nothing fancy, I just grabbed upstream ubuntu image and apt-get installed mysql21:51
clarkbpabelanger: I think it mostly died? mordred was deving on it. Not sure it really solved the problem people had?21:52
clarkbI dunno I never used it beacuse I didn't like the idea of giving unittests root on my boxes21:52
pabelangerya, I never tried it21:52
pabelangerjust knew of it21:52
SpamapSDNS is really slow from inside my docker containers for some reason.21:55
SpamapSI think it's confused by my native ipv621:55
pabelangerokay, dox cloned21:57
pabelangernow to install docker21:57
clarkbSpamapS: did docker stop hardcoding 8.8.8.8 and 8.8.4.4?21:59
SpamapSclarkb:  I dunno21:59
SpamapSclarkb: the ubuntu docker image uses the host's resolv.conf21:59
SpamapSwhich for me right now is the IBM vpn22:00
SpamapSwhich is actually probably where much of the confusion comes from, not IPv622:00
SpamapSWith needing to install build-essential, zookeeperd, and mysql-server ... this is a big docker image.22:00
SpamapSinteresting with dox...22:01
SpamapSreads .travis.yml22:01
SpamapSFetched 122 MB in 5min 41s (356 kB/s)22:04
SpamapSslow ubuntu mirror.. argh22:05
pabelangerclarkb: I see what you mean about root22:05
pabelangerdocker: Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock22:05
pabelangerneed to figure that out22:05
SpamapSpabelanger: you just need to be in the docker group22:06
clarkbpabelanger: you add yourself to the docker group but once you've done that the user effectively has root as long as the docker daemon is running22:06
clarkbsince docker can do all the things!22:06
pabelangerah22:06
pabelangerya22:06
SpamapSclarkb: that's a bit of an oversimplification22:06
clarkbI mean in as much as you can't just sudo foo22:06
clarkbthere are steps between here and there22:07
SpamapSa lot22:07
clarkbbut they are straightforward and not difficult22:07
clarkbyou just make a container with root in the hosts user namespace aiui22:07
SpamapSknown steps.. defined steps.. but.. you're going to have to invite untrusted code to run in your containers first.22:07
SpamapSyeah don't do that22:07
SpamapSthat's a privileged container22:07
SpamapSdon't do that :)22:07
clarkbbut I don't control what $randomtestsuite does22:08
SpamapSjust like don't have passwordless sudo22:08
clarkbwhich is why I have found it safer to run in VM as unprivileged user22:08
SpamapSclarkb: You trust yourself not to have any local root priv escalation too though, so... not that far off.22:08
clarkbSpamapS: the problem isn't in the specific zuul case22:08
clarkbSpamapS: its in the general use dox as test runner tool22:08
SpamapSLike, you invite code to run on your box all the time.22:08
SpamapSSame reason I don't use my dev box to access my PGP keys... I have so much garbage code in and out of this thing..22:09
pabelangerhey, that worked22:09
clarkbSpamapS: right, but there is a difference between trusting my distro provided code and grabbing random openstack project and running `dox`22:09
clarkbso what I do is run VM with kvm and run tox in there as ubuntu22:10
pabelangercool, dox doing thing22:10
SpamapSsame danger running 'tox' IMO22:10
pabelangerwe'll need to update this to use DIB22:10
SpamapSclarkb: ah yeah you're sealing that off22:10
clarkbSpamapS: its less because there isn't a direct line to root22:10
SpamapSI'm not22:10
clarkbmy workstation stays pristine22:10
SpamapSjust accepting they might pwn me.. and this is a throw away box without secrets on it.22:10
clarkbin as much as unprivilged user can escape kvm22:10
SpamapS(well.. except my GH/Gerrit SSH keys.. but I can revoke those)22:11
SpamapSclarkb: yeah, your way is way safer22:11
SpamapSand WAY more of a PITA22:11
SpamapSthough.. I imagine you have a host->guest mount so you can edit local?22:11
SpamapSthat might make local vms more attractive22:11
clarkbI've done that, but currently I just copy patches around22:12
SpamapSyeah, f that! ;)22:12
clarkb(either via git push or git apply whatever)22:12
SpamapSpush.. apply.. copy.. edit.. ggggggggggahhhh22:12
clarkbeh22:12
SpamapSno thanks :)22:12
SpamapSbut a host->guest would work too22:12
clarkbits trivial to do the other thing22:12
clarkbI found that further separating and not nuking my actual repos was desireable22:13
clarkbjust sshfs done22:13
jlkI wind up using a github fork as a staging place to pass changes around22:13
jlkgit push -f and git fetch && git reset --hard22:13
SpamapSjlk: yeah I've done that hell too22:13
SpamapSdamnit22:13
SpamapSwasn't going to spend all afternoon dockering22:14
jlkprice one pays for using OS X22:14
SpamapSbut ubuntu's slow mirror is killin me22:14
clarkbalso I found virt install22:14
clarkbwhich makes the boot VM process simple22:15
SpamapSoh good work docker.. you just fell back to Readline but you can't read lines from me.22:15
pabelangerokay, so dox isn't that bad!22:21
pabelangerI have to rebuild base image, but it still works22:21
SpamapSpabelanger: why not just write a Dockerfile?22:21
SpamapShttp://paste.openstack.org/show/60369922:21
SpamapSthats the one I'm testing right now22:22
mordredpabelanger: I don't think any of us are working on dox anymore - but I'd be more than happy to give you core on it immediately if you wanted to22:22
mordred(out of curiosity, what are we dockering?)22:22
SpamapSmordred: zuul testing22:22
SpamapSsince now I have to run mysql-server and zookeeperd to run tests22:22
clarkbSpamapS: oh I remember the other reason for not mounting git repos into VMs, it was devstack doing things and I realized I couldn't trust tests to not break me either22:23
SpamapSwhich I think is cool22:23
SpamapSnot a bad thing, but a great thing, to test against the real things22:23
SpamapSbut my laptop doesn't need all those things most of the time22:23
clarkbbasically that was sufficient surface area for me to have a bad day so I removed it22:23
SpamapSclarkb: yeah I'd make that mount RO22:23
mordredSpamapS: cool22:23
clarkbSpamapS: but then you can't commit your changes? it sort of defeats the sync back and forth use case22:24
clarkboh you do it on the other end derp22:24
clarkbgotcha22:24
SpamapSI'd be done with that Dockerfile but for some reason archive.ubuntu.com has always been really slow for me over ipv4. In fact, when it got fast was when I noticed I had native v6 from Spectrum.22:24
SpamapSclarkb: I edit on my local box, with my vim, and my settings.22:24
* jlk tries to get his laptops docker to work again22:25
SpamapSclarkb: that's the main reason I want to mount things in the VM22:25
jlkI"m sure this is all going to go sideways because OSX22:26
jlkbut worth a try22:26
SpamapSanother annoyance with both docker and vms is that I have to tell them how to find my local squid cache that has all these stupid debs in it already22:26
SpamapSI remember there was a thing for a while.. squid-deb-proxy.. that would make your squid an avahi service22:27
SpamapSuntil IIRC somebody MITM'd some debs at a UDS22:27
pabelangerHey it worked22:27
pabelangerdox.tests.config.test_commands.TestCommands.test_commands(dox_yaml)            0.02122:28
SpamapSjust to see if anyone would install anyway22:28
pabelangerthat was in my infra/trusty container22:28
SpamapSpabelanger: it's 2017... time to xenial22:30
SpamapS;)22:30
pabelangerya, need to patch dox :)22:36
pabelangermordred: going to try working with it, see how it goes. So, expect some patches22:37
jlkSpamapS: isn't that docker file missing the tools/test-setup.sh ?22:38
SpamapSjlk: it is, because I'm still just trying to get bindep to work22:38
jlkah22:38
SpamapSit's failing for some reason22:39
mordredpabelanger: you are already core on it, since infra-core is core on it - but I added you explicitly anyway22:39
SpamapSaha!22:40
SpamapSbindep requires lsb_release22:40
SpamapSa reasonable req22:41
pabelangermordred: WFM22:41
jlkhow else does it reliably determine which Linux you are?22:42
pabelangerHAHA22:43
pabelangerTrying again with ssh://pabelanger@review.openstack.org:29418/stackforge/dox.git22:43
SpamapSjlk: yeah entirely reasonable22:44
jlkimage caching for the win, right?22:45
pabelangerokay, will continue tomorrow22:45
jlkoh oh what22:46
mordredSpamapS: I love that distro cloud images don't always have lsb_release22:46
jlkbindep only shows that you need to install things, it doesn't actually install things?22:46
mordredit's the ultimate in "we stuck in a bunch of crazy vendor shit but left out the tiny utility that gives you metadata about your system"22:46
mordredjlk: yes.22:46
clarkbjlk: it will print the command though iirc22:46
clarkbso you take its output and run it22:46
jlkmordred: well, on Fedora to get lsb-release ou have to install a bunch of shit22:46
jeblairmordred: i like that policy: if you can run dox you're core.22:46
jlk# bindep -f /zuul/bindep.txt test22:47
jlkMissing packages:22:47
jlk    libjpeg-dev mysql-client mysql-server22:47
jlkthat's all it says for me22:47
mordredjlk: that's not true ... there is a minimal package you can install to just get lsb_release that isn't tied to all the X requirements22:47
clarkbthere is a switch22:47
SpamapScrap22:47
clarkb-b?22:47
SpamapSI forgot docker's lovely "we don't really do pid 1" way of being22:47
jlkthat just spits out the packges one per line22:48
jlkSpamapS: oh right, "running" mysql may be... diffiult22:48
SpamapSso to get something behind /var/run/mysql.sock one has to somehow get mysqld running :-P22:48
mordredjlk: https://github.com/ansible/ansible/pull/22159/files see line 6022:48
jlkyou may have to run a mysql container linked to your tox container22:48
clarkbjlk ya looks like you have to feed the -b output to your package manager22:48
SpamapSif only we had shell scripts we could just run when we need something like that22:48
jlkand a zookeeper container, and...22:48
mordredSpamapS: you may want to poke at mysql-sandbox from guiseppe - I think it has docker support now :)22:48
SpamapSjlk: except I need /var/run/mysql.sock, not 127.0.0.1:330622:48
jlkSpamapS: you can do linked volumes!22:49
SpamapSI'm just going to safe_mysqld22:49
mordredoh - I never pushed up my fix for the 2.4 issue on the bindep module PR22:49
SpamapSor not22:49
SpamapSI'm spent22:49
clarkbSpamapS: we that shell script is why I have never interacted with bindep directly22:50
clarkbs/we//22:50
SpamapSjlk: linked volumes sounds insane. So what, I can make the mysql container export its /var/run ?22:50
mordredSpamapS: yes22:50
jlkmordred: lsb_release comes from redhat-lsb-core-4.1-33.fc25.x86_6422:51
jlkwhich drags in a bunch of stuff on a stock fedora 25 docker image22:51
* SpamapS tries '/usr/sbin/mysqld_safe & ..'22:52
jlkSpamapS: yeah you can do volumes_from22:52
mordredjlk: right. well, that's what I meant by 'it's the ultimate in "we stuck in a bunch of crazy vendor shit but left out the tiny utility that gives you metadata about your system"'22:54
mordredsince packaging lsb_release in a package that includes a giant ton of other useless crap is an insane choice22:54
mordredredhat-lsb-core is the "minimal" version compared to redhat-lsb which pulls in a CRAZY amount of things22:55
jlkah. I see what you mean. Well, the whole world of LSB is full of useless crap and insane choices22:55
mordredyah22:55
mordredfo sho22:55
mordredbut the lsb_release utility is tiny and essential and it not being places is one of the most frustrating parts of being a person who consumes more than one distro22:55
jlkwhy isn't zookeeperd in the bindep list?22:56
jlkoh it's in platform:dpkg ?22:56
clarkbbecause it doesn't exist on red hat distros22:56
mordredyah22:56
mordredsomeone is working on it22:56
jlkweird seems there for Fedora 2522:56
mordreds/red hat distros/centos/22:57
mordredI think adding a bindep line for it for fedora would be great22:57
mordredjlk: I think pabelanger was working with someone on getting in to a spin or something for centos/rhel land22:58
clarkboh cool looks like f24 has it too22:58
jlkI'll look at the syntax22:58
SpamapSok, I figured out how to get a ghetto mysqld up22:58
mordredjlk, pabelanger: btw - updated the bindep pr for ansible with docstring fixes - it should be green this time22:58
SpamapSnow just a ghetto zookeeperd22:58
jlkkk22:59
jlkSpamapS: what did you do in your docker file to ghetto mysql?22:59
jlksafe_mysqld & ?  as a RUN ?22:59
jlkor maybe build a .sh that does it all22:59
SpamapShttp://paste.openstack.org/show/60370322:59
SpamapSyep22:59
SpamapSyeah I think I'll put that in tools22:59
SpamapSwith the zookeeperd23:00
SpamapS"daemons.sh"23:00
SpamapSoops, forgot python-dev23:00
SpamapSyay.. zookeeper kept its init script23:01
SpamapSso /etc/init.d/zookeeper start _works_23:01
* jlk eyeballs "/usr/bin/mysqld_pre_systemd"23:04
* clarkb wonders if that included all the self daemonization code and now it lets systemd do it23:04
SpamapSjlk: lol!23:05
SpamapSamazing23:05
SpamapSclarkb: yeah probably23:06
jlkit's a bash script23:06
*** harlowja has quit IRC23:06
openstackgerritJamie Lennox proposed openstack-infra/zuul feature/zuulv3: Re-add the ability to set username on zuul-executor  https://review.openstack.org/44630823:07
jlkthis is a pain23:10
SpamapSaye, 'tis23:11
SpamapSsince we depend on cryptography, we should add python2.7-dev to our bindep23:11
SpamapSor.. should we?23:11
jlk¯\_(ツ)_/¯23:12
clarkbSpamapS: ya a lot of projects have left it off because its implied on our test machines for other reasons (well zuul is installed on them in avirtualenv)23:13
clarkbbut I think it would be good to be explicit about the python versions required and whether or not you need the headers too23:13
SpamapSis there a way to call tox where it runs the _setup_ for the venvs, but not the commands?23:14
clarkbyes there is a setup only flag23:15
clarkbI forget what it is though23:15
SpamapSI'll look23:15
SpamapS#lazyirc23:15
SpamapSweird.. pep8 is trying to run with python3.5 .. but otherwise it seems to be workingish23:17
jesusaurSpamapS: not that weird, since it's set in tox.ini23:18
SpamapSoh!23:18
SpamapShaha23:18
SpamapSthat seems.. odd?23:18
jesusaurya, I'm not sure why it's set there, but I noticed it a couple days ago23:18
jlkhrm, struggling to get zookeeper up23:24
jlkah23:25
SpamapSI almost have it now23:27
SpamapSlast step is to run as not-root :)23:28
SpamapSand to figure out how to rebuild w/ new context23:29
jlkheh23:30
*** jamielennox is now known as jamielennox|away23:35
jlkwell you can docker run --no-cache23:38
jlker docker build --no-cache23:38
jlkto rebuild the image without caching23:38
*** jamielennox|away is now known as jamielennox23:39
SpamapSjlk: but I want all the caching _except_ the context23:39
SpamapSbasically everything up to the ADD23:39
jlkdefine "content"23:39
SpamapScontext23:39
SpamapSthe git repo23:39
SpamapSand working dir23:39
jlkoh I'm not putting that in the image at all23:39
jlkthat's just a volume mount23:39
SpamapSI guess that's the better way23:40
jlkso you literally change it on your host and it's changed in the container23:40
SpamapSbut I kind of liked the context thing23:40
jlkwhat's context?23:40
SpamapSby default everything in the dir with the Dockerfile23:41
SpamapSand then you can ADD things from it23:41
SpamapSbut maybe that's borken23:41
jlkI figured you'd want to have an image that's capable of running tox, and in our case, it has mysql and zookeeper running. You'd docker run with the volume of your zuul checkout23:46
jlkand the default command would be just to run tox, or you can override it and do a docker run <image> <volume> tox <whatever>23:46
jlk(a compose file may make this more easy)23:47
jesusauraha, I *finally* figured out why my test wasn't holding the builds, hold_jobs_in_queue != hold_jobs_in_build23:52
SpamapSjlk: I want an image with the tox venvs already populated :)23:52
jlkthey will be23:52
jlkbecause the tox build will be building in your zuul/ dir23:52
jlkwhich is on your host23:52
SpamapSYeah true, but that only works by happenstance because I'm running Xenial.23:52
jlkso every time you run, you already have .tox/ with populated content23:52
jlkI mean, it'll get populated from within the container23:53
SpamapSYeah either way true23:53
jlkit may not _work_ outside the container23:53
SpamapSI still need the bindep file too23:53
SpamapSbut that's easy to add23:53
jlkwhy?23:53
jlkall it's good for is telling you what to install23:53
SpamapSso I can install bindeps during docker build23:53
SpamapSalso I don't see how to control the uid of a volume23:54
SpamapSbut I can probably learn that by playing23:54
jlkoh for non-root?23:55
jlkyeah I dunno23:55
jlkI haven't crossed that bridge yet23:55
SpamapSI don't want anything running as root23:55
jlkERROR:   py27: InvocationError: '/zuul/.tox/py27/bin/python /zuul/setup.py --name'23:56
jlkhuh, what does that mean.23:56
*** hashar has quit IRC23:56
jlkjust running tox -e py2723:56
jlkI don't get it23:56
mordredjlk: there may be logs in .tox23:57
mordredjlk: it's possible it's a directory permissions issue - or that you don't have setuptools installed perhaps23:58
jlkit created the venv, but no longs23:58
mordredah - awesome23:58
mordredthat's fantastic :)23:58
jlkno setuptools in pip freeze tho23:59
jlkeither inside or outside the venv tox created23:59

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!