| *** threestrands has quit IRC | 00:38 | |
| *** jiapei has joined #zuul | 01:20 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul master: Use mode 0o755 for executor server mkdirs https://review.openstack.org/585068 | 02:16 |
|---|---|---|
| SpamapS | six is a dependency of like, everything. Probably not reasonable to rely on it not being there. ;) | 04:49 |
| SpamapS | pip and virtualenv are the only things I ever install into system python :-P | 04:50 |
| openstackgerrit | Ian Wienand proposed openstack-infra/zuul master: Rename didAnyJobFail() to hasAnyJobFailed() https://review.openstack.org/585134 | 05:16 |
| *** nchakrab has joined #zuul | 05:27 | |
| *** quiquell|off is now known as quiquell | 05:38 | |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/zuul master: Add instructions on building static web components https://review.openstack.org/581256 | 05:46 |
| *** nchakrab has quit IRC | 06:22 | |
| *** nchakrab has joined #zuul | 06:31 | |
| *** pcaruana has joined #zuul | 06:34 | |
| *** quiquell is now known as quiquell|bbl | 06:54 | |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Scope all_known_hosts in mult-node-known-hosts https://review.openstack.org/584993 | 06:56 |
| *** hashar has joined #zuul | 06:58 | |
| *** bbayszczak has joined #zuul | 07:06 | |
| *** bbayszcz_ has joined #zuul | 07:23 | |
| *** bbayszczak has quit IRC | 07:26 | |
| *** nchakrab has quit IRC | 07:29 | |
| *** nchakrab has joined #zuul | 07:36 | |
| *** gchenuet has joined #zuul | 07:43 | |
| *** bbayszcz_ has quit IRC | 07:43 | |
| *** bbayszczak has joined #zuul | 07:44 | |
| *** gchenuet has quit IRC | 07:47 | |
| *** quiquell|bbl is now known as quiquell | 07:52 | |
| *** nchakrab_ has joined #zuul | 08:29 | |
| *** nchakrab has quit IRC | 08:31 | |
| *** sshnaidm|ruck is now known as sshnaidm|afk | 08:46 | |
| *** gouthamr has quit IRC | 09:02 | |
| *** dmellado has quit IRC | 09:03 | |
| *** panda|off is now known as panda | 09:25 | |
| *** goern has joined #zuul | 10:16 | |
| *** gouthamr has joined #zuul | 10:20 | |
| *** dmellado has joined #zuul | 10:24 | |
| *** nchakrab_ has quit IRC | 10:42 | |
| *** hashar has quit IRC | 10:47 | |
| *** quiquell has quit IRC | 10:48 | |
| *** panda is now known as panda|lunch | 11:10 | |
| tristanC | pabelanger: i've started an implementation of the "container that behave like machine" as specified by the container-build-resources spec, and it seems like it can't support bindep unless we give root capability to the pod | 11:23 |
| tristanC | pabelanger: which will be a similar regression you reported about the runC driver: you need to duplicate the bindep list to the container build recipe... | 11:24 |
| *** sshnaidm|afk is now known as sshnaidm|ruck | 11:29 | |
| *** GonZo2000 has joined #zuul | 11:39 | |
| *** GonZo2000 has quit IRC | 11:39 | |
| *** GonZo2000 has joined #zuul | 11:39 | |
| *** jiapei has quit IRC | 12:10 | |
| *** nchakrab has joined #zuul | 12:12 | |
| *** panda|lunch is now known as panda | 12:36 | |
| *** GonZo2000_ has joined #zuul | 12:38 | |
| *** rlandy has joined #zuul | 12:39 | |
| *** GonZo2000 has quit IRC | 12:40 | |
| *** samccann has joined #zuul | 13:00 | |
| mordred | tristanC: for the container to behave like a machine I would expect the code running in it to be able to run commands as root - otherwise it's not really a thing that behaves like a machine ... | 13:05 |
| mordred | tristanC: is the issue with bindep just that installing packages at runtime at all would be an issue? | 13:06 |
| mordred | or is there something specific that bindep is doing that requires some capability beyond 'install packages' | 13:06 |
| *** ianychoi has quit IRC | 13:08 | |
| *** ianychoi has joined #zuul | 13:08 | |
| tristanC | mordred: this is similar to the bwrap issue, iiuc we can't create container with root capabilities | 13:09 |
| tobiash | mordred: in openshift containers by default cannot run as root (aka uid 0 inside the container) | 13:10 |
| tobiash | mordred: that is distinct to running with privileges | 13:10 |
| mordred | gotcha. so I'd say that, by default, openshift cannot provide thething I would consider "container that behaves like a machine" | 13:10 |
| tobiash | mordred: but both restricted | 13:10 |
| mordred | but it's interesting informatoin, and means we might need to define 2 different things here | 13:10 |
| tristanC | though i don't think this is a regression, user just need to provide an image with the bits they need to run a job | 13:11 |
| mordred | yeah - but you can't test a patch that way | 13:11 |
| mordred | if the patch contains an update to bindep.txt | 13:11 |
| mordred | the testing of the speculative future state would be incorrect | 13:11 |
| mordred | (if the test itself cannot create the environment) | 13:11 |
| mordred | or? | 13:12 |
| * mordred might be making statements - but they're all also questions | 13:12 | |
| tobiash | mordred: in the container world you may want to build an updated image first | 13:13 |
| mordred | right. but that's container-native | 13:13 |
| tristanC | mordred: and openshift seems designed for container-native... | 13:14 |
| mordred | yes | 13:14 |
| mordred | container-as-machine is explicitly about serving people who are not container native but who maybe don't need full vm isolation | 13:14 |
| mordred | pep8 job in openstack is a good example | 13:14 |
| mordred | or tox-py35 | 13:15 |
| mordred | it does not need to be rethought as a container-native workload - it's running pep8 - just running commands in the container should be fine | 13:15 |
| tobiash | mordred: in that case the operator either must know that root is not a thing or explicitly allow root in his kubernetes/openshift | 13:15 |
| mordred | tobiash: yes, I believe I'm coming to agree with that | 13:16 |
| tristanC | my current implementation adds a label's type to request a pod or project. e.g. "labels: [{name: openshift-project, type: project}, {name: py36-pod, type: pod, image: docker.io/centos/python-36-centos7}] | 13:16 |
| tristanC | could we have a step between container-as-machine and container-native, e.g. container-image ? | 13:18 |
| mordred | maybe - I think this is worthy of further discussion, as I think "containers can't run things as root" is a piece of information that is new | 13:18 |
| tristanC | i mean a step in the container-build-resources spec | 13:19 |
| mordred | tristanC: I'm not sure I follow what you mean? | 13:20 |
| tristanC | mordred: container-as-machine, e.g. un-restricted env like k8s or docker can provide, container-image, e.g. a regular pod in openshift, container-native, e.g. a project to build speculative pod(s) | 13:21 |
| rcarrillocruz | are you folks aware of kubevirt? i've used it, is about making virtual machines first class citizens in pods | 13:22 |
| mordred | rcarrillocruz: yes, but I don't think it solves anything here | 13:22 |
| mordred | this isn't about being able to use k8s to get vms (we can already get vms) - it's about getting containers that are containers | 13:23 |
| tristanC | mordred: because atm, I can't implement the container-build-resources spec on openshift for the simple use-case, unless we create a distinction for containers without root access | 13:24 |
| rcarrillocruz | right, i guess my point is, if you want to do root things in k8s you have to set a privileged pod | 13:24 |
| rcarrillocruz | be a container | 13:24 |
| rcarrillocruz | or a kubevirt vm | 13:24 |
| mordred | I don't want to do root things in k8s | 13:24 |
| mordred | I want to do root things in $something that behaves like an actual machine rather than like a PaaS | 13:25 |
| rcarrillocruz | root in the pod i mean, for the context of a job | 13:25 |
| mordred | right. so it's possible the answer here is "the thing people have been saying will make some things easier is in fact impossible and so the use case is worthless" | 13:25 |
| rcarrillocruz | heh | 13:26 |
| rcarrillocruz | yeah, is retrofitting , not a good use case | 13:26 |
| mordred | because the idea here is to be able to use a container in place of a VM for jobs where the container vs. vm distinction is not useful | 13:26 |
| mordred | again, I'll reference the tox-py35 job | 13:26 |
| mordred | there is nothing container native or vm native about that job - it's a DEAD SIMPLE thing that runs, installs software and runs some unit tests | 13:26 |
| mordred | so the argument has been made for the last 5 years or so that we could have amajor win if we just grabbed ubuntu containers instead of spinning up ubuntu vms | 13:27 |
| mordred | it's starting to look like all of the people who made that argument so far were ignoring the fact that the things one gets containers from don't let you do basic things lke install packages | 13:27 |
| rcarrillocruz | right, it's the same argument about 'hey, i can do whatever in docker you do in vms' , 'oh really, can you do this thing that is not a web app and requires root' , 'oh ofc, i run it in privileged mode' | 13:28 |
| rcarrillocruz | which means you are running something in a less secure way | 13:28 |
| mordred | yup | 13:28 |
| rcarrillocruz | just for the sake 'i use containers' | 13:28 |
| mordred | yes - so that's what I'm learning so far this morning | 13:28 |
| mordred | yet again "containers make this easier/better" is a marketing falsehood | 13:28 |
| rcarrillocruz | is moving complexity from config management to image building, apart from being less secure... at least that's how i see it | 13:29 |
| mordred | well - it's also missing the step between the source and the workload | 13:29 |
| mordred | because it seems to be quite common, as was mentioned before, that someone "provides an image" | 13:29 |
| tristanC | it seems like just a separation of privilege between build and runtime | 13:29 |
| mordred | zuul, however, is a system that is designed to work on source code, so if a user had to build an image and upload it somewhere before they could run a test ... | 13:30 |
| tobiash | just to note, you *can* have root containers without privileges in openshift, you just need to allow that | 13:30 |
| mordred | that would be like the old-school pre-gating CI where people push patches to a branch and then test run to tell you whether it's red or green | 13:30 |
| mordred | tobiash: awesome. so it may just be that for container-as-machine it'll only work if you've allowed that? | 13:31 |
| mordred | and that we may need to document that requirement for that use case | 13:31 |
| tobiash | mordred: if your workload needs to install packages, yes | 13:31 |
| tobiash | mordred: however I don't know what's the default in kubernetes | 13:31 |
| mordred | I don't either | 13:32 |
| tobiash | maybe there it's more relaxed | 13:32 |
| mordred | I feel like I should write some long-form words about this ... but one of my goals is to be able to do the thing we think of as gating without any humans ever having to do the equiv of docker push | 13:33 |
| mordred | becaues docker push is just like git push ... if a human is doing it, something has gone wrong | 13:33 |
| tristanC | tobiash: don't you need cluster-admin access to enable any uid? | 13:33 |
| *** swest1 has quit IRC | 13:33 | |
| tobiash | mordred: there are different sccs (security context constraints) in openshift, one is privileged, one is any-uid and the default is restricted (which starts the pod with a random uid) | 13:34 |
| tobiash | tristanC: the cluster admin can allow specific sccs to specific users or groups | 13:34 |
| tobiash | then that user can make use of that scc | 13:35 |
| * mordred has learned many things this morning | 13:35 | |
| tobiash | mordred: the any-uid enables you to choose the uid inside the pod (including 0) without giving you privileges | 13:35 |
| tobiash | mordred: it's just an additional default safety mechanism in addition to dropping privileges | 13:36 |
| mordred | cool. that sounds like the thing that would be needed for container-as-machine to work as intended. and when I say "as intended" - what I mean is that I expect jobs in zuul-jobs to work in containers provided by container-as-machine | 13:36 |
| mordred | it's possible we might want to be more explicit about that and maybe name that somehow | 13:37 |
| mordred | then I think there is another but potentially similar thing, perhaps what tristanC is referring to as container-image which is like container-as-machine but does not need any-uid | 13:38 |
| rcarrillocruz | https://blog.zhaw.ch/icclab/the-intricacies-of-running-containers-on-openshift/ | 13:39 |
| rcarrillocruz | so yeah, the uid thing is 'special' in openshift | 13:39 |
| tristanC | mordred: on a shared kernel, that seems like a safe thing to do for unstrusted ci jobs | 13:39 |
| rcarrillocruz | i hit that on trying to get zuul control plane to work in openshift online | 13:39 |
| rcarrillocruz | which is why i asked tobiash how he did, and cos he was admin he could just mangle with sccs | 13:39 |
| mordred | however, I imagine this is going to be a similar thing we're going to need to look at for container-native anyway, since zuul is going to nede to be able to build containers | 13:40 |
| tristanC | mordred: with build-config object, s2i can build image for you, as a regular user | 13:40 |
| mordred | so a zuul install is going to need _some_ level of elevated privs to interact with container systems whether we're doing container-as-machine or container-native | 13:40 |
| mordred | tristanC: nope | 13:40 |
| tobiash | mordred: in openshift you by default have several service accounts where one of it is 'builder' | 13:41 |
| tobiash | mordred: I think we might be able to use that service account to create 'builder' pods that build images | 13:41 |
| tristanC | mordred: well yes, see https://review.openstack.org/#/c/570669/3/playbooks/openshift/build-project.yaml | 13:41 |
| mordred | so - I think part of the problem here is that Zuul is a service that wants to _provide_ the same services as openshift buildConfig does | 13:42 |
| tristanC | tobiash: yes, for container-native that is fine, but for the simpler use-case to run linters job on an pre-exisiting image, it would be nice to have a distinction between environment that allow root and the one that doesn't | 13:43 |
| mordred | there are almost zero jobs defined in openstack's zuul today that would be able to be run in containers without root | 13:44 |
| mordred | I suppose the javascript jobs probably could be defined to run in javascript base image containers | 13:45 |
| mordred | but not even the python unittest jobs can | 13:45 |
| mordred | so maybe instead of using linters as the example "simple" case, we need to use "python unittests" as an example "simple" use case as we think through this? | 13:46 |
| tristanC | mordred: hum, iiuc the container-build-resources spec, the image building step is part of the job, not the zuul service per se | 13:47 |
| mordred | but it's also possible that we need to take a step back and take another stab at defining the use cases and what we're trying to accomplish with them - because today's conversation has brought a giant piece of new information | 13:47 |
| tristanC | mordred: so it should be up to the base job owner to use or not the build config service | 13:47 |
| *** dkranz has joined #zuul | 13:47 | |
| mordred | tristanC: that would depend on the base job owner knowing that they were running in openshift | 13:48 |
| mordred | and would mean that **zuul** would not be providing the appropriate primitives to allow someone to write a job | 13:48 |
| mordred | maybe that' sok | 13:48 |
| mordred | I don't know | 13:48 |
| mordred | it's just a ton of new information | 13:48 |
| mordred | I know it does not sound like how a system I would like to use would work - so it might take me a while to wrap my head around it | 13:49 |
| tristanC | mordred: well the zuul-base-job could be part of zuul too, and the primitives could be there too | 13:49 |
| mordred | maybe so | 13:49 |
| mordred | I've just been considering zuul as a replacement for the openshift provided buildConfig system, since AIUI that uses jenkins on the backend to do its work, no? | 13:50 |
| tristanC | mordred: buildconfig doesn't need jenkins afaik | 13:51 |
| mordred | ok | 13:52 |
| mordred | well - I feel like I've learned many things - so thanks ... I definitely need to digest for a bit | 13:52 |
| tobiash | mordred: openshift buildconfig uses build containers spawned in pods using the builder service account | 13:53 |
| mordred | ok | 13:53 |
| tobiash | mordred: and openshift also allows custom build containers | 13:53 |
| mordred | but does it allow for speculatively building build containers based on the git repo state of the repo that the build container would come from? | 13:54 |
| tobiash | mordred: it just supplies a few default builder containers (various ones for s2i and one for building using docker files) | 13:54 |
| mordred | because that's the thing we need to do to actually make that whole stack work | 13:54 |
| mordred | right? | 13:54 |
| tristanC | mordred: yes it can, but you need to give access to a git ref | 13:54 |
| tobiash | mordred: for that we might have to look if we can create builder containers without a build config | 13:54 |
| mordred | tristanC: right, which means you have to do the "I'm going to serve git refs over http" right? which is going to be hideously inefficient at scale | 13:55 |
| tristanC | mordred: well that's how the openshift-base ( https://review.openstack.org/570669 ) works atm. what's the difference between pull from http to push over exec connection? | 13:56 |
| mordred | I don't know. maybe nothing | 13:58 |
| *** maeca has joined #zuul | 13:58 | |
| mordred | it seems like it would make it very hard to keep a git repo cache mechanism that would obviate the need to push the entire contents every time | 13:58 |
| mordred | if we had zuul know how to build images itself natively, then it could also manage a git repo cache environment so that it cuts down on the overall traffic | 13:59 |
| mordred | which is why I've been expecting that zuul would not use the existing builder service inside of the COE but would have something it does itself | 14:00 |
| mordred | but again, that may just be a terrible assumption on my part | 14:00 |
| tristanC | mordred: iiuc, openshift user are already using the builder service... | 14:03 |
| pabelanger | tristanC: sorry, dealing with issues downstream but looks like you and mordred are already talking about solutions :) | 14:03 |
| tristanC | mordred: anyway, i don't think this discussion change the spec too much. the container-native is already enabling any building system as it is part of the job, not zuul | 14:04 |
| mordred | tristanC: nod. yeah- it's possible the buidler service is the right thing to do for that part of the spec | 14:04 |
| mordred | tristanC: and it's possible that the thing we've been calling container-as-machine has some caveats and restricts | 14:05 |
| mordred | resrictions | 14:05 |
| mordred | gah | 14:05 |
| mordred | restrictions | 14:05 |
| mordred | that just need to be documented | 14:05 |
| tristanC | mordred: ideally we should just add a mention regarding container-as-machine may not have root access (since that's not available to regular openshift user) | 14:05 |
| tristanC | or create a new category, container-as-restricted-machine ? | 14:06 |
| mordred | yeah. so - my concern is that I REALLY don't want to fall into the trap openstack did where different zuuls behave massively differently because of deployment details of the operator | 14:06 |
| mordred | I've been fixing that hell personally for many years now | 14:06 |
| mordred | I don't want some of the jobs in zuul-jobs to not work on some zuul installations because the operator decided to use containers for build resources and didn't have access to any-uid | 14:08 |
| tristanC | mordred: well, runc, lxc, docker, k8s and openshift all has a different feature set, so isn't that bound to happen? | 14:08 |
| mordred | tristanC: to me - all of those are ways to get access to linux | 14:08 |
| tristanC | mordred: zuul-jobs already doesn't work if you can't prepare-workspace, and afaik this role only works through ssh or docker atm | 14:08 |
| mordred | right - but the spec for container-as-machine is written so that we could update prepare-workspace to work on them | 14:09 |
| tristanC | or more generally, any task using synchronize | 14:09 |
| mordred | right - that's why all of that work went in to fleshing out ways to do synchronize in container-as-machine | 14:10 |
| mordred | if container-as-machine only works with prepare-workspace on _some_ zuul insalls, I think that's very confusing | 14:10 |
| tobiash | mordred: in that case I think the best we can do is to clearly state that anyuid is needed if using containers (and maybe even refusing to work if that's not satisfied) | 14:12 |
| mordred | tobiash: yeah. that's my gut-feeling right now (and we need special things for the executors anyway currently) ... but I also think my thoughts my change the more we think and talk about this | 14:13 |
| tristanC | tobiash: which exclude openshift user to use simple container image... could we please add a new container-as-restricted-machine to support user that do not need root access? | 14:13 |
| mordred | which is to say - thanks to both of you ... this has been *very* helpful for my brain :) | 14:13 |
| tobiash | tristanC: we might leave that option open but clearly state it as unsupported because of default jobs that won't work | 14:14 |
| mordred | tristanC: maybe? it's literally a **completely** new idea I've never even thought of before this morning so I don't really have much in the way of opinions on what the zuul user experience is for such a thing yet | 14:14 |
| tristanC | i mean, 90% of our jobs run fine on a common image that have a few devel package backed-in | 14:15 |
| mordred | maybe 1% of our jobs do | 14:15 |
| mordred | tristanC: but I hear what you're saying and I could see how it could be a thing that people would want - I just don't know how to think about it yet | 14:18 |
| tristanC | s/90%/most/ (sorry, that number doesn't make sense, i meant images just need bindep installed, and new bindeps could be installed preemptively) | 14:20 |
| tristanC | mordred: sure, i just hope this doesn't delay the container spec too much :) | 14:22 |
| mordred | tristanC: that just still sounds like container-native to me - because instaling the bindeps preemptively would mean the base job would need to submit something to buildConfig to get a base image with the bindep depends installed, and then would need to boot another container based on that new base image to run the rest of the job | 14:22 |
| mordred | tristanC: because it's a key part of the jobs that they install the bindep depends as part of the job | 14:23 |
| mordred | also - for things like openstack, we have 2k repos and all of them have different bindep depends list - so we'd either have to maintain base images with pre-installed deps for every repo+branch+distro combo ... and lose the ability to fail a job if someone proposes an update to the bindep list | 14:24 |
| tristanC | mordred: well maybe it's just a matter to define label that support dynamic bindep (e.g. root access required or container-native) from the one that doesn't (e.g. regular openshift user pod) | 14:24 |
| mordred | or we need to have the base job able to build a base image that's only for that job | 14:24 |
| mordred | indeed - that might be a possibility ... "this label provides root access" | 14:25 |
| mordred | but I think we'd need to think about how we want to express that or where | 14:25 |
| mordred | do you want to put a flag on a job that says "this job requires root" | 14:25 |
| mordred | and have zuul/nodepool be able to provide a useful error message if someone tries to run a job-needs-root on a label that doens't provide it? | 14:26 |
| tristanC | mordred: it's more the project that should indicate that, because one tox job would need to run bindep, but another would not | 14:26 |
| mordred | that sounds confusing to me - the tox job in zuul-jobs runs bindep, so the tox job in zuul-jobs needs root | 14:27 |
| mordred | that would be the first time that we'd have a job in zuul-jobs that requires a specific capability in nodepool label | 14:27 |
| mordred | it's written to be cross-distro because it's not tied to qualities of the label | 14:28 |
| mordred | also - users of the zuul may have *no idea* about the details of the underlying COE - and they may not know that the base job they are using needs root | 14:28 |
| mordred | because the way tox runs bindep may not be a thing they are aware of | 14:28 |
| mordred | running on an openshift without privs is a zuul operator choice and a thing a zuul operator could be reasonably expected to understand | 14:29 |
| mordred | but it doesn't seem like a pieceof information that a zuul user should need to know | 14:29 |
| tristanC | mordred: we should support user that doesn't need root access to run job | 14:30 |
| tristanC | mordred: otherwise, regular openshift user won't be able to use zuul | 14:30 |
| mordred | regular openshift user already can't use zuul | 14:30 |
| tristanC | well they can't deploy it atm, but they can use a vm for that... | 14:31 |
| mordred | :) | 14:31 |
| tristanC | (i'll eod soon) | 14:32 |
| mordred | tristanC: seriously - thanks for the nice long in-depth conversation - I know it's super helpful for my understanding - and I imagine the scollback will be very interseting and useful to people as they wake up | 14:32 |
| rcarrillocruz | (it is) | 14:33 |
| mordred | \o/ | 14:33 |
| tristanC | hope that helps :-) | 14:34 |
| corvus | good morning! thanks for the conversation in scrollback :) i've read it, and will digest as well. | 14:36 |
| corvus | in other news... over in #openstack-infra we found a nodepool regression | 14:36 |
| corvus | it looks like the launchers may not be reporting API times | 14:37 |
| corvus | i wonder if that's related to the openstacksdk switch? | 14:37 |
| corvus | (reporting -> reporting to statsd) | 14:37 |
| tobiash | corvus: sounds most likely | 14:37 |
| mordred | corvus: oh good. yeah - it certainly could be | 14:37 |
| mordred | corvus: I shall start looking in to that right now | 14:38 |
| tobiash | corvus: re job pause, should we make inventory merging into child inventory optional? | 14:39 |
| corvus | tobiash: inventory merging? i don't follow. | 14:40 |
| tobiash | corvus: the idea of buildset resources is to make the parent job resources available in the child jobs | 14:40 |
| Shrews | corvus: pabelanger: mordred: so a regression from the shade to sdk move? | 14:41 |
| tobiash | I think I'd like that to be optional | 14:41 |
| Shrews | is it the rate limiting? | 14:41 |
| pabelanger | Shrews: not sure, still looking | 14:41 |
| mordred | Shrews, corvus, pabelanger: in the logs, are we seeing (queue: ) in the "Manager %s running task" lines? | 14:42 |
| corvus | tobiash: i did not think that the idea was to *directly* to that, but rather, to allow services running on the parent jobs to keep running and allow the child jobs to indirectly use those services. so a parent could run, say, an image server, and then return the address of that image server using zuul_return and the child job could access the image server through the network. not that the child job would | 14:42 |
| corvus | get that node in its inventory. | 14:42 |
| tobiash | that would support a use case like the parent spins up a service (containing possibly secrets) that can be used by the child but the child job should not be able to ssh-access theparent | 14:42 |
| corvus | Shrews, pabelanger, mordred: i have tcpdumped on a host and see this: E..c..@.@./.....h........O.0nodepool.task.packethost-us-west-1.compute.DELETE.servers:779.000000|ms | 14:42 |
| corvus | that sure looks like a task time | 14:42 |
| tobiash | corvus: ah oh, so this would be needed to setup by the parent | 14:43 |
| corvus | tobiash: yeah... do you have any use case that requires the parent nodes in the child inventory? | 14:43 |
| tobiash | like create ssh key and supply it via zuul_return too to make rsync available | 14:43 |
| mordred | Shrews, corvus, pabelanger: ok, we are. that means we are properly running the nodepool TaskManager at least (first panic) | 14:43 |
| tobiash | corvus: no, that was just my first idea to make it available to the synchronize task | 14:43 |
| tobiash | without adding it to the inventory we cannot use synchronize and need to call rsync directly (which is not a problem for me) | 14:44 |
| corvus | tobiash: ah, good, then i think we're mostly imagining the same thing. yeah, i think that may be the best approach for now | 14:44 |
| mordred | Shrews, pabelanger, corvus: did the key names somehow change? | 14:44 |
| tobiash | corvus: ok, so that probably was a bad idea, thanks | 14:44 |
| pabelanger | mordred: I don't see "Manager %s ran task" in logs | 14:45 |
| pabelanger | mordred: which is from post_run_task | 14:45 |
| corvus | mordred: yes | 14:45 |
| corvus | nodepool.task.$provider.ComputeDeleteServers.mean | 14:45 |
| corvus | that's what grafana is expecting | 14:45 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: WIP: Support parent job pause https://review.openstack.org/585389 | 14:46 |
| corvus | mordred: er, also, the tasks all seem to be "GET" "DELETE" "POST" | 14:46 |
| tobiash | corvus: that's a first proof of concept that passes a simple test case and doesn't break other things | 14:46 |
| corvus | tobiash: heh, that makes it sound ready to merge ;) | 14:47 |
| mordred | corvus: well crap | 14:47 |
| mordred | corvus: yes, that is where the regression is - lemme see if I can figure out a good way to fix it | 14:48 |
| corvus | mordred: ack, thx | 14:48 |
| tobiash | corvus: yeah, it needs to be polished a bit but I think I got the event flow changes very small | 14:48 |
| mordred | corvus: sdk produces the name "compute.DELETE.servers" instead of ComputeDeleteServers | 14:49 |
| mordred | corvus: and I totally forgot about that when workingon the sdk patch | 14:49 |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool master: Make statsd key look like the keys from shade https://review.openstack.org/585397 | 14:53 |
| mordred | corvus, Shrews, pabelanger: ^^ that should fix the current regression | 14:53 |
| *** nchakrab has quit IRC | 14:54 | |
| corvus | mordred: that will produce ComputeGetServersDetail for example? | 14:55 |
| corvus | (along with ComputeGetServers ) | 14:55 |
| mordred | corvus: yes - although I'm verifying the details one real quick | 14:56 |
| tobiash | mordred: that regression sounds like we have a gap in the tests | 14:56 |
| mordred | also ... | 14:56 |
| mordred | pabelanger: we're not logging 'ran task' anymore | 14:57 |
| corvus | tobiash: indeed, i was thinking we should port over zuul's statsd fixture | 14:57 |
| mordred | ++ | 14:57 |
| tobiash | ++ | 14:57 |
| mordred | corvus: yes. ComputeGetServersDetail and ComputeGetServers | 14:57 |
| corvus | erm. we do have a statsd fixture in nodepool. | 14:59 |
| corvus | test_launcher.py: self.assertReportedStat('nodepool.nodes.ready', '1|g') | 14:59 |
| corvus | test_launcher.py: self.assertReportedStat('nodepool.nodes.building', '0|g') | 14:59 |
| corvus | that's the only assertion we do. we probably just need to add some more. | 15:00 |
| mordred | pabelanger: oh - hrm. that gets logged in the sdk taskmanager - so should happen in the super call - but sdk taskmanager is using self._log and nodepool's is using self.log - so we're also missing that log ine to the wrong logger | 15:00 |
| * mordred works on more patches | 15:00 | |
| corvus | hrm, since the taskmanager is inside openstacksdk, the normal test path doesn't use that. so we're not exercising the task manager in the unit tests. | 15:08 |
| corvus | (and therefore, not generating these metrics) | 15:09 |
| mordred | corvus: oh, hrm | 15:09 |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool master: Emit 'Manager ran task' log lines https://review.openstack.org/585401 | 15:09 |
| mordred | that'll fix the other regression | 15:10 |
| clarkb | mordred: is the logging all that the parent class post_run_task does? | 15:13 |
| mordred | clarkb: yup | 15:13 |
| mordred | clarkb: if it ever grew something else it would be statsd-emitting | 15:14 |
| mordred | corvus: I think we'd have to pull in requests-mock and the mock out openstack things at the http layer instead of the FakeOpenStackCloud to be able to get a test of that specific thing - which would be a non-trivial change in the way we're testing things that I'm not sure is worth it ... | 15:14 |
| corvus | yeah. we could look at adding statsd to the functional tests | 15:15 |
| mordred | corvus: perhaps we should just make sure there are tests in sdk that test what task.name is and make sure it's clear that that's part of the API there | 15:15 |
| mordred | and also functional tests | 15:16 |
| clarkb | mordred: question on the child change but feel free to approve if you don't want to update that | 15:17 |
| mordred | clarkb: I could make a followup for that - or respin that patch | 15:18 |
| mordred | corvus: thoughts? | 15:18 |
| mordred | corvus: (re clarkb's suggestion in https://review.openstack.org/#/c/585401/1/nodepool/task_manager.py@81) | 15:19 |
| corvus | oh, hrm, yeah i think that might be best | 15:19 |
| mordred | respin then? | 15:19 |
| corvus | (i'm not too worried about backwards compat in log lines, it'd be more to keep nodepool internally consistent) | 15:20 |
| corvus | yeah i think so | 15:20 |
| mordred | ok. how about I squash both patches and update it with that then | 15:22 |
| corvus | wfm | 15:22 |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool master: Make statsd key look like the keys from shade https://review.openstack.org/585397 | 15:23 |
| mordred | corvus, clarkb: ^^ | 15:24 |
| *** nchakrab has joined #zuul | 15:30 | |
| *** nchakrab has quit IRC | 15:32 | |
| *** pcaruana has quit IRC | 15:32 | |
| Shrews | yay patches | 15:36 |
| Shrews | thx mordred | 15:36 |
| Shrews | mordred: found an issue | 15:38 |
| Shrews | after I +W'd. slow brain | 15:39 |
| mordred | Shrews: yay! | 15:41 |
| *** rlandy is now known as rlandy|brb | 15:42 | |
| mordred | Shrews: ooh - thanks! | 15:43 |
| Shrews | mordred: also, should that be ".".join() ? | 15:44 |
| clarkb | Shrews: no it is converting foo.Bar.baz to FooBarBaz | 15:44 |
| Shrews | ah | 15:44 |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool master: Make statsd key look like the keys from shade https://review.openstack.org/585397 | 15:44 |
| mordred | Shrews: I updated the comment to make the second thing clearer for reading | 15:45 |
| Shrews | danke | 15:45 |
| Shrews | +2 | 15:45 |
| *** nchakrab has joined #zuul | 16:13 | |
| *** nchakrab has quit IRC | 16:15 | |
| *** panda is now known as panda|off | 16:18 | |
| *** rlandy|brb is now known as rlandy | 16:28 | |
| clarkb | should we enqueue 585397 directly to the gate so that we can restart nodepool launchers before shrews calls it a day? | 16:32 |
| *** sshnaidm|ruck is now known as sshnaidm|bbl | 16:37 | |
| mordred | clarkb: yah - maybe so | 16:40 |
| Shrews | clarkb: i really shouldn't need to be around for the restart. the only other things going in are very minor | 16:41 |
| clarkb | ok | 16:41 |
| Shrews | logging changes and mordred's statsd fix | 16:41 |
| *** jlk has quit IRC | 16:48 | |
| *** jlk has joined #zuul | 16:48 | |
| *** jlk has quit IRC | 16:48 | |
| *** jlk has joined #zuul | 16:48 | |
| *** rlandy is now known as rlandy|afk | 16:49 | |
| *** myoung is now known as myoung|lunch | 16:58 | |
| *** bbayszcz_ has joined #zuul | 17:05 | |
| *** bbayszczak has quit IRC | 17:09 | |
| *** bbayszcz_ has quit IRC | 17:09 | |
| jtanner | mordred: http://bullshitipsum.com/ for your enjoyment | 17:18 |
| mordred | jtanner: ++ | 17:27 |
| rcarrillocruz | folks, can't remember if there was a talk about serving the nodepool images in some central location | 18:10 |
| rcarrillocruz | i.e. dib images | 18:10 |
| mordred | rcarrillocruz: we've talked about it before, but we haven't done it at all yet | 18:19 |
| rcarrillocruz | ok | 18:19 |
| rcarrillocruz | re: 'i want to repro this job that failed in the gate locally' | 18:19 |
| rcarrillocruz | i.e. there should be a way to mimic that by using the same image | 18:19 |
| mordred | yah' | 18:19 |
| rcarrillocruz | (providing the image allows ssh key injection) | 18:20 |
| mordred | so - part of the issue is that our current images are HUGE because they are also serving a git repo caches | 18:20 |
| rcarrillocruz | in some of ours it will not (yay ! ) | 18:20 |
| mordred | there some thoughts that came out of the container spec about some other ways to do that | 18:20 |
| clarkb | rcarrillocruz: mordred openstack infra is serving the images now | 18:24 |
| clarkb | https://nb01.openstack.org/images for example | 18:24 |
| clarkb | we just put apache in front of the dir that dib builds them in and filter out everything but qcow2 (bceause qcow2 are smallest) | 18:25 |
| rcarrillocruz | ah sweet | 18:25 |
| *** myoung|lunch is now known as myoung | 18:37 | |
| clarkb | mordred: http://logs.openstack.org/97/585397/3/check/tox-pep8/093d4e5/job-output.txt.gz#_2018-07-24_17_45_16_774059 fyi | 18:48 |
| mordred | clarkb: sigh | 18:55 |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool master: Make statsd key look like the keys from shade https://review.openstack.org/585397 | 18:57 |
| *** rlandy|afk is now known as rlandy | 19:03 | |
| *** sshnaidm|bbl is now known as sshnaidm|ruck | 20:11 | |
| *** GZ2k has joined #zuul | 20:15 | |
| *** GonZo2000_ has quit IRC | 20:18 | |
| *** samccann has quit IRC | 20:35 | |
| *** dkranz has quit IRC | 20:36 | |
| openstackgerrit | Merged openstack-infra/nodepool master: Make statsd key look like the keys from shade https://review.openstack.org/585397 | 20:44 |
| openstackgerrit | Ian Wienand proposed openstack-infra/zuul master: Add variables to project https://review.openstack.org/584230 | 20:48 |
| openstackgerrit | Ian Wienand proposed openstack-infra/zuul master: Use definition list for job status https://review.openstack.org/584780 | 20:48 |
| corvus | mordred: http://paste.openstack.org/show/726561/ | 22:00 |
| mordred | corvus: well then! | 22:08 |
| mordred | corvus: that is ludicrously fantastic | 22:08 |
| corvus | it's pretty groovy! i'm tidying up the boilerplate; should be done soon | 22:09 |
| clarkb | ianw: re https://review.openstack.org/#/c/547889/ did the desired zuul change get pushed? | 22:14 |
| *** GZ2k has quit IRC | 22:21 | |
| *** rlandy has quit IRC | 22:37 | |
| ianw | clarkb: i haven't got back to that yet | 22:40 |
| clarkb | ok not a rush, just noticed it when doing a pass over nodepool changes to see if there was anything else worth trying to get into the next release | 22:41 |
| clarkb | speaking of I think based on current nodepool behavior mordred's last commit that merged is taggable | 22:49 |
| corvus | cool. let's tag it tomorrow? | 22:50 |
| corvus | i'm happy with zuul's mimory usage in infra. should we tag that commit tomorrow as well? | 22:50 |
| clarkb | wfm | 22:50 |
| mordred | clarkb, clarkb: ++ | 22:55 |
| mordred | gah | 22:55 |
| mordred | corvus: ++ | 22:55 |
| clarkb | pabelanger: did you see my comment on https://review.openstack.org/#/c/584978/ ? | 22:55 |
| clarkb | pabelanger: ianw has a good comment on https://review.openstack.org/#/c/585068/ too | 22:56 |
| pabelanger | clarkb: haven't looked yet | 22:57 |
| clarkb | ianw: is the change in header levels in https://review.openstack.org/#/c/584779/1/doc/source/user/jobs.rst intentional? Change items et al were under Zuul Variables but are now at the same level? | 23:01 |
| clarkb | http://logs.openstack.org/79/584779/1/check/build-sphinx-docs/7c39d99/html/user/jobs.html#change-items for example | 23:02 |
| clarkb | otherwise that organziational change makes a lot of sense to me, but I think Change Items et all should be under Zuul Variables as they extend the base set defined in Zuul Variables | 23:03 |
| ianw | clarkb: hrm, i thought i just moved the zuul.* jobs into a section, which would have "raised" it a level | 23:08 |
| clarkb | ianw: Change Items and friends have the same underline string as Zuul Variables so are rendered at the same level now | 23:09 |
| ianw | clarkb: hrm, it should be | 23:11 |
| ianw | zuul | 23:11 |
| ianw | ---- | 23:11 |
| ianw | change | 23:11 |
| ianw | ~~~~ | 23:11 |
| clarkb | ianw: you know I may just be blind, I looked at the rendered output too and they looked very similar, but that is probably a style css thing more than rst | 23:12 |
| ianw | yeah, i htink if you look at http://logs.openstack.org/79/584779/1/check/build-sphinx-docs/7c39d99/html/user/jobs.html#id1 | 23:12 |
| ianw | you can see there "working directory" is *slightly* smaller than the "SSH Keys" below it | 23:13 |
| corvus | TOC in top left can confirm | 23:13 |
| ianw | yes good point, the TOC reflects what i thought i was doing, anyway :) | 23:14 |
| clarkb | yup, I'm just blind | 23:14 |
| ianw | i've never found an emacs mode/magic that can roll up sections. i'm sure it exists | 23:15 |
| ianw | i even installed atom at one point to see if it did that effectively, but i couldn't deal with it :) | 23:15 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul master: Fix zuul reporting build failure with only non-voting jobs https://review.openstack.org/584990 | 23:16 |
| clarkb | ianw: did it use all the memory? | 23:16 |
| pabelanger | corvus: clarkb: tobiash: we also should include 584990 for zuul release tomorrow | 23:17 |
| pabelanger | if you'd like to review | 23:17 |
| corvus | oh yes, we were blocking on that | 23:19 |
| pabelanger | clarkb: ianw: re: 585068 yah, I can change umask, will look at that for the morning | 23:20 |
| corvus | pabelanger: left a comment on 584990 | 23:21 |
| pabelanger | corvus: okay, I'll pick it up first thing in the morning, if that is okay | 23:22 |
| corvus | yep | 23:23 |
| clarkb | ianw: left a thought on https://review.openstack.org/#/c/584230/4 I don't know that it is correct but something to consider | 23:29 |
| ianw | clarkb: yeah, i'll take advice if people want to see it differently. after re-reading this morning a clarification that the variables are merged from templates into the project vars was what i just added in the /4 change | 23:35 |
| clarkb | ianw: probably a decision for jeblair, mostly if I was reading the config naively I would assume that those project vars only apply to the jobs defined under them, however it is a "project-template" | 23:36 |
| ianw | note the job can override them in it's var: section too, so it is "leaky" as it were :) | 23:37 |
| corvus | that's a really good question. i'm going to think on it overnight. | 23:41 |
| *** GonZo2000 has joined #zuul | 23:45 | |
| *** GonZo2000 has quit IRC | 23:45 | |
| *** GonZo2000 has joined #zuul | 23:45 | |
| *** yolanda_ has joined #zuul | 23:56 | |
| *** yolanda has quit IRC | 23:58 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!