Tuesday, 2018-03-20

*** rlandy is now known as rlandy|bbl00:02
jlkWoo, good response on the github3.py bug.00:13
* fungi cheers00:36
tristanCcorvus: ianw: the nodepool-web interface i proposed on zuul-web could be moved to nodepool when it grew the tooling to support it00:37
tristanCcorvus: ianw: in the meantime, i added it to zuul-web to get the ball moving00:37
tristanCor maybe i misunderstood what you said :)00:55
*** odyssey4me has quit IRC00:59
*** odyssey4me has joined #zuul00:59
openstackgerritTristan Cacqueray proposed openstack-infra/zuul-jobs master: DNM: negative test  https://review.openstack.org/52243801:16
*** myoung|bbl is now known as myoung02:39
*** yolanda has quit IRC02:44
*** yolanda has joined #zuul02:45
*** yolanda has quit IRC02:53
*** yolanda has joined #zuul02:54
*** rlandy|bbl is now known as rlandy03:22
*** rlandy has quit IRC03:24
*** bhavik1 has joined #zuul03:28
*** bhavik1 has quit IRC03:53
*** myoung is now known as myoung|afk04:28
*** CrayZee has joined #zuul05:49
*** flepied has quit IRC06:27
openstackgerritTristan Cacqueray proposed openstack-infra/nodepool master: Implement an Azure driver  https://review.openstack.org/55443207:07
*** chrnils has joined #zuul07:11
*** flepied has joined #zuul07:31
*** hashar has joined #zuul08:05
*** ekan has joined #zuul08:08
*** jpena|off is now known as jpena08:11
*** fbouliane has quit IRC08:31
*** fbouliane has joined #zuul08:33
*** electrofelix has joined #zuul08:38
openstackgerritTristan Cacqueray proposed openstack-infra/nodepool master: Implement a VMWare driver  https://review.openstack.org/55446309:03
openstackgerritTristan Cacqueray proposed openstack-infra/nodepool master: Implement an Azure driver  https://review.openstack.org/55443209:05
*** ekan has quit IRC10:35
*** sshnaidm|afk is now known as sshnaidm|sick10:35
*** jpena is now known as jpena|brb10:38
*** JasonCL has quit IRC10:44
*** JasonCL has joined #zuul10:45
*** JasonCL has quit IRC10:50
*** jpena|brb is now known as jpena11:37
*** odyssey4me has quit IRC12:11
*** odyssey4me has joined #zuul12:11
*** dkranz has quit IRC12:19
*** Wei_Liu has quit IRC12:27
*** Wei_Liu has joined #zuul12:28
*** openstackgerrit has quit IRC12:33
*** rlandy has joined #zuul12:33
*** jpena is now known as jpena|lunch12:44
*** myoung|afk is now known as myoung|rover12:55
*** myoung|rover is now known as myoung|rover|mtg13:00
*** JasonCL has joined #zuul13:04
*** JasonCL has quit IRC13:09
dmsimardtobiash: you're running executors on openshift right ?13:18
dmsimardtobiash: is that the actual (long lived) executors themselves ? or are you spawning a new executor for every job ?13:19
*** dkranz has joined #zuul13:26
*** Wei_Liu has quit IRC13:29
*** Wei_Liu has joined #zuul13:29
pabelangerI thought they were long lived, IIRC13:40
mordreddmsimard: long lived13:40
*** jpena|lunch is now known as jpena13:42
*** openstackgerrit has joined #zuul13:49
openstackgerritMonty Taylor proposed openstack-infra/zuul master: dashboard: add /{tenant}/job.html page to display job details  https://review.openstack.org/53554513:49
openstackgerritMonty Taylor proposed openstack-infra/zuul master: dashboard: add /{tenant}/projects.html web page  https://review.openstack.org/53787013:49
openstackgerritMonty Taylor proposed openstack-infra/zuul master: Fix indentation and renable the eslint rule  https://review.openstack.org/54567113:49
openstackgerritMonty Taylor proposed openstack-infra/zuul master: Shift html templates into components  https://review.openstack.org/55132713:49
openstackgerritMonty Taylor proposed openstack-infra/zuul master: Upgrade to webpack 4  https://review.openstack.org/55198713:49
openstackgerritMonty Taylor proposed openstack-infra/zuul master: Upgrade from angularjs (v1) to angular (v5)  https://review.openstack.org/55198913:49
*** JasonCL has joined #zuul13:50
*** dmellado has quit IRC13:55
*** JasonCL has quit IRC13:55
dmsimardpabelanger, mordred: tristanC and I were brainstorming about short lived executors the other day.. I feel like it might be a good fit. Has anyone brought that up before ?13:56
dmsimardIt would be a way to spin a precise version of Ansible and Zuul, for example13:56
pabelangeryah, SpamapS / jlk have talked about that before using k8s13:57
dmsimardIncreased isolation, etc.13:57
pabelangeryah13:57
pabelangerI know a few people are interested in that13:57
pabelangerit might have even been discussed in the bwrap spec for zuulv313:58
pabelangeron review.o.o13:58
pabelangerI think of the items is how do we deal with git repos, since there would be a cost / overhead in building them each time13:59
dmsimardIf we want to "offer" something like this for the OpenStack community (assuming we have the features we need to be able to do that, of course) it would also mean standing up a k8s/OpenShift cluster I guess. We could also make it available to the community for actual workloads ? e.g, kolla-k8s14:00
dmsimardpabelanger: building what each time ? The executors is just runtime code right ? There's no persistence like the scheduler ?14:01
mordreddmsimard: cache14:02
pabelangeryah, git cache14:02
mordreddmsimard: the hard part about dynamic executors is git cache with the mergers14:02
dmsimardI didn't realize there was a git cache on the executors.. it's for speeding up zuul merger ?14:03
mordreddmsimard: dynamic executors is certainly a thing that's been discussed - but it's a thing we need to dig in on the design of because there are some hard parts14:03
mordreddmsimard: yah - if we didn't have persistent cache on the executors zuul would completely not scale to openstack size14:04
dmsimardIs that an actual process ? Or is the cache simply "warmed up" by the fact that the executor just merges things all the time ?14:05
mordreddmsimard: the second14:06
mordreddmsimard: also - we've got a bunch of notes on zuul+k8s/openshift that need to be written up and shared out14:06
mordreddmsimard: let's circle up and get those shared first before you get too far down that path14:07
mordreddmsimard: since I think we're likley sharing some thoughts14:07
*** JasonCL has joined #zuul14:07
*** JasonCL has quit IRC14:07
dmsimardoh, I'm nowhere deep.. it's just an idea that I had and tristanC had also thought about it before14:07
mordreddmsimard: ++14:07
dmsimardIt's not like I have time to work on this or anything :(14:07
*** JasonCL has joined #zuul14:08
mordreddmsimard: :(14:08
*** JasonCL has quit IRC14:11
*** JasonCL has joined #zuul14:12
*** elyezer has joined #zuul14:14
*** myoung|rover|mtg is now known as myoung|rover14:23
tristanCwe could probably uses a configmap per build to store all the git data, but we likely still need to spin a pod per playbook to isolate secrets from untrusted playbooks14:23
dmsimardOr maybe there could be a central pod for git cache? I dunno. I'll wait for the write up :)14:26
openstackgerritMonty Taylor proposed openstack-infra/zuul-jobs master: Uninstall and reinstall siblings one at a time  https://review.openstack.org/55429714:27
dmsimarda read-only git cache volume shared on all pods?14:29
* dmsimard stops thinking about this14:29
tristanCdmsimard: let's not distract #zuul with non 3.0 release topic :-)14:30
openstackgerritMonty Taylor proposed openstack-infra/zuul master: Upgrade from angularjs (v1) to angular (v5)  https://review.openstack.org/55198914:37
mordredtristanC: I don't know if you've gotten a chance to look at ^^ - it's also a non-3.0 release topic, so no rush, but it's definitely a patch I'd want your review on before landing14:38
mordredtristanC: it'll require an update to how you build the js stack for softwarefactory (if, that is, the latest version of the patch actually works)14:39
tristanCmordred: i'll give it a try then. note that the zuul-webui package we use is built using the same command as zuul-build-dashboard14:43
mordredtristanC: awesome - so the main difference for softwarefactory is that you'll need to add a --define ZUUL_BASE_HREF="/zuul" to the build:dist command14:44
*** dmellado has joined #zuul14:54
*** electrofelix has quit IRC14:54
*** dmellado has quit IRC14:56
*** JasonCL has quit IRC14:59
*** JasonCL has joined #zuul15:00
*** dmellado has joined #zuul15:01
*** dmellado has quit IRC15:04
*** dmellado has joined #zuul15:05
*** flepied_ has joined #zuul15:12
*** JasonCL has quit IRC15:13
*** JasonCL has joined #zuul15:14
*** flepied has quit IRC15:15
*** dmellado has quit IRC15:16
*** JasonCL has quit IRC15:20
*** JasonCL has joined #zuul15:20
*** JasonCL has quit IRC15:28
*** JasonCL has joined #zuul15:28
*** JasonCL has quit IRC15:29
*** JasonCL has joined #zuul15:30
*** JasonCL has quit IRC15:32
*** JasonCL has joined #zuul15:32
*** JasonCL has quit IRC15:33
*** bhavik1 has joined #zuul15:34
*** JasonCL has joined #zuul15:34
*** JasonCL has quit IRC15:46
*** bhavik1 has quit IRC15:46
*** JasonCL has joined #zuul15:48
clarkbYesterday when debugging zuul's config loading I noticed this exception http://paste.openstack.org/show/706233/ I thought I could quickly debug it this morning but as I dig into it more I'm just more confused15:49
clarkbthe code explicitly checks if that value is None but seems to move ahead and lookup the None value anyways (based on the log message saying it is looking for a hold for None)15:50
clarkbI'm left thinking either that None valid is being stringified before the is not None check or we are running older scheduler code and this was somehow fixed along the way?15:50
clarkboh hey15:50
clarkbgit log -p says we are running older scheduler code15:51
clarkbexcept the traceback shows the check for is not None in it15:51
clarkbcorvus: Shrews ^ any ideas on that?15:51
corvusclarkb: i think that's fixed in master15:52
clarkbthere is 876d55fd5420944a5169f32d247984cc411d87c215:53
clarkbwhich added the is not None check15:53
corvusclarkb: i believe the traceback shows the code on the filesystem, not the code that's running15:53
clarkbcorvus: aha15:53
clarkbthat also explains why that line appears to be off by one15:53
corvusyup15:54
clarkbok I guess I'll want to restart zuul (and reboot the host) sometime soon to make that annoying exception go away (it makes it hard to filter for exceptions when there are many "expected" exceptions)15:54
*** JasonCL has quit IRC15:55
openstackgerritAndreas Jaeger proposed openstack-infra/zuul-jobs master: Uninstall and reinstall siblings one at a time  https://review.openstack.org/55429716:01
*** jamielennox has quit IRC16:01
clarkbnot to jinx it but we haven't had to restart zuul scheduler in a while16:03
*** myoung|rover is now known as myoung|rover|mtg16:05
*** JasonCL has joined #zuul16:09
*** JasonCL has quit IRC16:13
*** hashar is now known as hasharAway16:15
*** JasonCL has joined #zuul16:20
SpamapSdmsimard: pabelanger mordred yes yes yes to k8s for executors. And the git cache is actually pretty easy to solve with volumes or some other methods.16:22
SpamapSIt would be kind of weird to use bwrap inside k8s though... so that might be one valid use case for nullwrap.16:22
clarkbfwiw tobiash does run his executors under k8s16:23
clarkbso I'm not sure there is more that needs to be solved directly?16:23
clarkbSpamapS: in tobiash apparently a new openshift had to be spun up that allowed the executors to run in privileged containers16:23
clarkb*in tobiash's case16:23
clarkbso it is doable but yes clunky16:23
mordredclarkb: I think there are two takes - one is "run zuul executors in k8s as a human like tobiash is currently doing" - and "have zuul manage its executors in an elastic fashion"16:24
*** flepied_ has quit IRC16:24
clarkbI see, that to me is less about containers and k8s in particular and more make zuul scale up and down executors16:25
mordredclarkb: yes, that's right16:25
clarkbsince the containers and k8s part (mostly) works16:25
*** elyezer has quit IRC16:25
mordredclarkb: using k8s is one way in which zuul could implementing zuul-managed-elastic-executors, but is certainly not the only way16:25
*** JasonCL has quit IRC16:26
clarkbmordred: in particular nodepool is currently a required component and happens to scale up and down based on demand16:26
pabelangerauto-scaling of zuul services, would be so freaking awesome! :D16:26
pabelangerjust saying16:26
clarkb(and then you can point your nodepool at k8s or openstack or whatever16:26
clarkbbut I might be overthinking that16:26
mordredgiven that one can already, as clarkb points out, run executors in k8s successfully, I'd argue that we should focus attention more on k8s build resources and distributed scheduler - since those are existing pain points - before we tackle managed-elastic-executors16:26
*** JasonCL has joined #zuul16:27
pabelangerwfm16:27
corvusthese conversations will be much more productive if we can have them in a structured manner.  once we get the release out of the door, i'll write up a summary of all the discussions we've had about this so far in spec form, and we can iterate from there16:27
mordredcorvus: ++16:27
mordredclarkb, pabelanger: have a second to be a second set of eyes on a build failure real quick?16:28
clarkbmordred: sur16:29
mordredclarkb, pabelanger: nevermind- prepping the links to paste to you made me see the issue differently - so thanks!16:29
*** JasonCL has quit IRC16:31
openstackgerritMonty Taylor proposed openstack-infra/zuul master: Upgrade from angularjs (v1) to angular (v5)  https://review.openstack.org/55198916:33
*** jamielennox has joined #zuul16:38
*** elyezer has joined #zuul16:38
mordredtristanC: you have a patch somewhere to add swagger docs for the api yeah?16:38
*** JasonCL has joined #zuul16:39
mordredtristanC: found it - also, nice use of depends-on a gh pr :) https://review.openstack.org/#/c/535541/16:39
*** JasonCL has quit IRC16:43
*** JasonCL has joined #zuul16:45
*** JasonCL has quit IRC16:51
*** JasonCL has joined #zuul16:58
*** JasonCL has joined #zuul16:58
*** myoung|rover|mtg is now known as myoung|rover17:01
*** flepied_ has joined #zuul17:11
*** JasonCL has quit IRC17:26
*** JasonCL has joined #zuul17:27
*** JasonCL has quit IRC17:27
*** JasonCL has joined #zuul17:28
*** JasonCL has quit IRC17:31
tobiashI also like the idea of running an executor per job but have to second mordred that build resources and scale out scheduler are probably more important atm17:31
*** pwhalen has quit IRC17:34
*** JasonCL has joined #zuul17:34
*** pwhalen has joined #zuul17:38
*** pwhalen has joined #zuul17:38
jlkexecutor per job sounds cool, but what does it really buy us?17:39
openstackgerritMonty Taylor proposed openstack-infra/zuul master: Upgrade from angularjs (v1) to angular (v5)  https://review.openstack.org/55198917:41
tobiashright, let me rephrase, an ansible container per job17:42
tobiashis there currently some v3.0 stuff to review?17:43
*** jimi|ansible has quit IRC17:46
corvustobiash: i don't think so -- next things we're waiting on is the security patch, and i'm going to write some remote tests for the current log stream system17:47
corvus(and possibly a change from mordred related to js stuff if he determines we need it before 3.0)17:47
tobiashcorvus: I'll be working on this tomorrow, it's too late today for doing such kind of things ;)17:48
corvusyeah, don't patch vulnerabilities when tired :)17:48
*** JasonCL has quit IRC17:53
SpamapSjlk: since the cache for an executor is critical to reducing job latency and repeated work, not much. But if one can leverage a merger's cache, it makes the executor more elastic.17:54
SpamapSDef a good thing to get into that spec. :)17:54
jlkSo, this exists https://github.com/pricing/business-hosted it's GitHub hosted for business. An upgrade for an organization, which adds SAML sign on and stuff. I'm not sure if it changes anything with the API, but I should probably dig into it a bit more.17:54
SpamapSFor me, at small scale, it doesn't help at all. :)17:54
jlkSpamapS: but what is the benefit of the elasticity?17:55
SpamapSjlk: public cloud cost17:55
SpamapSif you are having a big day and running a lot of concurrent jobs.. spend money.. if not.. spend nothing.17:55
SpamapScan do it with a nodepool of executors too ;)17:55
jlkhow far are we from that now though? What does it take to spin up an executor?17:56
SpamapSNot much17:56
jlkI thought you could spin one up with out having to "notify" or touch other parts of the system, like it just plugs into the pool17:56
SpamapSespecially if you have containerized it and have a kubernetes17:56
jlkand if it drops off, same, that's just one less worker17:56
jlkyeah, or at least have a good "boot this image that self-configures" system17:57
SpamapSanother thing though, for jobs that can run without VM17:57
SpamapSspin single use executor.. run job local.. executor drops.17:57
jlkyeah... I'm somewhat less comfortable with those kind of jobs17:57
SpamapSunittest jobs would fall in that category17:57
jlkexecution on MY infra vs execution on the runtime capacity I provide17:57
jlkI wouldn't necessarily mix the pool I use to RUN zuul with the pool I use to run zuul JOBS17:58
SpamapSagreed17:58
jlkzuul jobs may run in somebody else's infra17:58
SpamapSsingle use executors would be runtime17:58
jlknot mine17:58
SpamapSbut they might just run inside a kubernetes17:58
jlksortof? zuul executors have credentials17:58
jlkwhich means I may want them on MY infra17:58
jlkand away from the infra where rando code runs17:59
SpamapSYeah, I am with you.17:59
SpamapSJust falling into devil's advocate18:00
SpamapSThe reality is, the thing we need a spec for, and that I would like, is a way to run a job without a VM and without making it trusted.18:00
SpamapSI don't much like the idea of spinning up sshd's in containers, but it does sort of achieve that.18:00
clarkbSpamapS: in theory something like nodepool + k8s and ansible k8s connector does that for you right?18:01
jlkSpamapS: right, I don't want ssh in the container either18:03
jlkI think the container shouldn't expose any ports, just boot. We should use k8s API to cause things to happen inside the container.18:03
SpamapSclarkb: nodepool + k8s?18:03
clarkbSpamapS: nodepool assigning compute resources out of a k8s cluster18:04
openstackgerritFabien Boucher proposed openstack-infra/zuul master: Add bootstrap alert message when conf loading errors exists  https://review.openstack.org/55465218:04
jlkSpamapS: we met at PTG to talk through what it would look like to have Zuul + Nodepool work with k8s as a capacity source18:04
SpamapSclarkb: eh, that's not really how k8s works. k8s is pretty much a nodepool.18:04
jlkSomebody has the notes on it online I think.18:04
jlkNodepool would ask for a pod of certain types.18:04
clarkbjlk: corvus mentioned earlier that there would be a writeup, I don't thik that has been done yet18:04
corvusyep.  i'll do it soon.18:05
corvusor if someone wants to write those remote tests for log streaming, i could write up the spec now.  :)18:05
jlkhahah18:05
SpamapSOne thought I have is to just have jobs that have a k8s-object instead of a nodeset.18:05
jlkI suggested a pod, with at least 2 containers18:05
*** myoung|rover is now known as myoung|biab18:05
jlkone for git content (sidecar) and one for the job itself18:05
jlkthe sidecar so that we can push up the git content and the job container would have access to the source code18:06
SpamapSTeach either nodepool or zuul-scheduler to talk to k8s. Hand jobs the kubeconfig they need to access the deployment they asked for.18:06
jlkcorrect, nodepool would talk to k8s18:06
jlkBut we also talked about two different use cases18:06
SpamapSyeah we're thinking similarly.18:06
jlk1) a job wants to natively talk k8s18:06
jlkand 2) a job just wants to run something and doesn't care where18:06
SpamapSI care a lot more about 2)18:07
jlkhanding jobs direct access to k8s is... dangerous18:07
SpamapSbut really 2) is enabled by a parent that is 1)18:07
*** JasonCL has joined #zuul18:07
jlkk8s isn't well set up for multi-tenant18:07
SpamapSyeah, untrusted would only be able to do a k8s_exec or something like that.18:08
jlkSpamapS: well, we can teach Zuul's ansible how to run tasks on a "host" that is a k8s pod/container. That gets you to 2 without necessarily exposing 118:08
SpamapSgah18:08
SpamapSthis rat hole18:08
SpamapSit's SO NICE18:08
SpamapSlike I want to stay here18:08
SpamapSbut18:08
SpamapSthe rats18:08
mordredyah18:08
* SpamapS has to go back to work ;)18:08
clarkband if you use gke every pod has root!18:08
jlkthat's totally fine..18:10
jlkit's not like somebody could write a job that spawns a outbound connection to create a reverse proxy into the container, granting shell access to the nefarious user...18:10
clarkbjlk: clsuter root/admin I mean18:10
jlkright18:11
clarkbthe default in gke for ease of use is that all pods have cluster admin aiui and all gce VMs (that run gke containers) have account admin in the cloud18:11
jlkI guess you could just write jobs to poke at the k8s API from within the container18:11
clarkbjlk: yes that18:11
clarkband potentially to your cloud account18:11
jlkthisisfine.jpg18:12
*** JasonCL has quit IRC18:12
*** JasonCL has joined #zuul18:13
fungidunno if anybody noticed yet (i'm not caught up on scrollback) but as of today zuul is a free agent and officially no longer under openstack tc governance18:15
*** JasonCL has quit IRC18:17
*** JasonCL has joined #zuul18:17
fungithere is no openstack, only zuul18:17
jlkfreebird.mp318:17
SpamapSfungi: oh my18:18
*** jpena is now known as jpena|away18:18
*** JasonCL has quit IRC18:19
*** JasonCL has joined #zuul18:20
*** JasonCL_ has joined #zuul18:22
*** harlowja has joined #zuul18:23
*** JasonCL has quit IRC18:25
*** JasonCL_ has quit IRC18:26
*** chrnils has quit IRC18:51
*** JasonCL has joined #zuul18:58
*** JasonCL has quit IRC18:59
*** JasonCL has joined #zuul19:01
*** JasonCL has quit IRC19:03
*** JasonCL has joined #zuul19:03
openstackgerritFabien Boucher proposed openstack-infra/zuul master: Add a loading_errors info to {tenant}/info endpoint  https://review.openstack.org/55387319:04
*** JasonCL has quit IRC19:07
*** myoung|biab is now known as myoung|rover19:14
*** JasonCL has joined #zuul19:43
*** JasonCL has quit IRC19:47
*** smyers has quit IRC20:02
*** smyers has joined #zuul20:03
*** smyers has quit IRC20:18
*** smyers has joined #zuul20:19
*** JasonCL has joined #zuul20:21
*** JasonCL has quit IRC20:30
*** openstackgerrit has quit IRC20:33
*** JasonCL has joined #zuul20:55
*** dmellado has joined #zuul20:56
*** dmellado has quit IRC20:58
*** JasonCL has quit IRC21:00
*** JasonCL has joined #zuul21:00
*** dmellado has joined #zuul21:02
*** JasonCL has quit IRC21:02
*** JasonCL has joined #zuul21:04
*** dmellado has quit IRC21:08
*** dmellado has joined #zuul21:12
*** dmellado has quit IRC21:13
*** myoung|rover is now known as myoung|off21:13
*** myoung|off is now known as myoung|bbl21:13
*** dkranz has quit IRC21:14
*** dmellado has joined #zuul21:15
*** myoung|bbl is now known as myoung|rover21:16
*** JasonCL has quit IRC21:16
*** JasonCL has joined #zuul21:16
*** dmellado has quit IRC21:32
*** ssbarnea has quit IRC21:46
*** myoung|rover is now known as myoung|afk21:46
*** openstackgerrit has joined #zuul21:48
openstackgerritJames E. Blair proposed openstack-infra/zuul master: Add zuul-stream remote tests  https://review.openstack.org/55471421:48
*** ssbarnea has joined #zuul21:52
*** dmellado has joined #zuul22:04
*** CrayZee has quit IRC22:41
*** harlowja has quit IRC22:57
*** hasharAway has quit IRC23:01
*** rlandy is now known as rlandy|bbl23:03
*** jpena|away is now known as jpena|off23:21
openstackgerritJames E. Blair proposed openstack-infra/zuul master: Add zuul-stream remote tests  https://review.openstack.org/55471423:23
*** harlowja has joined #zuul23:49

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!