| dmsimard | Asking the opinion of pros on openstack-dev about how the new ARA API should behave, I don't think there's an actual [ara] tag on the ML filters so just in case anyone here has advice: http://lists.openstack.org/pipermail/openstack-dev/2017-August/120855.html | 04:09 |
|---|---|---|
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul feature/zuulv3: Try to early terminate streaming on ansible errors https://review.openstack.org/491027 | 05:16 |
| tobiash | mordred, jeblair: tried ^^^ in my local deployment and the delay in case of task parse errors is gone: http://paste.openstack.org/show/617875/ | 06:17 |
| tobiash | (note the missing streamer didn't terminate string) | 06:17 |
| *** amoralej|off is now known as amoralej | 07:26 | |
| *** TheJulia has quit IRC | 08:47 | |
| *** robled has quit IRC | 08:47 | |
| *** Shrews has quit IRC | 08:47 | |
| *** jkilpatr has quit IRC | 08:47 | |
| *** smyers has quit IRC | 08:47 | |
| *** jamielennox has quit IRC | 08:47 | |
| *** amoralej has quit IRC | 08:47 | |
| *** mmedvede has quit IRC | 08:47 | |
| *** olaph has quit IRC | 08:47 | |
| *** dmsimard has quit IRC | 08:47 | |
| *** EmilienM has quit IRC | 08:47 | |
| *** zigo has quit IRC | 08:47 | |
| *** yolanda has quit IRC | 08:47 | |
| *** tristanC has quit IRC | 08:47 | |
| *** toabctl has quit IRC | 08:47 | |
| *** jeblair has quit IRC | 08:47 | |
| *** eventingmonkey has quit IRC | 08:47 | |
| *** jlk has quit IRC | 08:47 | |
| *** SotK has quit IRC | 08:47 | |
| *** fbouliane has quit IRC | 08:47 | |
| *** adam_g has quit IRC | 08:47 | |
| *** mattclay has quit IRC | 08:47 | |
| *** mnaser has quit IRC | 08:47 | |
| *** pbrobinson has quit IRC | 08:47 | |
| *** persia has quit IRC | 08:47 | |
| *** _ari_ has quit IRC | 08:47 | |
| *** robcresswell has quit IRC | 08:47 | |
| *** patrickeast has quit IRC | 08:47 | |
| *** zaro has quit IRC | 08:47 | |
| *** tflink has quit IRC | 08:47 | |
| *** mgagne has quit IRC | 08:47 | |
| *** fungi has quit IRC | 08:47 | |
| *** leifmadsen has quit IRC | 08:47 | |
| *** kklimonda has quit IRC | 08:47 | |
| *** bstinson has quit IRC | 08:47 | |
| *** clarkb has quit IRC | 08:47 | |
| *** jhesketh has quit IRC | 08:47 | |
| *** tobiash has quit IRC | 08:47 | |
| *** pleia2 has quit IRC | 08:47 | |
| *** timrc has quit IRC | 08:47 | |
| *** lennyb has quit IRC | 08:47 | |
| *** mordred has quit IRC | 08:47 | |
| *** morgan has quit IRC | 08:47 | |
| *** xinliang has quit IRC | 08:47 | |
| *** openstackgerrit has quit IRC | 08:47 | |
| *** cinerama has quit IRC | 08:47 | |
| *** jesusaur has quit IRC | 08:47 | |
| *** rcarrillocruz has quit IRC | 08:47 | |
| *** ajafo has quit IRC | 08:47 | |
| *** SpamapS has quit IRC | 08:47 | |
| *** ChanServ has quit IRC | 08:47 | |
| *** rfolco has quit IRC | 08:47 | |
| *** gothicmindfood has quit IRC | 08:47 | |
| *** openstack has joined #zuul | 13:56 | |
| jeblair | tobiash, mordred: that all sounds good, but why put the password on the executor and not also pass it through nodepool? | 14:16 |
| tobiash | jeblair: the concern was about putting the pw into zookeeper | 14:17 |
| tobiash | jeblair: I would also be ok with putting that to zookeeper | 14:17 |
| tobiash | (where the connection can hopefully be encrypted) | 14:18 |
| jeblair | tobiash, mordred: oh, yeah, zookeeper has support for encryption and authentication; and if we're going to use it for everything in zuulv4, we'll have to get comfortable with privileged info in there, so now seems like a good time to make it secure and use it. | 14:19 |
| tobiash | jeblair: then that's the way to go (maybe also in future for the master node key?) | 14:20 |
| jeblair | tobiash: maybe so; the idea of passing everything through nodepool is growing on me :) | 14:20 |
| tobiash | :) | 14:21 |
| mordred | jeblair, tobiash: ok - I'm good with that | 14:21 |
| *** amoralej|lunch is now known as amoralej | 14:27 | |
| *** jkilpatr has quit IRC | 14:33 | |
| *** jkilpatr has joined #zuul | 14:34 | |
| *** jkilpatr has quit IRC | 15:03 | |
| tobiash | jeblair, mordred: I'll look into this after my vacation (whicj is the week before the ptg) | 15:20 |
| jeblair | tobiash: are you leaving for vacation now, or starting next week? | 15:21 |
| tobiash | jeblair: planned was next week but I have to reduce overtime starting with now | 15:27 |
| jeblair | tobiash: okay. enjoy your hiking! | 15:33 |
| tobiash | jeblair: thanks :) | 15:33 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Try to early terminate streaming on ansible errors https://review.openstack.org/491027 | 15:36 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Remove base job https://review.openstack.org/491907 | 15:56 |
| jeblair | pabelanger: when you have a minute, if you can take a look at https://review.openstack.org/491635 and see if the examples look okay, we can start to spiff up the job docs a bit i think. | 16:22 |
| pabelanger | jeblair: ya, sorry. I did start reviewing it yesterday. let me finish | 16:23 |
| pabelanger | +2 | 16:25 |
| mordred | jeblair: +A | 16:39 |
| openstackgerrit | Merged openstack-infra/zuul-sphinx master: Import directives/roles from zuul https://review.openstack.org/491635 | 16:41 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-sphinx master: Mark zuul-sphinx as supporting python3 https://review.openstack.org/492203 | 16:57 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-sphinx master: Add CONTRIBUTING file to repo and to docs https://review.openstack.org/492204 | 16:57 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-sphinx master: Add dist dir to .gitignore https://review.openstack.org/492205 | 16:57 |
| *** jkilpatr has joined #zuul | 16:59 | |
| mordred | jeblair: ^^ those are tiny little things from reviewing the other thing | 17:02 |
| pabelanger | mordred: did you want add https://review.openstack.org/491093 to your review pipeline. Our tarball publisher job | 17:28 |
| *** jkilpatr has quit IRC | 17:38 | |
| *** jkilpatr has joined #zuul | 17:42 | |
| jeblair | mordred: thanks! | 17:50 |
| *** electrofelix has quit IRC | 17:57 | |
| pabelanger | mordred: how did we want to handle jenkins/scripts/pypi-extract-name.py is that something we still need or should we expect the job to setup the values | 18:04 |
| pabelanger | mordred: left -1 on 491915 syntax error | 18:06 |
| jeblair | pabelanger: that's not a syntax error, that's a nitpicky pep8 whitespace style "error" :) | 18:09 |
| pabelanger | ah, yes | 18:14 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: Allow launcher to stop quicker when asked https://review.openstack.org/492219 | 18:26 |
| *** xinliang has quit IRC | 18:26 | |
| jeblair | pabelanger: was there a particular reason .ansible/fact-cache was created in $jobdir rather than $jobdir/work ? | 18:40 |
| pabelanger | jeblair: I think we wanted to to be the same as local_tmp and remote_tmp, for directory structure. I also think we didn't want untrusted jobs to have access to it either, since it was possible to leak executor facts | 18:43 |
| *** morgan is now known as kmalloc | 18:46 | |
| jeblair | pabelanger: okay, that helps, thanks | 18:47 |
| kklimonda | jeblair: looks like I'm already here - let me give you a quick introduction into what I'm trying to achieve. | 18:53 |
| kklimonda | erm, fungi ^ | 18:53 |
| kklimonda | sorry, nick colors have confused me | 18:54 |
| fungi | no problem ;) | 18:55 |
| jeblair | i mix us up all the time too :) | 18:55 |
| kklimonda | fungi: I'm working on OpenContrail CI which is currently based on OpenStack CI from the past, with zuul 2.0. We're planning on catching up with the current OpenStack CI stack, upgrading zuul to 2.5 (and then to 3.0 when you deem it stable and production-ready). | 18:55 |
| kklimonda | however, at the same time we find ourselves on a deadline to integrate Windows builders into the live CI, to merge test and release windows-related features. | 18:56 |
| fungi | oh, okay, so not actually trying to port zuul services to run under windows, just trying to get zuul to run jobs on windows servers | 18:57 |
| fungi | that's probably far easier | 18:57 |
| jeblair | kklimonda: zuul 2.5 still supports jenkins, so you can upgrade to the latest 2.x and still use jenkins to run jobs. that might be the quickest way forward. | 18:57 |
| kklimonda | jeblair: sigh, removal of Jenkins during 2.5 upgrade has been made a priority. | 18:58 |
| jeblair | kklimonda: which removal of jenkins? | 18:58 |
| kklimonda | it doesn't scale for us from what I've heard | 18:58 |
| jeblair | kklimonda: oh, so you have an internal constraint not to use jenkins? | 18:58 |
| kklimonda | yes, that's been a main driver for 2.5 release from what I understand - there's been issues with scaling it to 90+ slaves | 18:59 |
| fungi | kklimonda: depending on how you need to scale, we ran zuul with 8 (i think it was?) jenkins masters launching jobs on over 100 slaves per master for a while | 18:59 |
| fungi | zuul 2.x came about specifically because we needed to scale past a single jenkins master | 18:59 |
| pabelanger | ah, weekly jenkins master reboots. How I don't miss that :) | 19:00 |
| fungi | which was the point where we added the gearman coordination between zuul's scheduler and multiple jenkins masters | 19:00 |
| kklimonda | hmm, I think the powers-that-be will not reconsider jenkins at this point - we want to have a state of the art CI infrastructure based on what you guys are doing at the moment. | 19:01 |
| jeblair | that makes sense. i can see why you wouldn't want to invest more in setting up multiple jenkins masters if you aren't already doing that and want to converge on modern zuul | 19:02 |
| kklimonda | yes, that's a good way to put it :) | 19:03 |
| jeblair | anyway, just wanted to make sure that we covered that option, because it's probably the least risky and the fastest path to working jobs | 19:03 |
| jeblair | kklimonda: if you want to move to ansible-based zuul soon, probably the best bet would be to start working with zuul v3. it's not ready for widesrpead use yet (we still make breaking changes occasionally), but its use of ansible is more flexible that zuul v2.5, so whatever you would need to do to work with windows is going to be easier to do in v3 than v2. | 19:03 |
| jeblair | kklimonda: and we're so focused on developing v3 where we know we want to have support for windows, etc, that we don't really have time to review large changes like that for v2.5 | 19:04 |
| kklimonda | what's the timeline for v3? | 19:04 |
| jeblair | kklimonda: we're planning on switching openstack-infra to it in the 2nd week of september. we still have some more polishing after that before we do a real 3.0 release and start recommending openstack third-party ci systems use it, so that would come a little later. | 19:05 |
| jeblair | kklimonda: we're running it in a test mode for openstack now, so it does work. and there's at least one other user running it experimentally. | 19:06 |
| kklimonda | ha, we need to have a stop-gap measure (windows integration + all the existing reviews) by the end of august. | 19:06 |
| jeblair | kklimonda: if you're interested in that, you can expect occasional breaking changes, having to ask a lot of questions in here, and incomplete documentation (especially around setting up an environment). | 19:07 |
| jeblair | kklimonda: on the positive side, much of the documentation on how to actually use it is complete, and it does generally work reliably at this point. | 19:07 |
| jeblair | kklimonda: i just noticed that our irc logging bot was on the wrong side of a netsplit this morning, and missed our conversation about windows in this channel, so i've pasted it here: http://paste.openstack.org/show/617974/ | 19:08 |
| kklimonda | I've been thinking now to hack windows support into zuul launcher for v2.5, and then after we get out of the fire fighting mode focus on v3 - hopefully by the end of the year we'd be able to deploy a new infrastructure based on it. I'm worried that if we now start the discussion on v2.5 vs v3 we'll not do anything (in hindsight I should have that conversation with you guys three weeks back, but if wishes were pigs..) | 19:09 |
| jeblair | kklimonda: that's probably worth reading to see what would need to be done to actually run windows jobs (and unfortunately, it looks like our volunteer for implementing that won't be able to do so until september) | 19:09 |
| kklimonda | thanks, I'll read it in a moment | 19:09 |
| kklimonda | windows support is definitely something we need, and Juniper was fine with us contributing changes back to the upstream | 19:10 |
| kklimonda | so if you need help implementing that, either me or someone from my team will probably be happy to help/review/code some of it | 19:10 |
| jeblair | kklimonda: why don't you read through that conversation and let me know what you think -- it's possible you may be able to get going with v3 fairly quickly (it seems like most of the work would actually be in nodepool) | 19:12 |
| jeblair | kklimonda: i have to break for lunch; i'll be back in a bit | 19:12 |
| pabelanger | I know windows for ansible is pretty hot ATM. IIRC, ansible even started a windows SIG | 19:18 |
| mordred | pabelanger: yah- and I'm pretty sure it's important to tobiash too | 19:21 |
| mordred | kklimonda: from what I understand from the morning's conversation with tobiash that jeblair pasted above, I don't think it'll be super hard to add windows support to v3 - although it will be at least some work - I think it will be much harder to add to v2.5, and that combined with 2.5 being a develpment dead-end, I'd certainly recommend working with tobiash | 19:22 |
| mordred | kklimonda: there's a few things that need to be added to the core of zuul and nodepool - and then after that point it'll really just be about writing some additional ansible to make windows-aware versions of some of the functions handled by the base job | 19:24 |
| kklimonda | jeblair: thanks for the log, it basically matches what I've read about today - plaintext is the simplest, and doesn't require any additional setup (AD, or Kerberos). Pure Kerberos is an interesting idea, but a) it depends on having correct time and b) at least partially depends on DNS (I think RevDNS can be skipped if you are careful, but I don't think A records can). It is probably the *correct* way to do that, but will require some | 19:28 |
| kklimonda | careful thoughts and additional operational knowledge. | 19:28 |
| kklimonda | mordred: yes, I'll be recommending that we start working on v3 as soon as possible, but realistically speaking we need a stop-gap measure for v2.5 - at this point I'm just hoping to put as little work as possible into it, just to get it to work and avoid blocking a large merge of windows-specific features. | 19:32 |
| * tobiash saw his name on his android irc client | 19:34 | |
| *** amoralej is now known as amoralej|off | 19:34 | |
| kklimonda | hi :) | 19:35 |
| tobiash | mordred: I'm trying to avoid windows as much as possible but some stuff still need windows here... ;) | 19:35 |
| kklimonda | tell me about it :) | 19:35 |
| kklimonda | we have to support our existing Jenkins-based CI for windows folks, and every day is an adventure. | 19:36 |
| tobiash | kklimonda: currently I'm running a productive zuulv2 (single jenkins) setup with linux and windows slaves and an experimental v3 setup where I also need to support windows soon | 19:38 |
| tobiash | kklimonda: soon means starting on support for this at the beginning of september (I'll be back from vacation on 9/4) | 19:39 |
| tobiash | s/starting on support/starting on working to support/ | 19:39 |
| tobiash | kklimonda: my vision is to build windows images in diskimage builder (probably using ansible and openstack as tools as locally building a windows image is not that great) such that nodepool can rebuild these too | 19:41 |
| kklimonda | tobiash: didn't know diskimage builder supports windows | 19:43 |
| tobiash | kklimonda: well this would be a hack using a custom base element which triggers ansible to spawn a new vm, configure it, shut it down and downloading the image from openstack | 19:44 |
| tobiash | kklimonda: but important is that it *looks* like diskimage builder to nodepool ;) | 19:45 |
| tobiash | kklimonda: together with the changes mentioned in the conversation jeblair linked I think this could become a pretty smooth integration of windows support in zuulv3 | 19:45 |
| kklimonda | ah, I like it already - a way to create a base image, potentially with all the dependencies to speed up builds even more is something I've been thinking about | 19:45 |
| openstackgerrit | Merged openstack-infra/zuul-sphinx master: Mark zuul-sphinx as supporting python3 https://review.openstack.org/492203 | 19:46 |
| tobiash | kklimonda: additionally on config side there would be a separate base-windows job which pushes the repos using the windows modules | 19:46 |
| tobiash | mordred, jeblair: what strikes me now as I write this is that console streaming might become a challenge with windows nodes | 19:48 |
| kklimonda | mhm, not sure how well it will work for us - current pipeline uses android repo to pull all repositories into a correct project structure. | 19:48 |
| kklimonda | heh, console streaming is something I've been looking at today | 19:48 |
| tobiash | kklimonda: with windows? | 19:49 |
| kklimonda | yes | 19:49 |
| tobiash | cool :) | 19:49 |
| tobiash | kklimonda: you're using the android repo tool? | 19:51 |
| tobiash | kklimonda: with v3 you also can tell zuul to clone several repos | 19:52 |
| tobiash | kklimonda: but currently it has no feature yet to define the structure of this like the clonemap in zuul-cloner | 19:53 |
| kklimonda | I've been thinking of writing a small powershell script/app that does the job of a daemon spawned by zuul_console - I've looked into Start-Job (will probably not work given that ansible does new connection per task) and then creating a bare bones service in PowerShell - I've found an article about it, but ran out of time to actually test and see if it works. That would probably (not a windows expert here) require us to run it as | 19:53 |
| kklimonda | administrator, which is fine for our usecase, but may not work for everybody. | 19:53 |
| kklimonda | tobiash: yes, our current CI/build pipeline is based around android repo tool - developers also use it to create local environmentsâ„ | 19:54 |
| kklimonda | if zuul can be extended with project structure, we would have to discuss if we want to go this way - we'd probably still need repo for developers, especially if my idea of splitting the project into even more repositories gets some ground. | 19:57 |
| tobiash | kklimonda: maybe it is sufficient to write a small wrapper (as part of the job, maybe even a reusable ansible role) which links/moves the zuul created repo-layout into the repo defined layout (and eventually clone missing repos after that) | 19:57 |
| jeblair | kklimonda: in v3, zuul creates the proposed future state of all the repos involved in a change. the idea is then to push them onto the test nodes for the test. but you can further manipulate the repos if needed before a job (like if you need them in a different structure) ... and i was about to suggest what tobiash just said :) | 19:58 |
| tobiash | kklimonda: this wouldn't need a change to zuul and devs and ci system still can use the repo tool | 19:58 |
| kklimonda | hmm, that could work | 19:58 |
| tobiash | we also use the repo tool heavily and until 2 minutes ago I didn't have a good solution to that with v3 :) | 19:58 |
| tobiash | now I think that an ansible role which rearranges the zuul-prepared repos would be cool for that | 19:59 |
| jeblair | tobiash: ++ | 20:01 |
| kklimonda | thanks guys, I'll start slowly digging into v3 architecture, to see if I can wrap my head around all the moving parts. tobiash: if you have windows-related discussion, ping me on the channel - I'm in CET timezone, and would love to be part of it. | 20:02 |
| tobiash | jeblair: do you think such a role makes sense for zuul-jobs or is this too specialized? | 20:02 |
| kklimonda | now I need to get some sleep, it's been a long day - good night :) | 20:02 |
| tobiash | kklimonda: yay, almost same time zone :) | 20:02 |
| tobiash | ahem exactly the same time zone... | 20:03 |
| jeblair | kklimonda: take a look at the docs and let me know what helps and what doesn't, please: https://docs.openstack.org/infra/zuul/feature/zuulv3/ | 20:03 |
| jeblair | kklimonda: thanks, and good night! :) | 20:03 |
| jeblair | tobiash: probably depends on how it rearranges it? repo has a config file that specifies the arrangement, right? that probably does make sense in zuul-jobs, in that case (as it's potentially usable by anyone using repo) | 20:04 |
| tobiash | jeblair: yes, repo has a file named manifest.xml (urg, xml...) which defines the arrangement | 20:05 |
| tobiash | jeblair: so that could be generic (with variables like src, dest roots and copy/move/link) | 20:05 |
| jeblair | tobiash: sounds like a good fit i think | 20:06 |
| tobiash | :) | 20:06 |
| tobiash | that also will be something for september | 20:06 |
| tobiash | jeblair: I'm also heading to bed now. Maybe I'll read the scrollback from time to time during vacation... | 20:08 |
| tobiash | cya | 20:09 |
| jeblair | tobiash: goodnight! | 20:10 |
| *** dkranz_ has quit IRC | 20:28 | |
| mordred | pabelanger: I think pypi_extract_name can be a nice little python module in that role - I don't really think there's much need to make people put that info into the job when it's already in the git content, yeah? | 20:29 |
| pabelanger | mordred: ya, I guess it all depends on if we want to push git repos to the upload-to-pypi job | 20:30 |
| pabelanger | I mean, we do by default today | 20:30 |
| pabelanger | okay, I'll do that first, see how it works | 20:31 |
| mordred | sweet! lemme know if I can help - but it seems like you've got it well in hand | 20:37 |
| *** jkilpatr has quit IRC | 21:01 | |
| jeblair | pabelanger: question for you on 491093 | 21:35 |
| mordred | pabelanger, jeblair: +2 from me - but I agree with jeblair's question - and I also left a comment that we may want to consider | 21:37 |
| jeblair | pabelanger: what was the reason we couldn't use secrets for the tarball job? | 21:37 |
| jeblair | mordred: ^ maybe you recall? | 21:38 |
| pabelanger | jeblair: we only have secrets enabled on release pipeline ATM, so needed to tag to use it | 21:38 |
| pabelanger | to just created a dummy account for now to confirm it worked | 21:38 |
| pabelanger | but that is the next step, secrets | 21:39 |
| jeblair | pabelanger: oh, so there wasn't another blocker for that? (we can enable secrets on the post pipeline for branch tarballs) | 21:41 |
| jeblair | pabelanger: we wouldn't be able to use the inheritance structure you have now -- we'd have to disallow inheritance of the secret and explicitly use it in both the tarball and branch-tarball jobs. | 21:42 |
| jeblair | pabelanger: and any other jobs which uploaded to tarballs | 21:42 |
| jeblair | pabelanger: maybe that was the reason? because we want to let any project upload content to tarballs.o.o without us having to make a new job for it? | 21:42 |
| pabelanger | jeblair: oh, you mean the SSH key? | 21:42 |
| jeblair | pabelanger: that's the only secret the tarball job would use, right? | 21:42 |
| pabelanger | Ya, I thought you were talking about pypi. Right, we could add it to a secret and create post pipeline, I would be good with that. | 21:43 |
| jeblair | pabelanger: well, i mean, see the stuff i just wrote about why that wouldn't work. :) | 21:44 |
| jeblair | i thought we chose not to use secrets for tarballs for a reason and i need to understand that reason | 21:44 |
| pabelanger | right, cross pipelines was the main reason I didn't add it as a secret for now. Since I didn't think we wanted to allow the secret in check | 21:44 |
| jeblair | pabelanger: did you put tarballs in check? | 21:45 |
| pabelanger | Ya, trusted job in check for testing ATM | 21:45 |
| pabelanger | on sandbox project | 21:45 |
| jeblair | okay i find this confusing | 21:45 |
| jeblair | i'm not sure what we're testing in that case | 21:45 |
| jeblair | pabelanger: if that's what you need, why don't you make an experimental pipeline that requires a code review +2 before running jobs? | 21:46 |
| jeblair | pabelanger: then you can test the job you're writing in the way you want to write it | 21:46 |
| pabelanger | No reason why I didn't do that. I just deciced to use existing pipeline we had | 21:48 |
| pabelanger | and right now, only openstack-dev/sandbox is testing job in WIP review | 21:48 |
| pabelanger | so, happy to change it if you'd like | 21:48 |
| jeblair | pabelanger: i'm convinced we talked about it and decided it wouldn't work, so no, i don't think you should do that. i suggested one reason it wouldn't work above. what do you think of that? | 21:50 |
| pabelanger | jeblair: thinking | 21:51 |
| jeblair | (to be clear, the pipeline idea will work, but it will require that you completely change how you've written the job -- that's what i'm trying to work through now without having to go through all the trouble of doing it) | 21:51 |
| pabelanger | let me pull up the etherpad we did last week | 21:53 |
| pabelanger | https://etherpad.openstack.org/p/0BO00tBBUi | 21:53 |
| pabelanger | I think you are saying, new job is what you were expecting right? | 21:54 |
| pabelanger | jeblair: ^ | 21:54 |
| jeblair | pabelanger: yes. if you want to use secrets, you will have to use the hierarchy in "new job" because it's not safe for those secrets to be inherited. | 21:56 |
| pabelanger | jeblair: right, so I did the hybrid approch to make testing much easier. Because with new job, I would need to have that as a trusted playbook in project-config, if I understand correctly | 21:56 |
| jeblair | pabelanger: what you wrote though is the "hybrid" hierarchy which won't work with secrets because it requires inheritance. | 21:56 |
| pabelanger | So, hybrid was much easier to iterate on and test, because I didn't need to land code. But if we want new-job format, I'm happy to move to that. Just a little rewrite | 21:57 |
| jeblair | pabelanger: how will, say, kolla publish images to tarballs.o.o? | 21:58 |
| pabelanger | jeblair: right now, inheirt publish-openstack-tarball. But the 'new job' approach, we'd have to create a new project-config job? | 22:01 |
| jeblair | pabelanger: that's my understanding. does that make the hybrid approach more desirable? | 22:02 |
| pabelanger | maybe, I mean it was much easier to test the hybrid approch in this case | 22:02 |
| jeblair | pabelanger: well, i guess what i'm asking is -- do we want any project to be able to publish stuff to their corner of tarballs.o.o without us having to make a new job for it. | 22:04 |
| mordred | I think that would be nice | 22:04 |
| jeblair | if so, then the hybrid version is something we should desire to support, i think, as it would allow anyone to inherit from the upload job and get that behavior | 22:05 |
| mordred | especially if the job has a project-name prefix baked in to it or something - so that kolla could just add "publish-stuff-to-tarballs.o.o" and it would publish whatever they'd put in their publish dir (or something) | 22:05 |
| jeblair | mordred: yeah, that is in fact the way pabelanger wrote it | 22:06 |
| pabelanger | Ya, good question. I think if a job put something into the artifacts folder, we might want to auto publish it | 22:06 |
| pabelanger | however, I see the opposite of this, making a policy where the project needs to write the job and have project-config approve it | 22:06 |
| jeblair | okay. well, the reason i asked this is so that i could make sure i understood the current defects with secrets | 22:07 |
| jeblair | i think this is one of them | 22:08 |
| jeblair | are there any other reasons we didn't use secrets for tarballs? | 22:08 |
| pabelanger | no | 22:08 |
| mordred | I don't think so | 22:08 |
| jeblair | (i know why we didn't use secrets for logs -- it's because it needs to run in check) | 22:08 |
| pabelanger | I think we should write the job still | 22:08 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Bind secrets to their playbooks https://review.openstack.org/492307 | 22:09 |
| pabelanger | and see how it operates | 22:09 |
| mordred | jeblair: and binding to playbook allows us to maybe use secrets for logs, yeah? | 22:09 |
| jeblair | okay, that changes secrets *significantly* and should address both of those cases | 22:09 |
| jeblair | ya | 22:09 |
| jeblair | (i haven't run that through a full pep/tox pass yet, but it does pass spot tests) | 22:10 |
| mordred | woot | 22:10 |
| *** smyers has quit IRC | 22:13 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Remove 'auth' dict from jobs https://review.openstack.org/492309 | 22:14 |
| pabelanger | jeblair: so, if a job inherited a secret, in a pipeline with secrets disabled, we should be protected right? | 22:15 |
| jeblair | pabelanger: currently, or in my proposed change? | 22:16 |
| pabelanger | currently | 22:16 |
| pabelanger | just reading 492307 now | 22:17 |
| jeblair | pabelanger: that's still pretty dangerous. it wouldn't run in check, but a rogue project could add their own gate job to expose the inherited secret. | 22:17 |
| jeblair | pabelanger: basically, inheriting secrets in the current system is pretty much always bad. | 22:17 |
| pabelanger | k | 22:17 |
| jeblair | pabelanger: (inheritable auth stuff was envisioned for things like swift tokens which would change with every job, but we're thinking of doing that in a completely different way now) | 22:18 |
| pabelanger | Ya, binding secrets to a playbook seems better in this instance | 22:20 |
| *** jkilpatr has joined #zuul | 22:22 | |
| *** smyers has joined #zuul | 22:24 | |
| *** maxamillion has quit IRC | 22:24 | |
| pabelanger | jeblair: do you mind if I review that in the morning? | 22:25 |
| mordred | jeblair: those lgtm on first read | 22:26 |
| *** maxamillion has joined #zuul | 22:26 | |
| jeblair | pabelanger: not at all | 22:27 |
| jeblair | tobiash: do you mind if i un-abandon your expose final change? | 22:27 |
| *** jkilpatr has quit IRC | 22:29 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Expose final job attribute https://review.openstack.org/479382 | 22:32 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Prohibit inheritance with final https://review.openstack.org/492314 | 22:32 |
| jeblair | okay, there's the whole story, i think. | 22:32 |
| *** jkilpatr has joined #zuul | 22:42 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!