| *** rlandy has quit IRC | 00:03 | |
| *** JasonCL has joined #zuul | 00:12 | |
| *** JasonCL has quit IRC | 00:21 | |
| *** odyssey4me has quit IRC | 00:34 | |
| *** odyssey4me has joined #zuul | 00:35 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul master: Update metadata for pypi https://review.openstack.org/553642 | 00:45 |
|---|---|---|
| *** harlowja has quit IRC | 00:49 | |
| *** JasonCL has joined #zuul | 01:00 | |
| *** JasonCL has quit IRC | 01:14 | |
| *** dtruong_ has joined #zuul | 01:14 | |
| *** dtruong has quit IRC | 01:17 | |
| tristanC | corvus: could it be related to the test fixtures? fwiw to make 553596 work i had to use 553144 | 01:20 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool master: zk: use kazoo retry facilities https://review.openstack.org/535537 | 01:35 |
| *** JasonCL has joined #zuul | 02:08 | |
| *** JasonCL has quit IRC | 02:12 | |
| *** swest has quit IRC | 02:20 | |
| *** swest has joined #zuul | 02:35 | |
| openstackgerrit | Merged openstack-infra/zuul master: Generate symlinks during tests https://review.openstack.org/553316 | 02:54 |
| *** adam_g has quit IRC | 03:11 | |
| *** adam_g has joined #zuul | 03:17 | |
| *** harlowja has joined #zuul | 04:35 | |
| *** harlowja has quit IRC | 05:22 | |
| *** Diabelko has quit IRC | 06:40 | |
| *** bhavik1 has joined #zuul | 06:50 | |
| LinuxJedi | clarkb: yea, I've somehow ended up with 3 bottles of it at home... I do know when to stop drinking that stuff, last time it was when I couldn't feel my legs any more :D | 06:52 |
| *** yolanda_ is now known as yolanda | 07:16 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Don't count non-live queue items in tenant list https://review.openstack.org/553702 | 07:17 |
| tobiash | corvus: small fixup for the queue count in the tenant list ^ | 07:18 |
| *** Diabelko has joined #zuul | 07:26 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Don't count non-live queue items in tenant list https://review.openstack.org/553702 | 08:11 |
| *** electrofelix has joined #zuul | 08:18 | |
| *** hashar has joined #zuul | 08:49 | |
| *** jpena|off is now known as jpena | 08:49 | |
| *** hashar has quit IRC | 09:15 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Ignore node_modules in flake8 https://review.openstack.org/553725 | 09:22 |
| *** hashar has joined #zuul | 09:22 | |
| hashar | tobiash: thank you for a Zuul backport you did a year ago "Fix missing mutex release when aborting builds" https://review.openstack.org/#/c/432211/ | 10:10 |
| hashar | I got it by that on my first try using a mutex with zuul 2.x | 10:10 |
| tobiash | hashar: :) | 10:11 |
| tobiash | in fact I did the fix in v2 and forward ported that to v3 at that time ;) | 10:12 |
| hashar | my fault really for still running 2.5.1 :D | 10:16 |
| openstackgerrit | Matthieu Huin proposed openstack-infra/nodepool master: webapp: add optional admin endpoint https://review.openstack.org/536319 | 10:19 |
| *** bhavik1 has quit IRC | 10:24 | |
| electrofelix | Something that might be of interest here, I have been working on some testing code for another project to do an integration run against github enterprise and it seems that newer versions of github3.py won't work with github enterprise | 10:32 |
| electrofelix | but further than that, it might be difficult to use well scoped tokens with it as well in the future https://github.com/sigmavirus24/github3.py/issues/794 | 10:33 |
| electrofelix | I'll continue discussions with upstream on this area, but thought it might be of interest around the zuul <-> github integration for any further enhancements | 10:34 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/zuul-jobs master: Add vos examine before release https://review.openstack.org/553749 | 10:47 |
| tobiash | electrofelix: yes, we pinned it to an older version because of that | 11:20 |
| tobiash | But I didn't have time yet to look into that, so thanks :) | 11:21 |
| tobiash | corvus: I fear we'll have to vendor github3.py in zuul for the release due to ^ | 11:27 |
| tobiash | jlk: I think that's our problem ^ | 11:28 |
| electrofelix | the alternative is to fork it, change the behaviour and publish under a different package name? | 11:55 |
| *** hashar has quit IRC | 11:56 | |
| tobiash | both forking and vendoring are awful and just an interims solution | 12:00 |
| tobiash | :( | 12:00 |
| *** hashar has joined #zuul | 12:01 | |
| *** odyssey4me has quit IRC | 12:04 | |
| *** odyssey4me has joined #zuul | 12:04 | |
| tobiash | what strikes me is this sentence: 'It's entirely plausible that supporting GitHubEnterprise will have to end.' | 12:11 |
| Shrews | oh wow, LinuxJedi has finally rejoined the party. \o/ | 12:17 |
| LinuxJedi | Shrews: just like a bad smell you can’t get rid of me it seems 😄 | 12:18 |
| *** jbCrazySane has joined #zuul | 12:23 | |
| *** jpena is now known as jpena|lunch | 12:27 | |
| *** rlandy has joined #zuul | 12:32 | |
| dmsimard | corvus: really nicely worded announcement about Zuul. I like the part about the best CI system in the world :) | 12:41 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Move tmpdir into work root https://review.openstack.org/546698 | 13:00 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Update to Ansible 2.4 https://review.openstack.org/535781 | 13:00 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Disable action and lookup plugins from 2.4 https://review.openstack.org/535839 | 13:00 |
| dmsimard | tobiash: so if my proposal about nodepool label priorities made some sense, I'll formalize that in a story -- maybe you can add your perspective so we can capture that as well ? | 13:07 |
| tobiash | dmsimard: yes, that makes sense | 13:07 |
| dmsimard | ok | 13:08 |
| dmsimard | tobiash: also, since you're a zuul web expert.. there's something I've been wanting to implement but I'd probably need to be pointed in the right direction first .. | 13:08 |
| tobiash | however I won't have time to work on implementing this | 13:08 |
| tobiash | ah ok | 13:08 |
| tobiash | ;) | 13:08 |
| dmsimard | I'd like to expose, even in a basic shape (that we can iterate on to make it better), the data from sqlreporter | 13:09 |
| Shrews | tobiash: do you have any tips on running tox-remote tests locally? | 13:09 |
| tobiash | Shrews: yes, I run it locally | 13:09 |
| dmsimard | tobiash: oh, I'm not asking you to do the nodepool thing -- I think it's worthwhile to formalize the use case, that's all | 13:09 |
| tobiash | you just need a test node to which you can ssh as zuul user | 13:09 |
| tobiash | Shrews: and set a few env vars | 13:09 |
| tobiash | Shrews: let me look that up real quick | 13:10 |
| Shrews | tobiash: thx | 13:10 |
| tobiash | Shrews: ZUUL_SSH_KEY=/workspace/id_test ZUUL_REMOTE_IPV4=xxxx | 13:10 |
| tobiash | Shrews: you can either run that through tox or (which is faster for me) directly with "python -m unittest test_remote_action_modules.TestActionModules" from the tests/remote dir | 13:11 |
| Shrews | ok. i guess the hard part is setting up the test node | 13:11 |
| tobiash | (using the env from the tox-remote) | 13:11 |
| tobiash | well, it's just a machine on which you do 'sudo adduser zuul' and install a ssh key into authorized keys | 13:11 |
| tobiash | could also be localhost | 13:12 |
| Shrews | ah, that's what i was hoping | 13:12 |
| tobiash | but the ZUUL_REMOTE_IPV4 must not be 127.0.0.1 or localhost | 13:12 |
| tobiash | if you use a real IP then localhost also work | 13:12 |
| tobiash | (which is how the tox-remote in openstack works) | 13:12 |
| tobiash | grr, tox-remote works locally for https://review.openstack.org/#/c/535839/ | 13:14 |
| tobiash | :/ | 13:14 |
| Shrews | tobiash: ah ha, got it working. thx again | 13:21 |
| tobiash | :) | 13:21 |
| SpamapS | You guys know what I want for zuul-web? nodepool support. | 13:21 |
| *** dkranz has joined #zuul | 13:21 | |
| SpamapS | would be quite nice to have a nodepool list and image-list in the web ui | 13:21 |
| tobiash | SpamapS: that would be cool | 13:22 |
| SpamapS | people are always wondering | 13:22 |
| *** myoung|afk is now known as myoung|rover | 13:22 | |
| SpamapS | if it could tie in with graphite or something too that would be nice.. like the old status page did at the bottom | 13:22 |
| Shrews | SpamapS: i believe it has been suggested to merge the two web thingys (in technical terms) | 13:24 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: DNM: tox-remote debug change https://review.openstack.org/553787 | 13:25 |
| tobiash | Shrews: the proposal from mordred was that it should be possible to include/import this somehow into zuul-web | 13:27 |
| tobiash | but I have no idea of how that works technically | 13:28 |
| tobiash | probably some webpack magic | 13:28 |
| *** JasonCL has joined #zuul | 13:29 | |
| *** eventingmonkey has quit IRC | 13:30 | |
| *** eventingmonkey has joined #zuul | 13:32 | |
| *** pwhalen has quit IRC | 13:32 | |
| *** jpena|lunch is now known as jpena | 13:33 | |
| *** pwhalen has joined #zuul | 13:37 | |
| *** pwhalen has joined #zuul | 13:37 | |
| *** eventingmonkey has quit IRC | 13:42 | |
| *** eventingmonkey has joined #zuul | 13:44 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: DNM: tox-remote debug change https://review.openstack.org/553787 | 13:45 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Move tmpdir into work root https://review.openstack.org/546698 | 14:02 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Update to Ansible 2.4 https://review.openstack.org/535781 | 14:02 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Disable action and lookup plugins from 2.4 https://review.openstack.org/535839 | 14:02 |
| dmsimard | tobiash: not sure if you saw my question about exposing the sql reporter data in zuul web -- do you think most of the pieces to do that are in place ? Would a noob be able to tackle it ? :p | 14:14 |
| tobiash | dmsimard: sorry, which question? | 14:16 |
| tobiash | dmsimard: you mean something other than https://cc-ci.bmwgroup.net/zuul/cc/builds.html ? | 14:16 |
| tobiash | ups, wrong link | 14:16 |
| tobiash | I meant http://zuul.openstack.org/builds.html ;) | 14:17 |
| dmsimard | Yes, some form of primitive reporting. Like execution counts for a given job for a timeframe | 14:18 |
| tobiash | ah, so you don't want to list something but aggregate data? | 14:19 |
| dmsimard | i.e, this job ran 5000 times in february, max duration was X, lowest duration was Y, average was Z. It succeeded 4000 times and failed 1000 times | 14:19 |
| dmsimard | This kind of data is already available in the sql reporter table, it's just not exposed for consumption | 14:19 |
| tobiash | well for the html/js part I'm the wrong guy to ask ;) | 14:19 |
| tobiash | mordred and tristanC are the experts there | 14:20 |
| dmsimard | Oh? I thought you did some frontend stuff, my bad :p | 14:20 |
| tobiash | the only frontend stuff I did were small fixes, reviews and https://review.openstack.org/#/c/548248/ ;) | 14:21 |
| dmsimard | I'll try and see if I can pattern off of what already exists.. I've never done angular before but there's a first time for everything | 14:21 |
| dmsimard | ¯\_(ツ)_/¯ | 14:21 |
| dmsimard | I'd really like this feature. It would also be a good way to highlight the scale at which we're running Zuul. | 14:22 |
| corvus | tristanC: i don't know why it doesn't work. i didn't observe the failure, so i can't do anything more than what you tell me. i need you to take that test and make it fail. | 14:23 |
| dmsimard | corvus: tristanC is out for the weekend already (mentioning in case this is time sensitive) | 14:23 |
| corvus | SpamapS, tobiash, Shrews: at the ptg we discussed the following: creating a nodepool-web service structured like zuul-web, so that nodepool (without zuul) will have web/api functionality similar to zuul. then adapting zuul-web to use the same api endpoints as nodepool-web to provide the information in a unified dashboard (possibly with some of the same javascript code). | 14:31 |
| dmsimard | corvus: in case you missed it, mhu has done some amount of work towards an API in nodepool: https://review.openstack.org/#/c/536319/ | 14:33 |
| dmsimard | (I mention since it doesn't seem like you've reviewed it yet) | 14:33 |
| corvus | dmsimard: focused on the v3 release. i'm not reviewing anything that isn't either urgent, trivial, or related to the release. | 14:38 |
| openstackgerrit | Merged openstack-infra/zuul master: Update metadata for pypi https://review.openstack.org/553642 | 14:38 |
| dmsimard | corvus: that's fair, I thought it was relevant to your statement about nodepool-web service | 14:38 |
| dmsimard | if it's not, please ignore it :) | 14:38 |
| corvus | dmsimard: at the ptg we discussed things we'll do after the release :) | 14:39 |
| dmsimard | I've been missing out :( | 14:39 |
| corvus | but now we need to set them aside for the moment until we get the release out the door. | 14:39 |
| tobiash | clarkb, pabelanger: trivial fix and related to v3: https://review.openstack.org/535864 | 14:39 |
| tobiash | corvus: is something left for the release except mordred's thing, github3.py and possibly the remaining security fixes? | 14:42 |
| corvus | tobiash: those are the blockers; if we can get re2 in that'd be great. | 14:43 |
| tobiash | so I should probably review SpamapS's changes | 14:44 |
| dmsimard | tobiash: there was the truncated json issue that either no longer reproduces or I haven't been able to figure out | 14:44 |
| *** hashar has quit IRC | 14:55 | |
| mhu | dmsimard, corvus yep I've been focusing on providing some form of simple admin API. I was thinking of adding some form of ui in front of it but 1. my js is terrible 2. seems like we're headed towards a unified zuul/nodepool front anyway | 14:57 |
| tobiash | corvus: shall I try to prepare vendoring github3.py? | 14:58 |
| corvus | mhu: well, i'd characterize it as nodepool having its own web api/ui, but zuul being able to use it or merge it into zuul's ui. | 14:58 |
| corvus | tobiash: let's maybe wait a little longer? we can wait and see what happens on that issue while we work on the security fixes and streaming. | 14:59 |
| tobiash | ok | 15:00 |
| dmsimard | corvus, mhu: +1 zuul should totally be able to consume the nodepool api | 15:01 |
| dmsimard | mhu: it's fine if the frontend is basic, we can iterate on it -- the hard work is doing the backend/api to expose the data :) | 15:01 |
| tobiash | corvus: just had the case when using the uri module that a job on the executor cannot validate ssl certs | 15:10 |
| tobiash | for that I can add /etc/ssl/certs to the trusted_ro_paths | 15:11 |
| rcarrillocruz | corvus: can you confirm this scenario: If I wanted a private zuul executor to *only* run jobs created by a private launcher, would implementing the executor affinity story *and* have unique labels on the private launcher solve that? | 15:11 |
| tobiash | do you think it makes sense to mount this path generally into bwrap? | 15:11 |
| tobiash | rcarrillocruz: that sounds correct | 15:12 |
| *** electrofelix has quit IRC | 15:12 | |
| *** electrofelix has joined #zuul | 15:12 | |
| mordred | mhu,c, dmsimard: agree re: nodepool/zuul dashboard and api things - I'll get structure in place for that as soon as I get this current js stack happy | 15:13 |
| mordred | gah. that c, was supposed to be corvus ... I guess I don't know how to work a tab key | 15:14 |
| *** electrofelix has quit IRC | 15:14 | |
| Shrews | mordred: your fingers must relearn typing after being away for so long | 15:14 |
| jlk | toabctl: thanks for the heads up and link. I'll book mark that, and pour some thought into how to better support older enterprise installs. | 15:16 |
| dmsimard | tobiash: only thing I wonder is if the /etc/ssl/certs path is consistent across distros | 15:18 |
| AJaeger | dmsimard: exists on openSUSE | 15:21 |
| corvus | tobiash: sounds reasonable | 15:22 |
| dmsimard | AJaeger: yeah /etc/ssl/certs appears to be a thing in both fedora and centos | 15:22 |
| dmsimard | AJaeger: is there *one* thing that distros actually agree on ?? :) | 15:22 |
| corvus | dmsimard: the bubblewrap driver has a provision for only mounting things that exist | 15:22 |
| dmsimard | corvus: right, but I meant that it might exist elsewhere instead (say, /etc/anotherplace/certs) | 15:23 |
| tobiash | ya, so if you need certs from a non-standard location you still can tell zuul to mount that in | 15:23 |
| dmsimard | It seems like it does exist across all distros though, just didn't know | 15:23 |
| tobiash | oh, it seems that at least within ubuntu you also need /usr/share/ca-certificates | 15:24 |
| tobiash | the certs in /etc/ssl/certs are symlinked to there | 15:24 |
| clarkb | ya it symlinks between the two locations iirc | 15:24 |
| mordred | and the system for managing updating/rehashing is different | 15:25 |
| dmsimard | does bwrap follow symlinks ? | 15:25 |
| tobiash | nope so we would have to mount both | 15:25 |
| tobiash | btw, alpine follows the same path with /etc/ssl/certs and /usr/share/ca-certificates | 15:26 |
| dmsimard | tobiash: you're using alpine? interesting | 15:26 |
| tobiash | dmsimard: my zuul runs within an alpine container | 15:27 |
| dmsimard | nice! | 15:27 |
| clarkb | I thought we already mounted both fwiw | 15:29 |
| clarkb | because openstack jobs failed without working certs on the executor | 15:30 |
| corvus | clarkb: openstack has it in our local config | 15:30 |
| corvus | clarkb: i think tobiash was suggesting we make it default | 15:30 |
| clarkb | gotcha ++ to that | 15:31 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Add standard ca certificate paths https://review.openstack.org/553828 | 15:31 |
| tobiash | oh actually /usr is already mounted | 15:33 |
| corvus | tobiash: yeah, we only have /etc/ssl/certs added to our local config | 15:34 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Add standard ca certificate paths https://review.openstack.org/553828 | 15:34 |
| tobiash | fixed ^ | 15:34 |
| pabelanger | tobiash: +3 | 15:46 |
| tobiash | :) | 15:47 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Fix builds queued forever after failure to get node request https://review.openstack.org/537335 | 15:54 |
| tobiash | corvus: ^ | 15:54 |
| tobiash | thanks | 15:54 |
| *** aluria` has joined #zuul | 16:02 | |
| *** fbo_ has quit IRC | 16:03 | |
| *** jlk has quit IRC | 16:03 | |
| *** aluria has quit IRC | 16:03 | |
| *** harlowja has joined #zuul | 16:03 | |
| *** dtruong_ has quit IRC | 16:04 | |
| *** dtruong has joined #zuul | 16:05 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/zuul master: Rework log streaming to use python logging https://review.openstack.org/541434 | 16:05 |
| openstackgerrit | Merged openstack-infra/zuul master: Correctly document default git dirs https://review.openstack.org/535864 | 16:08 |
| *** fbo_ has joined #zuul | 16:09 | |
| openstackgerrit | Merged openstack-infra/zuul master: Ignore node_modules in flake8 https://review.openstack.org/553725 | 16:16 |
| *** jlk has joined #zuul | 16:18 | |
| openstackgerrit | Merged openstack-infra/zuul master: Add standard ca certificate paths https://review.openstack.org/553828 | 16:24 |
| * mordred apologizes to everyone for the upcoming patch-bomb - also please feel free to ignore this stack for the time being | 16:26 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Split status and stream into typescript modules https://review.openstack.org/551989 | 16:27 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Add typing to getSourceUrl https://review.openstack.org/551990 | 16:27 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Break build list out into its own module https://review.openstack.org/551991 | 16:27 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Break job out into its own module https://review.openstack.org/551993 | 16:27 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Break job list out into its own module https://review.openstack.org/551994 | 16:27 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Break tenant list out into its own module https://review.openstack.org/551995 | 16:27 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Break project detail and list out into their own module https://review.openstack.org/551996 | 16:27 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Move webpack html template to web/config https://review.openstack.org/551997 | 16:27 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Migrate webpack config to typescript https://review.openstack.org/551998 | 16:27 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Use glyphicons for status balls https://review.openstack.org/551992 | 16:27 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Migrate status, stream and builds to angular https://review.openstack.org/553845 | 16:27 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Migrate project and projects to angular https://review.openstack.org/553846 | 16:27 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Migrate job and jobs to angular https://review.openstack.org/553847 | 16:27 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Migrate tenants list to angular https://review.openstack.org/553848 | 16:27 |
| dmsimard | welcome back mordred | 16:30 |
| clarkb | http://logs.openstack.org/35/537335/6/check/zuul-tox-py35/0057247/job-output.txt.gz looks like a flaky test in the py35 job. There was also a governor slow start test that failed yesterday on the change to move symlinks into the code | 16:33 |
| clarkb | are these known issues with the test suite right now? or do they need investigation? | 16:33 |
| corvus | clarkb: i saw another slow start failure on an unrelated change, so i think that one is flaky | 16:34 |
| corvus | test_reconfigure_window_fixed failures are new to me | 16:34 |
| clarkb | I've rechecked 537335 and approved it. I don't think this test failure is related to the change | 16:35 |
| clarkb | but if it is, it should fail again and not merge | 16:36 |
| * mordred waves to dmsimard | 16:44 | |
| *** yolanda has quit IRC | 16:55 | |
| openstackgerrit | Fabien Boucher proposed openstack-infra/zuul master: Make Zuul able to start with a broken config https://review.openstack.org/535511 | 16:58 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Fix plugin injection vulnerability https://review.openstack.org/553854 | 17:00 |
| tobiash | corvus, clarkb: ^ | 17:00 |
| tobiash | corvus, clarkb: regarding the retry limit comment, it was actually mean for helping debugging this test case if it returns retry limit | 17:01 |
| tobiash | the intent is that one should not try to fix retry limit into success but fix the other thing | 17:02 |
| tobiash | so I'm open for better wording on that ;) | 17:02 |
| clarkb | tobiash: I think the piece I was missing was how it relates to testv3 not actyually running real jobs. Maybe just say the result will be retry_limit because we can't run against a real host here. | 17:03 |
| corvus | erm | 17:05 |
| *** harlowja has quit IRC | 17:05 | |
| corvus | what failure causes the retry limit? | 17:05 |
| tobiash | I have to check | 17:06 |
| clarkb | I'm guessing an inability to run the pre run for the job? so it hits that 3 times or whatever the default is and then returns retry limit | 17:06 |
| corvus | but there's no pre-run? | 17:06 |
| clarkb | oh right its just run | 17:08 |
| corvus | so we're all on the same page: this test does actually run ansible. it just does it only with localhost in the inventory. | 17:08 |
| tobiash | oh, actually my test setup causes retry limit: http://paste.openstack.org/show/703054/ | 17:09 |
| clarkb | corvus: gotcha | 17:09 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: web: add /{tenant}/jobs/{job_name} route https://review.openstack.org/550978 | 17:10 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: web: add /{tenant}/projects routes https://review.openstack.org/550979 | 17:10 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: web: add /{tenant}/pipelines route https://review.openstack.org/541521 | 17:10 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Rename javascript package to zuul-dashboard https://review.openstack.org/551999 | 17:10 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: dashboard: add /{tenant}/job.html page to display job details https://review.openstack.org/535545 | 17:10 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: dashboard: add /{tenant}/projects.html web page https://review.openstack.org/537870 | 17:10 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Fix indentation and renable the eslint rule https://review.openstack.org/545671 | 17:10 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Shift html templates into components https://review.openstack.org/551327 | 17:10 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Use arrow functions for http callbacks https://review.openstack.org/551399 | 17:10 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Upgrade to webpack 4 https://review.openstack.org/551987 | 17:10 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Express the bootstrap css depend in css https://review.openstack.org/551988 | 17:10 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Upgrade from angularjs (v1) to angular (v5) https://review.openstack.org/551989 | 17:10 |
| tobiash | I'm running locally from mac in docker and my container has no python | 17:10 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Use glyphicons for status balls https://review.openstack.org/551992 | 17:10 |
| tobiash | so it runs ansible but fails to run /usr/bin/python to execute the modules in my test setup | 17:11 |
| tobiash | corvus: so shall I just remove this comment? | 17:11 |
| corvus | ya | 17:11 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Fix plugin injection vulnerability https://review.openstack.org/553854 | 17:13 |
| clarkb | tobiash: there is a python image publsihed to docker hub :) of course it probably doesn't have bwrap in it | 17:14 |
| tobiash | well, I'm just using a hand crafted quick and dirty image for local testing | 17:15 |
| tobiash | it's kind of a frankenstein setup | 17:15 |
| tobiash | I'm running pytharm on mac | 17:15 |
| tobiash | mysql, postgres in docker | 17:15 |
| tobiash | and with docker remote debugging also zuul in docker | 17:15 |
| tobiash | but I got most things working and can even step through the code | 17:16 |
| *** pabelanger has quit IRC | 17:17 | |
| *** pabelanger has joined #zuul | 17:17 | |
| clarkb | nice, does it use a pycharm specific remote debugger? or just pdb with a socket? | 17:19 |
| tobiash | pycharm has builtin support for remote debugging using docker, docker-compose and ssh | 17:20 |
| tobiash | so I use that | 17:21 |
| tobiash | just needed to tweak networking settings as with docker on mac using just host networking doesn;t work | 17:21 |
| tobiash | so I'm setting different hostnames for zk and mysql | 17:21 |
| tobiash | corvus, clarkb, fungi: https://etherpad.openstack.org/p/jFn1sX7osV | 17:33 |
| *** myoung|rover is now known as myoung|bbl | 17:34 | |
| tobiash | corvus: should we add that the severity is high? | 17:34 |
| openstackgerrit | Fabien Boucher proposed openstack-infra/zuul master: Make Zuul able to start with a broken config https://review.openstack.org/535511 | 17:34 |
| tobiash | due to the fact that this makes it possible to execute arbitrary code it could be in theory also be use for escaping bwrap (e.g. if there is some kernel vulnerability to exploit) | 17:36 |
| pabelanger | Yah, I would think we want zuul operators to update right away and soon as possible | 17:37 |
| fbo_ | corvus: hi, I updated the patch to make Zuul starts with a broken config and implemented your requests. | 17:39 |
| openstackgerrit | Merged openstack-infra/zuul master: Fix builds queued forever after failure to get node request https://review.openstack.org/537335 | 17:40 |
| fbo_ | I'll have a look to a follow up patch to expose the loading errors for zuul-web | 17:40 |
| clarkb | tobiash: etherpad lgtm | 17:44 |
| pabelanger | tobiash: minor update to etherpad | 17:44 |
| tobiash | thanks | 17:45 |
| *** weshay is now known as weshay_brb | 17:47 | |
| jlk | oh neat. Now GitHub PRs can require MULTIPLE required reviews. So there could be a requirement for human AND zuul. | 17:53 |
| jlk | Not that zuul is dropping reviews, but still interesting. | 17:56 |
| *** jpena is now known as jpena|off | 18:01 | |
| tobiash | but one could enforce an at least two maintainer votes now :) | 18:01 |
| tobiash | it hopefully lands also in the next ghe | 18:01 |
| *** harlowja has joined #zuul | 18:01 | |
| corvus | fbo_: thanks! i may not get to review it right away | 18:01 |
| corvus | tobiash: that looks good to me. i'd say we should ask fungi, but he indicated he'd be gone for a long lunch, so maybe let's just send it. | 18:03 |
| tobiash | corvus: your choice | 18:03 |
| corvus | i say send | 18:03 |
| tobiash | ok | 18:03 |
| *** harlowja_ has joined #zuul | 18:04 | |
| tobiash | ups, gate is still running | 18:05 |
| tobiash | corvus: you might want to wait until it merges before approving the mail | 18:06 |
| corvus | tobiash: will do | 18:06 |
| *** harlowja has quit IRC | 18:06 | |
| openstackgerrit | Merged openstack-infra/zuul master: Fix plugin injection vulnerability https://review.openstack.org/553854 | 18:09 |
| tobiash | \o/ | 18:09 |
| pabelanger | woot | 18:09 |
| openstackgerrit | Fabien Boucher proposed openstack-infra/zuul master: Add zuul-web endpoint for getting configuration errors for a tenant https://review.openstack.org/553873 | 18:19 |
| tobiash | clarkb, pabelanger: I'd have two easy reviews if one of you has time: https://review.openstack.org/537432 and https://review.openstack.org/544236 | 18:45 |
| clarkb | now that infra is patched against plugin thing I do have time :) | 18:46 |
| *** elyezer has joined #zuul | 18:48 | |
| *** elyezer has quit IRC | 18:55 | |
| *** elyezer has joined #zuul | 18:59 | |
| *** weshay_brb is now known as weshay | 18:59 | |
| pabelanger | yah, apparently I had a callback_plugins for human_log.py still install. I pushed up a patch a few days ago to remove it, since I didn't need it any more. Didn't realize it was actually an exploit for untrusted playbooks until I start looking at connection_plugins / stragegy_plugins yesterday | 19:00 |
| *** elyezer has quit IRC | 19:17 | |
| *** elyezer has joined #zuul | 19:18 | |
| fungi | glad you didn't wait for me... that was indeed a long lunch. catching up now | 19:45 |
| fungi | tobiash: corvus: speaking of "some kernel vulnerability to exploit" this is the most recent of that class (from today): http://www.openwall.com/lists/oss-security/2018/03/16/1 | 19:47 |
| fungi | 'This flaw can be exploited [...] by an attacker who is a privileged user (a "root" user) in a user+network namespace' | 19:48 |
| corvus | hopefully folks aren't running executor as root | 19:48 |
| fungi | indeed. they nmeed to combine that one with a local user privilege escalation i suppose | 19:49 |
| corvus | but of course it's entirely possible | 19:49 |
| *** dkranz has quit IRC | 19:49 | |
| fungi | but at that point, potential bwrap escape | 19:49 |
| tobiash | so that would enable escaping from bwrap using unprivileged user namespaces? | 19:49 |
| *** myoung|bbl is now known as myoung|rover | 19:50 | |
| tobiash | I'm surprised that nothing broke so far due to the fix ;) | 19:51 |
| fungi | tobiash: the way i read that one, yes. someone with context root inside brwap and ability to manipulate kernel network bridging syscalls within their assigned namespace could at least write to some limited kernel memory on the containing system (whether they can broker that into a container escape is another question of course) | 19:51 |
| fungi | my point wasn't so much that specific vulnerability, but rather that those are still almost daily occurrences | 19:52 |
| fungi | containers are far from being a complete risk isolation solution | 19:52 |
| corvus | this is a really good example of why we don't *just* want bwrap. preventing arbitrary code execution (even within a container) on the executor is important. | 19:54 |
| pabelanger | +1 | 19:57 |
| kklimonda | has there been any discussion about making zuul jobs uninterruptible? I'm thinking of a usecase when aborting a job will leave some external resources (that are being tested) in an undefined state. For example, we have a request to support jobs that run terraform - if it's interrupted, there is no easy way to clean-up what's been created. | 20:02 |
| dmsimard | clarkb, fungi, corvus, tobiash: created a private story with zuul-security | 20:03 |
| dmsimard | Please read asap | 20:03 |
| kklimonda | ok, looking at the backlog this is probably not the right moment to discuss it - carry on with your firedrill ;) | 20:03 |
| fungi | kklimonda: declaring any computing process "uninterruptible" seems like a pipe dream. my experience would suggest that there are all manner of events which could interrupt any running process, so best to design systems with the assumption that they may be interrupted at the worst possible times | 20:05 |
| fungi | for example, many cloud providers will have you agree to terms of service which say that they may delete servers out from under you with no advance warning | 20:07 |
| corvus | kklimonda: as an alternative, we have discussed 'cleanup' jobs, so that (assuming zuul is functioning) we would always launch the cleanup job, no matter what happened to earlier ones. | 20:07 |
| kklimonda | yeah, a cleanup job (or perhaps a phase of a zuul job?) would probably go a long way to solve that problem | 20:09 |
| fungi | cleanup phase or cleanup jobs (or both) are an interesting idea | 20:10 |
| kklimonda | fungi: I agree, but now it's pretty hard to test something that deals with external resources - it seems unavoidable to leave stuff hanging around if the job is aborted. | 20:10 |
| fungi | yep, i caon definitely sympathoze | 20:11 |
| fungi | sympathize | 20:11 |
| openstackgerrit | Merged openstack-infra/zuul master: Fix runtime stats reporting for noop job https://review.openstack.org/537432 | 20:11 |
| openstackgerrit | Merged openstack-infra/zuul master: Fix self fulfilling empty node requests https://review.openstack.org/544236 | 20:11 |
| kklimonda | terraform seems to be pretty bad at it anyway, I don't think it will even keep anything resembling a valid state if it's aborted in the middle of execution.. | 20:12 |
| pabelanger | kinda the same issue with heat stacks in tripleo-test-cloud-rh1, an external resource to nodepool but would sometime leak stacks, making it hard to clean up. Tended to be a manually process for the admin on the cloud | 20:13 |
| kklimonda | so now I'm thinking of creating a small task server that can be used to schedule terraform runs and query for their status, and use it from zuul jobs | 20:13 |
| pabelanger | a clean-up job would fit nice there | 20:14 |
| kklimonda | yeah | 20:14 |
| corvus | added a note to https://storyboard.openstack.org/#!/story/2001340 about this use case, which is subtly different than the original | 20:17 |
| fungi | seems similar (though maybe at a slightly different layer) to the way nodepool evolved.much of its complexity is devoted to cleaning up resources for aborted actions | 20:19 |
| fungi | openstack has met us halfway, simultaneously getting better at not leaking resources | 20:19 |
| corvus | yeah, generally speaking, having nodepool be able to provide more resources than just nodes may be relevant. but only if the resources are used by the test, and not the thing being tested. obviously then you want the zuul job doing the work. | 20:20 |
| kklimonda | hmm, I can't seem to add a story in storyboard - "Save changes" is greyed out | 20:22 |
| corvus | kklimonda: you may be missing a field; like you may need to add a project. | 20:23 |
| corvus | ("select a project" is right above the save changes button) | 20:24 |
| corvus | i think there's work happening so we can provide a link that pre-populates that | 20:24 |
| corvus | so there'd be a specific "report a zuul bug" link that would fill in openstack-infra/zuul there | 20:24 |
| kklimonda | huh | 20:26 |
| kklimonda | I can submit if I choose "openstack-infra/storyboard" but not for nodepool nor zuul | 20:26 |
| kklimonda | is there some sort of access control? | 20:26 |
| *** rlandy is now known as rlandy|biab | 20:27 | |
| corvus | kklimonda: shouldn't be... any chance you can send a screenshot? | 20:28 |
| kklimonda | sure | 20:28 |
| SotK | kklimonda: there is currently a bug where if you type the whole project name before selecting from the dropdown, the model isn't updated and you can't submit | 20:29 |
| SotK | it could be that you're hitting that? | 20:29 |
| corvus | kklimonda: oh one more thing -- when you type 'openstack-infra/zuul' into the field, you need to click the project name, even... | 20:29 |
| corvus | yeah that :) | 20:29 |
| kklimonda | ah, there you go - thanks | 20:29 |
| corvus | kklimonda: you may want to file a story about 'cleanup phase' while you're in there; i think that's probably distinct enough from 'cleanup job' to warrant separate consideration. | 20:31 |
| kklimonda | btw, would configuration of some static resources, for example switches/routers, something that nodepool could with the current driver interface, or would more work be done on that? I'm thinking of test layouts like "2 VMs connected to a router (with logical router created within) over two L2 networks" | 20:32 |
| corvus | kklimonda: i'm not sure; i've been trying to focus on release blocking things and haven't thought much about the next steps for the driver interface. | 20:35 |
| kklimonda | corvus: I've added a comment about cleanup phase and some sort of "pre main" phase that can't be aborted | 20:41 |
| kklimonda | also, created https://storyboard.openstack.org/#!/story/2001685 for supporting some extra networking stuff in nodepool | 20:46 |
| *** rlandy|biab is now known as rlandy | 21:06 | |
| dmsimard | tobiash, clarkb, corvus: just updated the story. | 21:39 |
| tobiash | dmsimard: replied | 21:47 |
| fungi | feels like this one is getting very close to the point where we can switch it to a normal public story | 21:50 |
| *** myoung|rover is now known as myoung|afk | 21:59 | |
| *** elyezer has quit IRC | 22:03 | |
| *** Diabelko has quit IRC | 22:07 | |
| *** adam_g has quit IRC | 22:07 | |
| *** AJaeger has quit IRC | 22:07 | |
| *** Wei_Liu has quit IRC | 22:07 | |
| *** Diabelko has joined #zuul | 22:07 | |
| *** adam_g has joined #zuul | 22:07 | |
| *** AJaeger has joined #zuul | 22:07 | |
| *** Wei_Liu has joined #zuul | 22:07 | |
| dmsimard | tobiash, clarkb, corvus: added a last comment, have family thing I need to attend | 22:34 |
| *** rlandy_ has joined #zuul | 22:35 | |
| *** rlandy has quit IRC | 22:37 | |
| corvus | dmsimard: if you have just a second, can you clarify your last comment? | 22:41 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: WIP: late bind pipelines https://review.openstack.org/553618 | 22:52 |
| *** rlandy__ has joined #zuul | 23:03 | |
| *** rlandy_ has quit IRC | 23:06 | |
| *** rlandy__ has quit IRC | 23:17 | |
| Diabelko | is there a way to completely skip nodepool part and run job directly from executor? | 23:45 |
| clarkb | Diabelko yes, use an empty nodeset | 23:45 |
| clarkb | you'll be restricted in what you can do though | 23:45 |
| Diabelko | I basically just need to do curl combined with with_items loop | 23:45 |
| clarkb | that should work. I think our rtfd job may even be an example of doing similar | 23:46 |
| Diabelko | oh, I'll take a look then, thank you! | 23:46 |
| Diabelko | :) | 23:46 |
| *** openstackgerrit has quit IRC | 23:48 | |
| Diabelko | ah, got it, thanks again | 23:50 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!