| *** tflink_ has quit IRC | 00:55 | |
| *** tflink has joined #zuul | 01:04 | |
| tristanC | mordred: that rings a bell, i'll have a look | 01:07 |
|---|---|---|
| tristanC | pabelanger: thanks, i replied that add_host can't be used by untrusted job, and added host doesn't stay between pre, run and post | 01:08 |
| openstackgerrit | Ian Wienand proposed openstack-infra/zuul-jobs master: Add openafs-client role https://review.openstack.org/589334 | 01:15 |
| openstackgerrit | Ian Wienand proposed openstack-infra/zuul-jobs master: Add kerberos-client role https://review.openstack.org/590591 | 01:15 |
| pabelanger | tristanC: yah, wonder if there is a way to dump in-memory inventory via ansible. But also thought we'd let add_host be untrusted, or maybe mis-remembering. I always thought we could add blacklist support of IPs for add_host to block address in control plane, for example | 01:16 |
| tristanC | pabelanger: there is also the difficulty to pass the pods' name from the pre run to the run phase | 01:20 |
| tristanC | pabelanger: but yes, if we can add_host from untrusted, 590092 is not necessary | 01:21 |
| tristanC | doing ip blacklist sounds tricky though | 01:22 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: web: fix job required-projects list https://review.openstack.org/590594 | 01:31 |
| *** tflink has quit IRC | 01:41 | |
| *** tflink has joined #zuul | 01:43 | |
| *** elyezer has quit IRC | 02:36 | |
| openstackgerrit | Merged openstack-infra/zuul master: Fix wrong matched project template https://review.openstack.org/588201 | 02:47 |
| *** elyezer has joined #zuul | 02:48 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/zuul-jobs master: Add kerberos-client role https://review.openstack.org/590591 | 03:19 |
| openstackgerrit | Ian Wienand proposed openstack-infra/zuul-jobs master: Add openafs-client role https://review.openstack.org/589334 | 03:19 |
| *** ianychoi has joined #zuul | 03:56 | |
| *** smyers has quit IRC | 04:53 | |
| *** smyers has joined #zuul | 04:55 | |
| *** pcaruana has joined #zuul | 06:43 | |
| *** goern has quit IRC | 06:57 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Add --check-config option to zuul scheduler https://review.openstack.org/542160 | 07:17 |
| *** jpena|off is now known as jpena | 07:55 | |
| *** gtema has joined #zuul | 08:25 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Fix stuck job caused by exception during repo update https://review.openstack.org/590697 | 08:27 |
| *** ssbarnea has quit IRC | 08:52 | |
| openstackgerrit | Matthieu Huin proposed openstack-infra/nodepool master: Do not abort node launch if failed node cannot be deleted https://review.openstack.org/589854 | 08:59 |
| tobiash | mhu: responded on ^ | 09:05 |
| mhu | tobiash thx! | 09:11 |
| mhu | tobiash so the minimal info needed for node deletion would be the external id, the ips and the provider right? | 09:13 |
| mhu | am I forgetting smth? | 09:13 |
| tobiash | mhu: I don't think you need the ips but you might need the pool | 09:14 |
| tobiash | but you can look into the DeletedNodeWorker to see which information it uses | 09:14 |
| *** ssbarnea has joined #zuul | 09:31 | |
| *** panda|ruck|off is now known as panda|ruck | 10:33 | |
| *** jpena is now known as jpena|lunch | 11:06 | |
| mordred | it should only need the external id - from a cloud api perspective | 11:28 |
| mordred | ips, if there are any, are deleted as part of the delete_server call | 11:28 |
| *** jpena|lunch is now known as jpena | 12:02 | |
| tobiash | mordred: you're an openstack workflow expert and maybe you have an idea/opinion | 12:28 |
| mordred | uhoh | 12:28 |
| tobiash | mordred: I have many projects that use basically the same image but with different cache content | 12:28 |
| tobiash | so the current workflow is to have many images and each project wants to have a min-ready | 12:29 |
| tobiash | -> bad | 12:29 |
| tobiash | mordred: is it possible to clone-and-attach a volume directly when booting an image? | 12:29 |
| tobiash | (and making it auto-delete) | 12:29 |
| tobiash | hrm, that won;t solve the min-ready | 12:30 |
| tobiash | nevermind | 12:30 |
| tobiash | we probably would need to boot the images with min-ready and attach a clone of a cache volume on assignment to a node request | 12:30 |
| pabelanger | yah, would be interested in hearing how images are managed for others with cache (git repos) content in them. Also consider doing something like volume / AFS, but so far learning toward dedicated images for teams | 12:31 |
| pabelanger | but, min-ready 0 in most cases | 12:31 |
| tobiash | pabelanger: that's our current approach but the teams are not happy with min-ready 0 | 12:32 |
| tobiash | :/ | 12:32 |
| pabelanger | The other things I started thinking about, is how to stop a user in tenant A use image in tenant B. thinking the first task in a pre-run job would be to use some regex to validate nodepool.label, based on something like tenantA-centos-7 / tenantB-centos-7. Then fail the job if they use the wrong one. | 12:33 |
| pabelanger | tobiash: yah, if a fast cloud, min-ready 0 isn't an issue | 12:33 |
| tobiash | pabelanger: we discussed a whitelist for labels in the tenant config | 12:34 |
| tobiash | a few months ago | 12:34 |
| pabelanger | tobiash: yah, remember, haven't looked into where that landed | 12:34 |
| tobiash | pabelanger: our linux images typically boot in 50-60 seconds, but windows take 5 minutes | 12:34 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Fix permanently broken git cache https://review.openstack.org/590761 | 12:45 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Fix permanently broken git cache https://review.openstack.org/590761 | 12:54 |
| tobiash | that fixes an issue with permanently broken caches we have almost every time a compute node with an executor on it crashes ^ | 12:58 |
| tobiash | and this currently happens more often than I want it to happen atm :/ | 13:00 |
| mordred | tobiash: well, it IS possible to do what you're looking for with volumes, but I agree it doesn't help with min-ready | 13:01 |
| mordred | tobiash: we've been pondering if it would be better to stop having in-image cache if we could have per-cloud-region git mirrors (oh, pabelanger said that already) | 13:02 |
| mordred | tobiash: but I'm guessing that cloning from your local git infrastructure every time isn't workable for you | 13:02 |
| tobiash | mordred: regarding docker images we definitely want to have in image caching | 13:03 |
| tobiash | mordred: atm we don't do git caching (other than what the executor does) | 13:03 |
| tobiash | but what would kill us is to not do docker caching | 13:03 |
| tobiash | (we have projects that use 15GB docker images) | 13:03 |
| mordred | tobiash: have you suggested to them that 15G docker images are rather large? :) | 13:04 |
| tobiash | mordred: of course... | 13:05 |
| mordred | :) | 13:05 |
| tobiash | but the toolchains they use are let's say not optimized for being used in docker | 13:05 |
| mordred | well - as mentioned above - you could do clone/attach of a cache volume when an node is assigned | 13:06 |
| tobiash | e.g. pull in cuda and you get the compiler + 5gb sdk, ui tools etc | 13:06 |
| mordred | for you personally that sholnd't be too bad, because IIRC you're using ceph there -so those volume clones should be near-instant COW operations | 13:06 |
| tobiash | yes, that would work | 13:07 |
| mordred | and you can make a volume from an image | 13:07 |
| tobiash | we would have to reconfigure and restart docker then to use the graph from vdb | 13:07 |
| mordred | oh - yeah. hrm | 13:07 |
| mordred | wel - you could do that in the pre-playbook of your base job | 13:07 |
| tobiash | yes that would be possible | 13:07 |
| mordred | we could teach nodepool how to do this probably without too much pain | 13:08 |
| mordred | especially if we let nodepool manage the cache volumes by managing them as images | 13:08 |
| tobiash | the only problem is that we might need to restructure the nodepool data structure | 13:08 |
| tobiash | the min-ready nodes would need to have multiple labels attached | 13:09 |
| mordred | yah. it'll take some thought - but from an openstack workflow POV it should be both possible and not terribly inefficient | 13:09 |
| tobiash | cool | 13:09 |
| tobiash | so maybe a friday afternoon project in a few months ;) | 13:10 |
| pabelanger | yah, regional git mirror might also be cool too | 13:17 |
| *** samccann has joined #zuul | 13:22 | |
| *** dmsimard has quit IRC | 13:26 | |
| *** dmsimard has joined #zuul | 13:31 | |
| mordred | pabelanger: ++ | 13:54 |
| openstackgerrit | Merged openstack-infra/zuul master: web: fix job required-projects list https://review.openstack.org/590594 | 14:47 |
| corvus | tobiash: https://review.openstack.org/589975 is ready now -- let me know if it matches your thinking | 15:00 |
| tobiash | corvus: great, will have a look after dinner | 15:00 |
| clarkb | I can't vouch for tobiash's thinking but the change lgtm | 15:15 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: Add nodepool info to test-emit-job-header https://review.openstack.org/557947 | 15:15 |
| pabelanger | clarkb: mordred: corvus: could I get a review of the stack at ^, adds nodepool info into job logs now. | 15:20 |
| pabelanger | https://logs.rdoproject.org/15/15415/1/check/tox-docs/0478753/job-output.txt.gz#_2018-08-10_15_17_40_255434 shows what the output would be | 15:20 |
| pabelanger | from rdoproject | 15:20 |
| clarkb | pabelanger: does that work when running gainst localhost? | 15:21 |
| clarkb | oh its a !localhost | 15:21 |
| pabelanger | clarkb: yah, we skip that | 15:21 |
| clarkb | pabelanger: does that role run only against localhost? otherwise I think we will print all that inventory info for each node right? (since its looping overall of them directly) | 15:23 |
| clarkb | pabelanger: looks like the role runs against only localhost then we loop over all hosts in inventory that are not localhost | 15:26 |
| clarkb | that seems odd, but given our history trying t oget this right also seems to be correct | 15:26 |
| corvus | pabelanger: what if one of those pieces of info is missing? (ie, is a container?) | 15:27 |
| pabelanger | clarkb: yah, we should only run it against localhost | 15:29 |
| pabelanger | corvus: good point, we'll have checks for fields that will be missing | 15:30 |
| Shrews | should we stop using with_* for loops and begin using the recommended 'loop' syntax? | 15:31 |
| Shrews | per https://docs.ansible.com/ansible/latest/user_guide/playbooks_loops.html#migrating-from-with-x-to-loop | 15:31 |
| Shrews | i don't know of any planned deprecation, fwiw | 15:32 |
| pabelanger | maybe, that patchset is a little old | 15:32 |
| pabelanger | but also don't see replacement for with_inventory_hostname on that page | 15:33 |
| Shrews | pabelanger: https://docs.ansible.com/ansible/latest/user_guide/playbooks_loops.html#looping-over-the-inventory | 15:33 |
| corvus | Shrews: my understanding is that they are planning the deprecation now :) | 15:33 |
| pabelanger | Shrews: ++ | 15:33 |
| mordred | yah - I've been using loop: for the new stuff I've been writing for update-cfg-mgmt just to be future proof | 15:34 |
| mordred | we've got a while yet before we need to, you know, hard-migrate or anything | 15:34 |
| pabelanger | corvus: containers should have a nodepool.provider / nodepool.label right? and interface_ip is likely optional | 15:44 |
| corvus | pabelanger: it was the ip that caught my attention, yes | 15:45 |
| pabelanger | ack | 15:46 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: Add nodepool info to test-emit-job-header https://review.openstack.org/557947 | 15:46 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: Add nodepool info to test-emit-job-header https://review.openstack.org/557947 | 15:51 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: Add nodepool info to test-emit-job-header https://review.openstack.org/557947 | 15:57 |
| pabelanger | corvus: mordred: clarkb: Shrews: ^ update for condition check and new loop syntax | 16:01 |
| pabelanger | https://logs.rdoproject.org/15/15415/4/check/tox-docs/2ce861f/job-output.txt.gz#_2018-08-10_15_59_21_231353 is output | 16:01 |
| clarkb | TIL there are three different ways to do that lookup that are identical | 16:03 |
| clarkb | actually, 4 | 16:04 |
| pabelanger | first time using query for me | 16:05 |
| corvus | pabelanger: has your base job rework happened yet (is that being dynamically tested)? | 16:05 |
| pabelanger | oh, query is same as lookup, except it will return list | 16:05 |
| pabelanger | corvus: not for zuul.o.o yet, but have done it in rdoproject, how we can test | 16:06 |
| clarkb | with_inventory_hostnames, lookup('inventory_hostnames', foo, wantlist=True), query('inventory_hostnames' and q('inventory_hostnames'. But now I have learned about ansible loops | 16:06 |
| pabelanger | https://review.rdoproject.org/r/15415 was patch in rdo | 16:06 |
| corvus | pabelanger: so that's a live test of the zuul-jobs change in rdo? | 16:07 |
| pabelanger | corvus: yes | 16:08 |
| corvus | pabelanger: (a) that is awesome. (b) that is sufficient testing for me. +3 :) | 16:08 |
| pabelanger | yay | 16:08 |
| tobiash | corvus: I'm +3 with comments on 589975 | 16:17 |
| *** jpena is now known as jpena|off | 16:22 | |
| corvus | tobiash: okay. i'd be okay with those changes; i just didn't want to reach beyond my comfort area | 16:22 |
| tobiash | corvus: I think I can do these changes together with cache update on pr event after my vacation | 16:24 |
| tobiash | corvus: I'll be officially back on 3rd of september | 16:30 |
| tobiash | but I might be doing some reviews from time to time | 16:30 |
| clarkb | tobiash: any concern that github could allow a branch to be created as protected? | 16:31 |
| clarkb | as a future change? might be worthwhile being cautious in zuul about that? | 16:31 |
| tobiash | clarkb: I think the api cannot do that | 16:32 |
| tobiash | a git push to a new branch must be unprotected by definition | 16:32 |
| clarkb | tobiash: the git push yes, but as mordred pointed out you can also create them via the web ui? | 16:32 |
| tobiash | the only method would be to allow branch creates in the ui to be protected from the beginning | 16:32 |
| tobiash | I don't think that fits into githubs usage model as branch protection settings are pretty complex | 16:34 |
| mordred | yah. and even creating through the webui it's a simple "create" button - there's no options on it | 16:34 |
| tobiash | and as long as github doesn't deliver branch protection events our only option to add new protected branches on the fly is to get that information on events related to this branch and update the cache accordingly | 16:35 |
| tobiash | like pr updated should later check the branch protection and update the cache + eventually trigger a tenant reconfig before delivering the event to the scheduler loop | 16:36 |
| tobiash | clarkb: when we have that it actually doesn't matter if we add a new branch as protected or unprotected to the cache as it gets updated with real information before usage | 16:37 |
| openstackgerrit | Merged openstack-infra/zuul master: Cache branches in connections/sources https://review.openstack.org/589975 | 16:39 |
| clarkb | ya | 16:40 |
| *** panda|ruck is now known as panda|ruck|off | 16:48 | |
| *** gtema has quit IRC | 16:49 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Tolerate missing project https://review.openstack.org/579872 | 16:52 |
| mordred | pabelanger, tristanC, corvus, tobiash: I just left a comment on https://review.openstack.org/#/c/590092 - which just happened to coincide with a thing I was pondering anyway | 17:31 |
| corvus | mordred: does the ssh key stuff described in http://specs.openstack.org/openstack-infra/infra-specs/specs/zuulv3.html#continuous-deployment help? | 17:36 |
| mordred | corvus: oh - yeah - it does | 17:37 |
| corvus | mordred: though, probably something something config-projects or post-review or something :) | 17:38 |
| pabelanger | reading | 17:38 |
| mordred | corvus: we could them put the git.openstack.org/openstack-infra/system-config public key into the zuul user on bridge.openstack.org | 17:38 |
| mordred | and simply allowing add_host: to be used by users would complete the circle | 17:39 |
| mordred | corvus: but yes, also something something config-projects / post-reivew or something :) | 17:39 |
| corvus | mordred: yep. and as long as that key is only used in a trusted context, that should be safe | 17:39 |
| mordred | ++ | 17:39 |
| corvus | what was the hangup on add_host again? | 17:39 |
| mordred | there wasn't one | 17:39 |
| mordred | we just need to unrestrict it | 17:39 |
| corvus | i think pabelanger pointed out an issue in email? | 17:40 |
| mordred | but I didn't get around to doing that yet because of the ssh key | 17:40 |
| mordred | oh? /me goes to read | 17:40 |
| pabelanger | I completely for got about http://specs.openstack.org/openstack-infra/infra-specs/specs/zuulv3.html#continuous-deployment ++ | 17:40 |
| corvus | oh the ssh key issue *was* the hangup? | 17:40 |
| mordred | yup | 17:40 |
| corvus | http://lists.zuul-ci.org/pipermail/zuul-discuss/2018-June/000458.html | 17:41 |
| corvus | yes that was it :) | 17:41 |
| mordred | so - it seems if we implement that part of the spec, we'll be good to do the thing that that part of the spec was writen to allow us to do | 17:41 |
| corvus | mordred: \o/ | 17:41 |
| corvus | i actually don't think it'll be that hard at this point. the basic patterns for almost all of that have been established (zuul generating and serving keys, trusted contexts, ssh-agent actions) | 17:42 |
| pabelanger | +1 | 17:43 |
| *** ianychoi has quit IRC | 18:10 | |
| *** ianychoi has joined #zuul | 18:11 | |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Add test-emit-job-header roles https://review.openstack.org/557946 | 18:16 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Add nodepool info to test-emit-job-header https://review.openstack.org/557947 | 18:16 |
| *** pcaruana has quit IRC | 19:19 | |
| fungi | anybody happen to have a foot in the door with https://www.aswf.io/community/ ? | 20:09 |
| fungi | seems very ci/cd focused | 20:09 |
| fungi | another fine product of the linux foundation | 20:09 |
| fungi | the tie-in with the academy of motion picture arts and sciences eludes me | 20:10 |
| pabelanger | never heard of it, seems to have launched today | 20:16 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Map file comment line numbers (alternate) https://review.openstack.org/590442 | 20:22 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Map file comment line numbers https://review.openstack.org/590442 | 20:23 |
| mordred | fungi: so - I was confused at first - but I think teh aswf is about open source software associated with motion picture making: https://www.aswf.io/about/ | 20:29 |
| fungi | agreed | 20:30 |
| mordred | fungi: and that community page is just indicating that one of the pieces of their dev community is a shared ci infrastruture for their projects | 20:30 |
| * mordred originally thougth that the aswf was a new CI/CD project and was super confused about the name | 20:30 | |
| fungi | they seem quite invested in ci/cd though, which is quite cool | 20:30 |
| mordred | ++ | 20:30 |
| *** samccann has quit IRC | 20:31 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Map file comment line numbers https://review.openstack.org/590442 | 20:48 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Fix web content copying in multi dashboard job https://review.openstack.org/591089 | 21:17 |
| mordred | clarkb, fungi: ^^ that fixes an issue seen here: https://review.openstack.org/#/c/586552/ | 21:17 |
| mordred | the path exclusions on the change to do file copying caused that change to not be self-testing | 21:18 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Map file comment line numbers https://review.openstack.org/590442 | 21:21 |
| corvus | "TypeError: expected string or bytes-like object" would be even more helpful if it also included "...but received $something" | 22:27 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Map file comment line numbers https://review.openstack.org/590442 | 22:42 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Fix race in test_crd_check_unknown https://review.openstack.org/591106 | 22:42 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!