| *** jamesmcarthur has joined #zuul | 00:08 | |
| *** jamesmcarthur has quit IRC | 00:18 | |
| *** jamesmcarthur has joined #zuul | 00:19 | |
| *** jamesmcarthur has quit IRC | 00:25 | |
| *** jamesmcarthur has joined #zuul | 00:52 | |
| *** jamesmcarthur has quit IRC | 00:58 | |
| *** jamesmcarthur has joined #zuul | 01:28 | |
| *** jamesmcarthur has quit IRC | 01:33 | |
| *** jamesmcarthur has joined #zuul | 01:45 | |
| *** jamesmcarthur has quit IRC | 01:49 | |
| *** jamesmcarthur has joined #zuul | 02:12 | |
| *** jamesmcarthur has quit IRC | 02:17 | |
| *** irclogbot_3 has quit IRC | 02:26 | |
| *** irclogbot_0 has joined #zuul | 02:30 | |
| *** jamesmcarthur has joined #zuul | 02:35 | |
| *** jamesmcarthur has quit IRC | 03:05 | |
| *** rlandy|bbl is now known as rlandy | 03:14 | |
| *** bhavikdbavishi has joined #zuul | 03:16 | |
| *** rlandy has quit IRC | 03:20 | |
| *** jamesmcarthur has joined #zuul | 03:26 | |
| *** bhavikdbavishi has quit IRC | 03:45 | |
| *** bhavikdbavishi has joined #zuul | 03:45 | |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul master: Add nodesets API route https://review.opendev.org/605877 | 03:48 |
|---|---|---|
| *** jamesmcarthur has quit IRC | 03:48 | |
| *** jamesmcarthur has joined #zuul | 03:56 | |
| *** bhavikdbavishi has quit IRC | 04:06 | |
| *** bhavikdbavishi has joined #zuul | 04:07 | |
| *** jamesmcarthur has quit IRC | 04:13 | |
| *** jamesmcarthur has joined #zuul | 04:40 | |
| *** jamesmcarthur has quit IRC | 04:51 | |
| *** swest has joined #zuul | 04:54 | |
| *** jamesmcarthur has joined #zuul | 04:57 | |
| *** swest has quit IRC | 04:59 | |
| *** swest has joined #zuul | 05:00 | |
| *** jamesmcarthur has quit IRC | 05:04 | |
| *** jamesmcarthur has joined #zuul | 05:12 | |
| openstackgerrit | Felix Schmidt proposed zuul/zuul master: Differentiate between queued and waiting jobs in zuul web UI https://review.opendev.org/660878 | 05:12 |
| *** raukadah is now known as chandankumar | 05:25 | |
| *** saneax has joined #zuul | 05:28 | |
| *** zbr has joined #zuul | 05:34 | |
| openstackgerrit | Felix Schmidt proposed zuul/zuul master: Differentiate between queued and waiting jobs in zuul web UI https://review.opendev.org/660878 | 05:37 |
| *** jamesmcarthur has quit IRC | 05:57 | |
| *** jamesmcarthur has joined #zuul | 05:58 | |
| openstackgerrit | Felix Schmidt proposed zuul/zuul master: Differentiate between queued and waiting jobs in zuul web UI https://review.opendev.org/660878 | 06:21 |
| *** themroc has joined #zuul | 06:54 | |
| *** jamesmcarthur has quit IRC | 06:54 | |
| *** bjackman has joined #zuul | 07:01 | |
| *** themroc has quit IRC | 07:05 | |
| *** pcaruana has joined #zuul | 07:07 | |
| openstackgerrit | Fabien Boucher proposed zuul/zuul master: Pagure driver - https://pagure.io/pagure/ https://review.opendev.org/604404 | 07:12 |
| *** rf0lc0 has joined #zuul | 07:23 | |
| *** rfolco has quit IRC | 07:26 | |
| *** themroc has joined #zuul | 07:30 | |
| *** jpena|off is now known as jpena | 07:45 | |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: [wip] zone file vaildator https://review.opendev.org/660916 | 07:45 |
| *** gtema has joined #zuul | 07:48 | |
| *** bjackman has quit IRC | 07:48 | |
| *** bjackman has joined #zuul | 07:50 | |
| openstackgerrit | Markus Hosch proposed zuul/zuul master: Evaluate CODEOWNERS settings during canMerge check https://review.opendev.org/644557 | 07:50 |
| openstackgerrit | Merged zuul/zuul-jobs master: Switch to TOX_CONSTRAINTS_FILE https://review.opendev.org/657886 | 07:51 |
| *** themroc has quit IRC | 07:57 | |
| *** themroc has joined #zuul | 08:03 | |
| *** saneax has quit IRC | 08:07 | |
| *** saneax has joined #zuul | 08:08 | |
| *** bhavikdbavishi has quit IRC | 08:11 | |
| *** hashar has joined #zuul | 08:21 | |
| *** sanjayu_ has joined #zuul | 08:27 | |
| *** saneax has quit IRC | 08:29 | |
| *** themroc has quit IRC | 08:32 | |
| *** themroc has joined #zuul | 08:39 | |
| *** panda|rover has quit IRC | 08:40 | |
| *** themroc has quit IRC | 08:42 | |
| *** panda has joined #zuul | 08:42 | |
| openstackgerrit | Felix Schmidt proposed zuul/zuul master: Report job depedencies via MQTT reporter https://review.opendev.org/660940 | 08:44 |
| openstackgerrit | Felix Schmidt proposed zuul/zuul master: Report job depedencies via MQTT reporter https://review.opendev.org/660940 | 08:47 |
| *** themroc has joined #zuul | 08:48 | |
| *** bhavikdbavishi has joined #zuul | 08:49 | |
| *** jamesmcarthur has joined #zuul | 08:51 | |
| *** jangutter has joined #zuul | 08:54 | |
| *** jamesmcarthur has quit IRC | 08:55 | |
| *** bhavikdbavishi1 has joined #zuul | 08:59 | |
| *** bhavikdbavishi has quit IRC | 08:59 | |
| *** bhavikdbavishi1 is now known as bhavikdbavishi | 08:59 | |
| *** themroc has quit IRC | 09:02 | |
| *** jamesmcarthur has joined #zuul | 09:04 | |
| *** themroc has joined #zuul | 09:06 | |
| *** panda is now known as panda|rover | 09:11 | |
| openstackgerrit | Felix Schmidt proposed zuul/zuul master: Report job depedencies via MQTT reporter https://review.opendev.org/660940 | 09:13 |
| *** themroc has quit IRC | 09:15 | |
| *** gtema has quit IRC | 09:15 | |
| *** gtema has joined #zuul | 09:16 | |
| *** themroc has joined #zuul | 09:17 | |
| *** electrofelix has joined #zuul | 09:38 | |
| *** jamesmcarthur has quit IRC | 09:44 | |
| *** jamesmcarthur has joined #zuul | 09:45 | |
| *** fbo has joined #zuul | 09:49 | |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul master: trigger: add job filter event https://review.opendev.org/639905 | 09:54 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul master: webtrigger: add initial driver and event https://review.opendev.org/555153 | 09:54 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul master: webtrigger: add web route and rpclistener https://review.opendev.org/554839 | 09:54 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul master: web: add build button to trigger job https://review.opendev.org/635716 | 09:54 |
| openstackgerrit | Felix Schmidt proposed zuul/zuul master: Show retry attempts in UI https://review.opendev.org/660960 | 09:57 |
| *** jamesmcarthur has quit IRC | 09:58 | |
| *** jamesmcarthur has joined #zuul | 10:10 | |
| *** electrofelix has quit IRC | 10:12 | |
| *** electrofelix has joined #zuul | 10:13 | |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: Zone file validation role https://review.opendev.org/660916 | 10:15 |
| *** jamesmcarthur has quit IRC | 10:16 | |
| *** jamesmcarthur has joined #zuul | 10:21 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/zone-zuul-ci.org master: Add zone-check job https://review.opendev.org/660967 | 10:22 |
| *** jamesmcarthur has quit IRC | 10:26 | |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: Zone file validation role https://review.opendev.org/660916 | 10:32 |
| *** jamesmcarthur has joined #zuul | 10:39 | |
| *** paladox has quit IRC | 10:41 | |
| *** jamesmcarthur has quit IRC | 10:43 | |
| *** paladox has joined #zuul | 10:45 | |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Log github requests with annotated events https://review.opendev.org/660800 | 10:47 |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Annotate logs around build completion and cancellation https://review.opendev.org/660806 | 10:47 |
| *** jamesmcarthur has joined #zuul | 10:48 | |
| *** jamesmcarthur has quit IRC | 10:52 | |
| openstackgerrit | Felix Schmidt proposed zuul/zuul master: Report job depedencies via MQTT reporter https://review.opendev.org/660940 | 10:57 |
| *** gtema has quit IRC | 10:57 | |
| *** gtema has joined #zuul | 10:58 | |
| *** AshBullock has joined #zuul | 11:00 | |
| *** jpena is now known as jpena|lunch | 11:01 | |
| openstackgerrit | Mark Meyer proposed zuul/zuul master: Create a basic Bitbucket event source https://review.opendev.org/658835 | 11:10 |
| openstackgerrit | Mark Meyer proposed zuul/zuul master: Upgrade formatting of the patch series. https://review.opendev.org/660683 | 11:10 |
| *** jamesmcarthur has joined #zuul | 11:16 | |
| AshBullock | Hey guys, I'm trying to get the logging working with zuul from scratch guide. On my base job I have a post logging playbook as described in the logging guide here: https://zuul-ci.org/docs/zuul-jobs/log-roles.html I'm getting this ssh error: http://paste.openstack.org/show/751980/ | 11:19 |
| AshBullock | This is when it's trying to upload the logs from the executor box to the logging box I have provisioned. I'm using the following secrets from the add-fileserver role as shown here: http://paste.openstack.org/show/751981/ and my playbook is shown here: http://paste.openstack.org/show/751982/ | 11:19 |
| AshBullock | The zuul executor is running as the zuul user, I created a zuul user on the logging box, is there any other step I am missing here? | 11:19 |
| AshBullock | I made sure both the fedora user and zuul user had the relevant authorised keys on the target logging box, and I can login manually via ssh using the private key from the secrets file | 11:21 |
| pabelanger | AshBullock: you may want to confirm your ssh_known_hosts attribute is correct on site_logs | 11:23 |
| pabelanger | wonder if you are getting hostkey mismatch | 11:23 |
| pabelanger | which will prevent ansible from connecting to remote node | 11:23 |
| pabelanger | or | 11:23 |
| pabelanger | dns isn't setup on zuul-executor | 11:23 |
| openstackgerrit | Flavio Percoco proposed zuul/zuul master: Read multiple *.conf files from directory https://review.opendev.org/660977 | 11:24 |
| pabelanger | and cannot resolve ip-10-xxx-xx-xx.eu-west-1.compute.internal | 11:24 |
| pabelanger | maybe try IP address first | 11:24 |
| *** AshBullock has quit IRC | 11:26 | |
| *** bhavikdbavishi has quit IRC | 11:29 | |
| *** sshnaidm|afk is now known as sshnaidm | 11:42 | |
| *** panda|rover is now known as panda|rover|eat | 11:59 | |
| tobiash | clarkb or pabelanger: want to review https://review.opendev.org/616306 (tenant and project specific resource usage stats)? | 12:10 |
| tobiash | that might be interesting to you as well | 12:11 |
| *** gtema has quit IRC | 12:17 | |
| *** jamesmcarthur has quit IRC | 12:18 | |
| *** jamesmcarthur has joined #zuul | 12:18 | |
| *** rf0lc0 is now known as rfolco | 12:18 | |
| *** rlandy has joined #zuul | 12:29 | |
| *** jpena|lunch is now known as jpena | 12:30 | |
| *** jamesmcarthur has quit IRC | 12:30 | |
| *** sanjayu_ has quit IRC | 12:30 | |
| *** flepied has joined #zuul | 12:31 | |
| *** AshBullock has joined #zuul | 12:34 | |
| pabelanger | tobiash: sure, will look shortly | 12:34 |
| pabelanger | tobiash: yah, looks extremely helpful! +2 | 12:39 |
| pabelanger | let clarkb get some eyes on it | 12:39 |
| tobiash | yupp | 12:39 |
| pabelanger | especially when you have to pay for nodepool resources :) | 12:39 |
| *** gtema has joined #zuul | 12:47 | |
| *** jamesmcarthur has joined #zuul | 12:52 | |
| *** panda|rover|eat is now known as panda|rover | 13:00 | |
| *** sanjayu_ has joined #zuul | 13:01 | |
| *** jamesmcarthur has quit IRC | 13:05 | |
| *** bjackman has quit IRC | 13:05 | |
| *** bjackman_ has joined #zuul | 13:05 | |
| openstackgerrit | Fabien Boucher proposed zuul/zuul master: Set the DiskAccountant log when size > self.limit as warning level https://review.opendev.org/660996 | 13:12 |
| fbo | to ease grep usage in executor logs ^ | 13:17 |
| *** gtema has quit IRC | 13:27 | |
| *** bjackman_ has quit IRC | 13:41 | |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: WIP: Fix case sensitivity in codeowners check https://review.opendev.org/661003 | 13:43 |
| openstackgerrit | Merged zuul/zuul-jobs master: Zone file validation role https://review.opendev.org/660916 | 13:58 |
| *** yolanda__ has joined #zuul | 14:02 | |
| *** yolanda__ is now known as yolanda | 14:03 | |
| *** pcaruana has quit IRC | 14:10 | |
| *** AshBullock has quit IRC | 14:27 | |
| *** pcaruana has joined #zuul | 14:29 | |
| *** gtema has joined #zuul | 14:32 | |
| mordred | flaper87: left comment on https://review.opendev.org/#/c/660977 | 14:57 |
| *** sanjayu_ has quit IRC | 14:58 | |
| pabelanger | tobiash: do you have any thoughts on maybe update our quota checks in nodepool to be per nodeset, not just a node? The issue I am seeing, since we have pretty small quota available in our nodepool regions, is when we do multi-node testing, we check provider for quota but we end up only launch 1 or 2 nodes. And the ready node is locked, for lets say 1 hours until there is more room to launch the 2nd node in the | 14:58 |
| pabelanger | nodeset. | 14:58 |
| fungi | yeah, that seems like an inefficient use of resources | 14:59 |
| pabelanger | This is more an optimization I think of the nodes, since that ready locked node could service another single nodeset | 14:59 |
| pabelanger | fungi: yah, my thoughts too. | 14:59 |
| pabelanger | for now, it isn't a huge deal | 14:59 |
| fungi | deferring a larger request which won't be fulfilled right away to that smaller requests can be served in the meantime would be tough though, because you're liable to just end up never running the jobs with the larger requests | 14:59 |
| pabelanger | but something nice to have | 14:59 |
| fungi | we should probably recommend a minimum quota of 2x the size of the nodeset max size limit | 15:01 |
| fungi | or something along those lines | 15:01 |
| pabelanger | fungi: right, or in this case, we have more regions that do have the quota, but nodepool doesn't know about that. IIRC, the launcher won't ask if quota is available in another region right now, and release the request to it | 15:01 |
| fungi | ahh, yeah i suppose that could be an alternative to solve the deadlock | 15:01 |
| pabelanger | I also think, the launcher is too aggressive at grabbing nodesets (just a guess), I was thinking we maybe could do what we do in zuul-executor when a new request comes online. That way another launcher has an option to try the node-request. Often, if a project has multiple multi-node jobs, they seem to all be run in the same region. I understand that is needed for the nodeset, but a group of nodesets could | 15:03 |
| pabelanger | be split up more | 15:03 |
| pabelanger | some sort of back-off | 15:03 |
| pabelanger | Shrews: you likely have thoughts too^ | 15:06 |
| Shrews | those ideas require fundamental changes to how nodepool works. basically a redesign | 15:07 |
| Shrews | all great ideas, but not easily hackable into the current system | 15:08 |
| flaper87 | mordred: thanks... totally missed the return | 15:16 |
| *** chandankumar is now known as raukadah | 15:17 | |
| *** jamesmcarthur has joined #zuul | 15:21 | |
| *** jamesmcarthur_ has joined #zuul | 15:32 | |
| *** jamesmcarthur has quit IRC | 15:36 | |
| *** hashar has quit IRC | 15:42 | |
| *** themroc has quit IRC | 15:44 | |
| mordred | Shrews: jfdi | 15:46 |
| mordred | Shrews: (that was me making a joke about the people who like to ignore design issues in favor of Writing Moar Code - but I just realized it could have come across without the humor) | 15:48 |
| clarkb | pabelanger: Shrews what we likely can do with less major redesign is grab request, notice we would block, release the request without adding current provider to the tried list, then block | 15:52 |
| clarkb | that would allow other providers to service it but the workflow in nodepool doesn't fundamentally change. | 15:53 |
| fungi | though could end up indefinitely starving larger requests if there is a constant stream of smaller ones | 15:56 |
| clarkb | if we process them serially I don't think it would | 15:58 |
| clarkb | though there is a race there if you move ahead because one provider has the lock on a request that it eventually gives us | 15:58 |
| clarkb | s/gives us/gives up/ | 15:58 |
| clarkb | This is where the request proxy idea I've had would help a lot but that is far bigger of a design change | 15:59 |
| clarkb | basically don't have requests get directly handled by providers. Instead have them get delegated to providers as necessary by an intermediary | 15:59 |
| clarkb | this would allow us to fifo requests always giving the next node of type foo to the next request that needs type foo nodes | 16:00 |
| *** tosky has joined #zuul | 16:00 | |
| *** pcaruana has quit IRC | 16:09 | |
| *** gtema has quit IRC | 16:13 | |
| *** mattw4 has joined #zuul | 16:16 | |
| pabelanger | clarkb: yah, I think that would also work | 16:28 |
| pabelanger | also, just hit https://zuul-ci.org/docs/zuul/admin/components.html#attr-executor.disk_limit_per_job for first time | 16:29 |
| pabelanger | time to double it | 16:29 |
| clarkb | pabelanger: keep in mind that limit is there to prevent jobs from creating many gigs of logs/artificats | 16:30 |
| clarkb | you might have a runaway job | 16:30 |
| pabelanger | clarkb: yah, in this case, it is ARA | 16:30 |
| pabelanger | because were generating static html to upload to swift | 16:31 |
| pabelanger | we have something like 40+ plays that ansible-test is doing | 16:31 |
| pabelanger | so I suspect talking up a bit of space | 16:31 |
| pabelanger | I'd guess none of that data is compress coming back to zuul-executor | 16:31 |
| pabelanger | should see if I can fix that | 16:32 |
| corvus | pabelanger: if you change that, it will be a little tricky in that we want swift to deal with log files in their original format (though compressed in storage and transit) | 16:33 |
| pabelanger | yah | 16:33 |
| pabelanger | upping to 500MB shouldn't be a huge issue ATM | 16:34 |
| corvus | (in other words, we want swift to have an object called "log.txt" with content-type text/plain and content-transfer-encoding gzip, not "log.txt.gz" with content-type application/gzip) | 16:35 |
| clarkb | fungi: Shrews pabelanger simpler way to deal with the large requests starvation may be to record required quota for request that will cause provider to block, unlock the request, block then only unblock when you have space for that many resources? | 16:35 |
| corvus | (the current system of keeping things uncompressed until log upload is the easiest way of doing that) | 16:35 |
| clarkb | this could still haev races depending on the state of the requests db when you unlock but at least you should have enough quota to serve a request at least as large as the one that would've caused you to block | 16:35 |
| pabelanger | corvus: do you have major concerns of having the nodes do the upload directly to swift? over executor? I know we'd have to share the password with the node | 16:36 |
| clarkb | I don't think you want those creds on the test node | 16:36 |
| pabelanger | yah, that's the downside | 16:37 |
| *** mgoddard has quit IRC | 16:38 | |
| tjgresha | nodepool questions appropriate on this IRC channel? | 16:39 |
| *** mgoddard has joined #zuul | 16:39 | |
| clarkb | tjgresha: yes | 16:40 |
| tjgresha | so i told nodepool to keep min ready 10 - to test my hooks to my openstack provider.. so it runs the dib, uploads the image, then it spawns instances and as soon as they are done, deletes them | 16:43 |
| clarkb | tjgresha: as soon as what is done? | 16:44 |
| tjgresha | sorry soon as instances are in ready state | 16:44 |
| pabelanger | sounds like nodepool can't reach them via SSH? | 16:45 |
| pabelanger | maybe check your launcher debug log | 16:45 |
| *** zbr has quit IRC | 16:45 | |
| tjgresha | exception that is being thrown is "instance set to active before it has an IP" -- but i wasn't sure if that was a red herring or is it was like pabelanger suggested -ssh issue | 16:46 |
| tjgresha | will NP tear down an instance immediately if it can't SSH? | 16:47 |
| clarkb | tjgresha: not immediately. It should try to connect via ssh repeatedly up to a timeout. However if the instance is active without an IP that might be a short circuit failure | 16:47 |
| pabelanger | yah, maybe also attaching to incorrect network? | 16:48 |
| tjgresha | its is instant teardown .. i tried adjusting the timeouts | 16:49 |
| tjgresha | just on the demo-net for now. | 16:49 |
| tobiash | mordred: fyi, I responded on 660977 | 16:50 |
| mordred | tobiash: thanks! | 16:53 |
| tjgresha | oddly i can see some of the instances are getting IP's and even floating - yet even those get torn down.. i will poke on the ssh keys for a bit and see if that changes things.. | 16:53 |
| * SpamapS did too | 16:54 | |
| *** tjgresha has quit IRC | 16:55 | |
| *** tjgresha has joined #zuul | 16:55 | |
| pabelanger | tjgresha: you might want to look at your provider security rules, maybe SSH is being blocked there | 16:56 |
| tjgresha | pabelanger - will double check that.. | 16:57 |
| SpamapS | tjgresha:Also there are a few options for nodepool that affect how networking is handled... it might make sense to tweak them. | 16:59 |
| SpamapS | https://zuul-ci.org/docs/nodepool/configuration.html#attr-providers.[openstack].pools.auto-floating-ip comes to mind | 17:00 |
| *** pcaruana has joined #zuul | 17:00 | |
| SpamapS | tjgresha: ^^ also https://zuul-ci.org/docs/nodepool/configuration.html#attr-providers.[openstack].pools.networks | 17:00 |
| *** AshBullock has joined #zuul | 17:01 | |
| AshBullock | Hey guys, thanks for the help earlier, I now have logs being sent from the zuul executor to the logging box, and this looks great. However the job-output.json file is blank for all jobs http://paste.openstack.org/show/751996/ | 17:02 |
| AshBullock | whereas the jobs-output.txt shows the whole task run as I'd expect. I expected the json file to show the verbose output of the run. Is there a setting I'm missing for the JSON file to get populated? Thanks in advance, again ! | 17:03 |
| clarkb | pabelanger: in write out psuedo code I've realized by idea won't solve your thing because I would still block until N node quota is available (to avoid starving larger requests) | 17:04 |
| clarkb | pabelanger: what my thing will do is allow other providers to pick up a request and they may not block | 17:04 |
| *** armstrongs has joined #zuul | 17:05 | |
| *** AshBullock has quit IRC | 17:07 | |
| clarkb | pabelanger: your thing needs the request intermediary so that it can do a straight fifo of all the nodes (though if you are single cloud provider then you may already effectively be that fifo) | 17:07 |
| *** tosky has quit IRC | 17:08 | |
| *** tosky has joined #zuul | 17:12 | |
| clarkb | ashbullock is gone, but I don't understand how that json file could be empty unless no plays were actually run or the v2_playbook_* callbacks aren't working for some reason | 17:15 |
| clarkb | zuul does take different paths if setting up ara callbacks | 17:18 |
| armstrongs | im in ash team | 17:18 |
| clarkb | however it appends the ara callbacks to the other callbacks path | 17:18 |
| armstrongs | so we are running playbooks and are using ansible 2.6 | 17:18 |
| *** jpena is now known as jpena|off | 17:19 | |
| armstrongs | its a very simple hello world debug command at this point | 17:19 |
| armstrongs | but no matter what we put the json output is empty | 17:20 |
| armstrongs | the txt one is populated | 17:20 |
| armstrongs | how would we debug the v2_playbook_* callbacks with zuul? | 17:21 |
| clarkb | hrm the callbacks are installed via symlinks to zuul/ansible/base/callbacks. The link points to a valid path in the zuul tree and the stream callback is in the same dir as the json callback so I don't think it is an installation problem | 17:21 |
| clarkb | mordred: tobiash ^ any ideas? | 17:22 |
| *** tosky has quit IRC | 17:22 | |
| tobiash | json output is broken for nodeless jobs | 17:22 |
| *** tosky__ has joined #zuul | 17:23 | |
| tobiash | there had been a fix recently | 17:23 |
| tobiash | https://review.opendev.org/659802 | 17:23 |
| clarkb | armstrongs: ^ I guess try updating to a zuul version that includes that commit? There isn't a release that includes that commit. Will have to install from git. | 17:24 |
| armstrongs | Zuul version: 3.8.2.dev49 f5d6b9bd | 17:25 |
| armstrongs | is what im on | 17:25 |
| clarkb | armstrongs: that is the commit before the one that should fix your issue | 17:26 |
| armstrongs | haha awesome | 17:26 |
| armstrongs | ok thanks so much again guys | 17:26 |
| *** armstrongs has quit IRC | 17:31 | |
| *** themroc has joined #zuul | 17:46 | |
| pabelanger | clarkb: ack | 18:09 |
| pabelanger | clarkb: having other providers try, could be an improvement | 18:10 |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Allow to select the merge method in GitHub https://review.opendev.org/617949 | 18:10 |
| pabelanger | tobiash: exciting^ | 18:11 |
| tobiash | pabelanger: that's an old change I need to dust off | 18:12 |
| tobiash | (and refactor) ;) | 18:12 |
| pabelanger | tobiash: does switching to say squash change how a dependent pipeline would work? or are we in the same boat as merge commits today | 18:13 |
| *** klindgren has quit IRC | 18:13 | |
| tobiash | pabelanger: I think it should work the same if we teach the merger how to do squash merge | 18:14 |
| *** klindgren has joined #zuul | 18:14 | |
| tobiash | pabelanger: that would be a follow-up to this then | 18:14 |
| pabelanger | okay, yah, that's what I was guessing too | 18:14 |
| tobiash | the merger already supports different merge modes and my plan is to leverage this and plumb through to the reporter | 18:14 |
| tobiash | afterwards the merger can learn additonal merge modes pretty easy | 18:15 |
| pabelanger | nice | 18:15 |
| pabelanger | I know there has been some ansible folks asking for squash, since that is what they are used too | 18:15 |
| tobiash | however I'm not sure if anyone uses different merge modes at all so not sure how well this is tested | 18:15 |
| pabelanger | would be cool thing to say zuul now supported :) | 18:16 |
| tobiash | jepp | 18:16 |
| *** rlandy is now known as rlandy|brb | 18:16 | |
| pabelanger | so, I kinda hate myself for asking this, but where did we land on adding a slack reporter to zuul? eg: Along side database / email. | 18:17 |
| pabelanger | in the mean time, I am going to poke at SpamapS role for slack | 18:18 |
| *** hashar has joined #zuul | 18:29 | |
| *** bjackman_ has joined #zuul | 18:34 | |
| *** bjackman_ has quit IRC | 18:41 | |
| *** rlandy|brb is now known as rlandy | 18:42 | |
| *** electrofelix has quit IRC | 18:47 | |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Allow to select the merge method in Github https://review.opendev.org/617949 | 19:06 |
| tobiash | pabelanger: this lays the foundation so we can implement rebase and squash-merge on top of that ^ | 19:07 |
| Shrews | corvus: I'm looking to add a test to zuul that just tests zk interaction (no need to start any daemons) but doesn't look like such unit tests currently exist. Is that correct, or am I blind? | 19:10 |
| corvus | Shrews: i think test_nodepool does that | 19:17 |
| SpamapS | pabelanger: Please do! It works great, we use it every day.. just never could get a real test going. | 19:17 |
| SpamapS | tobiash: zomg I *so* need squash merge. | 19:18 |
| SpamapS | our git history is... interesting. | 19:18 |
| fungi | does that make some attempt at reconciling the committer and author fields? | 19:19 |
| Shrews | corvus: ah so it does, but nodepool specific. Would you be opposed to a new test_zk.py unit test file for zk-only interactions? | 19:20 |
| Shrews | corvus: similar to what i have in nodepool | 19:20 |
| clarkb | fungi: I do know that if you do it by hand it lets the person doing the merge/squash edit things which I think would make you committer at least. Not sure who becomes committer/author when done via api | 19:23 |
| SpamapS | worth some experimentation and assertion in the docs | 19:30 |
| corvus | Shrews: go for it! :) | 19:33 |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Allow to select the merge method in Github https://review.opendev.org/617949 | 19:33 |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Allow to select the merge method in Github https://review.opendev.org/617949 | 19:35 |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Support squash merge in Github https://review.opendev.org/661096 | 19:40 |
| daniel2 | So it seems the way Zuul CI works is that github is suppose to send a webhook push to Zuul to trigger the tests? Is there a way to poll for changes with Zuul? | 19:40 |
| tobiash | SpamapS: that adds squash merge ^ | 19:40 |
| corvus | daniel2: no | 19:42 |
| fungi | the git driver could check for updates to a branch, but wouldn't trigger on pull requests | 19:46 |
| SpamapS | daniel2: it can also be triggered by a comment on a PR | 19:48 |
| SpamapS | but, yeah, no webhook, no workie. | 19:49 |
| SpamapS | There's a webhook proxy somewhere out there that can help if you have zuul behind draconian firewalls. | 19:49 |
| clarkb | github also publishes their IP ranges so you could punch firewall holes that are specific | 19:50 |
| pabelanger | Woah, if I am reading https://help.github.com/en/articles/repository-permission-levels-for-an-organization right, a new group for humans (not write access) to use labels | 19:58 |
| pabelanger | that's one more step to removing write access for humans on github repos | 19:58 |
| fungi | filthy humans. the machines will have their day | 20:00 |
| SpamapS | daniel2:https://github.com/BonnyCI/github-webhook-proxy | 20:01 |
| SpamapS | pabelanger:I did see that, and I'm quite happy about it. | 20:01 |
| pabelanger | SpamapS: yah, same. I've been asking every github human for this for a while, I am happy it has happened | 20:02 |
| *** themroc has quit IRC | 20:07 | |
| mordred | only 8 years after we asked the first time | 20:15 |
| pabelanger | ++ | 20:16 |
| *** jamesmcarthur_ has quit IRC | 20:16 | |
| *** tosky__ is now known as tosky | 20:18 | |
| daniel2 | SpamapS: https://github.com/stakater/GitWebhookProxy | 20:24 |
| daniel2 | This one seems more up to date | 20:24 |
| daniel2 | Just out of curiosity, is nodepool really only designed for OpenStack stuff? Could it be used to build say FreeBSD nodes? | 20:29 |
| mordred | daniel2: it works with openstack, ec2 and pre-built static nodes with support coming for gce and azure (hopefully) | 20:30 |
| mordred | daniel2: and we have people using it to boot windows vms and network appliance vms, so I see no reason it couldn't do freebsd | 20:30 |
| daniel2 | i guess its more of building freebsd nodes with disk image builder. | 20:30 |
| clarkb | I don't think anyone has added elements to do that, but I imagine it would work. Biggest gotcha is probably fs differences? | 20:31 |
| mordred | yah. so for that - there's still nothing stopping dib from making freebsd images - although I don't htink we have a freebsd base element | 20:31 |
| clarkb | so you may end up wanting to run dib on freebsd to build freebsd | 20:31 |
| clarkb | (because zfs and ufs on linux is weird) | 20:31 |
| mordred | that said - nodepool can boot vms in clouds from already-existing images | 20:31 |
| mordred | so you could use an existing freebsd image - or manually upload one and then just reference that from the nodepool config if you weren't interested in making a base freebsd element in dib | 20:32 |
| mordred | which is all to say - it should TOTALLY be possible - but none of us have done it :) | 20:32 |
| daniel2 | I might give it a try | 20:33 |
| clarkb | daniel2: at a high level the way dib works is to spin up a chroot then run-parts a bunch of elements' scripts in the chroot then end with an image conversion | 20:33 |
| clarkb | daniel2: I expect that with a few tweaks that would work for freebsd (at least running on freebsd too) | 20:33 |
| daniel2 | I might look into creating a port for nodepool and disk image builder for FreeBSD this weekend. | 20:34 |
| mordred | cool! | 20:34 |
| daniel2 | Probably do one for zuul too, it really shouldnt be hard because they are python applications. | 20:37 |
| daniel2 | That makes them extremely portable. | 20:37 |
| clarkb | yes, though zuul (and maybe nodepool) both use a select alternative that doesn't exist on OS X iirc | 20:37 |
| clarkb | that may be the other gotcha | 20:37 |
| clarkb | https://docs.python.org/3/library/select.html#select.poll | 20:39 |
| clarkb | internet seems to imply that actually works on freebsd but not osx | 20:40 |
| clarkb | so may be fine | 20:40 |
| daniel2 | clarkb: I wonder if some modifications can be done to use kqueue. | 20:40 |
| openstackgerrit | David Shrewsbury proposed zuul/zuul master: WIP: Store hold requests in zookeeper https://review.opendev.org/661114 | 20:40 |
| daniel2 | or kevent, im not looking much into it right this moment | 20:40 |
| clarkb | if select.poll works I think it is probably fine as is | 20:41 |
| daniel2 | This should be fun. Now I'm invested, I'm going to create FreeBSD ports (or at least try) for Zuul/Nodepool and disk image-builder | 20:41 |
| daniel2 | If anything it will be a good learning experience. | 20:42 |
| fungi | also running a zuul executor on freebsd could be "interesting" as i expect bubblewrap is linux-specific so you may need to do something with a jail instead if you want to have similar protections around ansible forks | 20:47 |
| *** pcaruana has quit IRC | 20:47 | |
| daniel2 | fungi: I could possibly tie it into the linux emulation | 20:48 |
| fungi | i've been out of touch with the state of linux emulation in *bsd, didn't realize it had grown support for things like network namespaces and cgroups | 20:59 |
| daniel2 | fungi: I'm not entirely sure, which is why I said "possibly" | 21:00 |
| SpamapS | I doubt diskimage-builder would work on freebsd without *heavy* modification. | 21:03 |
| SpamapS | Unless they have the same loopback manipulation tools. | 21:04 |
| fungi | loopback filesystem support is basically the same, only the names of the utilities/options and virtual devices are different | 21:05 |
| fungi | er, loopback block storage i mean | 21:05 |
| SpamapS | That's what I mean. losetup, etc. etc... lots of work to do | 21:08 |
| SpamapS | whoa, they've been busy | 21:08 |
| SpamapS | https://opendev.org/openstack/diskimage-builder/src/branch/master/diskimage_builder/block_device/plugin.py | 21:08 |
| SpamapS | https://opendev.org/openstack/diskimage-builder/src/branch/master/diskimage_builder/block_device/level0/localloop.py probably just need to cargo-cult that | 21:09 |
| *** hashar has quit IRC | 21:21 | |
| *** flepied has quit IRC | 22:57 | |
| tristanC | corvus: re: having a base job create a containers on a static host so that untrusted job run in a restricted environment similar to the nodepool runc driver (i propose we call that feature pledge like the openbsd syscall) | 22:58 |
| *** panda|rover has quit IRC | 22:59 | |
| tristanC | it seems like we can almost do that with https://review.opendev.org/590092 by having the job start with no nodeset | 22:59 |
| *** panda has joined #zuul | 22:59 | |
| tristanC | however the issue i'm facing is taking care of abort/cancel operation that may leak contexts over time... | 23:00 |
| tristanC | would it be possible to have a cleanup action performed before re-using a static node? | 23:00 |
| clarkb | tristanC: there has been talk of adding a cleanup run phase that always runs (before post? maybe after post?) | 23:02 |
| *** tosky has quit IRC | 23:05 | |
| *** rlandy has quit IRC | 23:06 | |
| tristanC | clarkb: it seems like abort/cancel operation are fire-and-forget in zuul model. Wouldn't it be more simple to implement it in nodepool side when doing the "in-used" -> "ready" transition? | 23:06 |
| clarkb | tristanC: we removed all of the functionality in nodepool that "managed" instances internally | 23:06 |
| clarkb | tristanC: because the idea was zuul could do it. Zuul mostly can except in the abort case | 23:07 |
| SpamapS | I'd like to see it as a distinct phase in zuul buildsets. | 23:07 |
| SpamapS | Ideally, zuul would always run a cleanup job after any buildset ended. | 23:07 |
| SpamapS | That's not just for static nodes, it helps with anything where you need to allocate resources during jobs. | 23:08 |
| clarkb | SpamapS: it will have to be per job not buildset due to job retries | 23:08 |
| clarkb | (could possibly also have per buildset cleanups) | 23:08 |
| SpamapS | For instance, I'd like to spin up an entire EKS cluster in a zuul job, to test the stuff that initializes them. I want to be 100% sure that EKS cluster is always killed, no matter what job that runs after the spin up of the cluster fails. | 23:08 |
| SpamapS | The other path to that, is the idea of an ansible nodepool provider, which could do that. | 23:09 |
| clarkb | SpamapS: that could still be a job cleanup action in the paused job | 23:09 |
| clarkb | (doesn't need to be buildset) | 23:09 |
| SpamapS | clarkb:yeah I don't fully understand paused jobs yet. | 23:09 |
| SpamapS | is that spelled out? Maybe that's all I want. | 23:09 |
| clarkb | I think it is documented /me looks | 23:10 |
| clarkb | https://zuul-ci.org/docs/zuul/user/jobs.html#pausing-the-job | 23:10 |
| corvus | what's still missing for that use-case is something that runs if the job (the parent job -- the paused job) is aborted, say, due to a new patchset. that's what a 'cleanup' phase would do i think. | 23:11 |
| clarkb | yup we need a new run phase to properly close that gap | 23:12 |
| clarkb | (but it doesn't need to be per buildset and I think it does need to be per job to handle retries) | 23:12 |
| corvus | agree | 23:13 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul master: encrypt_secret: display the full_url on error https://review.opendev.org/661134 | 23:18 |
| tristanC | great, iiuc, a "cleanup" job attribute with a playbook that is always executed would be great for the job pledge feature | 23:21 |
| corvus | i think that will get us 99% of the way there, with a small hole involving executor failure -- but we can probably close that with the zuulv4/everything-in-zookeeper work | 23:25 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: validate-zone-db : add job and make more generic https://review.opendev.org/661138 | 23:43 |
| SpamapS | corvus:the 1% have enough money to solve it themselves. | 23:45 |
| SpamapS | ;) | 23:45 |
| SpamapS | wow | 23:45 |
| SpamapS | this pause thing | 23:45 |
| SpamapS | how did I miss that? | 23:45 |
| SpamapS | I've been trying to bend a bunch of jobs into weird places and all I needed was that. ;) | 23:46 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: validate-zone-db : add job and make more generic https://review.opendev.org/661138 | 23:49 |
| *** mattw4 has quit IRC | 23:50 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!