| @cidlik:matrix.org | Hello there! | 06:31 |
|---|---|---|
| I want to run Zuul jobs for testing embedded devices which are not available directly on SSH by Zuul. But I had a face with some problems. I described them more complete in the mail which was sent to zuul-discuss@lists.zuul-ci.org but I don't sure that the mail was reviewed by moderator. The header of the mail is "Zuul and unreachable devices". Can someone check it please? | ||
| @cidlik:matrix.org | * Hello there! | 06:33 |
| I want to run Zuul jobs for testing embedded devices which are not available directly on SSH by Zuul. But I had a face with some problems. I described them more complete in the mail which was sent to zuul-discuss@lists.zuul-ci.org but I'm afraid the the letter may get lost. The header of the mail is "Zuul and unreachable devices". Can someone check it please? | ||
| @cidlik:matrix.org | * Hello there! | 06:33 |
| I want to run Zuul jobs for testing embedded devices which are not available directly on SSH by Zuul. But I had a face with some problems. I described them more complete in the mail which was sent to zuul-discuss@lists.zuul-ci.org, but I'm afraid the the letter may get lost. The header of the mail is "Zuul and unreachable devices". Can someone check it please? | ||
| @cidlik:matrix.org | * Hello there! | 06:34 |
| I want to run Zuul jobs for testing embedded devices which are not available directly on SSH by Zuul. But I had a face with some problems. I described them more complete in the mail which was sent to zuul-discuss@lists.zuul-ci.org, but I'm afraid the the mail may get lost. The header of the mail is "Zuul and unreachable devices". Can someone check it please? | ||
| @cidlik:matrix.org | * Hello there! | 06:34 |
| I want to run Zuul jobs for testing embedded devices which are not available directly on SSH by Zuul. But I had a face with some problems. I described them more complete in the mail which was sent to zuul-discuss@lists.zuul-ci.org, but I'm afraid the the mail can get lost. The header of the mail is "Zuul and unreachable devices". Can someone check it please? | ||
| @flaper87:matrix.org | Morning folks! reaching out to see if someone has faced this issue and have recommendations on how to debug it better. | 07:55 |
| For a while, we started seeing more "MODULE ERRORS" in the logs, which I believe refers to an error running an ansible module/task. Correct? | ||
| I'm a bit unsure on what causes these errors an what the best way to debug them is. It happens for various tasks (so not 1 specific task) | ||
| @flaper87:matrix.org | > <@flaper87:matrix.org> Morning folks! reaching out to see if someone has faced this issue and have recommendations on how to debug it better. | 08:25 |
| > | ||
| > For a while, we started seeing more "MODULE ERRORS" in the logs, which I believe refers to an error running an ansible module/task. Correct? | ||
| > | ||
| > I'm a bit unsure on what causes these errors an what the best way to debug them is. It happens for various tasks (so not 1 specific task) | ||
| Some more context here: Jobs are running in Kubernetes PODs, nodes should have enough resources (trying to exclude podd re-scheduling or something like that) and I've noticed this behavior on both spot and non-spot nodes | ||
| @flaper87:matrix.org | 2024-10-16 08:10:00,434 DEBUG zuul.AnsibleJob.output: [e: 66b826e0-8b94-11ef-8b79-1633a44cbff6] [build: 63489bb7deb44c0ebfa155b659fc162d] Ansible output: b'[WARNING]: Failure using method (v2_runner_item_on_ok) in callback plugin' | 08:35 |
| @flaper87:matrix.org | just noticed that error message, which is just a warning but may be causing issues | 08:35 |
| @flaper87:matrix.org | * 2024-10-16 08:10:00,434 DEBUG zuul.AnsibleJob.output: [e: 66b826e0-8b94-11ef-8b79-1633a44cbff6] [build: 63489bb7deb44c0ebfa155b659fc162d] Ansible output: b'[WARNING]: Failure using method (v2_runner_item_on_ok) in callback plugin' | 08:39 |
| 2024-10-16 08:10:00,435 DEBUG zuul.AnsibleJob.output: [e: 66b826e0-8b94-11ef-8b79-1633a44cbff6] [build: 63489bb7deb44c0ebfa155b659fc162d] Ansible output: b'(<ansible.plugins.callback.zuul_stream.CallbackModule object at' | ||
| 2024-10-16 08:10:00,435 DEBUG zuul.AnsibleJob.output: [e: 66b826e0-8b94-11ef-8b79-1633a44cbff6] [build: 63489bb7deb44c0ebfa155b659fc162d] Ansible output: b"0x7a18938ba190>): 'item'" | ||
| @flaper87:matrix.org | I can't find anything in the executor's logs for these specific failures | 10:25 |
| @flaper87:matrix.org | I was hoping to find a stacktrace or actual error to debug. The error_dict seems to be empty | 10:26 |
| -@gerrit:opendev.org- Simon Westphahl proposed: | 12:08 | |
| - [zuul/zuul] 929273: Add support for 'min-ready' provider nodes https://review.opendev.org/c/zuul/zuul/+/929273 | ||
| - [zuul/zuul] 931779: Implement 'max-ready-age' handling https://review.opendev.org/c/zuul/zuul/+/931779 | ||
| - [zuul/zuul] 931780: Implement re-use of ready nodes w/o request https://review.opendev.org/c/zuul/zuul/+/931780 | ||
| - [zuul/zuul] 932170: Store state time for provider nodes https://review.opendev.org/c/zuul/zuul/+/932170 | ||
| - [zuul/zuul] 932179: Only expire nodes when no pending layout update https://review.opendev.org/c/zuul/zuul/+/932179 | ||
| @karlatec:matrix.org | Hello! I am experimenting with ZUUL and I am trying to move the environment described in Quick Start / Tutorial section from Docker Compose to Kubernetes. | 13:17 |
| I have a k3s cluster installed on a single node. Gerrit and ZUUL components (web, nodepool, scheduler, etc) are all deployed inside the cluster. I can run example test jobs without issue using Nodepool's static driver, but am experiencing issues when trying to use Kubernetes driver. | ||
| @karlatec:matrix.org | Test playbook runs (as in the tutorial) two roles: | 13:19 |
| - add-build-sshkey | ||
| - prepare-workspace | ||
| I can see a pod created in the cluster after scheduler picks up the task. Looking at the build log it seems there's no problem with `add-build-sshkey`, but there's an issue with `prepare-workspace` | ||
| @karlatec:matrix.org | ``` | 13:20 |
| 2024-10-16 12:58:19.132710 | TASK [add-build-sshkey : Remote setup ssh keys (linux)] | ||
| 2024-10-16 12:58:19.188351 | ubuntu-jammy | ok | ||
| 2024-10-16 12:58:19.205872 | ubuntu-jammy | included: /var/lib/zuul/builds/7801181bcff44a0f94ee753cfc6681a0/trusted/project_1/opendev.org/zuul/zuul-jobs/roles/add-build-sshkey/tasks/remote-linux.yaml | ||
| 2024-10-16 12:58:19.217066 | | ||
| 2024-10-16 12:58:19.217149 | TASK [add-build-sshkey : Remove previously added zuul-build-sshkey] | ||
| 2024-10-16 12:58:19.242496 | ubuntu-jammy | skipping: Conditional result was False | ||
| 2024-10-16 12:58:19.252974 | | ||
| 2024-10-16 12:58:19.253086 | TASK [add-build-sshkey : Enable access via build key on all nodes] | ||
| 2024-10-16 12:58:21.347923 | ubuntu-jammy | changed | ||
| 2024-10-16 12:58:21.360436 | | ||
| 2024-10-16 12:58:21.360574 | TASK [add-build-sshkey : Make sure user has a .ssh] | ||
| 2024-10-16 12:58:23.324509 | ubuntu-jammy | ok | ||
| 2024-10-16 12:58:23.338236 | | ||
| 2024-10-16 12:58:23.338320 | TASK [add-build-sshkey : Install build private key as SSH key on all nodes] | ||
| 2024-10-16 12:58:25.468741 | ubuntu-jammy | ok | ||
| 2024-10-16 12:58:25.481147 | | ||
| 2024-10-16 12:58:25.481241 | TASK [add-build-sshkey : Install build public key as SSH key on all nodes] | ||
| 2024-10-16 12:58:27.611529 | ubuntu-jammy | ok | ||
| 2024-10-16 12:58:27.624664 | | ||
| 2024-10-16 12:58:27.624779 | TASK [add-build-sshkey : Remote setup ssh keys (windows)] | ||
| 2024-10-16 12:58:27.658759 | ubuntu-jammy | skipping: Conditional result was False | ||
| 2024-10-16 12:58:27.679202 | | ||
| 2024-10-16 12:58:27.679313 | TASK [remove-zuul-sshkey : Remove master key from local agent] | ||
| 2024-10-16 12:58:28.075477 | ubuntu-jammy -> localhost | changed | ||
| 2024-10-16 12:58:28.109697 | | ||
| 2024-10-16 12:58:28.109819 | TASK [add-build-sshkey : Add back temp key] | ||
| 2024-10-16 12:58:28.432594 | ubuntu-jammy -> localhost | Identity added: /var/lib/zuul/builds/7801181bcff44a0f94ee753cfc6681a0/work/7801181bcff44a0f94ee753cfc6681a0_id_rsa (zuul-build-sshkey) | ||
| 2024-10-16 12:58:28.433099 | ubuntu-jammy -> localhost | ok: Runtime: 0:00:00.021332 | ||
| 2024-10-16 12:58:28.459606 | | ||
| 2024-10-16 12:58:28.459739 | TASK [add-build-sshkey : Verify we can still SSH to all nodes] | ||
| 2024-10-16 12:58:30.433605 | ubuntu-jammy | ok | ||
| 2024-10-16 12:58:30.447085 | | ||
| 2024-10-16 12:58:30.447203 | TASK [add-build-sshkey : Verify we can still SSH to all nodes (windows)] | ||
| 2024-10-16 12:58:30.481248 | ubuntu-jammy | skipping: Conditional result was False | ||
| 2024-10-16 12:58:30.512440 | | ||
| 2024-10-16 12:58:30.512525 | TASK [prepare-workspace : Start zuul_console daemon.] | ||
| 2024-10-16 12:58:32.509329 | ubuntu-jammy | ok | ||
| 2024-10-16 12:58:32.523646 | | ||
| 2024-10-16 12:58:32.523757 | TASK [prepare-workspace : Synchronize src repos to workspace directory.] | ||
| 2024-10-16 12:58:32.590584 | ubuntu-jammy | Output suppressed because no_log was given | ||
| failure | ||
| 2024-10-16 12:58:32.593255 | | ||
| 2024-10-16 12:58:32.593374 | PLAY RECAP | ||
| 2024-10-16 12:58:32.593465 | ubuntu-jammy | ok: 12 changed: 4 unreachable: 0 failed: 1 skipped: 3 rescued: 0 ignored: 0 | ||
| 2024-10-16 12:58:32.593510 | | ||
| ``` | ||
| @karlatec:matrix.org | * Hello! I am experimenting with ZUUL and I am trying to move the environment described in Quick Start / Tutorial section from Docker Compose to Kubernetes. | 13:20 |
| I have a k3s cluster installed on a single node. Gerrit and ZUUL components (web, nodepool, scheduler, etc) are all deployed inside the cluster. I can run example test jobs without issue using Nodepool's static driver, but am experiencing issues when trying to use Kubernetes driver. Can anyone help? | ||
| @karlatec:matrix.org | Any suggestions on what might be the problem? | 13:21 |
| @karlatec:matrix.org | Also - is there any way I can override `no_log` flag? I found a copy of the prepare-workspace role in the Executor pod and changed it there to `false`, but looks like a fresh version is getting pulled from opendev.org every time I run the test. | 13:27 |
| @jim:acmegating.com | flaper87: you can run `zuul-executor verbose` to enable ansible verbose logging and you may get more helpful output from ansible | 13:30 |
| @jim:acmegating.com | Karol: try the prepare-workspace-openshift role (yes even for k8s): https://zuul-ci.org/docs/zuul-jobs/latest/general-roles.html#role-prepare-workspace-openshift | 13:31 |
| @jim:acmegating.com | * Karol: try the prepare-workspace-openshift role (yes even for k8s) instead of prepare-workspace: https://zuul-ci.org/docs/zuul-jobs/latest/general-roles.html#role-prepare-workspace-openshift | 13:33 |
| @flaper87:matrix.org | > <@jim:acmegating.com> flaper87: you can run `zuul-executor verbose` to enable ansible verbose logging and you may get more helpful output from ansible | 13:35 |
| Is runnuing `zuul-executor verbose` enough to enable it? Or do I need to enable some other settings? | ||
| @jim:acmegating.com | flaper87: that's it. takes effect for jobs that start after that point. need to run it on each executor since it's a local command. stays in place until restarted or until you run `zuul-executor unverbose`. | 13:37 |
| @yodakv:matrix.org | I guess you don't have network connectivity between executor and created pod. The "Synchronize src repos to workspace directory" role try to execute git push command to sync local build with remote worker (in use case pod). So check for: | 17:32 |
| 1. launcher nodepool configuration | ||
| .If pod has ssh and ssh work property | 00:00 | |
| 3. network connectivity between executor and pod by run ssh -v pod_ip | ||
| Something is broken between these two component | ||
| @karlatec:matrix.org | corvus: I tried `verbose` mode for my problem too, but it did not provide enough information as to why the prepare-workspace role was failing. Guess `no_log` option must be affecting that. Fortunately using `prepare-workspace-openshift` worked as the pipeline was able to complete successfully. Many thanks! | 17:51 |
| -@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed: [zuul/zuul] 932573: WIP: Allow multiple errors for a single config item https://review.opendev.org/c/zuul/zuul/+/932573 | 23:45 | |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!