| @bridgefan:matrix.org | I have been working with an example zuul instance using this as a template for while: https://opendev.org/zuul/zuul/src/branch/master/doc/source/examples | 16:01 |
|---|---|---|
| @bridgefan:matrix.org | However, I recently wanted to try adding additional node containers to it. | 16:02 |
| @bridgefan:matrix.org | I've duplicated the Dockerfile, added it to the docker-compose, etc_nodepool/nodepool.yaml, and select it as the proper node in the job definition | 16:03 |
| @bridgefan:matrix.org | I have found that if I simply copy the ubuntu 22.04 node-Dockerfile it'll work with this new node | 16:03 |
| @bridgefan:matrix.org | but if I try to switch to a different distro such as debian (tried 11, 12, 13, stable, unstable) I seem to get RETRY_LIMIT failures | 16:04 |
| @bridgefan:matrix.org | As far as I can tell the host key public key is correct as for testing Iit is the same as the ubuntu | 16:04 |
| @bridgefan:matrix.org | I seem to see the executor connect to the sshd: Accepted publickey for root from 172.19.0.7 port 39368 ssh2: RSA SHA256... | 16:05 |
| @bridgefan:matrix.org | I'm trying to figure out what else I should check | 16:05 |
| @bridgefan:matrix.org | The only thing I'm changing is the FROM line of the dockerfile | 16:06 |
| @clarkb:matrix.org | you can add flags to the sshd process like `-d` to put it in debug mode (it will only accept a single connection if you do but log more info). I would double check that permitrootlogin is enabled if you are ssh'ing as root. You can also ssh manually using -vvv on the client side to get a better sense of what the client isseeing | 16:09 |
| @bridgefan:matrix.org | Thanks, I've beeing launching sshd with -D -e with LogLevel set higher to get more information. | 16:14 |
| @bridgefan:matrix.org | I see: permitrootlogin without-password | 16:15 |
| @bridgefan:matrix.org | I'll try ssh -vvv on the executor (I assume RETRY_LIMIT is due to executor not being able to connect to teh node?) | 16:16 |
| @clarkb:matrix.org | it could be or it could be due to failures in the job itself | 16:17 |
| @clarkb:matrix.org | maybe debian doesn't install things that ubuntu does that the job relies on | 16:17 |
| @clarkb:matrix.org | that may be worth checking first. Look at the build log to see what the actual failure is | 16:18 |
| @bridgefan:matrix.org | Ok. The failure is at the very beginning. I actually can't see any logs (I am familiar with the Zuul interface under normal working circumstances). I just see: "This build does not provide any results" | 16:19 |
| @bridgefan:matrix.org | It's like the initial connection the the node doesn't work | 16:19 |
| @clarkb:matrix.org | look in your executor logs | 16:19 |
| @bridgefan:matrix.org | I haven't been able to see an error directly. | 16:23 |
| @bridgefan:matrix.org | I see: | 16:23 |
| @bridgefan:matrix.org | 2025-08-21 16:22:04,802 INFO zuul.AnsibleJob: [e: 6e81d8b99d2f46acb287edf3c1cef29c] [build: aa4d0442bb5e4a56ad050e228ea6ca69] Job execution took: 1.934 seconds | 16:23 |
| 2025-08-21 16:22:07,767 INFO zuul.AnsibleJob: [e: 6e81d8b99d2f46acb287edf3c1cef29c] [build: fdb73c7cd43845d7892914ae36970422] Checkout workspace repos: True | ||
| 2025-08-21 16:22:07,794 INFO zuul.ExecutorServer: [e: 6e81d8b99d2f46acb287edf3c1cef29c] [build: fdb73c7cd43845d7892914ae36970422] Started SSH Agent, {'SSH_AUTH_SOCK': '/tmp/ssh-WGNMVmtSPZnS/agent.11223', 'SSH_AGENT_PID': '11224'} | ||
| 2025-08-21 16:22:07,806 INFO zuul.ExecutorServer: [e: 6e81d8b99d2f46acb287edf3c1cef29c] [build: fdb73c7cd43845d7892914ae36970422] Added SSH Key /var/ssh/nodepool | ||
| 2025-08-21 16:22:07,810 INFO zuul.AnsibleJob: [e: 6e81d8b99d2f46acb287edf3c1cef29c] [build: fdb73c7cd43845d7892914ae36970422] Beginning job appimage-get-wine for ref refs/changes/02/1402/3 (change http://192.168.5.71:9007/1402) | ||
| @bridgefan:matrix.org | it then just repeats the update repo steps | 16:25 |
| @bridgefan:matrix.org | Is there a way to get more debug information from the executor container? | 16:25 |
| @clarkb:matrix.org | You can run the executor with debug logging if it isn't already doing that. Doing so includes all of the Ansible logs | 16:26 |
| @jangutter:matrix.org | Is "zuul-executor keep" and "zuul-executor nokeep" still a thing? I used that plenty of times to keep the build dir and do some low level debuggery. | 16:31 |
| @jangutter:matrix.org | https://www.softwarefactory-project.io/docs/3.1/operator/zuul_operator.html <--- back in the day the two options 'zuul-executor verbose' and 'zuul-executor keep' was the trick I used to debug logging problems. | 16:33 |
| @bridgefan:matrix.org | I'm trying the zuul-executor -d option now. Maybe I'll find something. The current test setup already keeps the containers in a state I can see the results. I'm not sure if keep does more though? | 16:33 |
| @bridgefan:matrix.org | for: command: "sh -c '/var/playbooks/wait-to-start-certs.sh && exec zuul-executor -f -d'" | 16:34 |
| @bridgefan:matrix.org | should I instead use command: "sh -c '/var/playbooks/wait-to-start-certs.sh && exec zuul-executor -f verbose'" ? | 16:34 |
| @bridgefan:matrix.org | or am I execing into the zuul-executor and running zuul-executor verbose? | 16:35 |
| @bridgefan:matrix.org | nevermind, I think -d may have found that the container needs python3 installed | 16:39 |
| @bridgefan:matrix.org | I'm going to explicitly install it. I guess Ubuntu default install python | 16:39 |
| @fungicide:matrix.org | debian by default, if minimized, may only include python-minimal which doesn't cover the python stdlib | 16:41 |
| @bridgefan:matrix.org | haha looks like missing python was it | 16:41 |
| @bridgefan:matrix.org | thanks very much for the help Clark, jangutter, and fungi | 16:42 |
| @fungicide:matrix.org | and yeah, ansible will need the python stdlib | 16:42 |
| @jangutter:matrix.org | bridgefan: if you did a "docker exec" to set verbose and keep, just be sure to turn them off otherwise the executor disk space will get full very quickly. | 17:06 |
| @fungicide:matrix.org | (or make sure you dedicate a ton of space to logging) | 17:08 |
| @fungicide:matrix.org | we actually run with debug logging turned on in production, but a day's worth of debug logging across our executors is around 4.5gb uncompressed | 17:11 |
| @jangutter:matrix.org | You do get really good compression ratio on it though. | 17:14 |
| @fungicide:matrix.org | yeah, looks like we get around a 7:1 compression ratio from gzip on those | 17:17 |
Generated by irclog2html.py 4.0.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!