| opendevreview | Cyril Roelandt proposed openstack/pbr master: Python 3.14: do not use the onerror parameter in shutil.rmtree() https://review.opendev.org/c/openstack/pbr/+/924803 | 00:01 |
|---|---|---|
| opendevreview | Clark Boylan proposed openstack/project-config master: Set xenial min ready to 0 https://review.opendev.org/c/openstack/project-config/+/924806 | 00:28 |
| opendevreview | Merged openstack/project-config master: Set xenial min ready to 0 https://review.opendev.org/c/openstack/project-config/+/924806 | 00:55 |
| *** bauzas_ is now known as bauzas | 06:04 | |
| *** bauzas_ is now known as bauzas | 09:08 | |
| *** bauzas_ is now known as bauzas | 09:21 | |
| jakeyip | hi, sorry to bother, is there any admins around who can help me check why is https://opendev.org/openstack/magnum-capi-helm-charts/ not replicating to https://github.com/openstack/magnum-capi-helm-charts ? it may be due to a force-push we previously did to gerrit, so may need to wipe github and let gerrit push everything again... | 11:04 |
| tonyb | jakeyip: I can look after the meeting I'm in finishes | 11:06 |
| jakeyip | thanks tonyb, appreciate it. it isn't urgent. | 11:08 |
| tonyb | jakeyip: It doesn't look like that's been configured to run at all? | 11:38 |
| jakeyip | tonyb: not sure what you mean? | 11:38 |
| tonyb | jakeyip: I think you need something like: https://opendev.org/openstack/project-config/src/branch/master/zuul.d/projects.yaml#L2193-L2196 for the openstack/magnum-capi-helm-charts/ project | 11:38 |
| tonyb | The key thing is the official-openstack-repo-jobs template that includes the job to publish and sync the repos | 11:39 |
| tonyb | jakeyip: I'm looking at the 3555,11: - maintain-github-openstack-mirror jobs now, maybe that's where it's supposed to happen | 11:40 |
| jakeyip | hmm I see, trying to trace | 11:42 |
| jakeyip | I think you are right, there's a 'openstack-upload-github-mirror' job which sounds like what we want :) | 11:47 |
| tonyb | jakeyip: I'm about to sign off for now but ad me as a reviewer and I'll check it out first thing tomorrow | 11:50 |
| jakeyip | tonyb: sure, thanks for your help | 11:50 |
| opendevreview | Jake Yip proposed openstack/project-config master: Sync magnum-capi-helm-charts repo to GitHub mirror https://review.opendev.org/c/openstack/project-config/+/924846 | 11:53 |
| fungi | jakeyip: once it merges, the next change to merge in the magnum-capi-helm-charts repo should trigger replication | 11:55 |
| jakeyip | fungi: great, thanks :) | 11:55 |
| opendevreview | Merged openstack/project-config master: Sync magnum-capi-helm-charts repo to GitHub mirror https://review.opendev.org/c/openstack/project-config/+/924846 | 12:10 |
| noonedeadpunk | hey folks! do you happen to know if in nodepool vm root auth is permitted with a key? | 13:13 |
| noonedeadpunk | I'm trying to investigate why https://review.opendev.org/c/openstack/openstack-ansible-tests/+/921434 fails in CI but not locally | 13:13 |
| noonedeadpunk | or maybe restriction on key length? | 13:14 |
| frickler | I would hope we disallow root login in the ssh config | 13:19 |
| noonedeadpunk | aha | 13:23 |
| noonedeadpunk | that would explain it :D | 13:23 |
| noonedeadpunk | and seems we do override it.... | 13:24 |
| frickler | actually I'm wrong, we don't. if we hold a node, the login is as root | 13:28 |
| frickler | but also I can set up a hold for your patch and we can try to check in place. assuming the infra root login isn't broken by what you are doing | 13:29 |
| noonedeadpunk | it's in check right now, but will fail 99.9% | 13:31 |
| noonedeadpunk | so if you can make a hold - that might be helpful | 13:31 |
| noonedeadpunk | but in the patch I've added storing /etc/ssh directory, so can check for the content when it fail | 13:32 |
| noonedeadpunk | yeah, `PermitRootLogin yes` | 13:39 |
| noonedeadpunk | so if you make a hold - would be pretty much appreciated | 13:50 |
| clarkb | side note: when you want to refer to a failure I personally find it most helpful if you link directly to the failure. Not the change, not the top level of the build, not the top level of the log file, etc. https://zuul.opendev.org/t/openstack/build/b6fee14ad4484afebb5f7b7cc53e971f/log/job-output.txt#5371-5373 or similar in this case. It is very easy to work backward from that | 14:12 |
| clarkb | point to the other details like what job what change etc. But working forward can often be a slog as I'm not familiar with everyone's jobs | 14:12 |
| clarkb | if I had to guess maybe localhost in that context is not the localhost you expect it to be and there is no sshd running there | 14:14 |
| clarkb | for example if within a container with its own network namespace | 14:14 |
| opendevreview | Clark Boylan proposed openstack/project-config master: Limit what volvocars loads from opendev/base-jobs https://review.opendev.org/c/openstack/project-config/+/924858 | 14:20 |
| clarkb | noonedeadpunk: perhaps the regression here (if there is one) is that you were previously using a local connection for ansible but now it is trying to ssh to localhost in the container and there is no sshd? | 14:25 |
| clarkb | seems like the sort of thing with an ansible version bump that might change subtly for reasons because ansible | 14:26 |
| frickler | I was too late with the hold, need to recheck once the current run finishes | 14:26 |
| noonedeadpunk | clarkb: well yes, it was using local connection, and now tries to SSH. But at this point it should have had a SSH key generated and placed for that, ie https://zuul.opendev.org/t/openstack/build/cd9da1010d33419b8c1395283b963089/log/job-output.txt#4199-4218 | 14:31 |
| noonedeadpunk | and task for that is https://opendev.org/openstack/openstack-ansible-tests/src/branch/master/test-prepare-keys.yml#L20-L63 | 14:31 |
| clarkb | but does it have an sshd server? the error is unreachable not authentication failure | 14:31 |
| clarkb | oh its pubkey failure and unreachable | 14:31 |
| clarkb | nice ansible | 14:32 |
| noonedeadpunk | yeah, so it's specifically smth with auth. and on a local VM this passes nicely each time | 14:32 |
| noonedeadpunk | I've added /etc/ssh to logs but don't see anything obvious either | 14:33 |
| noonedeadpunk | it indeed could be that `localhost` is smth very different now | 14:33 |
| clarkb | I would expect ot see logs indicating that auth failed at least. If you don't see that then ya maybe ist takling to something else | 14:34 |
| noonedeadpunk | yeah, I actually also was slightly surprised not seeing that | 14:35 |
| noonedeadpunk | frickler: I can run recheck now if you're around to put a hold? | 14:38 |
| clarkb | noonedeadpunk: the existing hold should still be valid if it didn't catch anything. You just have to wait for the current run to report otherwise the recehck will be ignored | 14:39 |
| clarkb | basically go ahead and recheck once gerrit gets a comment for the last run | 14:39 |
| noonedeadpunk | I think it's reported? | 14:39 |
| clarkb | ya I don't see it in the current status dashboard and there is a report from 15 minutes or so ago. I think you can recheck now | 14:40 |
| frickler | noonedeadpunk: I have the held node now. "ssh root@localhost" as root works just fine. I added your key if you want to check yourself root@104.130.124.131 | 15:14 |
| noonedeadpunk | ugh.... | 15:17 |
| noonedeadpunk | thanks a lot | 15:17 |
| noonedeadpunk | but I actually don't see ansible's ssh key added there | 15:19 |
| noonedeadpunk | so probably there's smth that's different | 15:19 |
| noonedeadpunk | you didn't wipe out /root/.ssh/authorized_keys when pulling keys? | 15:20 |
| noonedeadpunk | wonder if zuul does that though... | 15:21 |
| clarkb | zuul doesn't. | 15:23 |
| clarkb | glean sets the authorized keys at first boot then they should never be touched again automatically | 15:23 |
| clarkb | there may be some zuul job content that tries to manage them though | 15:23 |
| clarkb | but the service itself shouldn't | 15:24 |
| noonedeadpunk | ok, yeah, so I apparently don't see self-generated SSH key that is supposed to be used by internal ansible there | 15:24 |
| noonedeadpunk | which can explain issue easily | 15:24 |
| frickler | I do see the key there | 15:27 |
| frickler | at least the one from /root/.ssh/id_rsa.pub which is what I think we're talking about? | 15:28 |
| noonedeadpunk | ah, right, I've changed a way to generate the key recently, so local sandbox differs a bit | 15:31 |
| noonedeadpunk | was expecting to see a specific comment | 15:33 |
| noonedeadpunk | so, running same tox job now doesn't bring any auth issues /o\ | 15:42 |
| jrosser | maybe we need a meta: reset_connection after dropping the ssh keys? | 15:46 |
| noonedeadpunk | yeah, thinking about that already | 15:54 |
| noonedeadpunk | but then, it's weird as you'd try to connect differently | 16:03 |
| noonedeadpunk | i really clueless why it fails | 16:22 |
| noonedeadpunk | reset_connection obviously does not help | 16:25 |
| fungi | noonedeadpunk: and you're sure the localhost ssh attempt is happening in the scope of the test node not on the executor? | 16:46 |
| noonedeadpunk | so that looks like nested ansible to me: https://zuul.opendev.org/t/openstack/build/7faa9d9a289149d3a1ad58587eb41345/log/job-output.txt#4150 | 16:50 |
| noonedeadpunk | and yeah - ubuntu-jammy is a label of the test vm | 16:50 |
| noonedeadpunk | so I'd say it's not on executor | 16:50 |
| noonedeadpunk | jrosser: I was just able to reproduce on my test VM by running the tox as ubuntu user | 16:51 |
| jrosser | oooh | 16:51 |
| noonedeadpunk | pretty much - git clone; ./run_tests.sh functional | 16:52 |
| noonedeadpunk | I think you can release a hold now - thanks | 16:52 |
| noonedeadpunk | frickler: ^ | 17:47 |
| clarkb | I suspect frickler may have called it a day. I can clean that up later after I'm done with this gitea maintenance | 17:52 |
| noonedeadpunk | ++ thanks | 18:01 |
| frickler | deleted | 18:04 |
| *** bauzas_ is now known as bauzas | 19:17 | |
| opendevreview | Clark Boylan proposed openstack/project-config master: Limit what volvocars loads from opendev/base-jobs https://review.opendev.org/c/openstack/project-config/+/924858 | 20:45 |
| *** bauzas_ is now known as bauzas | 21:20 | |
| opendevreview | Merged openstack/project-config master: Limit what volvocars loads from opendev/base-jobs https://review.opendev.org/c/openstack/project-config/+/924858 | 21:55 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!