| *** prometheanfire has joined #opendev | 00:21 | |
| *** prometheanfire has quit IRC | 00:24 | |
| *** prometheanfire has joined #opendev | 00:27 | |
| *** JayF has quit IRC | 01:23 | |
| *** brinzhang has joined #opendev | 01:35 | |
| *** iurygregory has joined #opendev | 02:07 | |
| *** JayF has joined #opendev | 02:40 | |
| *** cloudnull has quit IRC | 03:38 | |
| *** ykarel has joined #opendev | 04:31 | |
| *** rosmaita has left #opendev | 04:43 | |
| *** marios has joined #opendev | 06:22 | |
| *** cloudnull has joined #opendev | 06:53 | |
| *** ysandeep|away is now known as ysandeep | 07:03 | |
| *** ralonsoh has joined #opendev | 07:40 | |
| *** eolivare has joined #opendev | 07:43 | |
| *** jpena|off is now known as jpena | 07:50 | |
| *** slaweq has joined #opendev | 07:51 | |
| *** hashar has joined #opendev | 08:01 | |
| *** andrewbonney has joined #opendev | 08:13 | |
| *** fressi has joined #opendev | 08:21 | |
| *** sboyron has joined #opendev | 08:22 | |
| *** ykarel is now known as ykarel|lunch | 08:30 | |
| *** sboyron has quit IRC | 08:30 | |
| *** rpittau|afk is now known as rpittau | 08:33 | |
| *** ralonsoh has quit IRC | 08:38 | |
| *** ykarel|lunch has quit IRC | 08:39 | |
| *** ralonsoh has joined #opendev | 08:41 | |
| *** lpetrut has joined #opendev | 08:45 | |
| *** yoctozepto has quit IRC | 08:48 | |
| *** kevinz has quit IRC | 08:53 | |
| *** sboyron has joined #opendev | 08:57 | |
| *** sshnaidm|afk is now known as sshnaidm|ruck | 08:57 | |
| *** DSpider has joined #opendev | 09:04 | |
| *** sboyron has quit IRC | 09:48 | |
| *** sboyron has joined #opendev | 10:26 | |
| *** ykarel has joined #opendev | 10:31 | |
| *** wiktri has joined #opendev | 10:34 | |
| *** dtantsur|afk is now known as dtantsur | 10:37 | |
| openstackgerrit | Merged opendev/elastic-recheck master: Add query for bug 1882521 https://review.opendev.org/c/opendev/elastic-recheck/+/767948 | 10:54 |
|---|---|---|
| openstack | bug 1882521 in OpenStack Compute (nova) ussuri "Failing device detachments on Focal" [Undecided,New] https://launchpad.net/bugs/1882521 | 10:54 |
| *** ykarel is now known as ykarel|afk | 11:50 | |
| *** ykarel|afk has quit IRC | 11:55 | |
| *** ykarel|afk has joined #opendev | 12:07 | |
| *** dviroel has joined #opendev | 12:11 | |
| openstackgerrit | Merged zuul/zuul-jobs master: Better error output for update-test-platforms.py https://review.opendev.org/c/zuul/zuul-jobs/+/766754 | 12:17 |
| *** Oriz has joined #opendev | 12:20 | |
| *** Oriz has quit IRC | 12:20 | |
| *** ykarel|afk has quit IRC | 12:22 | |
| *** jpena is now known as jpena|lunch | 12:30 | |
| *** ykarel has joined #opendev | 13:06 | |
| *** redrobot has joined #opendev | 13:19 | |
| *** ykarel_ has joined #opendev | 13:21 | |
| *** ykarel has quit IRC | 13:23 | |
| *** auristor has quit IRC | 13:24 | |
| *** auristor has joined #opendev | 13:24 | |
| *** ykarel_ is now known as ykarel | 13:25 | |
| *** jpena|lunch is now known as jpena | 13:29 | |
| *** yoctozepto has joined #opendev | 13:45 | |
| *** ralonsoh_ has joined #opendev | 13:59 | |
| *** ralonsoh has quit IRC | 13:59 | |
| *** ralonsoh_ is now known as ralonsoh | 14:01 | |
| *** DSpider has quit IRC | 14:24 | |
| *** zul has joined #opendev | 14:33 | |
| *** ralonsoh has quit IRC | 14:37 | |
| *** ralonsoh has joined #opendev | 14:38 | |
| *** tosky has joined #opendev | 14:39 | |
| *** ralonsoh has quit IRC | 14:40 | |
| *** spotz has joined #opendev | 14:57 | |
| *** lpetrut has quit IRC | 15:10 | |
| *** fressi has quit IRC | 15:10 | |
| *** ralonsoh has joined #opendev | 15:16 | |
| *** d34dh0r53 has joined #opendev | 15:22 | |
| *** ykarel has quit IRC | 15:44 | |
| *** brinzhang has quit IRC | 15:55 | |
| *** hashar has quit IRC | 16:09 | |
| *** cloudnull has quit IRC | 16:11 | |
| *** cloudnull has joined #opendev | 16:14 | |
| *** wiktri has quit IRC | 16:30 | |
| *** ysandeep is now known as ysandeep|dinner | 16:37 | |
| *** mlavalle has quit IRC | 16:41 | |
| *** mlavalle has joined #opendev | 16:41 | |
| *** jpena is now known as jpena|off | 16:50 | |
| *** DSpider has joined #opendev | 17:00 | |
| *** marios is now known as marios|out | 17:08 | |
| *** rpittau is now known as rpittau|afk | 17:10 | |
| *** tosky has quit IRC | 17:24 | |
| *** tosky has joined #opendev | 17:25 | |
| *** ysandeep|dinner is now known as ysandeep | 17:32 | |
| *** openstackgerrit has quit IRC | 17:37 | |
| redrobot | Hi #opendev friends! | 17:59 |
| redrobot | Do y'all own git-review? | 17:59 |
| fungi | we maintain it, yes | 18:02 |
| fungi | what's up? | 18:02 |
| fungi | we're the first order maintainers of anything in our opendev/ git namespace, so that includes https://opendev.org/opendev/git-review | 18:03 |
| redrobot | Well, I upgraded my system, and now git-review does not seem to be able to find my SSH key | 18:03 |
| fungi | fedora 33? | 18:04 |
| redrobot | Yeah | 18:04 |
| fungi | fedora decided to disable ssh-rsa keytypes | 18:04 |
| fungi | there are a number of potential workarounds. note this isn't a git-review specific problem, it's impacting ssh and git over ssh more generally | 18:05 |
| fungi | redrobot: i answered similar questions in this openstack-discuss ml thread: http://lists.openstack.org/pipermail/openstack-discuss/2020-December/019567.html | 18:06 |
| redrobot | I see ... I didn't notice any ssh failures, just git-review so far. I also have a kind of not-standard setup in that I'm using a Yubikey as a GPG card and using an RSA subkey as SSH key. | 18:06 |
| redrobot | fungi thanks, I'll give that thread a read | 18:06 |
| fungi | another option i forgot to mention in that ml post is you can switch to using https instead of ssh for connecting to gerrit | 18:07 |
| clarkb | fungi: did we ever figure out if clearing existing known hosts entries would cause it to find the ssh-rsa2 options? | 18:07 |
| clarkb | because review.opendev.org does do ssh-rsa2 and its really confusing to me why that doesn't just work on fedora 33 unless old known host keys are set or ssh-rsa is entirely disabled | 18:07 |
| fungi | clarkb: good question, if redrobot wants to try deleting his known_hosts entries for review.o.o that could be a good experiment | 18:08 |
| fungi | i don't know whether anyone has tested that explicitly | 18:08 |
| fungi | though in theory the UpdateHostKeys option would have a similar effect | 18:08 |
| * redrobot is happy to be a guinea pig | 18:09 | |
| fungi | redrobot: please do let us know how you get on. i've already seen folks work around this in several different ways, so any additional info you can render as to what works is appreciated | 18:09 |
| redrobot | No dice. It did ask to add the fingerprint to my known_hosts but then failed again in the same manner. | 18:11 |
| redrobot | Will get back to y'all after I read that thread an sort out my configuration. Thanks again. | 18:11 |
| clarkb | infra-root I think https://review.opendev.org/c/opendev/system-config/+/769226 is ready for review now (let me know if I can dig up the test instance that was held for you). As noted previously the gitea upgrade from 1.12 to 1.13 is big ish as gitea upgrades go since they added a bunch of new features like kanban boards. Definitely give it a reasonable review | 18:11 |
| clarkb | separately fungi and I have talked to smarcet et al on the foundation side of openstackid and it sounds like we can spin down the extra capacity we turned on for the summit. | 18:12 |
| clarkb | fungi: I'm thinking maybe we shutdown those two servers today, then in a couple of days delete them if nothing breaks? | 18:13 |
| clarkb | openstackid02.openstack.org and openstackid03.openstack.org are the two servers in vexxhost which I will shutdown if that plan seems reasonabel | 18:14 |
| fungi | yeah, in theory nothing is hitting those servers, but i agree in proceeding with some measure of caution just in case | 18:16 |
| *** hamalq has joined #opendev | 18:16 | |
| *** eolivare has quit IRC | 18:16 | |
| clarkb | fungi: I think we need to put those servers in the emergency file when we do that too (otherwise ssh will fail to them) | 18:17 |
| clarkb | then completely remove them from inventory once deleted? | 18:17 |
| clarkb | I Guess we can remove them from inventory now too | 18:17 |
| fungi | or delete them from the inventory first? | 18:17 |
| fungi | right, that. jinx | 18:17 |
| fungi | one or the other | 18:18 |
| clarkb | yup working on that change now | 18:18 |
| *** sboyron has quit IRC | 18:18 | |
| fungi | i'd be fine with deleting them from inventory/dns in advance of shutting them down. then all that remains in a couple of days is deleting the server instances from the provider | 18:18 |
| *** sboyron has joined #opendev | 18:19 | |
| clarkb | I don't even know if we added them to dns since haproxy used addrs? | 18:19 |
| clarkb | actually no would've added to dns because its in the old domain which uses the rest client | 18:20 |
| fungi | `host openstackid02.openstack.org` | 18:20 |
| fungi | et cetera | 18:20 |
| fungi | so yeah, there are openstack.org dns entries | 18:21 |
| fungi | no opendev.org entries i'm aware of though | 18:21 |
| clarkb | we lost gerritbot | 18:21 |
| clarkb | remote: https://review.opendev.org/c/opendev/system-config/+/770167 Cleanup openstackid02 and openstackid03 | 18:22 |
| clarkb | fungi: once that lands we can remove from dns (again to avoid ansible ssh problems) | 18:22 |
| clarkb | now to restart gerritbot (it left saying it was changing servers according to my scrollback) | 18:22 |
| clarkb | fungi: also looks like we may be leaking ansible processes again due to logstash worker 18 and paste | 18:34 |
| clarkb | ssh fails to both and paste http isn't accessible either | 18:35 |
| clarkb | fungi: do you think we should check console on paste or just restart them (I'm not worried about console on the logstash worker) | 18:35 |
| fungi | i'll pull up th econsole for paste.o.o now | 18:36 |
| fungi | thanks for noticing | 18:37 |
| clarkb | logstash-worker18 has been restarted | 18:37 |
| clarkb | let me know if you think I should do paste (or you can reboot it too) | 18:37 |
| fungi | i have to wonder if we're the only rackspace users who see server instances regularly getting stuck like this | 18:37 |
| clarkb | ya I was thinking about that and wonder if it has to do with us cleaning up the nova agent or something like that | 18:38 |
| clarkb | like maybe we need to do an arp after being requested post migration or something? | 18:38 |
| fungi | judging from cacti, paste.o.o hung sometime around 00:15 today | 18:39 |
| fungi | yeah, hung kernel tasks around 20216760 seconds after the last reboot | 18:41 |
| fungi | it's rebooting now | 18:41 |
| clarkb | thanks. I'll start cleaning up the stale ansible processes shortly | 18:42 |
| fungi | #status log rebooted paste.o.o just now in order to recover from a hung userspace earlier around 00:15 utc today | 18:42 |
| openstackstatus | fungi: finished logging | 18:42 |
| *** andrewbonney has quit IRC | 18:43 | |
| redrobot | fungi sorted my ssh issues with a config change to PubkeyAcceptedKeyTypes http://paste.openstack.org/show/801539/ | 18:46 |
| clarkb | redrobot: ok so that does seem to imply fedora 33 killed all ssh-rsa including the sha2 variants | 18:46 |
| clarkb | which seems like a completely broken way to do ssh | 18:46 |
| clarkb | only the sha1 variant is deprecated upstream (and it isn't even removed tehre yet either) | 18:47 |
| fungi | paste uptime is 10 minutes, and i can get to the webui again | 18:53 |
| fungi | i think we're all good there | 18:53 |
| clarkb | I've got ansible processes from yseterday and prior cleaned up | 18:55 |
| clarkb | need to look more closely at those from today to avoid killing happy processes | 18:55 |
| clarkb | ok now there are only a couple of processes from just under a couple of hours ago that I suspect have leaked. I'll let them sit around a bit long er to feel stronger in that suspicion | 19:03 |
| *** openstackgerrit has joined #opendev | 19:04 | |
| openstackgerrit | Merged opendev/system-config master: Cleanup openstackid02 and openstackid03 https://review.opendev.org/c/opendev/system-config/+/770167 | 19:04 |
| *** marios|out has quit IRC | 19:04 | |
| *** _mlavalle_1 has joined #opendev | 19:05 | |
| fungi | clarkb: ^ i guess you can shut them down now | 19:08 |
| *** mlavalle has quit IRC | 19:08 | |
| clarkb | thanks. Kids just started lunch break so I'll grab that in a bit. I'll do the dns cleanup then too | 19:08 |
| redrobot | fungi, clarkb, I found an interesting thread on the Fedora mailing list that made me dig into this ssh config rabbit hole some more. | 19:10 |
| redrobot | I was able to narrow down the needed config change to just this: http://paste.openstack.org/show/801540/ | 19:10 |
| redrobot | It appears that adding a "+" symbol to the value for PubkeyAcceptedKeyTypes bypasses the fedora crypto policy and enables the default openssh policy instead | 19:11 |
| redrobot | which pulls in ssh-rsa | 19:11 |
| redrobot | so that's why my earlier paste worked | 19:11 |
| clarkb | redrobot: well you actually don't want to use ssh-rsa if you can avoid it | 19:11 |
| clarkb | you want to use the ssh-rsa sha2 options which your first paste attempted to do | 19:12 |
| clarkb | our gerrit does support those and they are not deprecated by upstream openssh | 19:12 |
| clarkb | I think we can disable ssh-rsa in our gerrit config too | 19:13 |
| fungi | yeah, i think that's what he's saying, so my earlier post adding sha2 rsa worked but also caused the crypto policy restrictions to be completely undone? | 19:13 |
| clarkb | maybe we should consider that and see if that helps fedora clients find the right type | 19:13 |
| redrobot | clarkb yeah, it's weird because just enabling those two does not work (unless you use + , which also pulls uin ssh-rsa) | 19:14 |
| redrobot | e.g this does not work http://paste.openstack.org/show/801541/ | 19:14 |
| clarkb | huh I know I tested this at one time but on my suse machien just manually setting options | 19:14 |
| redrobot | may ultimately be a bug in Fedora. Seems the discussion is still ongoing https://bugzilla.redhat.com/show_bug.cgi?id=1881301 | 19:15 |
| openstack | bugzilla.redhat.com bug 1881301 in openssh "openssh-clients do not accept PubkeyAcceptedKeyTypes rsa-sha2-512/256" [Unspecified,Closed: errata] - Assigned to jjelen | 19:15 |
| *** dtantsur is now known as dtantsur|afk | 19:15 | |
| clarkb | fungi: maybe you can independently try to confirm that the sha2 rsa options work for you via debain? | 19:16 |
| clarkb | just as a way of double checking the new server is doing what we expect of it? | 19:16 |
| fungi | i would need to exclude sha1 ssh-rsa | 19:16 |
| fungi | debian doesn't block use of it, they're much more conservative about breaking userspace | 19:17 |
| clarkb | yup same thing with suse. iirc I used an option to set the specific type to use and set it to a sha2 version and it worked | 19:17 |
| clarkb | but I'd have to go through my notes and docs again | 19:17 |
| fungi | otherwise there are tons of folks who would be, like, "now i can't manage my 15-year-old load balancer which uses f-secure sshd 1.x) | 19:17 |
| *** gmann is now known as gmann_afk | 19:25 | |
| fungi | so sha2 for ssh rsa is covered by https://tools.ietf.org/html/rfc8332 | 19:26 |
| fungi | key algorithms would be listed as rsa-sha2-256 and rsa-sha2-512 | 19:27 |
| fungi | when i `ssh-keyscan -p 29418 review.opendev.org` i don't see those, only: ssh-rsa ecdsa-sha2-nistp256 ssh-ed25519 | 19:27 |
| fungi | maybe we need to recreate the public key from the current private key? | 19:37 |
| clarkb | hrm aren't these hashes of the public key? and not the raw public key material? | 19:50 |
| clarkb | ya I think the reason you don't see them on a keyscan is the key type is ssh-rsa but the hash type during connection negotiation can be ssh-rsa or rsa-sha2-256 etc | 19:52 |
| clarkb | they are different things | 19:52 |
| clarkb | fungi: from the bugzilla: "No, it is not. It generates the "same" rsa key as before. The digest is not reflected in the key at all." | 19:54 |
| clarkb | -oPubkeyAcceptedKeyTypes=rsa-sha2-256 seems to be the flag to ssh to force the sha2 option and that does indeed fail for me. I wonder if I used a + before and ran into the unexpected behavior of it pulling in defaults as a result | 19:57 |
| clarkb | I have confirmed that if I use a + that works | 19:59 |
| clarkb | looking at gerrit docs it seems we can configure which key exchange, mac, and cipher algrorithms are allowed but not host key? | 20:00 |
| clarkb | my next question is then why doesn't fedora use ecdsa or ed25119 since those are both fine on fedora right? | 20:01 |
| clarkb | could this be related to an existing rsa hostkey so it is looking for that? | 20:01 |
| clarkb | hrm looking at ssh -vvv output it seems the key used to authenticate impacts this as well | 20:03 |
| fungi | what am i missing? does this come down to support for server-sig-algs? https://tools.ietf.org/html/rfc8308 | 20:03 |
| clarkb | ok PubkeyAcceptedKeyTypes seems to affect both the client and the server. So if I tell it to use ecdsa exclusively it won't authenticate with my ssh-rsa key | 20:04 |
| clarkb | even if it can get a valid server pubkey via that method | 20:04 |
| fungi | fedora would likely use ecdsa or ed25119 if you have one of those for your client keys | 20:04 |
| clarkb | yup | 20:05 |
| fungi | that much i had already assumed | 20:05 |
| clarkb | fungi: I think gerrit isn't doing rsa-sha2-256 or rsa-sha2-512 to hash the ssh-rsa pubkey during connection setup. It supports sha2 variants for other rsa things according to the config docs (mac and kex) | 20:06 |
| clarkb | but the pubkey signatrue sent it always sha1 for the host key ? | 20:06 |
| fungi | that could explain it then | 20:06 |
| clarkb | redrobot: if you've got (or are willing generate) an ecdsa or ed25119 key to test auth to gerrit that may be useful to confirm that suspicion | 20:07 |
| clarkb | fungi: ya the thing that makes this confusing is the same (ish) terms are used to specify hashes used in a lot of different places and I think what is going on here is we support sha2 for a number of them but not the host pubkey (and probably not the client auth pubkey) excahgne | 20:11 |
| fungi | if so, seems like gerrit can't currently support rsa with fedora 33, and is worth trying to fix in gerrit upstream or mina-sshd? | 20:14 |
| clarkb | ya maybe start with a bug upstream and use `ssh -oPubkeyAcceptedKeyTypes=rsa-sha2-256 -p 29418 foo@gerrit` as the reproduction case | 20:18 |
| clarkb | in gerrit's initSignatures() method for sshd it calls setSignatureFactories(ServerBuilder.setUpDefaultSignatureFactories(false)); | 20:27 |
| clarkb | two things to note there. As metnioned before mac, cipher, and kex algorithrms seem configurable and the code to init those seems to confirm while this confirms signatures are not configurable. Also setUpDefaultSignatureFactories(false) implies we may be doing something to avoid certain defailts which might get us rsa-sha2-256 otherwise? | 20:28 |
| melwitt | from what I can tell, we have gerrit 3.2.5.1, but the diff ui doesn't look the way it does in the documentation https://gerrit-documentation.storage.googleapis.com/Documentation/3.2.5.1/user-review-ui.html#side-by-side does anyone know why? | 20:30 |
| clarkb | ok the false argument there is for a parameter named "ignoreUnsupported" not disabling defaults | 20:30 |
| clarkb | melwitt: if I had to guess the docs screenshots are from an older gerrit | 20:30 |
| clarkb | melwitt: the arrows in the top right indicate something like that | 20:31 |
| melwitt | yeah.. just seems so weird they'd include screenshots and ui instructions for old version in new documentation | 20:31 |
| fungi | i would not be surprised if they lag behind in replacing screenshots in their documentation | 20:31 |
| melwitt | I really want to do this https://gerrit-documentation.storage.googleapis.com/Documentation/3.2.5.1/user-review-ui.html#search but it doesn't work for me | 20:32 |
| fungi | the "new" documentation probably starts as "old" documentation, much like ours | 20:32 |
| clarkb | ya it looks like they have change the search key to mean put my cursor in the top level search box | 20:36 |
| clarkb | and not do a file level search (fwiw the file level search created a number of problems and was incredibly slow so maybe they ripped it out as not viable) | 20:37 |
| fungi | keyword searching in diffs is one of the main reasons i prefer gertty | 20:41 |
| fungi | i get annoyed by web pages overriding my in-browser search hotkeys | 20:41 |
| clarkb | https://github.com/apache/mina-sshd/blob/master/sshd-common/src/main/java/org/apache/sshd/common/signature/BuiltinSignatures.java the sshd software seems to support the sha2 rsa signature types | 20:42 |
| fungi | huh, so... | 20:42 |
| clarkb | https://github.com/apache/mina-sshd/blob/e021a6f4597cb7259162169851bb9e4cb85e1a39/sshd-core/src/main/java/org/apache/sshd/common/BaseBuilder.java#L118-L134 seems to default to include them too | 20:43 |
| clarkb | but that is master maybe the change is fairly recent? | 20:43 |
| *** elod has quit IRC | 20:43 | |
| fungi | setUpDefaultSignatureFactories(false) is the prob i guess | 20:43 |
| fungi | ? | 20:44 |
| *** ralonsoh has quit IRC | 20:44 | |
| redrobot | heya, sorry, was afk for a bit. I can help test/generate keys still if y'all need me to? | 20:45 |
| clarkb | redrobot: it might be good just to double check that ecdsa and/or ed25119 work fine (and if you're on fedora 33 it will be easier for you to confirm that than us) | 20:45 |
| clarkb | fungi: that is where we set up the defaults signatures | 20:45 |
| *** elod has joined #opendev | 20:45 | |
| clarkb | fungi: but that basically says don't ignore unknown signatures and use the defaults otherwise | 20:45 |
| redrobot | clarkb cool, I can do that now. | 20:46 |
| clarkb | fungi: https://github.com/apache/mina-sshd/blob/master/sshd-core/src/main/java/org/apache/sshd/server/ServerBuilder.java#L150-L153 | 20:47 |
| clarkb | and that seems independent of the Nio2 vs mina config option | 20:49 |
| redrobot | clarkb can confirm new ecdsa key works | 20:51 |
| clarkb | redrobot: thank you for confirming. | 20:52 |
| fungi | awesome, so at least switching to non-rsa keys is an available workaround which doesn't involve adjusting or weakening the client config | 20:53 |
| clarkb | https://github.com/apache/mina-sshd/blob/sshd-2.4.0/sshd-core/src/main/java/org/apache/sshd/server/ServerBuilder.java#L93-L109 I think that is closer to the version of what gerrit is running | 20:53 |
| clarkb | but still the sha2 rsa signatures are present | 20:53 |
| redrobot | clarkb ed25519 also works | 20:53 |
| clarkb | but the documentation note there says it is for HostKeyAlgorithms | 20:54 |
| fungi | mmm | 20:54 |
| clarkb | not public key? maybe there is another axis here that is in play? | 20:54 |
| fungi | host key is what we're dealing with though, right? | 20:58 |
| clarkb | well client side seems to matter too | 20:58 |
| clarkb | does the client send a signature of its pubkey too? | 20:58 |
| fungi | i'm not sure if client pubkey signature influences host pubkey signature | 21:00 |
| clarkb | it doesn't, but what if the server says " I can only accept the client pubkey in sha1 format" and then the fedora 33 client says "nope can't continue" ? | 21:00 |
| clarkb | the ultimate problem is a lack of mutual algorithm for some step in the process and there may be more than one that exchange a signature of pubkeys? | 21:01 |
| clarkb | (definitely the host key is exchagned via a signature) | 21:01 |
| clarkb | fungi: `ssh-keyscan -t ssh-rsa -p 29418 -v review.opendev.org` shows debug1: kex: host key algorithm: rsa-sha2-512 | 21:05 |
| clarkb | which has me thinking there is something weird about fedora again :/ | 21:05 |
| clarkb | redrobot: sorry to be a bother. But what if you set HostKeyAlgorithms=rsa-sha2-512 instead of PubkeyAcceptedKeyTypes=rsa-sha2-512 in your config? | 21:10 |
| clarkb | I think PubkeyAcceptedKeyTypes=rsa-sha2-512 interacts with local client side things as well and possibly in ways that don't make sense. But if this is only a host key problem then I think that other option may sort it out (and if it doesn't that is a good clue pointing to something else) | 21:11 |
| redrobot | clarkb no bother, I don't mind helping out to get this figured out | 21:11 |
| clarkb | fungi: fwiw gerrit openssh seems to use rsa-sha2-512 by default for the host key | 21:12 |
| clarkb | fungi: so that aspect is working how I would expect it especially after digging into the code a bit. I think some other aspect of the connection set up may be having problems | 21:13 |
| clarkb | or perhaps we were just using the wrong override | 21:13 |
| *** gmann_afk is now known as gmann | 21:13 | |
| clarkb | I'm going to proceed with shutting down those two openstackid servers using openstack server stop | 21:13 |
| clarkb | then will clean up dns | 21:13 |
| redrobot | clarkb Permission denied with HostKeyAlgorithms=rsa-sha2-512 | 21:15 |
| redrobot | although it did give me a conflict at first | 21:15 |
| redrobot | on a different key from known_hosts | 21:15 |
| fungi | this seems like it's either a bug in gerrit/mina-sshd or in fedora33/openssh | 21:16 |
| fungi | redrobot: i bet the "other" key is the sha1 hash for the same host key | 21:16 |
| clarkb | fungi: reading around more I wonder if it is your idea taht maybe we need to generate a new key? | 21:22 |
| clarkb | or rather regenerate to reencode it? | 21:22 |
| clarkb | I'm going to fiddle with ssh-keygen locally and see if things look different with different inputs | 21:22 |
| fungi | clarkb: well, it will be interesting if you find out that's the case, because my reading of the fedora bz is that it won't make any difference because the hash is not used for encoding the public key itself, only for exchanging it | 21:24 |
| clarkb | fungi: ya that is my understanding too. And the -vvv output of ssh seems to show that we're doing sha2 not sha1 with rsa for the host keys | 21:24 |
| clarkb | redrobot: are you able to use your old rsa pubkey to authenticate to other services/servers? | 21:25 |
| clarkb | the other wonder I've got is if it has to do with carrying forward an old key | 21:25 |
| clarkb | on the client side I mean | 21:25 |
| fungi | i wonder if https://bodhi.fedoraproject.org/updates/FEDORA-2020-413ab3bca3 is going to fix it, or if f33 users already have tha | 21:26 |
| fungi | t | 21:26 |
| clarkb | says it was pushed to stable 2 months ago? | 21:27 |
| fungi | related to https://bugzilla.redhat.com/show_bug.cgi?id=1881301 | 21:27 |
| openstack | bugzilla.redhat.com bug 1881301 in openssh "openssh-clients do not accept PubkeyAcceptedKeyTypes rsa-sha2-512/256" [Unspecified,Closed: errata] - Assigned to jjelen | 21:27 |
| clarkb | fungi: reading ssh-keygen man page says that -t specifcying the algorithm and not just the cipher only matters for using an RSA CA to sign things | 21:27 |
| clarkb | which implies to me that ya it shouldn't matter as we suspected | 21:27 |
| redrobot | clarkb I've only been kicking the tires on this F33 install today. So far the only issue was with gerrit. When I first was debugging I thought maybe my Yubikey->GPG->SSH setup was broken, but I have been able to ssh to other F33 hosts. | 21:28 |
| redrobot | I can try some centos hosts ... give me a few min. | 21:28 |
| clarkb | redrobot: I don't think the gpg -> ssh should really matter, unless that somehow encodes the signature stuff in hardware or similar to hide keys from the controlling computer | 21:29 |
| redrobot | looks like my F33 host and Centos8 host are both using ecdsa-sha-2nistp256 keys | 21:31 |
| clarkb | I wonder if openssh has an implied ssh-rsa pubkey type that is valid for rsa-sha2-* but fedora's config excludes ssh-rsa globally | 21:31 |
| redrobot | at least that's what known_hosts says | 21:31 |
| clarkb | fungi: just thinking out loud here, but what if the problem isn't our server but the updated config is too greedy and needs to enable ssh-rsa in places because the sha2 variants fall under that? | 21:32 |
| redrobot | FWIW, this is the known_hosts entry I have for RDO gerrit: [review.rdoproject.org]:29418,[38.102.83.25]:29418 ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLOLaO81l9MzOYM85DVm7RzDQp3GIPuf86F4gUbOQUYR8/1fGjZ6+5wrz3ZqIwx e4X5D43ZwRZD/DNFmvQPtYLg= | 21:32 |
| redrobot | note it's also ecdsa-sha2-nistp256 | 21:33 |
| fungi | redrobot: can you confirm | 21:33 |
| fungi | openssh-8.4p1-3.fc33 is what you're using? | 21:33 |
| redrobot | openssh-8.4p1-4.fc33.x86_64 | 21:33 |
| fungi | okay, cool | 21:35 |
| fungi | so rules out the -3 update having fixed it at least | 21:35 |
| clarkb | https://levelup.gitconnected.com/demystifying-ssh-rsa-in-openssh-deprecation-notice-22feb1b52acd scroll down to takeaway | 21:39 |
| clarkb | I think that may explain what we are seeing | 21:39 |
| clarkb | I bet gerrit's ssh server doesn't implement server-sig-algs so it is falling back to ssh-rsa there | 21:40 |
| clarkb | then we fail | 21:40 |
| clarkb | fedora likely hasn't updated their default fallback like that paragraph describs will be done upstream if they deprecate ssh-rsa for authentication | 21:40 |
| clarkb | yup I think this must be it | 21:45 |
| fungi | okay, so missing functionality in mina-sshd? | 21:46 |
| clarkb | the server-sig-algs log line shows up with this prefix debug1: kex_input_ext_info: so you get it at a -v level | 21:46 |
| clarkb | if you do that against some random openssh server that isn't anceitn you get that line in the debug log | 21:46 |
| clarkb | but do it against gerrit sshd and you don't get it | 21:46 |
| clarkb | fungi: yes, but also I think fedora should maybe change their fallback default if they are disabling ssh-rsa entirely | 21:46 |
| clarkb | fungi: there are a couple of issues here but reading that blog post it seems like if you are going to stop doing ssh-rsa for authentication then you also need to update your fallback | 21:47 |
| clarkb | upstream hasn't updated the fallback yet beacuse ssh-rsa is still the safer fallback for most systems | 21:47 |
| fungi | i can't seem to find where mina-sshd does its defect tracking | 21:48 |
| fungi | there are some pull requests in github, but none for that | 21:48 |
| clarkb | I think they have a jira | 21:49 |
| clarkb | fungi: https://github.com/apache/mina-sshd/blob/master/README.md grep server-sig-algs. Do you read that as them implying they don't support server-sig-algs and since people are disabling ssh-rsa you should switch to ecdsa or ed25119 | 21:49 |
| clarkb | ? | 21:49 |
| clarkb | its not very explicit but I take that as one reading there | 21:50 |
| fungi | RFC 8332 - Use of RSA Keys with SHA-256 and SHA-512 in the Secure Shell (SSH) Protocol: Note: - the server side supports these signatures by default. The client side requires specific initialization - see section 3.3 and also the above mentioned hooks for RFC 8308. | 21:50 |
| fungi | oh, is that what you were just referring to as well? | 21:51 |
| clarkb | I think it is related | 21:51 |
| clarkb | I think what fedora has done is break ssh-rsa for authentication in addition to host key signatures. Upstream is only deprecating it for host key signatures at this point | 21:52 |
| fungi | i noticed the readme had a link of reference rfcs implemented and was hunting for mention of the sha2 rfc (8332) | 21:52 |
| clarkb | the ssh protocol allows for server-sig-algs in negotiation to work around this but that only works if both sides know how to speak that extention and it appears our gerrit does not | 21:53 |
| clarkb | this means instead of using rsa-sha2-512 to authenticate the fedora client falls back to ssh-rsa and then fails | 21:53 |
| clarkb | I think our server could understand rsa-sha2-512 based on the DEFAULT_SIGNATURE_PREFERENCE values I linked previously | 21:53 |
| clarkb | but it isn't attempted beacuse the server doesn't say it can do it | 21:54 |
| clarkb | similarly when I do ssh -oPubkeyAcceptedKeyTypes=rsa-sha2-512 from my suse machine ti fails because the remote server isnt' saying it understands it | 21:54 |
| clarkb | (and so I think in my case it is falling back to ssh-rsa) | 21:55 |
| clarkb | I think gerrit should do server-sid-algs and ensure it supports authentication with rsa + sha2. However, fedora should also default to sha2 instead of ssh-rsa if they are jumping the gun since that is what the rfc seems to say they should do | 21:57 |
| clarkb | basically I think fedora half did their security update. But I also think gerrit sshd can be better too | 21:57 |
| clarkb | fungi: it looks like ServerSignatureAlgorithms does provide support for server-sig-algs in mina | 22:08 |
| fungi | but it's not firing? does gerrit need to explicitly call into it? | 22:08 |
| clarkb | ya I'm beginning to think this may be somethign gerrit needs to opt into when setting up the server? | 22:09 |
| *** sboyron has quit IRC | 22:11 | |
| clarkb | "**Note:** - the code contains **hooks** for implementing the RFC but beyond allowing convenient access to the required protocol details, it does not implement any logic that handles the messages." | 22:15 |
| clarkb | that was from a commit that added kex extension support in 2019 | 22:16 |
| clarkb | not sure if it has progressed from there, but certainly seems to look more and more like this may be the issue | 22:16 |
| clarkb | yup looks like they only ever implemented a default extension handler for the client side | 22:24 |
| clarkb | I think I've tracked this down far enough to file a gerrit bug. I'll work on that in a bit | 22:24 |
| clarkb | also this is another case of upstream doesn't use the software like anyone else and misses things like this (they only allow https access) :( | 22:25 |
| *** DSpider has quit IRC | 22:29 | |
| clarkb | fungi: redrobot: https://bugs.chromium.org/p/gerrit/issues/detail?id=13930 | 22:44 |
| clarkb | infra-root I think we can point fedora 33 users to ^ when issues come up. That captures taht using ecdsa or ed25119 to authenticate works or that you can explicitly enable ssh-rsa again | 22:46 |
| clarkb | and hopeflly soon it includes more info from upstream about what their thoughts on the subject are | 22:46 |
| clarkb | the last thing I'm wondering about is if openssh client lets you force the fallback to rsa-sha2-512 | 22:47 |
| clarkb | not seeing anything in ssh_config that might allow you to do that though | 22:49 |
| *** slaweq has quit IRC | 22:51 | |
| redrobot | clarkb, fungi, good work,y'all! I definitely learned a lot about ssh today. :) | 22:52 |
| clarkb | redrobot: thanks for helping with the debugging. I too learned a lot about ssh :) | 22:53 |
| *** tkajinam has joined #opendev | 22:57 | |
| *** icey has quit IRC | 23:08 | |
| *** icey has joined #opendev | 23:09 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!