Monday, 2021-01-11

*** prometheanfire has joined #opendev00:21
*** prometheanfire has quit IRC00:24
*** prometheanfire has joined #opendev00:27
*** JayF has quit IRC01:23
*** brinzhang has joined #opendev01:35
*** iurygregory has joined #opendev02:07
*** JayF has joined #opendev02:40
*** cloudnull has quit IRC03:38
*** ykarel has joined #opendev04:31
*** rosmaita has left #opendev04:43
*** marios has joined #opendev06:22
*** cloudnull has joined #opendev06:53
*** ysandeep|away is now known as ysandeep07:03
*** ralonsoh has joined #opendev07:40
*** eolivare has joined #opendev07:43
*** jpena|off is now known as jpena07:50
*** slaweq has joined #opendev07:51
*** hashar has joined #opendev08:01
*** andrewbonney has joined #opendev08:13
*** fressi has joined #opendev08:21
*** sboyron has joined #opendev08:22
*** ykarel is now known as ykarel|lunch08:30
*** sboyron has quit IRC08:30
*** rpittau|afk is now known as rpittau08:33
*** ralonsoh has quit IRC08:38
*** ykarel|lunch has quit IRC08:39
*** ralonsoh has joined #opendev08:41
*** lpetrut has joined #opendev08:45
*** yoctozepto has quit IRC08:48
*** kevinz has quit IRC08:53
*** sboyron has joined #opendev08:57
*** sshnaidm|afk is now known as sshnaidm|ruck08:57
*** DSpider has joined #opendev09:04
*** sboyron has quit IRC09:48
*** sboyron has joined #opendev10:26
*** ykarel has joined #opendev10:31
*** wiktri has joined #opendev10:34
*** dtantsur|afk is now known as dtantsur10:37
openstackgerritMerged opendev/elastic-recheck master: Add query for bug 1882521
openstackbug 1882521 in OpenStack Compute (nova) ussuri "Failing device detachments on Focal" [Undecided,New]
*** ykarel is now known as ykarel|afk11:50
*** ykarel|afk has quit IRC11:55
*** ykarel|afk has joined #opendev12:07
*** dviroel has joined #opendev12:11
openstackgerritMerged zuul/zuul-jobs master: Better error output for
*** Oriz has joined #opendev12:20
*** Oriz has quit IRC12:20
*** ykarel|afk has quit IRC12:22
*** jpena is now known as jpena|lunch12:30
*** ykarel has joined #opendev13:06
*** redrobot has joined #opendev13:19
*** ykarel_ has joined #opendev13:21
*** ykarel has quit IRC13:23
*** auristor has quit IRC13:24
*** auristor has joined #opendev13:24
*** ykarel_ is now known as ykarel13:25
*** jpena|lunch is now known as jpena13:29
*** yoctozepto has joined #opendev13:45
*** ralonsoh_ has joined #opendev13:59
*** ralonsoh has quit IRC13:59
*** ralonsoh_ is now known as ralonsoh14:01
*** DSpider has quit IRC14:24
*** zul has joined #opendev14:33
*** ralonsoh has quit IRC14:37
*** ralonsoh has joined #opendev14:38
*** tosky has joined #opendev14:39
*** ralonsoh has quit IRC14:40
*** spotz has joined #opendev14:57
*** lpetrut has quit IRC15:10
*** fressi has quit IRC15:10
*** ralonsoh has joined #opendev15:16
*** d34dh0r53 has joined #opendev15:22
*** ykarel has quit IRC15:44
*** brinzhang has quit IRC15:55
*** hashar has quit IRC16:09
*** cloudnull has quit IRC16:11
*** cloudnull has joined #opendev16:14
*** wiktri has quit IRC16:30
*** ysandeep is now known as ysandeep|dinner16:37
*** mlavalle has quit IRC16:41
*** mlavalle has joined #opendev16:41
*** jpena is now known as jpena|off16:50
*** DSpider has joined #opendev17:00
*** marios is now known as marios|out17:08
*** rpittau is now known as rpittau|afk17:10
*** tosky has quit IRC17:24
*** tosky has joined #opendev17:25
*** ysandeep|dinner is now known as ysandeep17:32
*** openstackgerrit has quit IRC17:37
redrobotHi #opendev friends!17:59
redrobotDo y'all own git-review?17:59
fungiwe maintain it, yes18:02
fungiwhat's up?18:02
fungiwe're the first order maintainers of anything in our opendev/ git namespace, so that includes
redrobotWell, I upgraded my system, and now git-review does not seem to be able to find my SSH key18:03
fungifedora 33?18:04
fungifedora decided to disable ssh-rsa keytypes18:04
fungithere are a number of potential workarounds. note this isn't a git-review specific problem, it's impacting ssh and git over ssh more generally18:05
fungiredrobot: i answered similar questions in this openstack-discuss ml thread:
redrobotI see ...  I didn't notice any ssh failures, just git-review so far.  I also have a kind of not-standard setup in that I'm using a Yubikey as a GPG card and using an RSA subkey as SSH key.18:06
redrobotfungi thanks, I'll give that thread a read18:06
fungianother option i forgot to mention in that ml post is you can switch to using https instead of ssh for connecting to gerrit18:07
clarkbfungi: did we ever figure out if clearing existing known hosts entries would cause it to find the ssh-rsa2 options?18:07
clarkbbecause does do ssh-rsa2 and its really confusing to me why that doesn't just work on fedora 33 unless old known host keys are set or ssh-rsa is entirely disabled18:07
fungiclarkb: good question, if redrobot wants to try deleting his known_hosts entries for review.o.o that could be a good experiment18:08
fungii don't know whether anyone has tested that explicitly18:08
fungithough in theory the UpdateHostKeys option would have a similar effect18:08
* redrobot is happy to be a guinea pig18:09
fungiredrobot: please do let us know how you get on. i've already seen folks work around this in several different ways, so any additional info you can render as to what works is appreciated18:09
redrobotNo dice.  It did ask to add the fingerprint to my known_hosts but then failed again in the same manner.18:11
redrobotWill get back to y'all after I read that thread an sort out my configuration. Thanks again.18:11
clarkbinfra-root I think is ready for review now (let me know if I can dig up the test instance that was held for you). As noted previously the gitea upgrade from 1.12 to 1.13 is big ish as gitea upgrades go since they added a bunch of new features like kanban boards. Definitely give it a reasonable review18:11
clarkbseparately fungi and I have talked to smarcet et al on the foundation side of openstackid and it sounds like we can spin down the extra capacity we turned on for the summit.18:12
clarkbfungi: I'm thinking maybe we shutdown those two servers today, then in a couple of days delete them if nothing breaks?18:13 and are the two servers in vexxhost which I will shutdown if that plan seems reasonabel18:14
fungiyeah, in theory nothing is hitting those servers, but i agree in proceeding with some measure of caution just in case18:16
*** hamalq has joined #opendev18:16
*** eolivare has quit IRC18:16
clarkbfungi: I think we need to put those servers in the emergency file when we do that too (otherwise ssh will fail to them)18:17
clarkbthen completely remove them from inventory once deleted?18:17
clarkbI Guess we can remove them from inventory now too18:17
fungior delete them from the inventory first?18:17
fungiright, that. jinx18:17
fungione or the other18:18
clarkbyup working on that change now18:18
*** sboyron has quit IRC18:18
fungii'd be fine with deleting them from inventory/dns in advance of shutting them down. then all that remains in a couple of days is deleting the server instances from the provider18:18
*** sboyron has joined #opendev18:19
clarkbI don't even know if we added them to dns since haproxy used addrs?18:19
clarkbactually no would've added to dns because its in the old domain which uses the rest client18:20
fungiet cetera18:20
fungiso yeah, there are dns entries18:21
fungino entries i'm aware of though18:21
clarkbwe lost gerritbot18:21
clarkbremote: Cleanup openstackid02 and openstackid0318:22
clarkbfungi: once that lands we can remove from dns (again to avoid ansible ssh problems)18:22
clarkbnow to restart gerritbot (it left saying it was changing servers according to my scrollback)18:22
clarkbfungi: also looks like we may be leaking ansible processes again due to logstash worker 18 and paste18:34
clarkbssh fails to both and paste http isn't accessible either18:35
clarkbfungi: do you think we should check console on paste or just restart them (I'm not worried about console on the logstash worker)18:35
fungii'll pull up th econsole for paste.o.o now18:36
fungithanks for noticing18:37
clarkblogstash-worker18 has been restarted18:37
clarkblet me know if you think I should do paste (or you can reboot it too)18:37
fungii have to wonder if we're the only rackspace users who see server instances regularly getting stuck like this18:37
clarkbya I was thinking about that and wonder if it has to do with us cleaning up the nova agent or something like that18:38
clarkblike maybe we need to do an arp after being requested post migration or something?18:38
fungijudging from cacti, paste.o.o hung sometime around 00:15 today18:39
fungiyeah, hung kernel tasks around 20216760 seconds after the last reboot18:41
fungiit's rebooting now18:41
clarkbthanks. I'll start cleaning up the stale ansible processes shortly18:42
fungi#status log rebooted paste.o.o just now in order to recover from a hung userspace earlier around 00:15 utc today18:42
openstackstatusfungi: finished logging18:42
*** andrewbonney has quit IRC18:43
redrobotfungi sorted my ssh issues with a config change to PubkeyAcceptedKeyTypes
clarkbredrobot: ok so that does seem to imply fedora 33 killed all ssh-rsa including the sha2 variants18:46
clarkbwhich seems like a completely broken way to do ssh18:46
clarkbonly the sha1 variant is deprecated upstream (and it isn't even removed tehre yet either)18:47
fungipaste uptime is 10 minutes, and i can get to the webui again18:53
fungii think we're all good there18:53
clarkbI've got ansible processes from yseterday and prior cleaned up18:55
clarkbneed to look more closely at those from today to avoid killing happy processes18:55
clarkbok now there are only a couple of processes from just under a couple of hours ago that I suspect have leaked. I'll let them sit around a bit long er to feel stronger in that suspicion19:03
*** openstackgerrit has joined #opendev19:04
openstackgerritMerged opendev/system-config master: Cleanup openstackid02 and openstackid03
*** marios|out has quit IRC19:04
*** _mlavalle_1 has joined #opendev19:05
fungiclarkb: ^ i guess you can shut them down now19:08
*** mlavalle has quit IRC19:08
clarkbthanks. Kids just started lunch break so I'll grab that in a bit. I'll do the dns cleanup then too19:08
redrobotfungi, clarkb, I found an interesting thread on the Fedora mailing list that made me dig into this ssh config rabbit hole some more.19:10
redrobotI was able to narrow down the needed config change to just this:
redrobotIt appears that adding a "+" symbol to the value for PubkeyAcceptedKeyTypes bypasses the fedora crypto policy and enables the default openssh policy instead19:11
redrobotwhich pulls in ssh-rsa19:11
redrobotso that's why my earlier paste worked19:11
clarkbredrobot: well you actually don't want to use ssh-rsa if you can avoid it19:11
clarkbyou want to use the ssh-rsa sha2 options which your first paste attempted to do19:12
clarkbour gerrit does support those and they are not deprecated by upstream openssh19:12
clarkbI think we can disable ssh-rsa in our gerrit config too19:13
fungiyeah, i think that's what he's saying, so my earlier post adding sha2 rsa worked but also caused the crypto policy restrictions to be completely undone?19:13
clarkbmaybe we should consider that and see if that helps fedora clients find the right type19:13
redrobotclarkb yeah, it's weird because just enabling those two does not work (unless you use + , which also pulls uin ssh-rsa)19:14
redrobote.g this does not work
clarkbhuh I know I tested this at one time but on my suse machien just manually setting options19:14
redrobotmay ultimately be a bug in Fedora.  Seems the discussion is still ongoing bug 1881301 in openssh "openssh-clients do not accept PubkeyAcceptedKeyTypes rsa-sha2-512/256" [Unspecified,Closed: errata] - Assigned to jjelen19:15
*** dtantsur is now known as dtantsur|afk19:15
clarkbfungi: maybe you can independently try to confirm that the sha2 rsa options work for you via debain?19:16
clarkbjust as a way of double checking the new server is doing what we expect of it?19:16
fungii would need to exclude sha1 ssh-rsa19:16
fungidebian doesn't block use of it, they're much more conservative about breaking userspace19:17
clarkbyup same thing with suse. iirc I used an option to set the specific type to use and set it to a sha2 version and it worked19:17
clarkbbut I'd have to go through my notes and docs again19:17
fungiotherwise there are tons of folks who would be, like, "now i can't manage my 15-year-old load balancer which uses f-secure sshd 1.x)19:17
*** gmann is now known as gmann_afk19:25
fungiso sha2 for ssh rsa is covered by
fungikey algorithms would be listed as rsa-sha2-256 and rsa-sha2-51219:27
fungiwhen i `ssh-keyscan -p 29418` i don't see those, only: ssh-rsa ecdsa-sha2-nistp256 ssh-ed2551919:27
fungimaybe we need to recreate the public key from the current private key?19:37
clarkbhrm aren't these hashes of the public key? and not the raw public key material?19:50
clarkbya I think the reason you don't see them on a keyscan is the key type is ssh-rsa but the hash type during connection negotiation can be ssh-rsa or rsa-sha2-256 etc19:52
clarkbthey are different things19:52
clarkbfungi: from the bugzilla: "No, it is not. It generates the "same" rsa key as before. The digest is not reflected in the key at all."19:54
clarkb-oPubkeyAcceptedKeyTypes=rsa-sha2-256 seems to be the flag to ssh to force the sha2 option and that does indeed fail for me. I wonder if I used a + before and ran into the unexpected behavior of it pulling in defaults as a result19:57
clarkbI have confirmed that if I use a + that works19:59
clarkblooking at gerrit docs it seems we can configure which key exchange, mac, and cipher algrorithms are allowed but not host key?20:00
clarkbmy next question is then why doesn't fedora use ecdsa or ed25119 since those are both fine on fedora right?20:01
clarkbcould this be related to an existing rsa hostkey so it is looking for that?20:01
clarkbhrm looking at ssh -vvv output it seems the key used to authenticate impacts this as well20:03
fungiwhat am i missing? does this come down to support for server-sig-algs?
clarkbok PubkeyAcceptedKeyTypes seems to affect both the client and the server. So if I tell it to use ecdsa exclusively it won't authenticate with my ssh-rsa key20:04
clarkbeven if it can get a valid server pubkey via that method20:04
fungifedora would likely use ecdsa or ed25119 if you have one of those for your client keys20:04
fungithat much i had already assumed20:05
clarkbfungi: I think gerrit isn't doing rsa-sha2-256 or rsa-sha2-512 to hash the ssh-rsa pubkey during connection setup. It supports sha2 variants for other rsa things according to the config docs (mac and kex)20:06
clarkbbut the pubkey signatrue sent it always sha1 for the host key ?20:06
fungithat could explain it then20:06
clarkbredrobot: if you've got (or are willing generate) an ecdsa or ed25119 key to test auth to gerrit that may be useful to confirm that suspicion20:07
clarkbfungi: ya the thing that makes this confusing is the same (ish) terms are used to specify hashes used in a lot of different places and I think what is going on here is we support sha2 for a number of them but not the host pubkey (and probably not the client auth pubkey) excahgne20:11
fungiif so, seems like gerrit can't currently support rsa with fedora 33, and is worth trying to fix in gerrit upstream or mina-sshd?20:14
clarkbya maybe start with a bug upstream and use `ssh -oPubkeyAcceptedKeyTypes=rsa-sha2-256 -p 29418 foo@gerrit` as the reproduction case20:18
clarkbin gerrit's initSignatures() method for sshd it calls setSignatureFactories(ServerBuilder.setUpDefaultSignatureFactories(false));20:27
clarkbtwo things to note there. As metnioned before mac, cipher, and kex algorithrms seem configurable and the code to init those seems to confirm while this confirms signatures are not configurable. Also setUpDefaultSignatureFactories(false) implies we may be doing something to avoid certain defailts which might get us rsa-sha2-256 otherwise?20:28
melwittfrom what I can tell, we have gerrit, but the diff ui doesn't look the way it does in the documentation does anyone know why?20:30
clarkbok the false argument there is for a parameter named "ignoreUnsupported" not disabling defaults20:30
clarkbmelwitt: if I had to guess the docs screenshots are from an older gerrit20:30
clarkbmelwitt: the arrows in the top right indicate something like that20:31
melwittyeah.. just seems so weird they'd include screenshots and ui instructions for old version in new documentation20:31
fungii would not be surprised if they lag behind in replacing screenshots in their documentation20:31
melwittI really want to do this but it doesn't work for me20:32
fungithe "new" documentation probably starts as "old" documentation, much like ours20:32
clarkbya it looks like they have change the search key to mean put my cursor in the top level search box20:36
clarkband not do a file level search (fwiw the file level search created a number of problems and was incredibly slow so maybe they ripped it out as not viable)20:37
fungikeyword searching in diffs is one of the main reasons i prefer gertty20:41
fungii get annoyed by web pages overriding my in-browser search hotkeys20:41
clarkb the sshd software seems to support the sha2 rsa signature types20:42
fungihuh, so...20:42
clarkb seems to default to include them too20:43
clarkbbut that is master maybe the change is fairly recent?20:43
*** elod has quit IRC20:43
fungisetUpDefaultSignatureFactories(false) is the prob i guess20:43
*** ralonsoh has quit IRC20:44
redrobotheya, sorry, was afk for a bit.  I can help test/generate keys still if y'all need me to?20:45
clarkbredrobot: it might be good just to double check that ecdsa and/or ed25119 work fine (and if you're on fedora 33 it will be easier for you to confirm that than us)20:45
clarkbfungi: that is where we set up the defaults signatures20:45
*** elod has joined #opendev20:45
clarkbfungi: but that basically says don't ignore unknown signatures and use the defaults otherwise20:45
redrobotclarkb cool, I can do that now.20:46
clarkband that seems independent of the Nio2 vs mina config option20:49
redrobotclarkb can confirm new ecdsa key works20:51
clarkbredrobot: thank you for confirming.20:52
fungiawesome, so at least switching to non-rsa keys is an available workaround which doesn't involve adjusting or weakening the client config20:53
clarkb I think that is closer to the version of what gerrit is running20:53
clarkbbut still the sha2 rsa signatures are present20:53
redrobotclarkb ed25519 also works20:53
clarkbbut the documentation note there says it is for HostKeyAlgorithms20:54
clarkbnot public key? maybe there is another axis here that is in play?20:54
fungihost key is what we're dealing with though, right?20:58
clarkbwell client side seems to matter too20:58
clarkbdoes the client send a signature of its pubkey too?20:58
fungii'm not sure if client pubkey signature influences host pubkey signature21:00
clarkbit doesn't, but what if the server says " I can only accept the client pubkey in sha1 format" and then the fedora 33 client says "nope can't continue" ?21:00
clarkbthe ultimate problem is a lack of mutual algorithm for some step in the process and there may be more than one that exchange a signature of pubkeys?21:01
clarkb(definitely the host key is exchagned via a signature)21:01
clarkbfungi: `ssh-keyscan -t ssh-rsa -p 29418 -v` shows debug1: kex: host key algorithm: rsa-sha2-51221:05
clarkbwhich has me thinking there is something weird about fedora again :/21:05
clarkbredrobot: sorry to be a bother. But what if you set HostKeyAlgorithms=rsa-sha2-512 instead of PubkeyAcceptedKeyTypes=rsa-sha2-512 in your config?21:10
clarkbI think PubkeyAcceptedKeyTypes=rsa-sha2-512 interacts with local client side things as well and possibly in ways that don't make sense. But if this is only a host key problem then I think that other option may sort it out (and if it doesn't that is a good clue pointing to something else)21:11
redrobotclarkb no bother, I don't mind helping out to get this figured out21:11
clarkbfungi: fwiw gerrit openssh seems to use rsa-sha2-512 by default for the host key21:12
clarkbfungi: so that aspect is working how I would expect it especially after digging into the code a bit. I think some other aspect of the connection set up may be having problems21:13
clarkbor perhaps we were just using the wrong override21:13
*** gmann_afk is now known as gmann21:13
clarkbI'm going to proceed with shutting down those two openstackid servers using openstack server stop21:13
clarkbthen will clean up dns21:13
redrobotclarkb Permission denied with  HostKeyAlgorithms=rsa-sha2-51221:15
redrobotalthough it did give me a conflict at first21:15
redroboton a different key from known_hosts21:15
fungithis seems like it's either a bug in gerrit/mina-sshd or in fedora33/openssh21:16
fungiredrobot: i bet the "other" key is the sha1 hash for the same host key21:16
clarkbfungi: reading around more I wonder if it is your idea taht maybe we need to generate a new key?21:22
clarkbor rather regenerate to reencode it?21:22
clarkbI'm going to fiddle with ssh-keygen locally and see if things look different with different inputs21:22
fungiclarkb: well, it will be interesting if you find out that's the case, because my reading of the fedora bz is that it won't make any difference because the hash is not used for encoding the public key itself, only for exchanging it21:24
clarkbfungi: ya that is my understanding too. And the -vvv output of ssh seems to show that we're doing sha2 not sha1 with rsa for the host keys21:24
clarkbredrobot: are you able to use your old rsa pubkey to authenticate to other services/servers?21:25
clarkbthe other wonder I've got is if it has to do with carrying forward an old key21:25
clarkbon the client side I mean21:25
fungii wonder if is going to fix it, or if f33 users already have tha21:26
clarkbsays it was pushed to stable 2 months ago?21:27
fungirelated to bug 1881301 in openssh "openssh-clients do not accept PubkeyAcceptedKeyTypes rsa-sha2-512/256" [Unspecified,Closed: errata] - Assigned to jjelen21:27
clarkbfungi: reading ssh-keygen man page says that -t specifcying the algorithm and not just the cipher only matters for using an RSA CA to sign things21:27
clarkbwhich implies to me that ya it shouldn't matter as we suspected21:27
redrobotclarkb I've only been kicking the tires on this F33 install today.  So far the only issue was with gerrit.  When I first was debugging I thought maybe my Yubikey->GPG->SSH setup was broken, but I have been able to ssh to other F33 hosts.21:28
redrobotI can try some centos hosts ... give me a few min.21:28
clarkbredrobot: I don't think the gpg -> ssh should really matter, unless that somehow encodes the signature stuff in hardware or similar to hide keys from the controlling computer21:29
redrobotlooks like my F33 host and Centos8 host are both using ecdsa-sha-2nistp256 keys21:31
clarkbI wonder if openssh has an implied ssh-rsa pubkey type that is valid for rsa-sha2-* but fedora's config excludes ssh-rsa globally21:31
redrobotat least that's what known_hosts says21:31
clarkbfungi: just thinking out loud here, but what if the problem isn't our server but the updated config is too greedy and needs to enable ssh-rsa in places because the sha2 variants fall under that?21:32
redrobotFWIW, this is the known_hosts entry I have for RDO gerrit: []:29418,[]:29418 ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLOLaO81l9MzOYM85DVm7RzDQp3GIPuf86F4gUbOQUYR8/1fGjZ6+5wrz3ZqIwx    e4X5D43ZwRZD/DNFmvQPtYLg=21:32
redrobotnote it's also ecdsa-sha2-nistp25621:33
fungiredrobot: can you confirm21:33
fungiopenssh-8.4p1-3.fc33 is what you're using?21:33
fungiokay, cool21:35
fungiso rules out the -3 update having fixed it at least21:35
clarkb scroll down to takeaway21:39
clarkbI think that may explain what we are seeing21:39
clarkbI bet gerrit's ssh server doesn't implement server-sig-algs so it is falling back to ssh-rsa there21:40
clarkbthen we fail21:40
clarkbfedora likely hasn't updated their default fallback like that paragraph describs will be done upstream if they deprecate ssh-rsa for authentication21:40
clarkbyup I think this must be it21:45
fungiokay, so missing functionality in mina-sshd?21:46
clarkbthe server-sig-algs log line shows up with this prefix debug1: kex_input_ext_info: so you get it at a -v level21:46
clarkbif you do that against some random openssh server that isn't anceitn you get that line in the debug log21:46
clarkbbut do it against gerrit sshd and you don't get it21:46
clarkbfungi: yes, but also I think fedora should maybe change their fallback default if they are disabling ssh-rsa entirely21:46
clarkbfungi: there are a couple of issues here but reading that blog post it seems like if you are going to stop doing ssh-rsa for authentication then you also need to update your fallback21:47
clarkbupstream hasn't updated the fallback yet beacuse ssh-rsa is still the safer fallback for most systems21:47
fungii can't seem to find where mina-sshd does its defect tracking21:48
fungithere are some pull requests in github, but none for that21:48
clarkbI think they have a jira21:49
clarkbfungi: grep server-sig-algs. Do you read that as them implying they don't support server-sig-algs and since people are disabling ssh-rsa you should switch to ecdsa or ed2511921:49
clarkbits not very explicit but I take that as one reading there21:50
fungiRFC 8332 - Use of RSA Keys with SHA-256 and SHA-512 in the Secure Shell (SSH) Protocol: Note: - the server side supports these signatures by default. The client side requires specific initialization - see section 3.3 and also the above mentioned hooks for RFC 8308.21:50
fungioh, is that what you were just referring to as well?21:51
clarkbI think it is related21:51
clarkbI think what fedora has done is break ssh-rsa for authentication in addition to host key signatures. Upstream is only deprecating it for host key signatures at this point21:52
fungii noticed the readme had a link of reference rfcs implemented and was hunting for mention of the sha2 rfc (8332)21:52
clarkbthe ssh protocol allows for server-sig-algs in negotiation to work around this but that only works if both sides know how to speak that extention and it appears our gerrit does not21:53
clarkbthis means instead of using rsa-sha2-512 to authenticate the fedora client falls back to ssh-rsa and then fails21:53
clarkbI think our server could understand rsa-sha2-512 based on the DEFAULT_SIGNATURE_PREFERENCE values I linked previously21:53
clarkbbut it isn't attempted beacuse the server doesn't say it can do it21:54
clarkbsimilarly when I do ssh -oPubkeyAcceptedKeyTypes=rsa-sha2-512 from my suse machine ti fails because the remote server isnt' saying it understands it21:54
clarkb(and so I think in my case it is falling back to ssh-rsa)21:55
clarkbI think gerrit should do server-sid-algs and ensure it supports authentication with rsa + sha2. However, fedora should also default to sha2 instead of ssh-rsa if they are jumping the gun since that is what the rfc seems to say they should do21:57
clarkbbasically I think fedora half did their security update. But I also think gerrit sshd can be better too21:57
clarkbfungi: it looks like ServerSignatureAlgorithms does provide support for server-sig-algs in mina22:08
fungibut it's not firing? does gerrit need to explicitly call into it?22:08
clarkbya I'm beginning to think this may be somethign gerrit needs to opt into when setting up the server?22:09
*** sboyron has quit IRC22:11
clarkb"**Note:** - the code contains **hooks** for implementing the RFC but beyond allowing convenient access to the required protocol details, it does not implement any logic that handles the messages."22:15
clarkbthat was from a commit that added kex extension support in 201922:16
clarkbnot sure if it has progressed from there, but certainly seems to look more and more like this may be the issue22:16
clarkbyup looks like they only ever implemented a default extension handler for the client side22:24
clarkbI think I've tracked this down far enough to file a gerrit bug. I'll work on that in a bit22:24
clarkbalso this is another case of upstream doesn't use the software like anyone else and misses things like this (they only allow https access) :(22:25
*** DSpider has quit IRC22:29
clarkbfungi: redrobot:
clarkbinfra-root I think we can point fedora 33 users to ^ when issues come up. That captures taht using ecdsa or ed25119 to authenticate works or that you can explicitly enable ssh-rsa again22:46
clarkband hopeflly soon it includes more info from upstream about what their thoughts on the subject are22:46
clarkbthe last thing I'm wondering about is if openssh client lets you force the fallback to rsa-sha2-51222:47
clarkbnot seeing anything in ssh_config that might allow you to do that though22:49
*** slaweq has quit IRC22:51
redrobotclarkb, fungi, good work,y'all!  I definitely learned a lot about ssh today. :)22:52
clarkbredrobot: thanks for helping with the debugging. I too learned a lot about ssh :)22:53
*** tkajinam has joined #opendev22:57
*** icey has quit IRC23:08
*** icey has joined #opendev23:09

Generated by 2.17.2 by Marius Gedminas - find it at!