Secure transport protocols all do some kind of handshake to set up a session, agree on keys, etc. Modern secure transport protocols (everything since SSL2) authenticate the handshake, so a MITM can't just edit the messages and, like, stick both sides on the NULL cipher (don't have a NULL cipher, though).
Ever since Kocher and SSL 3.0, the gold standard for handshake authentication has been to keep a transcript of all the handshake messages, and just hash them to fingerprint the entire handshake. You can look at Noise for a streamlined, slick version of the same concept.
SSH does something else: it looks at the handshake as a vehicle for setting up a DH-style key exchange; that's all it's for, everything else happens inside the secure transport that key exchange provides. So instead of doing a transcript hash, SSH picks out the handshake values that will end up being inputs to the key exchange, and hashes those.
The problem is: SSH also does implicit sequence numbers; receivers keep track of how many messages they've received, senders keep track of how many they've sent. Not only that, but SSH has (for reasons passing understanding) a NOP message (`IGNORE`). `IGNORE` carries no data used to do key generation, so it has no impact on the handshake authentication --- but it does impact sequence numbers.
Result: MITM attackers can set sequence numbers to arbitrary values (by injecting `IGNORE`s in the handshake), and then edit out subsequent messages (by just not sending them). If you're using ChaPoly (and often if you're using CBC), the protocol will sync up and keep going. You can use this to, for instance, snipe out extension messages (for things like keystroke timing mitigation) from the beginning of an SSH session.
This is a pretty obvious problem! It's absolutely not something you can just accept from a secure transport protocol. And you could look at SSH and SSL3 and see "SSH is doing something really different and less sophisticated than SSL3". But it took until 2023 for someone to do the legwork to figure out how broken it was.
I think that's a really good question. The way this worked out is worth studying in detail. What was the process with which the AES-GCM cipher suites for SSH were developed? What was the process with which the ChaCha20-Poly1305 cipher suites were developed? How did the difference in processes lead to the difference in results? Will anybody change their process based on these results?
Super, super interesting. What a cool bit of research, and as you said in your other comment also a interesting bit of living history as well.
As far as mitigations and Noise, I've been tunneling all my SSH connections through WireGuard or Nebula anyway primarily just because they're such easy reliable ways to reach hosts behind NAT with in secure fashion, and while there is certainly overhead in putting SSH through something else all the tunnels are fat and fast enough that for just console control it's been fine, haven't had to use mosh (does mosh have the same issue?). Even through Starlink it's never a problem. But one does wonder a bit anyway with all the really old protocols at this point, just feels like there have been a lot of fundamental shifts in thinking around security (simplicity of implementations, not having lots of buttons and switches and flexibility, etc) such that there are less likely to be hidden bugbears now. There is more scrutiny not just day 1 but in the whole process of design.
Not that SSH isn't still important to fix but I wonder if just tunneling everything is a decent default at this point. I use internal VPNs for everything management related but not air gapped at this point, not just external. Maybe that's overkill or foolish doubling up? But it's convenient, performant, and bypasses a lot of complexity in other layers.
This paper is the best protocol security research I've seen in a while. Thankfully the consequences aren't too big - everyone should update ssh (which will be a pain!) - but it likely doesn't need to happen at breakneck pace. This issue isn't quite as serious as prior issues that impacted TLS/SSL.
Two things stand out to me;
1. That the SSH protocol lacked a robust transcript hash all along, this seems like a staggering decision, and even though I'd read the RFCs, I'd never put this together. Turns out it's to allow for injection of messages designed to defeat traffic analysis? Not the right way to do it.
2. The SSH handshake had been formally verified ... and the proofs are still "true". But the researchers found non-obvious gaps between the environmental assumptions for the proofs and what is actually true in the real world. Formal verification is still the most comprehensive and exhaustive form of testing that we have, but gaps can really bite you. It is like relying Pythagoras' theorem for a triangle, but then it turns out that your triangles are on the surface of a sphere, which isn't a plane. The math itself is right, but the environment
was different all along.
Isn't this great? I can put myself "in the room" when they decided not to do a full transcript MAC. The SSL3 transcript hash is annoying! It was the first protocol I personally encountered that did that. And what are you trying to protect? A key exchange, right? Why not just create a mechanism that catches the inputs to the key exchange, authenticates them, and ignores everything else? Everything else is just a vehicle for communicating those inputs.
So like, you could imagine yourself looking at the SSL 3.0 "Finished" message and the SSH "Exchange Hash" and asking yourself, why do the complicated thing? The SSH thing seems to work!
And: if you asked yourself that, like, in 2013, you could go on believing for ten years that everything was fine, and SSH just found a clever way to do handshake integrity less cumbersomely than SSL 3.0.
There's like an obvious lesson here about PGP and the MDC, right?
ChaPoly was added in 2013, but the weird KEX is even older, dating back all the way to 1998 in SSHv2. And surprisingly, the attack only works with the "better" symmetric ciphers that do INT-CTXT instead of INT-PTXT.
>There's like an obvious lesson here about PGP and the MDC, right?
Not sure that all the readers will already know about our constant quibbling about the security of the MDC. I have in the past suggested that the fact that the MDC has remained unbroken for 20+ years is evidence that it is secure.
Interestingly enough, I have recently stumbled across a paper[1] that shows that preventing an attacker from computing the hash is important for insuring the security of hash than encrypt schemes like the MDC. OpenPGP does that by preseeding the hash with a random, encrypted value.
So does that mean that the MDC is secure? Unfortunately, when you come up with a set of requirements for security, you have to use your knowledge of existing attacks. You don't get to know about about future attacks. So the fact that a particular construct has never been broken over a long time is in a sense the only objective evidence available.
The MDC thing has nothing to do with you. It's a perennial complaint among cryptographers about how PGP's design has escaped modernity. It's weird that you think HN is somehow the center of that phenomenon.
It's an archaic premodern construction that cryptographers hate but that hasn't been demolished in a paper yet, in part because lots of people use SSH and very few people seriously use PGP. I'm confused, though: what would it have to do with you?
Uh, if it was that easy. I recently(!) audited a brand new system for deployment in what most nations consider critical infrastructure. They had just updated to the newest firmware supplied by a well-known and established manufacturer in that industry (not some garbage by a small company). That ran a dropbear ssh server version from 2016 with multiple CVEs, including an unauthenticated RCE. So essentially it's a root shell for anyone that can talk IP to that device.
I predict I will start seeing this terrapin in my Nessus reports soon, and stop seeing it when I enter pension.
it seems obvious that as long as there's no disclosure there's no real need to fix things. audits mean nothing if suddenly things can disappear from The Scope and appear somehow mItIgAtEd.
Strictly speaking audit is the wrong word for what I did (I'm not a certified auditor). I don't want to say too much in case I break some confidentiality agreement. Though I'm curious how that will be argued away, or if they get the manufacturer to get their shit together and build a new firmware...
> The SSH handshake had been formally verified ... and the proofs are still "true".
This seems unsurprising for me. Formal verification advocates believe it will prevent all issues, but it seemed obvious to me that the verification is still vulnerable to human fallibility in deciding _what_ to verify.
Short version of the net threat/risk of this appears to be a sophsticated attacker on the network may snipe ssh session negotiations with injected packets and cause the session to not-apply mitigations against keystroke timing analysis - which in turn may expose an ssh authN password via implementing that timing attack, or sudo credentials on the other side of the connection.
I'm interpreting the likelihood of encountering this near term seems very low, and the impact is high enough where an APT group uses it as part of their pivot and persistence strategy, but it's mitigated by the fact that it doesn't compromise a clients private key. There may be some future implementation of this attack that enables automated ssh session hijacking as a result of a key downgrade and an additional cryptographic weakness in that chosen weaker cipher. It implies the network IoCs would be for a bunch of NOP ssh packets and failed ssh session setups.
Actually, you cannot perform keystroke timing attacks on the password sent during SSH user authentication. This is because SSH transmits the password in its entirety, not the individual keystrokes as it does during a shell session.
The possible downgrade is usually limited to downgrading SHA2 to SHA1 when using RSA keys for user authentication. We are currently not aware of any attacks that benefit from this.
> The possible downgrade is usually limited to downgrading SHA2 to SHA1 when using RSA keys for user authentication
AFAIK this downgrade is not possible for OpenSSH, as it refuses RSA/SHA1 signatures by default at both the client and server. Moreover, deleting the EXT_INFO message would only remove the _advertisement_ that these messages are accepted and would change the fact of whether they are.
It might be possible to use this attack to trick an old client to making a RSA/SHA1 signature where it might mormally have made an RSA/SHA2 one, but I can't see any security impact from this as the signed data is still bound to the connection instance and it wouldn't change whether the peer would accept it anyway.
I don't think dropping the server-sig-algs packet gives any additional downgrade capability? If a server is going to accept a rsa sha1 signature it will accept it regardless of whether the client receives the server-sig-algs or not. Performing a rsa sha1 signature doesn't cause any exposure for a client (that I know of?).
server-sig-algs was only added for compatibility, so new clients wouldn't get kicked off old servers that don't accept rsa-sha2 pubkey auth packets. https://www.rfc-editor.org/rfc/rfc8308#page-7
Netgate is providing a pfsense patch that disables ETM MACs but leaves chacha20 enabled. The vuln announcement is arguably vague because of the way is uses "and" and "or" but from the vuln scanner source chacha20 is clearly not okay even without ETM.
Yes, I commented this over at netgate as well. If you'd like to work around the vulnerability, you will need to disable both, [email protected] encryption and [email protected] MACs. Keeping either of them enabled still allows for exploitation.
Nothing's really changed today. The workarounds have become smoother and some of the things we can do with networks of Unix machines are pretty impressive, but when ssh is the foundation of your security architecture, you know things aren't working as they should.
I feel like this is quoting out of context. Pike was saying that SSH is the absolute bare minimum of integration between systems on a network you could ask for, and that Plan9 and others have had much greater visions.
Where can I learn more about securing networks? SSH is something I know well by inertia (academic upbringing, and seems like SSH is ubiquitous!). What are the alternatives?
After that quote he says hardware getting smaller will disrupt Microsoft and make desktops and laptops seem old. The thing he describes next is very different from a smartphone (which to be clear, existed in 2004) but seems like a good insight.
Resolved in OpenSSH 9.6, which was released yesterday. Unfortunately it appears both client and server must be updated for the change to automatically be effective.
Would be nice if there was a way to configure the server to reject clients that don't support "strict KEX".
> OpenSSH 9.6 addresses this protocol weakness through a new "strict KEX" protocol extension that will be automatically enabled when both the client and server support it. This extension makes two changes to the SSH transport protocol to improve the integrity of the initial key exchange.
Terrapin-Attack Style Vulnerability Likely Exploited for 2 Years!!
I wanted to share some insights into a security concern that echoes the Terrapin-Attack scenario, highlighting a similar vulnerability that has been observed in other tools.
Recently, I came across a Pull Request on GitHub for the SSH-MITM tool, which sheds light on a critical aspect of SSH protocol security, specifically regarding RFC 4253 during the KEXINIT process. The Pull Request, available at GitHub https://github.com/ssh-mitm/ssh-mitm/pull/163, describes the necessity of discarding certain packages during the KEXINIT phase to prevent issues with intercepted clients.
Moreover, a look into the GitHub Blame for SSH-MITM reveals that these crucial changes in the KEXINIT step were integrated into SSH-MITM about 1-2 years ago. You can see the specific changes at this link: https://github.com/ssh-mitm/ssh-mitm/blame/4fc3ef418847c35d1...
An important note to add is that this information suggests that a similar form of attack, akin to the Terrapin-Attack, could potentially have been exploited for the last two years. This raises significant concerns about the historical vulnerability of systems to such attack techniques and emphasizes the importance of retroactive security analysis in addition to ongoing vigilance.
The approach by ssh-mitm differs quite significantly as already mentioned by Manfred Kaiser in the issue linked above. Most significantly, ssh-mitm basically establishes two connections (one as a server, one as a client) and forwards messages between the two. This however requires the client to accept the hostkey from the MitM (or the MitM to get a hold on the private key of the server's hostkey). In comparison, our attack manipulates a single connection, without the attacker knowing the hostkey or a client accepting a malicious hostkey. For more details, see the issue here: https://github.com/ssh-mitm/ssh-mitm/issues/165
> I am an admin, and it's not clear _how_ I might fix this at the server end.
You're just hearing about this early as it's the first hours of public disclosure. There are 3 CVE's assigned. You'll track those with your vendors or upstream and patch as fixes become available, which they already should be for at least some implementations that had earlier disclosure. The main issue is CVE-2023-48795. It does not appear there is any immediate need to implement compensating controls or workarounds at this time, but that assessment really depends on your own environment, security policy, and risk tolerance.
No. Mitigations were available for openssh before you wrote that comment. Follow the recommendations from ssh-audit (master version). [0]
For packages, openssh and openssh-portable 9.6 include the fix, but every vendor will need to backport if they maintain their own pile of patches against a particular version of -portable.
We updated the FAQ question to cover this. For most users, fixing can be done by installing patched versions of their SSH implementations as they come available. If you feel uncomfortable waiting for an update, you may disable the affected cipher modes temporarily and use other modes like AES-GCM instead. But keep in mind that a faulty configuration can cause you to loose access to the server.
Fwiw, I found a bunch of clients and libraries that didn't support AES-GCM at all, or only very recently. libssh2 is one example. That's made me nervous to make AES-GCM the only supported cipher as a mitigation.
Depends a bit on the MAC. CTR-EtM is technically vulnerable (i.e. cryptographically broken), but due to key stream desynchronization the attack will quickly lead to application errors, defeating the attacker. See Sect. 4.3.3.
No, because there may be other messages that are ignored, i.e. don't trigger a response message. Any such message can be used for injection. The details are implementation specific, though. The new strict-kex disallows all unexpected messages during the initial handshake, which helps a lot. (Even better would be to authenticate the complete handshake transcript). Another mitigation is resetting the sequence number. Both together give some redundancy.
This sounds like another reason to no longer put SSH on the bare internet and layer security by only exposing wireguard and having sshd listen on the wireguard network.
This is not helpful for currently vulnerable setups but hopefully will be a good reminder to think about defense in depth :)
On the other hand, if the WireGuard implementation has a vulnerability, the attacker does not need to attack SSH at all and simply punch through WireGuard to have kernel-level access to the system.
It's not defense-in-depth; it's a security chain : only as strong as its weakest link.
You can restrict most of these options both client- and server-side in SSH, which is basically always a winning move, regardless of protocol. You can't be downgraded to a protocol version or cipher that you've explicitly disabled.
Secure transport protocols all do some kind of handshake to set up a session, agree on keys, etc. Modern secure transport protocols (everything since SSL2) authenticate the handshake, so a MITM can't just edit the messages and, like, stick both sides on the NULL cipher (don't have a NULL cipher, though).
Ever since Kocher and SSL 3.0, the gold standard for handshake authentication has been to keep a transcript of all the handshake messages, and just hash them to fingerprint the entire handshake. You can look at Noise for a streamlined, slick version of the same concept.
SSH does something else: it looks at the handshake as a vehicle for setting up a DH-style key exchange; that's all it's for, everything else happens inside the secure transport that key exchange provides. So instead of doing a transcript hash, SSH picks out the handshake values that will end up being inputs to the key exchange, and hashes those.
The problem is: SSH also does implicit sequence numbers; receivers keep track of how many messages they've received, senders keep track of how many they've sent. Not only that, but SSH has (for reasons passing understanding) a NOP message (`IGNORE`). `IGNORE` carries no data used to do key generation, so it has no impact on the handshake authentication --- but it does impact sequence numbers.
Result: MITM attackers can set sequence numbers to arbitrary values (by injecting `IGNORE`s in the handshake), and then edit out subsequent messages (by just not sending them). If you're using ChaPoly (and often if you're using CBC), the protocol will sync up and keep going. You can use this to, for instance, snipe out extension messages (for things like keystroke timing mitigation) from the beginning of an SSH session.
This is a pretty obvious problem! It's absolutely not something you can just accept from a secure transport protocol. And you could look at SSH and SSL3 and see "SSH is doing something really different and less sophisticated than SSL3". But it took until 2023 for someone to do the legwork to figure out how broken it was.