John Bethencourt writes:
Here's the protocol in a more concise form:
[first authentication]
Client --> username --> Server Client <-- 'initial password?' <-- Server Client --> initialpass --> Server Client <-- r_0 <-- Server Client --> h(h(s.r_0)) --> Server
[subsequent authentications]
Client --> username --> Server Client <-- r_x <-- Server Client --> h(s.r_x) --> Server Client <-- r_y <-- Server Client --> h(h(s.r_y)) --> Server
One thing that is unclear in your protocol is whether s, which I think is the same as initialpass, is stored permanently on the server. If so, then when the server receives h(s.r_x), it can do two checks. The check you described is that the hash of this matches the value the client sent previously, h(h(s.r_x)). The other is that the server can construct s.r_x since it knows both of these values, and then compare h(s.r_x) with what the client sent. This has two problems. First, it makes the sending of h(h(s.r_x)) unnecessary since the server can verify the client's response just using s. And second, s is a sensitive value which if stolen could be used at some later time to impersonate the user. So I will assume that the server doesn't store s, in which case I don't see why it needs to see initialpass in the first place. In that case the protocol basically says, each time you log on you provide a hash value for which you will provide the hash input next time you log on. The purpose of the r_x values is to keep the user from having to save state; he can compute h(h(s.r_y)) and then forget it, because next time he logs on the server will remind him of what r_y is. The main security problem I see is that an eavesdropper sees r_x and h(s.r_x), allowing him to do a dictionary attack on s. You could use a really slow hash function to slow this attack down, but it is a serious vulnerability. Given that you are using client software to authenticate, I suggest pursuing SPEKE-style password-authenticated key exchange. There are a number of these systems (some patented) which feature immunity to dictionary attacks on the password.