Upgrading to SSL an existing socket connection

SSL support is in the JDK since 1.4 (and even before as a separate jar), so it is fairly common knowledge how to create and use SSLSocket and SSLServerSocket.

What is less known, or it was to me at least, is that it is possible to create a plain socket, connect to a server, exchange some message, then upgrade the existing socket connection to SSL, exchange some private message with confidentiality (for example a password that would have traveled in clear text), then downgrade back to the plain the socket connection; all this without closing the original socket nor creating a new one on a different port.

Let's see how.

The server

The server class creates a plain ServerSocket, and calls accept() on it; when a client connects, accept() returns a socket that is normally handed to a different thread (details omitted):

Socket socket = serverSocket.accept();
threadPool.execute(new Handler(socket));

class Handler implements Runnable
{
 private final Socket socket;
 Handler(Socket socket) { this.socket = socket; }
 public void run()
 {
   Socket currentSocket = this.socket;
   while (isConnectionOpen(currentSocket))
   {
     String message = readMessage(currentSocket);
     if (upgradeToSSL(message))
     {
         SSLSocketFactory sslSocketFactory =
             (SSLSocketFactory)SSLSocketFactory.getDefault();
         SSLSocket sslSocket =
             (SSLSocket)sslSocketFactory.
                 createSocket(currentSocket,
                              currentSocket.getInetAddress().getHostAddress(),
                              currentSocket.getPort(),
                              false);
         sslSocket.setUseClientMode(false);
         sslSocket.startHandshake();
         currentSocket = sslSocket;
       }
       else if (downgradeFromSSL(message))
       {
         currentSocket.close();
         currentSocket = this.socket;
       }
       else
       {
         handleNormalMessage(currenSocket, message);
       }
     }
  }
}

When upgrading, the server wraps the current (plain) socket using SSLSocketFactory.createSocket(socket, address, port, autoClose), then configures the SSLSocket to be a non-client one (for the purposes of the SSL handshake), then starts the SSL handshake to secure the connection.

When downgrading, the current (ssl) socket is just closed and, thanks to the autoClose argument set to false (which will not close the underlying plain socket), the original plain socket is still operative.

In order for this mechanism to work, you have to configure the SSL details as usual for a server. This means having created a private/public key pair in a keystore (and optionally having signed the public key with a CA root) and having set the relevant SSL system properties (or having done the equivalent using the SSL APIs).

The client

The client connects to the server using a plain Socket, then upgrades the socket connection (details omitted):

// Connects to server
this.socket = new Socket(serverAddress, serverPort);
...
Socket currentSocket = this.socket;
while (moreMessagesToSend)
{
  if (upgradeToSSL)
  {
    SSLSocketFactory sslSocketFactory =
      (SSLSocketFactory)SSLSocketFactory.getDefault();
    SSLSocket sslSocket =
      (SSLSocket)sslSocketFactory.
        createSocket(currentSocket,
                     currentSocket.getInetAddress().getHostAddress(),
                     currentSocket.getPort(),
                     false);
    sslSocket.setUseClientMode(true);
    sslSocket.startHandshake();
    currentSocket = sslSocket;
  }
  else if (downgradeFromSSL)
  {
    currentSocket.close();
    currentSocket = this.socket;
  }

  sendMessage(currentSocket);
  readReply(currentSocket);
}

Upgrading and downgrading the socket connection in the client works like in the server code, the only difference being that the SSLSocket is configured to be a client one (for the purposes of the SSL handshake). The autoClose argument prevents the original plain socket to be closed when the SSL socket is closed.

In order for this mechanism to work, you have once again to configure the SSL details as usual for a client. This means having setup properly a trust store (and optionally having imported into it the server certificate if it's not signed by a CA root) and having set the relevant SSL system properties (or having done the equivalent using the SSL APIs).

If you're writing custom network protocols that require confidentiality, the above may come handy. Enjoy !