Use JavaScript encryption module instead of SSL/HTTPS

[Note: See Correction below]

It is possible to securely communicate with an authenticated server on google app engine with a custom domain, but it is a hassle. As some of the other answers indicate, you must be very careful how you implement the encryption to prevent man-in-the-middle attacks.

Here are the basic instructions for python, but you could could modify for java. Expect to spend at least a day or two getting something like this up and running.

Prerequisites:

• A python RSA and AES library for the server. pyCrypto works well on GAE.
• A javascript RSA and AES library to be run on the client. I used stanford's RSA library and crypto-js for AES. I can't remember why I didn't just use one library.

Instructions:

• Step 1: Offline, create a RSA public and private key pair. Here are pyCrypto instructions.

• Step 2: Save the generated RSA public & private keys in a python file, be sure the private key is not publicly accessible.

• Step 3: On the server, create a request that generates a javascript file. The javascript file should only transmits the public key to the client. For example, it would only return a file with this:

  var public_key="[your public rsa key here]"

• Step 4: In your app.yaml file, make sure that the generated javascript file is only served over SSL (i.e. set secure: always). See instructions here.

• Step 5: On the client, load the javascript file using ssl. However, instead of using your custom domain, use the appspot domain. For example, add this to your html file:

<script src="https://example.appspot.com/publicKey.js"></script>

The client will now have an authenticated RSA public key preventing man-in-the-middle attacks. Note: accessing other domains is normally prohibited by browsers to prevent XSS, but there is a loophole which allows you to load javascript files.

• Step 6: On the client, generate a random private key. Use a javascript RSA library and the public key we got in step 4 to encrypt the random private key. Send the encrypted private key to the server.

• Step 7: On the server, decrypt the random key generated on the client using the RSA private key.

  At this point, both the server and the client have the same private key. Even better, because the original public key was transmitted over SSL, we can authenticate that the server is really who we believe it is (i.e. no man-in-the-middle).

• Step 8: Now the server and client can encrypt and decrypt any data they want using the randomly generated private key and their respective AES libraries.

-- EDIT: CORRECTION --

Bruno's comment below is 100% correct, the above steps are insecure. Although the steps above do work to setup an authenticated session between client and server, the only way that the user would really know it was authenticated is if they checked the code to ensure that the public key was being loaded using https. A man-in-the-middle could serve the initial html page modify the <script src="https://... code to point to something else.

Instead, take a look at wwwizer.com.