What ended up working eventually was to add:
import dj_database_url
DATABASES['default'] = dj_database_url.config(default='postgres://<user>:<password>@localhost:5432/<name>')
instead of just
import dj_database_url
DATABASES['default'] = dj_database_url.config()
like the tutorial states. Then when trying to run
python manage.py runserver
I still got the error
ImportError: DLL load failed: %1 is not a valid Win32 application.
I then was able to figure out that the stickpeople build i needed to use was the 64 bit instead of the 32 bit that the tutorial had me use, and the 64 bit build is:
easy_install http://www.stickpeople.com/projects/python/win-psycopg/psycopg2-2.4.5.win-amd64-py2.7-pg9.1.3-release.exe
I don't know if it was the best way, but in order to use the 64 bit build I started over from the beginning. Now it finally works. Hopefully this can also be used to help anyone else who is also stuck. There are a lot of holes in the heroku tutorial.