I found out that it's impossible to install NumPy/SciPy via installers on Windows 64-bit, that's only possible on 32-bit. Because I need more memory than a 32-bit installation gives me, I need the 64-bit version of everything.
I tried to install everything via Pip and most things worked. But when I came to SciPy, it complained about missing a Fortran compiler. I installed it via MinGW/MSYS. But you can't install SciPy right away after that, you need to reinstall NumPy. So I tried that, but now it doesn't work anymore via Pip nor via easy_install. Both give these errors:
Numpy (as also some other packages like Scipy, Pandas etc.) includes lot's of C-, Cython, and Fortran code that needs to be compiled properly, before you can use it. This is, btw, also the reason why these Python-packages provide such fast Linear Algebra.
To get precompiled packages for Windows, have a look at Gohlke's Unofficial Windows Binaries or use a distribution like Winpython (just works) or Anaconda (more complex) which provide an entire preconfigured environment with lots of packages from the scientific python stack.