I find the main benefit of project-specific site-package directories is simply that it's easy to see what my project depends on and what it doesn't. If everything's mixed together in the global dir then it's not always obvious. This becomes important when you want to deploy your project somewhere other than your personal PC.

It's also very useful for managing different versions of packages - e.g. I can upgrade to the latest Django for one project without risking breaking all of the others.

Having said that, I install the stuff that I consider to be "core" globally - for me that's primarily numpy, but also a handful of others.