Pulling Permission Sets only through IDE polls a trimmed version of them

The Force.com IDE (eclipse plugin) is built on top of the Metadata API and uses it to populate the various files. The contents of a Permission Set or Profile depend directly on all of the other content requested. So, when you have a project that has close to no other metadata component types other than the Permission Set, the retrieved Permission Set will likely have close to nothing in it. When you have a project with all metadata component types in it, you will likely have a larger Permission Set file.

From the common migration issues documentation:

The contents of a Permission Set or Profile returned by Metadata API depends on the content requested in the RetrieveRequest message. For example, profiles only include field-level security for fields included in custom objects returned in the same RetrieveRequest as the profiles.

There are more examples in the Profile documentation in the Metadata API docs as well and here is a good article on the Force.com IDE and Metadata.

There are a number of idiosyncrasies working with both profiles and permission sets in the Metadata API (MdAPI) as well as various clients that use this API like Force.com IDE and Change Sets.

While Peter is dead on with his answer, I also recommend reading the following blog posting for more detailed information than what the docs currently provide on migrating permissions with the MdAPI: http://www.salesforcehacker.com/2013/05/dude-wheres-my-permission.html