Management Studio System.OutOfMemoryException

This error indicates that Management Studio is running out of memory, not the SQL Server service. Even if you installed 64 bit SQL Server, the SQL Server Management Studio executable is a 32 bit application.

This is likely caused by the size of the result set that you are returning to Management Studio. Are you executing something like SELECT * FROM really_big_table? See http://support.microsoft.com/kb/2874903 for more.

Mike is right that the error message you're receiving is from the Management Studio application itself, and not from SQL Server. It is the memory on your local workstation that has been exhausted, likely due to trying to pull 16 billion rows into the client application (rendering that much data in a grid is quite expensive in terms of memory, so try to limit your queries using TOP etc. - I don't know what practical thing(s) you could possibly do with enough data to use up all of your local memory anyway).

But I do want to address another problem: using Task Manager to assess the amount of memory SQL Server is using. Don't do this; it's a bold-faced liar. Copying from this answer (your question is two-fold, so I can't really close it as a duplicate):

---

You could NEVER, EVER trust Task Manager to tell you how much memory SQL Server is using. Stop using Task Manager for this, period. Use the performance counter - you can also query the performance counter using DMVs:

SELECT object_name, cntr_value 
  FROM sys.dm_os_performance_counters
  WHERE counter_name = 'Total Server Memory (KB)';

You could save that as a query shortcut in Tools > Options > Environment > Keyboard > Query Shortcuts, and get accurate results in a query window much faster than getting inaccurate results from Task Manager.

You can also check for memory pressure (and whether you can do anything about it) using these queries:

SELECT object_name, cntr_value
  FROM sys.dm_os_performance_counters
  WHERE counter_name IN ('Total Server Memory (KB)', 'Target Server Memory (KB)');

-- SQL Server 2012:
SELECT physical_memory_kb FROM sys.dm_os_sys_info;

-- Prior versions:
SELECT physical_memory_in_bytes FROM sys.dm_os_sys_info;

EXEC sp_configure 'max server memory';

It was the same case with me. My SQL Server Management Studio was open for couple of days. I restarted it and it got solved.