Using Task Parallel Library with Multiple Computers

The default settings for TPL is to schedule tasks on the .NET thread pool and as such this is aimed at parallelism within a single process. However, you can implement your own TaskScheduler, which handles how tasks are actually run, so at least in theory you could move beyond the current scope. While this could be done, I am sure there are better options available as Reed Copsey and Chris Taylor point out.

The TPL is geared towards single computer, multiple processor core scenarios.

If you want to work across multiple systems, you'll need to use some type of clustering software, such as MPI (usable in .NET directly via MPI.NET) or one of the many options based on Windows HPC.

That being said, the TPL is very useful on each of the nodes of the cluster. It can be used to have each cluster node scale well across the cores available on that node.

No TPL focuses on local threads within a process. There are however existing projects that do tackle this area.

http://research.microsoft.com/en-us/projects/dryad/

And you can take a look at the answers to this SO question

Any good distributed agent/service models for .NET?