Aborting evaluation when the memory exceeds a certain limit

Since one may not always accurately predict when MemoryContrained is needed, I recommend setting up a watch-dog task. Belisarius described how to do this here in answer to my question. I will reproduce it below as answers that are merely links are discouraged.

---

In Mathematica 8 you could start a memory watchdog, something along the lines of:

maxMemAllowed        = 1.3 1024^3; (*1.3 GB*);
intervalBetweenTests = 1; (*seconds*)
iAmAliveSignal       = 0;
Dynamic[iAmAliveSignal]
RunScheduledTask[
       If[MemoryInUse[] > maxMemAllowed , Quit[], iAmAliveSignal++],      
       intervalBetweenTests];

Remember to run

RemoveScheduledTask[ScheduledTasks[]];

to disable it.

In addition to Mr.Wizard's answer.

In many cases it is very practical to stop the evaluation when the actual amount of free physical memory in your system becomes less than specified threshold. You can get the amount of free physical memory very efficiently via NETLink call to GlobalMemoryStatusEx function of kernel32.dll (which is available both under 32 bit and 64 bit Windows systems). Here I define getFreePhysMem[] function which returns the current amount of free physical memory:

Needs["NETLink`"];
getFreePhysMem::internalError = 
  "globalMemoryStatusEx[memorystatusex] has not returned True.";
If[$OperatingSystem === "Windows", 
  memorystatusex = Symbol["LoadedNETTypes"][];
  globalMemoryStatusEx = 
   Symbol["DefineDLLFunction"][
    "[StructLayout(LayoutKind.Sequential, CharSet=CharSet.Auto)]
            public class MEMORYSTATUSEX
            {public uint dwLength;
            public uint dwMemoryLoad;
            public ulong ullTotalPhys;
            public ulong ullAvailPhys;
            public ulong ullTotalPageFile;
            public ulong ullAvailPageFile;
            public ulong ullTotalVirtual;
            public ulong ullAvailVirtual;
            public ulong ullAvailExtendedVirtual;
            public MEMORYSTATUSEX()
            {this.dwLength = (uint) 
            Marshal.SizeOf(typeof( MEMORYSTATUSEX ));}}
            [return: MarshalAs(UnmanagedType.Bool)]
            [DllImport(\"kernel32.dll\", CharSet=CharSet.Auto,     \
SetLastError=true)]
            public static extern bool GlobalMemoryStatusEx([In, Out]  \
   MEMORYSTATUSEX lpBuffer);"];
  memorystatusex = 
   Complement[Symbol["LoadedNETTypes"][], memorystatusex][[1, 1]];
  memorystatusex = memorystatusex <> "+MEMORYSTATUSEX";
  memorystatusex = Symbol["NETNew"][memorystatusex];
  getFreePhysMem[] := 
   If[TrueQ[globalMemoryStatusEx[memorystatusex]], 
    memorystatusex@ullAvailPhys, 
    Message[getFreePhysMem::internalError]; Abort[]; $Failed]];

Timings:

Do[getFreePhysMem[], {100}] // AbsoluteTiming
(*=> {0.0312500, Null}*)

More information on this function: "Calling kernel.dll from Mathematica. 1", "Calling kernel.dll from Mathematica. 2".

You can make use of either TimeConstrained or MemoryConstrained to terminate evaluation when it runs out of time or memory respectively.

For example, if you have a function that has a reasonable memory footprint, but takes time to evaluate, you can abort evaluation after a certain amount of time (in seconds) has elapsed, as:

TimeConstrained[Eigenvalues@RandomReal[1, {5000, 5000}], 1]
Out[1]= $Aborted

On the other hand, if memory is the one that blows up first, you can wrap it in MemoryConstrained to abort when a certain amount of bytes have been consumed:

MemoryConstrained[Range /@ Range[100], 1000]
Out[2]= $Aborted