How can I visualise the memory (SRAM) usage of an AVR program?

Don't use the heap / dynamic allocation on embedded targets. Especially with a processor with such limited resources. Rather redesign your application because the problem will reoccur as your program grows.

The usual approach would be to fill the memory with a known pattern and then to check which areas are overwritten.

You can check RAM static usage using avr-size utility, as decribed in
http://www.avrfreaks.net/index.php?name=PNphpBB2&file=viewtopic&t=62968,
http://www.avrfreaks.net/index.php?name=PNphpBB2&file=viewtopic&t=82536,
http://www.avrfreaks.net/index.php?name=PNphpBB2&file=viewtopic&t=95638,
and http://letsmakerobots.com/node/27115

avr-size -C -x Filename.elf

(avr-size documentation: http://ccrma.stanford.edu/planetccrma/man/man1/avr-size.1.html )

Follows an example of how to set this on an IDE: On Code::Blocks, Project -> Build options -> Pre/post build steps -> Post-build steps, include:

avr-size -C $(TARGET_OUTPUT_FILE) or
avr-size -C --mcu=atmega328p $(TARGET_OUTPUT_FILE)

Example output at the end of build:

AVR Memory Usage
----------------
Device: atmega16

Program:    7376 bytes (45.0% Full)
(.text + .data + .bootloader)

Data:         81 bytes (7.9% Full)
(.data + .bss + .noinit)

EEPROM:       63 bytes (12.3% Full)
(.eeprom) 

Data is your SRAM usage, and it is only the amount that the compiler knows at compile time. You also need room for things created at runtime (particularly stack usage).

To check stack usage (dynamic RAM), from http://jeelabs.org/2011/05/22/atmega-memory-use/

Here’s a small utility function which determines how much RAM is currently unused:

int freeRam () {
  extern int __heap_start, *__brkval; 
  int v; 
  return (int) &v - (__brkval == 0 ? (int) &__heap_start : (int) __brkval); 
}

And here’s a sketch using that code:

void setup () {
    Serial.begin(57600);
    Serial.println("\n[memCheck]");
    Serial.println(freeRam());
}

The freeRam() function returns how many bytes exists between the end of the heap and the last allocated memory on the stack, so it is effectively how much the stack/heap can grow before they collide.

You could check the return of this function around code you suspect may be causing stack/heap collision.

You say malloc is failing and returning NULL:

The obvious cause which you should look at first is that your heap is "full" - i.e, the memory you've asked to malloc cannot be allocated, because it's not available.

There are two scenarios to bear in mind:

a: You have a 16 K heap, you've already malloced 10 K and you try and malloc a further 10K. Your heap is simply too small.

b: More commonly, you have a 16 k Heap, you've been doing a bunch of malloc/free/realloc calls and your heap is less than 50% 'full': You call malloc for 1K and it FAILS - what's up? Answer - the heap free space is fragmented - there isn't a contigous 1K of free memory that can be returned. C Heap managers can not compact the heap when this happens, so you're generally in a bad way. There are techniques to avoid fragmentation, but it's difficult to know if this is really the problem. You'd need to add logging shims to malloc and free so that you can get an idea of what dynamic memory operations are being performed.

EDIT:

You say all mallocs happen at startup, so fragmentation isn't the issue.

In which case, it should be easy to replace the dynamic allocation with static.

old code example:

char *buffer;

void init()
{
  buffer = malloc(BUFFSIZE);
}

new code:

char buffer[BUFFSIZE];

Once you've done this everywhere, your LINKER should warn you if everything cannot fit into the memory available. Don't forget to reduce the heap size - but beware that some runtime io system functions may still use the heap, so you may not be able to remove it entirely.