What can't the ATtiny do that the Arduino can?

Update: To better answer the question, ATTiny's can do anything that don't require hardware the ATTiny's are missing, which include the hardware serial port, hardware i2c, or multiple interrupts. Since this is hidden away in the Arduino Libraries, unless you have a good understanding of how they work or read the actual library files, things like the Wire library (hardware i2c) will not work.

From http://forum.arduino.cc/index.php/topic,5582.0.html:

For example a ATTINY84 has 1 Interrupt, no Hardware UART (Serial or i2c) and a small handful of GPIO pins, less program space and much less ram but comes in a 14 pin package.

A ATMEGA328 has 2 Interrupts, 1 Hardware UART, more ram more program space and more GPIO, which is great, if you really need it.

Additionally, most ATTiny's have no hardware multiplication, so a compiler would have to do non-base-2 multiplication in software.

Finally, Arduinos rely on a bootloader instead of spi programming, so you need an ATTiny with atleast 4~8kb flash for them to work with the Arduino bootloader.

ATTiny44/45/84/45/2313 are popular as micro-Arduinos. They have about a fourth or eighth of the ram and flash of a ATMega328 that the official Arduino Uno uses.

There are many ATTiny's, so no single statement can be said about them all.

Also see here: http://hlt.media.mit.edu/?p=1695

The most attinys don't have a hardware USART module, so you won't be able to communicate with them via serial. For some attinys instead of dedicated I2C and SPI hardware peripherals there is a simplified USI peripheral for which you have to write a bit of glue code to implement SPI or I2C communication. Some attinys don't have ADC.

In general, you have to be more careful when choosing attiny for your project, because those tend not to have full set of features available in Arduino's atmega328. For example attiny2313 does have an UART but does not have ADC, attiny44 does have ADC, but doesn't have an UART, etc.

Just check the Atmel microcontroller selection table and a datasheet at the beginning.