What do I do about a caravan that is being attacked?

Caravans under attack are always located in the Frontier. Head there and you'll see a shield icon on the map - that's your caravan:

Get there, help fend off the Redcoats, and you're good.