Can your settlements actually get attacked?

Yes - I've had multiple instances where my turrets have opened fire on Deathclaws and other wasteland wildlife, the moment one of your defences start attacking (whether that be a settler shooting something or a turret opening fire). The attacks will kill settlers and destroy (reduce durability to zero) defensive structures and crops. You'll get the opportunity to fix any damaged structures or crops but if settlers die, that's final.

This means you'll want to ensure the approaches your settlements are well guarded and that you should position your defences for optimal coverage. However, I have noticed that in some instances when you don't turn up to a help a settlement that is under attack an unexplainable amount of damage will occur to your defences anyway... they do seem to help anyway and it might just be a coverage thing, and they're certainly helpful if you do respond to the settlement's call for help.

One of the earlier tooltips in the game indicate that your chance to get attacked is based on a your resource generation in comparison to your defence rating, and that you should keep your defence rating equal or better than your combined resource generation.