How can I make breaking walls less tedious?

Attack speed (Slow) is not the same as wall-breaking speed. You know the 'ticks' you hear when you are mining or chopping or breaking down walls? With the meteorite hamaxe, it reduces the 'ticks' to 2 for destroying a wall tile.. This is, in my opinion, a significant upgrade from the gold hammer (What I had previously).

So end of the day, even though the attack speed for hammers is slow, the rate at which they can affect the environment is faster.

UPDATE: With the 1.1 update, as well as before this, a few passes have been made to make destruction and building of the background walls to be slightly less tedious.

Since 1.3.0.1 update, it is possible to place player placed walls directly on natural ones (or other player placed) in order to replace them. Should replaced wall be obtainable, it will drop as if mined. When replacing natural walls, it is still required that you start from a boundary/corner area of the wall, replacing at center of natural wall is not possible.

As noted on Gamepedia Wiki:

Now you can place walls in front of each other.

the formulation is misleading though, it's replacement, no wall is actually placed in front of other wall.