Magic damage vs Physical damage based on level

You are 100% correct in your misgivings. Due to destruction magic not scaling, you will lag behind in terms of damage done late game (especially if you up the difficulty). The only way you can make up for this is via the enchanting perks that lower destruction magic costs so you can cast and spam more spells. If you can get it down to 10% or less cost via multiple enchanted items, it is still viable due to the fact that you have instant stagger at your beck and call with the dual casting and impact perks, not to mention heavy aoe with fireball and such.

If you don't mind modding, there are already mods out there that improves destruction magic.

Finally, you'll still need a backup plan in case you run into some highly magic resistant foes (Breton with dragonskin activated, for instance.)

On the other side of the equation, while physical attacks get more powerful, the armor they must overcome also gets more powerful. Magic resistances stay flat.

Don't forget about resistance poisons for a bit of extra kick.