Should I sell to Nook or Re-Tail?

Always, always sell to Re-Tail. They will always buy your items at a higher price than the Nook stores will.

While the Nookling store does upgrade in this game, they upgrade only based on the amount you have spent in their store, not the combined buy/sell amount. You can see all shop upgrades here and their requirements on their respective pages, where you can see that the only requirement is Bells spent.

So you're not missing out on anything if you buy from Nook's and sell to Re-Tail.