Is Starbound the sequel to Terraria? or what is their relationship?

Starbound is from a different developer, only @Tiyuri worked on both games. Starbound is quite similar to Terraria, but there are a few key differences such as races and the fact that the world doesn't end, it wraps around (planets are round, you can walk in one direction and end up where you started).

Starbound is not a sequel, but many people call it a spiritual successor to Terraria.

The gameplay is similar, and the key differences come from Starbound's questing, spaceships and the ability to travel between planets within the game. This is extremely useful for public multiplayer servers, who don't need to worry about players stealing from others anymore and most random people are unlikely to even see each other by chance. Other than that, most mechanics are just the same old polished up a bit with a new look.

No, Starbound is not a sequel to Terraria.