Land loop and land branch
They define the division of the world. There are some test images floating around that I have included for convenience. The random world generator creates chunks of land and determines where to place them relative to previously placed land and these settings tell it how to decide to go about that.
Branch
Per the wiki:
Determine the degree to which the land will branch off. Available options are least, default and most.
Causes the world's land mass to branch out like a tree. Branch essentially breaks the world into smaller masses and connect them by bridges or more specifically, it tends to place land pieces to the sides and connect several of them into chains while avoiding touching previous branches. Higher settings give you more branches.
Branch Only
Default
Most
Loop
Per the wiki:
Determine the degree to which the roads will tend to lead you in circles, in addition to determine the degree to which the land will curve. Available options are none, default and always.
Causes the world to have holes in the center of land mass, forming loops. It essentially tries to make sure land masses connect in a way to form one or more loops by placing land on the same diagonals, trying to loop back to connect with previous land which it may or may not do.
Having only loops creates wormholes.
Loop Only
Default
Always
Neither
If both are turned off, you get a more connected land mass although it still looks to branch some because of random generation, but the branches are noticeably shorter.
Both
Combinations lead to some interesting results. Branching takes priority so you won't see donuts, but rather curved masses with curvy branches.
Loop Default Branch Least
Loop Default Branch Default
Loop Default Branch Most
Loop Always Branch Default
Loop Always Branch Most
Per the Klei forum on Land Loops and Land Branches:
Land branches are bridges between land masses. Land loops are the gaps between them. Having loops but not branches will generate wormholes. Having neither will create one giant flat land mass.
The forum also has this series of test maps showing the effects. It shows the various modifiers in combination.
To add a bit of confusion: The Don't Starve Wiki says:
Land Loops: Determine the degree to which the roads will tend to lead you in circles, in addition to determine the degree to which the land will curve. Available options are none, default and always.
Land Branches: Determine the degree to which the land will branch off. Available options are least, default and most.
More information from the developer forum:
Land Branch: most setting creates "starfish" shaped worlds with a central(ish) hub and peninsulas jutting out.
Land Branch: never creates "salamander" shaped worlds, with each region coming one after the other.
Land Loop determines how curved everything is -- so the starfish arms would be curly, or the salamander would curl all the way around to almost (but not quite) form a ring shaped world.