How are high scores achieved?

In the vanilla combat challenges (no DLC), there are a limited amount of bad guys per combat challenge. Each combat challenge is comprised of four rounds. Each consecutive round adds more bad guys and, in general, difficulty. At the end of each round, which occurs when all enemies have been dispatched, it tallies up the points that you've earned so far. Variations bonuses and such apply per round, so you may have gotten hit or lost your combo streak in one round, but you can still get bonuses for those in the other rounds. This combat mechanics guide will help.

The fewer enemies that there are, the harder it is to fit in all variations without incapacitating too many to prevent being able to perform all variations.
You'll want to build your combo streak with the weakest hits as possible. Squeeze every hit out of each baddy. I usually say to "juice". Beatdown finisher tenderizing punches means each punch after a stun, in what is called a beatdown, not including the finishing blow. My tests show that the weakest hits come from beatdown finisher tenderizing punches.
Perform finisher variations with increasingly higher modifier-multiplier moves. For example, let's say that you have two enemies left and the beatdown finisher and ground pound variations left to do. Beatdown finisher has a x50 modifier and ground pound has x75. You should do the beatdown finisher first.

Step by step: I made acronyms to help me memorize the best order in which to perform the variations. My latest iteration (I modify this even as I post it), in its full form, is: CBSAGDLMFIUG. I would pronounce this as cob-sag-delem-fee-ug. Counter, Batarang/Bolas, Projectile, Lieutenant, Titan, Strike, Aerial assault, Gadgets, Disarm & destroy, Lesser-lethal special combos, Multi-ground takedown, beat-down Finisher, Instant takedown, Ultra-stun, Ground-pound. The routine is largely the same for Catwoman, who only ignores D.
C & B are for Counter and Batarang/Bolas. These moves are first because the first two hits of a combo don't deal damage, i.e. when the combo count is below x3. Some moves still deal damage regardless of the lowness of the combo streak, like aerial assault and strike, which is why they're not first, despite how much damage that they do. You'll also only want to counter one attack to only get a +x1. Placing it first gives you more control to manipulate the mob for that, because you don't have to worry about dropping a combo that you don't have yet. You may want to evade/redirect after this.
S is for Strike. This is third because, at this point, it can't be a critical strike. Like I said, as little damage as possible. And again, this is a move that deals damage regardless of combo lowness, so there's no extra benefit to doing it first, not to mention that it won't freeflow at sub-x3.
A is for Aerial assault. This is fourth because it has a x25 modifier. Around this point, it's possible to incapacitate a baddy early, unintentionally, so you may want to beware of who you've been hitting. A directed aerial assault is safer the farther you go, and can be performed by moving in a direction and attacking during an aerial assault.
G is for Gadgets. I tried to avoid using the same letters, but I couldn't. Batman: batarangs are already done, batclaw, explosive gel (which must hit something), remote electrical charge (REC), and freeze blast. Catwoman: whip and caltrops. You may want to throw in evades/redirects to make this safer.
D is for special combo Disarm & destroy. This begins the special combo portion. This is a special combo move that only Batman has. Disarming doesn't deal damage.
L is for Lesser lethal special combos. Lesser lethal special combos comprise of batswarm, for Batman, and whiplash ("whip around") and whip trip, for Catwoman. Unlike disarm & destroy, these lesser lethal special combos do a bit of damage, hence "lesser" lethal. So, disarm should be performed before batswarm.
M is for special combo Multi-ground takedown. Make sure you only take down one. This can immediately follow up a strike or batarang.
F is for beatdown Finisher. Batman's starts counting hits for the combo and actually dealing damage after the first four punches. Catwoman's starts after five. This begins the final phase. You'll be doing beatdowns on much of the remaining enemies. You can juice more hits out of an enemy by going for one "actual" hit at a time, then restarting (as by stunning), instead of completing a beatdown finisher in one go. Picture this: three enemies left and these last variations left to go. Tenderize the remaining three in rotation to juice out every last multiplier. Freeze blast one, stun second, stun third, tenderize third, freeze blast third, stun second, tenderize second, freeze blast second, stun first, etc. until one of the thugs inevitably goes down immediately after the beatdown start-up. Then...
I is for special combo Instant takedown. Finally...
U is for Ultra-stun. Ultra-stun the last guy and...
G is for Ground pound. x75 modifier. You can freeze blast a grounded enemy before ground pounding them, for good measure.
Now, there's one variation that I haven't mentioned: Redirection. Unlike what the guide says, it is not Evade but Redirection. I assume that you'll do this anyway.

Optimally, you're only finishing any enemy with special combo takedown, knife evade takedown or ground pound (because those finishing moves have the highest modifier (x100, x100, and x75, respectively)) and you're only finishing them when they have a sliver of health left. However, there aren't always knives to evade and electrified walls to move enemies into, and more importantly, unless you're super mindful, you can't keep track of how much damage each enemy has received or how much health that each has to maximize the points from each enemy. So, rotating beatdowns is the next best thing.

Putting it all together: Now, that acronym is hella long, and I find myself evading/redirecting until I run through the letters in my head. Here's a break-down by group: AGLF. Attack moves, Gadgets, Special combos, Finishers.
A is for Attack moves. Counter, Batarang, Strike, and Aerial assault.
G is for Gadgets.
L is for Lesser lethal special combos. Disarm & destroy and batswarm, or whiplash and whip-trip.
F is for Finishers. Special combo Multi-ground takedown, beatdown Finisher, special combo Instant takedown, and Ground pound.
This is more of a semantic mnemonic. This helps to perform the moves fluently.

Is it ever better to just juice rather than fit in variations? After gathering data and running numbers, I've concluded little except that it's plausible that pure juicing, instead of variations, can be more profitable if your combo exceeds a certain point. I tried to craft a formula for calculating the difference in points earned from the two methods, but it's complicated. If you want to take a crack at the numbers, I post my jumbled analysis here: http://pastebin.com/5GryuHZz

Environmental projectiles: Of the projectiles that are thrown by thugs, gas canisters are set in certain combat challenge stages. Some only become allowed to be picked up by the enemy after a certain round. These can still be blown up at any time via batarang or bolas, even before their "activation" round. This is useful because blowing one up adds +1 to the combo count. Beware: Blowing it up when you're too close to it causes you to get hit by the explosion thus threatening to be a c-c-c-combo breaker.
I don't know if blowing one up via batarang/bolas fulfills the batarang/bolas gadget variation.

I recently learned that there's an indefinite juice loop only with Catwoman and her bolas. Normally, moves don't count for points unless they damage, but the bolas won't deal damage at sub-x3 combo and award points on the hit that goes from x1 to x2.

Opinion: Once I learned that juicing beatdowns was necessary, I was like, "say it ain't so!" I had the same reaction to discovering the Catwoman indefinite. What I mean is, yes, getting a high score by juicing beatdowns is frustrating and the system seems too arbitrary. The case changes for the DLC combat challenges which are based on time or survival.
In a time-based, you'll probably want to go simply crazy with critical strikes.

Update: From research that I conducted here, I've learned that it's also possible to indefinitely juice Titans.