what exactly is a "register machine"?

A register machine almost always has a stack, also.

But a stack machine rarely has architecturally visible registers, or it may only have one or two.

A register machine may have some stack ops and may even have a stack addressing mode.

The difference is one of orientation. The register machine will mostly have instructions that operate on registers, and will have a handful of ops for loading and storing between the registers and the stack or memory.

A stack machine .. and these are very rare as actual hardware devices .. will operate directly on the stack with its instructions and wll have a handlful of ops for loading and storing between the stack and memory.

Now, the reasons that hardware register machines are faster than hardware stack machines are possibly unrelated to the reasons that software "register" VM's are faster, according to the cited paper, than software "stack" machines.

For the software VM's, it's apparently the case that fewer instructions need to be executed. This was determined empirically according to claims in the cited paper, but I imagine it's because far fewer overhead instructions like push, pop, and exchange need to be done in the register machine, and because the register machine can reuse operands easily if they are still lying around in the register file, without needing load or push ops. Of course, it's all just memory really; they are virtual registers.

A register machine is a hardware or software unit that when working with data takes it from memory, puts it in a location where it can work with it quickly, and then returns the result.

For example a regular CPU is a register machine. Since the ALU (the unit that works with numbers in a CPU) can only work with numbers in a register.

A stack based machine adds the data onto a stack and then either pops or pushes stuff onto it.

For example, adding two numbers would be

Push 2 // Push 2 onto the stack
Push 3 // Push 3 onto the stack
Add // Add the top two things on the stack.

When in a register machine it would be something like this.

Load x, r0 // Load x onto register 0
Load y, r1 // Load y onto register 1
Add r0, r1, r2 // Add 1 and 2 and store the result in register 2

A register machine uses a fixed number of registers or buckets for storing intermediate values for computation. For example the "add" instruction could add the values in two specific registers and store the result in another register.

A stack based machine uses a stack for storing intermediate values during computation. For example, to add two numbers the "add" instructions pops off two values from the stack, adds them, and pushes the result back onto the stack.