PCB equivalent of wire twisting

Think about why you want to twist wires in the first place. It is to keep the loop area as small as possible. With the loop area small, the opposing currents generate two opposite magnetic fields that cancel each other out. On a PCB, this is usually done by what is called 'differential pair'-traces. What it basically means for your PCB is two straight traces, as close to each other as possible.

The traces can be side by side, but using two layers the loop area is possibly even smaller by routing them on top of each other.

You have two connections to the motor: the motor leads, and the encoder with its own power. What you should try to do is treat each connection as separate, and minimize the loop area of the leads. That’s what ‘wire twisting’ does: makes a separation between pairs of wires and couples each pair as a loop.

On a PCB, you define pairs by running them alongside each other. This is called ‘micro strip’ or ‘stripline’ routing.

If you were to use a 1 layer board, a sequence of wires Iike this would be good:

• Motor + (big fat trace)
• Motor - (big fat trace)
• (Extra space or possibly motor frame ground)
• encoder ground
• encoder cha+
• encoder cha-
• encoder power
• encoder chb+
• encoder chb-
• encoder ground

I would also add a common-mode filter to the motor leads to suppress the PWM noise. Preferably this would be on your driver PCB, but an additional one on this board could help.

BONUS: A two-layer version

Top:               Back:
  Motor + (fat)      Motor- (fat)
  Motor - (fat)      Motor+ (fat)
          ---- space ----
  encoder cha+       encoder ground
  encoder cha-       encoder ground
          ---- space ----
  encoder chb+       encoder power
  encoder chb-       encoder power

The idea is for the motor wires to couple as much to each other as possible, both side-by side and top-bottom. The encoder signals each have a continuous plane, either power or ground.