How to map two matrix by each element and perform simple mathematical operation by each row?

I didn't immediately give the full answer, in the hope someone would follow up on the hint in my comment. Anyway, the missing piece is to use Ordering[] to rearrange list B, like so:

MapThread[#1[[Ordering[#2]]] &, {B, A}].{{1, -1}, {0, 1}, {0, 0}, {-1, 0}}
   {{-5, 7}, {1, -7}, {8, -11}}

A second method is to convert the data to an association, which can then be used for lookups:

{#[a] - #[d], #[b] - #[a]} & /@ MapThread[AssociationThread, {A, B}]
   {{-5, 7}, {1, -7}, {8, -11}}

How about:

MapThread[Block[{a, b, c, d}, # = #2; {a - d, b - a}] &, {A, B}]
(* {{-5, 7}, {1, -7}, {8, -11}} *)

Here is a way using ReplaceAll (/.):

{a - d, b - a} /. MapThread[Rule, {A, B}, 2]

(* {{-5, 7}, {1, -7}, {8, -11}} *)