Binary Convolution

Python 2, 35 bytes

lambda m,n:n^m&(n^n*257/2^n*2^n>>7)

JavaScript (ES6), 31 bytes

(n,m)=>n^m&(n^(n*=257)>>1^n>>7)

J, 34 bytes

2#.[:({"_1],._1&|.~:1&|.)/(8#2)#:]

Straight-forward approach that applies the process defined in the challenge. Takes input as an array [n, m].

Forms seven smileys, [:, :(, :1, (8, 8#, #:, and :].

Usage

   f =: 2#.[:({"_1],._1&|.~:1&|.)/(8#2)#:]
   f 150 59
171
   f 59 150
166
   f 209 242
178
   f 17 1
0
   f 139 189
181