For given two integers A and B, find a pair of numbers X and Y such that A = X*Y and B = X xor Y

You know that at least one factor is <= sqrt(A). Let's make that one X.

The length of X in bits will be about half the length of A.

The upper bits of X, therefore -- the ones higher in value than sqrt(A) -- are all 0, and the corresponding bits in B must have the same value as the corresponding bits in Y.

Knowing the upper bits of Y gives you a pretty small range for the corresponding factor X = A/Y. Calculate Xmin and Xmax corresponding to the largest and smallest possible values for Y, respectively. Remember that Xmax must also be <= sqrt(A).

Then just try all the possible Xs between Xmin and Xmax. There won't be too many, so it won't take very long.

The other straightforward way to solve this problem relies on the fact that the lower n bits of XY and X xor Y depend only on the lower n bits of X and Y. Therefore, you can use the possible answers for the lower n bits to restrict the possible answers for the lower n+1 bits, until you're done.

I've worked out that, unfortunately, there can be more than one possibility for a single n. I don't know how often there will be a lot of possibilities, but it's probably not too often if at all, so this may be fine in a competitive context. Probabilistically, there will only be a few possibilities, since a solution for n bits will provide either 0 or two solutions for n+1 bits, with equal probability.

It seems to work out pretty well for random input. Here's the code I used to test it:

public static void solve(long A, long B)
{
    List<Long> sols = new ArrayList<>();
    List<Long> prevSols = new ArrayList<>();
    sols.add(0L);
    long tests=0;
    System.out.print("Solving "+A+","+B+"... ");
    for (long bit=1; (A/bit)>=bit; bit<<=1)
    {
        tests += sols.size();
        {
            List<Long> t = prevSols;
            prevSols = sols;
            sols = t;
        }
        final long mask = bit|(bit-1);
        sols.clear();
        for (long prevx : prevSols)
        {
            long prevy = (prevx^B) & mask;
            if ((((prevx*prevy)^A)&mask) == 0)
            {
                sols.add(prevx);
            }
            long x = prevx | bit;
            long y = (x^B)&mask;
            if ((((x*y)^A)&mask) == 0)
            {
                sols.add(x);
            }
        }
    }
    tests += sols.size();
    {
        List<Long> t = prevSols;
        prevSols = sols;
        sols = t;
    }
    sols.clear();
    for (long testx: prevSols)
    {
        if (A/testx >= testx)
        {
            long testy = B^testx;
            if (testx * testy == A)
            {
                sols.add(testx);
            }
        }
    }

    System.out.println("" + tests + " checks -> X=" + sols);
}
public static void main(String[] args)
{
    Random rand = new Random();
    for (int range=Integer.MAX_VALUE; range > 32; range -= (range>>5))
    {
        long A = rand.nextLong() & Long.MAX_VALUE;
        long X = (rand.nextInt(range)) + 2L;
        X|=1;
        long Y = A/X;
        if (Y==0)
        {
            Y = rand.nextInt(65536);
        }
        Y|=1;
        solve(X*Y, X^Y);
    }
}

You can see the results here: https://ideone.com/cEuHkQ

Looks like it usually only takes a couple thousand checks.

Here's a simple recursion that observes the rules we know: (1) the least significant bits of both X and Y are set since only odd multiplicands yield an odd multiple; (2) if we set X to have the highest set bit of B, Y cannot be greater than sqrt(A); and (3) set bits in X or Y according to the current bit in B.

The following Python code resulted in under 300 iterations for all but one of the random pairs I picked from Matt Timmermans' example code. But the first one took 231,199 iterations :)

from math import sqrt

def f(A, B):
  i = 64
  while not ((1<<i) & B):
    i = i - 1
  X = 1 | (1 << i)

  sqrtA = int(sqrt(A))

  j = 64
  while not ((1<<j) & sqrtA):
    j = j - 1

  if (j > i):
    i = j + 1

  memo = {"it": 0, "stop": False, "solution": []}

  def g(b, x, y):
    memo["it"] = memo["it"] + 1
    if memo["stop"]:
      return []

    if y > sqrtA or y * x > A:
      return []

    if b == 0:
      if x * y == A:
        memo["solution"].append((x, y))
        memo["stop"] = True
        return [(x, y)]
      else:
        return []

    bit = 1 << b

    if B & bit:
      return g(b - 1, x, y | bit) + g(b - 1, x | bit, y)
    else:
      return g(b - 1, x | bit, y | bit) + g(b - 1, x, y)

  g(i - 1, X, 1)
  return memo

vals = [
  (6872997084689100999, 2637233646), # 1048 checks with Matt's code
  (3461781732514363153, 262193934464), # 8756 checks with Matt's code
  (931590259044275343, 5343859294), # 4628 checks with Matt's code
  (2390503072583010999, 22219728382), # 5188 checks with Matt's code
  (412975927819062465, 9399702487040), # 8324 checks with Matt's code
  (9105477787064988985, 211755297373604352), # 3204 checks with Matt's code
  (4978113409908739575,67966612030), # 5232 checks with Matt's code
  (6175356111962773143,1264664368613886), # 3756 checks with Matt's code
  (648518352783802375, 6) # B smaller than sqrt(A)
]

for A, B in vals:
  memo = f(A, B)
  [(x, y)] = memo["solution"]
  print "x, y: %s, %s" % (x, y)
  print "A:   %s" % A
  print "x*y: %s" % (x * y)
  print "B:   %s" % B
  print "x^y: %s" % (x ^ y)
  print "%s iterations" % memo["it"]
  print ""

Output:

x, y: 4251585939, 1616572541
A:   6872997084689100999
x*y: 6872997084689100999
B:   2637233646
x^y: 2637233646
231199 iterations

x, y: 262180735447, 13203799
A:   3461781732514363153
x*y: 3461781732514363153
B:   262193934464
x^y: 262193934464
73 iterations

x, y: 5171068311, 180154313
A:   931590259044275343
x*y: 931590259044275343
B:   5343859294
x^y: 5343859294
257 iterations

x, y: 22180179939, 107776541
A:   2390503072583010999
x*y: 2390503072583010999
B:   22219728382
x^y: 22219728382
67 iterations

x, y: 9399702465439, 43935
A:   412975927819062465
x*y: 412975927819062465
B:   9399702487040
x^y: 9399702487040
85 iterations

x, y: 211755297373604395, 43
A:   9105477787064988985
x*y: 9105477787064988985
B:   211755297373604352
x^y: 211755297373604352
113 iterations

x, y: 68039759325, 73164771
A:   4978113409908739575
x*y: 4978113409908739575
B:   67966612030
x^y: 67966612030
69 iterations

x, y: 1264664368618221, 4883
A:   6175356111962773143
x*y: 6175356111962773143
B:   1264664368613886
x^y: 1264664368613886
99 iterations

x, y: 805306375, 805306369
A:   648518352783802375
x*y: 648518352783802375
B:   6
x^y: 6
59 iterations