Possible bug with UpTo

Turning my comment into an answer, this bug has been fixed as of Mathematica 10.4.0.

Take[{{1}}, All, {UpTo@3}]

During evaluation of Take::seqs: Sequence specification (+n, -n, {+n}, {-n}, {m, n}, 
or {m, n, s}) expected at position 3 in Take[{{1}},All,{UpTo[3]}]. >>

(* Take[{{1}}, All, {UpTo[3]}] *)

I indeed got the out-of-memory error in your original example.

This works:

Take[{{1}}, All, UpTo@3]

I think putting the last argument in braces makes Mathematica assume that its contents is an integer, but UpTo[3] is not an integer.