How to avoid "if statement" in Mathematica?

pts = RandomReal[1.0, {100, 2}];
vec = {1, 1};

Rotate the elements of pts such that you want the one with small positive first coordinate (or maximal negative first coordinate if no one has positive first coordinate):

rotated = Transpose[RotationMatrix[{vec, {0, 1}}].Transpose[pts]];

Add the vector {0, 0} at position 1 and make a list with the vector indices ordering the points from left to right:

order = Ordering[Prepend[rotated, {0, 0}]]

Now Ordering[order, 1] yields the position of {0, 0}

pts[[Extract[Append[order, order[[-2]]], Ordering[order, 1] + 1] - 1]]

You can define a function that returns the positive angle between two vectors:

positiveAngle[{u1_, u2_}, {v1_, v2_}] := 
 N@Mod[ArcTan[u2, u1] - ArcTan[v2, v1], 2 π]

Then, if your input is pts and vec:

pts = RandomReal[1.0, {10, 2}];
vec = {1, 1};

You can simply use:

min = First[MinimalBy[pts, positiveAngle[#, vec] &]]

to get the first vector with the smallest positive angle to vec

If we order the points by positiveAngle, we can see the order in which they would be picked:

enter image description here

Graphics[{
  {Dashed, Line[{{0, 0}, vec}]},
  {LightGray, Line[{{0, 0}, #}] & /@ pts},
  {Red, Arrow[{{0, 0}, min}]},
  MapIndexed[
   Text[#2[[1]], #1] &, SortBy[pts, positiveAngle[#, vec] &]]
  }]

If your points are in all 4 quadrants, they would be ordered like this:

enter image description here

pts = CirclePoints[.9, 24];    
Graphics[{{Dashed, Line[{{0, 0}, vec}]}, {LightGray, 
   Line[{{0, 0}, #}] & /@ pts}, {Red, Arrow[{{0, 0}, min}]}, 
  MapIndexed[Text[#2[[1]], #1] &, 
   SortBy[pts, positiveAngle[#, vec] &]]}]

Different orderings should be easy to achieved using e.g. Piecewise functions


Here is an implementation of the algorithm that you describe:

pts = RandomReal[2, {50, 2}];
vec = {1, 1};

rhsQ[vec_, pt_] := Sign@Last@Cross[Append[vec, 0], Append[pt, 0]] == -1

onRHS = rhsQ[vec, #] & /@ pts;

rhs = Pick[pts, onRHS];
lhs = Pick[pts, onRHS];

rightmost[v1_, v2_] := If[rhsQ[v2, v1], v2, v1]
findRightmost[vecs_] := Fold[rightmost, vecs]

res = If[
   rhs === {},
   findRightmost[lhs],
   findRightmost[rhs]
   ];

Graphics[{
  Line[{{0, 0}, Normalize[#]}] & /@ pts,
  Red, Line[{{0, 0}, 1.1 Normalize[vec]}],
  Green, Line[{{0, 0}, 1.1 Normalize[res]}]
  }]

Mathematica graphics

There are two ifs in there, but I don't see it as a problem. I do however favor the type of solution that Coolwater posted.