The nth numerator

MATL, 17 13 bytes

:tt!/XR6#uG))

Try it online! Or verify all test cases.

Input size may be limited by floating point accuracy. All test cases give the correct result.

Explanation

This generates all fractions k/m with k, m in [1 2 ...n], as an n×n matrix. The row indicates the numerator and the column indicates the denominator. Actually the matrix entry contains the inverse fraction m/k, instead of k/m, but this is irrelevant and can be ignored in the rest of the explanation.

Matrix entries are implicitly considered to be sorted in column-major order. In this case this corresponds to the required order: denominator, then numerator.

Three types of entries need to be disregarded from this matrix:

1. Entries k/m, k>m, that have the same value as a previous entry (for example, 2/4 is disregarded because it is the same as 1/2)
2. Entries k/k, k>1. Entries that have a numerator exceeding the denominator
3. Entries k/m, k<m (these are not part of the problem).

Disregarding entries is done with a unique function, which stably removes duplicate values and outputs the indices of the surviving entries. With this, entries of type 1 above are automatically removed. To deal with types 2 and 3, the matrix entries at the diagonal and below are set to 0. This way, all zero entries will be removed except the first (corresponding to the valid fraction 1/1).

Consider input 4 as an example.

:     % Input n implicitly. Push range [1 2 ...n]
      % STACK: [1 2 3 4]
t     % Duplicate
      % STACK: [1 2 3 4], [1 2 3 4]
t!    % Duplicate and transpose
      % STACK: [1 2 3 4], [1 2 3 4], [1; 2; 3; 4]
/     % Divide element-wise with broadcast: gives matrix with all pairs
      % STACK: [1 2 3 4], [1       2       3       4;
                           0.5000  1       1.5000  2;
                           0.3333  0.6667  1       1.3333;
                           0.2500  0.5000  0.7500  1     ]
XR    % Upper triangular part above the diagonal. This sets to 0 all entries
      % corresponding to fractions that equal or exceed 1. (Since the matrix
      % actually contains the inverse fractions, nonzero entries will contain
      % values greater than 1)
      % STACK: [1 2 3 4], [0       2       3       4;
                           0       0       1.5000  2;
                           0       0       0       1.3333;
                           0       0       0       0     ]
6#u   % Indices of first appearance of unique elements
      % STACK: [1 2 3 4], [1; 5; 9; 10; 13; 15]
G     % Push input n again
      % STACK: [1 2 3 4], [1; 5; 9; 10; 13; 15], 4
)     % Index: get the n-th entry from the array of indices of unique elements
      % STACK: [1 2 3 4], 10
)     % Index (modular): get the corresponding real part. Display implicitly
      % STACK: 2

Jelly, 11 9 bytes

gRỊTµ€Fị@

Try it online! or verify all test cases.

How it works

gRỊTµ€Fị@  Main link. Argument: n

    µ€     Map the monadic chain to the left over [1, ..., n]; for each k:
 R           Range; yield [1, ..., k].
g            Compute the GCD of k and each j in [1, ..., k].
  Ị          Insignificant; yield 1 for 1; 0 for 2, ..., k.
   T         Truth; yield all indices of 1's, i.e., all coprimes with k.
      F      Flatten the resulting 2D array.
       ị@    At-index swapped; return the n-th element.

Wolfram Language (Mathematica), 53 49 bytes

(Join@@Array[Pick[r=Range@#,r~GCD~#,1]&,#])[[#]]&

Try it online!