determinant of a matrix with binomial coefficient entries

Answer: The determinant of $A$ is $\dfrac{H\left(k\right) H\left(l-k\right) H\left(n\right) H\left(l+n\right)}{H\left(l-k+n\right) H\left(n+k\right) H\left(l\right)}$, where we are using the notation $H\left(m\right)$ for the hyperfactorial $\left(m-1\right)! \left(m-2\right)! \cdots 1! 0!$ of a nonnegative integer $m$.

(There are various other ways to express the answer, but the above is the most compact.)

1st proof. Theorem 8 in my note A hyperfactorial divisibility says that for every nonnegative integers $a$, $b$ and $c$, we have \begin{align*} & \dfrac{H\left( a\right) H\left( b\right) H\left( c\right) H\left( a+b+c\right) }{H\left( b+c\right) H\left( c+a\right) H\left( a+b\right) }\\ & =\det\left( \left( \dbinom{a+b+i-1}{a+i-j}\right) _{1\leq i\leq c,\ 1\leq j\leq c}\right) =\det\left( \left( \dbinom{a+b}{a+i-j}\right) _{1\leq i\leq c,\ 1\leq j\leq c}\right) . \end{align*} Applying this to $a = k$, $b = l-k$ and $c = n$ (noticing that $b$ is nonnegative because $k \leq l$), we obtain \begin{align*} &\dfrac{H\left(k\right) H\left(l-k\right) H\left(n\right) H\left(l+n\right)}{H\left(l-k+n\right) H\left(n+k\right) H\left(l\right)} \\ & =\det\left( \left( \dbinom{l+i-1}{k+i-j}\right) _{1\leq i\leq n,\ 1\leq j\leq n}\right) =\det\left( \left( \dbinom{l}{k+i-j}\right) _{1\leq i\leq n,\ 1\leq j\leq n}\right) . \end{align*} But the matrix on the right(most) side of this equality is exactly your $A$; so the Answer is proven.

2nd proof (sketched). The question has a combinatorial interpretation in terms of nonintersecting lattice paths on a rectangular grid, or semistandard Young tableaux, or Schur functions. These three objects are essentially different languages for the same argument; I will use the third since it is the easiest to write about.

I'll assume that you are familiar with the concept of symmetric functions (see, e.g., Chapter 2 of Darij Grinberg and Victor Reiner, Hopf algebras in combinatorics, or Mark Wildon, An involutive introduction to symmetric functions, or Chapter 7 of Richard Stanley, Enumerative Combinatorics, volume 2). In particular, I'll use the notations common in this subject, such as $s_\lambda$ for a Schur function and $e_i$ for an elementary symmetric function.

Set $b = l-k$; this is a nonnegative integer since $k \leq l$.

Let $\lambda$ be the partition $\left(n^b\right)$, which is to be understood as a shorthand for $\left(\underbrace{n,n,\ldots,n}_{b\text{ terms}}\right)$. The transpose (aka conjugate) $\lambda^t$ of this partition $\lambda$ is then $\left(b^n\right)$ (understood similarly). The Jacobi-Trudi formula (the one that uses elementary symmetric functions) thus says that \begin{equation} s_\lambda = \det \left( \left( e_{b-i+j} \right) _{1\leq i\leq n,\ 1\leq j\leq n} \right) . \end{equation} Now, substitute $\underbrace{1,1,\ldots,1}_{l \text{ entries}},0,0,0,\ldots$ for the countably many indeterminates into this equality. This substitution transforms each elementary symmetric function $e_p$ into $e_p\left(\underbrace{1,1,\ldots,1}_{l \text{ entries}},0,0,0,\ldots\right) = \dbinom{l}{p}$. Thus, the equation transforms into \begin{equation} s_\lambda\left(\underbrace{1,1,\ldots,1}_{l \text{ entries}},0,0,0,\ldots\right) = \det \left( \left( \dbinom{l}{b-i+j} \right) _{1\leq i\leq n,\ 1\leq j\leq n} \right) . \end{equation} Since each $i$ and $j$ satisfy $\dbinom{l}{b-i+j} = \dbinom{l}{k+i-j}$ (indeed, this follows from the symmetry of Pascal's triangle, because $l - \left(b-i+j\right) = k+i-j$), this rewrites as \begin{equation} s_\lambda\left(\underbrace{1,1,\ldots,1}_{l \text{ entries}},0,0,0,\ldots\right) = \det \left( \left( \dbinom{l}{k+i-j} \right) _{1\leq i\leq n,\ 1\leq j\leq n} \right) = \det A . \end{equation} It remains to compute the left hand side.

But the combinatorial definition of $s_\lambda$ shows that $s_\lambda\left(\underbrace{1,1,\ldots,1}_{l \text{ entries}},0,0,0,\ldots\right)$ is just the number of semistandard Young tableaux of shape $\lambda$ with entries in $\left\{1,2,\ldots,l\right\}$. For this number, there is a formula (known as Weyl's character formula in type A; see, e.g., https://mathoverflow.net/questions/106606/new-formula-for-counting-ssyts ), which can be written as \begin{equation} s_\lambda\left(\underbrace{1,1,\ldots,1}_{l \text{ entries}},0,0,0,\ldots\right) = \dfrac{1}{H\left(l\right)} \prod_{1 \leq i < j \leq l} \left(\left(\lambda_i - i\right) - \left(\lambda_j - j\right)\right) , \end{equation} where $\lambda_p$ is the $p$-th entry of $\lambda$ (so, in our case, $\lambda_p = b$ if $p \leq n$, and otherwise $\lambda_p = 0$). It takes a while to massage the right hand side so as to look like the Answer given above (the first step is to observe that the only factors in the product that are distinct from $1$ are those with $1 \leq i \leq n$ and $n < j \leq l$, so that all other factors can be discarded), but it isn't difficult (you just need to check that two large products have the same factors).

I am sure that the full answer is given by Darij Grinberg in his comment. This is indeed a problem in combinatorics.

If, however, all you want to see is that your determinant is non-zero then this may help.

First, for convenience, swap $k$ and $\ell-k$. Set $N=n+k$. Label all rows, columns, bases starting at $0$.

Then your determinant is an $n\times n$ minor of the $N\times N$ matrix $M$ where $m_{ij}=\binom{\ell}{j-i}$; to be precise it is the minor corresponding to the pair $\{1,2,\dots,n\}\times \{N-n, \dots, N-2,N-1\}$.

Now the matrix $M$ is just $(1+N)^\ell$, where $N$ is the matrix with $1$'s on the superdiagonal and $0$ elsewhere. So $M^{(n)}$, the matrix formed from the $n\times n$ minors of $M$, can be expressed $$ M^{(n)}=\left((1+N)^\ell\right)^{(n)}=\left((1+N)^{(n)}\right)^\ell $$ by the usual rule.

Write $a_{I,J}$ for the entry of $(1+N)^{(n)}$ labelled by the pair $(I,J)$ of $n$-subsets of $\{0,\dots, N-1\}$, and $b_{I,J}$ for the corresponding entry of $M^{(n)}$. We have then $$ b_{I,J}= \sum_{I_1,I_2,\dots, I_{\ell-1}} a_{I,I_1}a_{I_1,I_2}\dots a_{I_{\ell-1},J}. $$

Now we need to say something about the $a_{I,J}$. It is easiest to proceed a bit more abstractly and so let $e_0,\dots,e_{N-1}$ be an ordered basis of the underlying vector space, and let $e_N=0$. We can suppose that our linear map $N$ acts as $e_i N=e_{i+1}$ for all $i=0,1,\dots, N-1$.

Now consider the action of $M=1+N$ on the $n$-th exterior products, ordered lexicographically. $$ e_{i_1}\wedge e_{i_2}\wedge \dots \wedge e_{i_{n}} (1+N) = (e_{i_1}+e_{i_1+1})\wedge (e_{i_2}+e_{i_2+1})\wedge \dots \wedge(e_{i_{n}}+e_{i_{n}+1}). $$

As $i_1<i_2<\dots<i_n$ we have $i_1<i_1+1\leqslant i_2<i_2+1\leqslant i_3<\dots<i_n$. This means that when we expand the product on the right hand side all the terms are in canonical form, we need not perform any swaps; and each basis element can occur at most once.

Hence each $a_{I,J}$ is either $0$ or $1$.

We then have that each $b_{I,J}$ is a natural number.

To prove that the one we are interested in, the one with $I=\{1,2,\dots,n\}$, $J= \{N-n, \dots, N-2,N-1\}$, is non-zero we merely have to find a chain $I,I_1,I_2,\dots I_{\ell -1}, J$ with the $a_{I_s,I_{s+1}}=1$. This is left as an exercise for the reader.