What is the difference between singular matrices and degenerate matrices?
Yes.
A singular matrix, also known as a degenerate matrix, is a square matrix whose determinate is zero.
That is, they do not have an inverse.
No. Degenerate matrices are not the same as singular matrices. A matrix is degenerate if and only if the determinants of all its principal submatrices are non-zero (where the principal submatrices are the square matrices obtained by choosing the same subset of both the rows and the columns of the original matrix).
See e.g. "On the number of solutions to the complementarity problem and spanning properties of complementary cones", Murty (1972) http://www.sciencedirect.com/science/article/pii/0024379572900195
Because lately I am reading many articles that is including degenerate instead of non singular or non invert-able then I assume there is a difference. As far as I understood from Wikipedia the singularity theories in this link, it is mentioned that singular sometimes is stable (non degenerate) and sometimes is not stable (degenerate).
The determinant explanation is common for both, if ${\textbf{A}_{n\times n}}$ $det(\textbf{A}) \neq 0$, then $\textbf{A}$ is non singular, non degenerate and as a result is invertable.
A biological definition of degenerate is to become less specialized. I have seen that explanation several times when columns of a matrix are called flat columns (high correlation between columns), each column is becoming less specialized or less informative and as a result the matrix becomes degenerate. This is directly related to the condition number of a matrix. When Condition no. of a matrix is too high then it is an ill conditioned matrix and there is a high correlation between its columns.
Another explanation that I liked for singular matrix is that when you transform a matrix to another shape of less dimension (plane to line), it becomes singular and you will not be able to invert it back as information is lost. Again this is mentioned in Wikipedia link saying 2 points becomes 1. Also a good explanation is in that youtube video
I just found an answered question that proves it,
Simply degeneracy is related to correlation between columns or between rows. Condition of a matrix is decided based on the correlation of the matrix so there is a direct connection between degeneracy and correlation between columns. Adding that the relation between rank of matrix (full rank or not) and its size is another definition of degeneracy. ex. ${\bf{A}}_{I \times J}$ with rank $R$
if $R \le (I,J)$ then it is a non degenerate matrix
if $R > (I,J)$ then it is a degenerate matrix (under-determined system) more unknowns than givens, high information to be found represented by rank while available data is limited represented by size.
if $\sigma_1 \gg \sigma_{R} $ then the matrix is degenerate, where $\sigma_n$ is the singular values of a matrix and $R$ is the highest index where $\sigma$ > 0. ex. $\sigma_1 = 100$ while $\sigma_R = 0.0001$
if ${\bf{A}}$ is not full rank matrix then ${\bf{A}}$ is a degenerate matrix. Rank{$\bf{A}$} $\neq$ min{size($\bf{A}$)}
if ${\bf{A}}$ is a singular matrix then $cond\{ {\bf{A}} \} = \infty $, so is also a degenerate matrix.
high condition number is an ill condition matrix and low condition no. is well condition matrix.