What is a type in Wolfram Mathematica programming language?

The nearest Mathematica has to "types" are Heads of expressions that are Atoms. For example:

Through[{AtomQ, Head}[2]]

{True, Integer}

Through[{AtomQ, Head}[2 + I]]

{True, Complex}

Through[{AtomQ, Head}["cat"]]

{True, String}

and so on...

There are also somewhat different "types" in the context of Compile.

In Mathematica, the type of a built-in object is represented by the Head. For example, Head[3] is Integer, Head[1.5] is Real and Head[a] is Symbol (assuming that a hasn't been assigned a value, of course, because in that case, you'll get the Head of that value).

Note that for expressions of the form foo[bar,baz], the head is foo. Most expressions are internally represented in such a form, as can be seen by applying FullForm, for example:

{1, 2, 3}//FullForm
(*
==> List[1, 2, 3]
*)
a + b //FullForm
(*
==> Plus[a, b]
*)

and consequently the Head gives List in the first case, and Plus in the second case.

Edit

Note that Mathematica evaluates the expression given to Head if possible, so the result of Head[1+2] is not Plus but Integer because Head is replaced only after evaluating 1+2 which gives the value 3 of type Integer. On the other hand, Head[a+b] gives Plus (assuming neither a nor b have any applicable definition), because a+b is "inert" under evaluation: There are no evaluation rules which could applied to it, therefore it is self-evaluating. Note that the same is true for head replacing (Apply, @@). Also note in that context that Mathematica, unlike the typical functional language, continues evaluation until it reaches an "inert" expression which cannot be further evaluated. This is best demonstrated by the following Mathematica code:

foo := bar;
bar := 3;
foo
(*
==> 3
*)

Here in the first step, Mathematica evaluates foo, which evaluates to bar. But it doesn't stop there, but continues to evaluate bar which gives the result 3. Now that result cannot be further evaluated, and therefore is given as the result. On the other hand, the seemingly equivalent Lisp code

(progn
  (setq foo 'bar)
  (setq bar 'baz)
  foo)

will evaluate to bar because after evaluating foo to bar, Lisp doesn't try to re-evaluate.

To see how this interacts with head replacing, consider the following code:

Plus@@Range[1,5]
(*
==> 15
*)

In the first step, Range[1,5] is evaluated, giving {1,2,3,4,5}, that is List[1,2,3,4,5] with head List. Now Apply replaces the head of that expression with Plus, resulting in Plus[1,2,3,4,5], that is 1+2+3+4+5. Now that is again an evaluatable expression, evaluating to 15 which as an "inert" expression is the final result.

Note that you can prevent Mathematica from evaluating an argument before calling a function by using Unevaluated. For example, Head[Unevaluated[Range[1,5]]] gives Range, and Plus@@Unevaluated[Range[1,5]] gives 6 (because the Range got replaced by Plus, and then the resulting expression is evaluated). In this respect, Unevaluated is similar to Lisp's quote. Note however that due to Mathematica's continuing evaluation, the effects can be drastically different. For example, consider the code

f[x_]:=x
f[Unevaluated[1+1]]
(*
==> 2
*)

You might expect this to return 1+1 unevaluated, just like the corresponding Lisp code with quote does. However while Unevaluated indeed causes 1+1 to be passed to f without further evaluation, applying the definition of f results in 1+1 (note: without the Unevaluated!), which Mathematica continues to evaluate, resulting in 2.

Since Mathematica 10, there is the TypeSystem` Context, that is nearly what you might be looking for. It is just a wrapper around patterns.

TypeSystem`ConformsQ[
  {1, 2, 3}, 
  TypeSystem`Vector[TypeSystem`IntegerT, 3]
]  (* --> True *)

It is the thing being used internally by Dataset-related functions. (Maybe one should say something like Dataset[{}] first to beable to use the TypeSystem; I’m too lazy to check again whether it is necessary.)

Unfortunately, it is not documented but you can figure it out looking at names in ?TypeSystem`* and at GeneralUtilities`PrintDefinitions @ TypeSystem`Validation`PackagePrivate`vtor.

TypeSystem`Either is (was?) not working as expected, but see my patch.