What does algebraic mean?

It means it's the root of a polynomial with coefficients in $F$.

For example $\sqrt{2}$ is algebraic over $\Bbb{Q}$ where $\pi$ isn't.

$a$ is algebraic over $F$ just when it is a root of a polynomial with coefficients from $F$. So $\sqrt{2}$ is algebraic over $\mathbb{Q}$ since it's a root of the polynomial $x^2 - 2$.

Edit: More examples, more context.

First a small finite example - essentially, the smallest example in the family suggested by @fleablood 's comment.

Here are the addition and multiplication tables for the four element field:

+ | 0 1 a b       * | 0 1 a b
-------------     -------------
0 | 0 1 a b       0 | 0 0 0 0
1 | 1 0 b a       1 | 0 1 a b
a | a b 0 1       a | 0 a b 1
b | b a 1 0       b | 0 b 1 a

Then $a$ is a root of the polynomial $x^2 + x + 1$ with coefficients in the two element subfield $F = \{0,1\}$ so it's algebraic over that subfield.

In order to understand what "algebraic" means you should also consider situations where it fails. (That's true for any new mathematical concept.) To see examples you have to move beyond finite. If a field $K$ containing a field $F$ is finite dimensional (not necessarily finite) as a vector space over $F$ then every element $a$ of $K$ is algebraic over $F$ since the set $\{1, a, a^2, \ldots \}$ must be linearly dependent.

As @ZacharySelk notes in his answer, $\pi$ is not algebraic over the rationals. But that's pretty deep. It's easier to prove that Liouville numbers (https://en.wikipedia.org/wiki/Liouville_number) are not algebraic.

There are examples that don't depend on analysis. For any field $F$ (think about the rationals) let $K$ be the field whose elements are quotients of polynomials with coefficients from $F$. Then the element $x$ of $K$ is not the root of any polynomial with coefficients in $F$, so it's not algebraic.