What is "Bra" and "Ket" notation and how does it relate to Hilbert spaces?

In short terms, kets are vectors on your Hilbert space, while bras are linear functionals of the kets to the complex plane

$$\left|\psi\right>\in \mathcal{H}$$

\begin{split} \left<\phi\right|:\mathcal{H} &\to \mathbb{C}\\ \left|\psi\right> &\mapsto \left<\phi\middle|\psi\right> \end{split}

Due to the Riesz-Frechet theorem, a correspondence can be established between $\mathcal{H}$ and the space of linear functionals where the bras live, thereby the maybe slightly ambiguous notation.

If you want a little more detailed explanation, check out page 39 onwards of Galindo & Pascual: http://www.springer.com/fr/book/9783642838569.

First, the $bra$c$ket$ notation is simply a convenience invented to greatly simplify, and abstractify the mathematical manipulations being done in quantum mechanics. It is easiest to begin explaining the abstract vector we call the "ket". The ket-vector $|\psi\rangle $ is an abstract vector, it has a certain "size" or "dimension", but without specifying what coordinate system we are in (i.e. the basis), all we know is that the vector $\psi$ exists. Once we want to write down the components of $\psi,$ we can specify a basis and see the projection of $\psi$ onto each of the basis vectors. In other words, if $|\psi\rangle$ is a 3D vector, we can represent it in the standard basis $\{e_1,e_2,e_3\}$ as $\psi = \langle e_1|\psi\rangle |e_1\rangle + \langle e_2|\psi\rangle|e_2\rangle + \langle e_3|\psi\rangle|e_3\rangle,$ where you notice that the $\langle e_i|\psi\rangle$ is simply the coefficient of the projection in the $|e_i\rangle$ direction.

If $|\psi\rangle $ lives in a function space (a Hilbert space is the type of function space used in QM - because we need the notion of an inner product and completeness), then one could abstractly measure the coefficient of $\psi$ at any given point by dotting $\langle x | \psi \rangle = \psi(x)$, treating each point $x$ as its own coordinate or its own basis vector in the function space. But what if we dont use the position basis? Say we want the momentum-frequency-fourier basis representation? Simple, we have an abstract ket vector, how do we determine its representation in a new basis? $\langle p | \psi \rangle = \hat{\psi}(p)$ where $\hat{\psi}$ is the fourier transform of $\psi(x)$ and $|p\rangle$ are the basis vectors of fourier-space. So hopefully this gives a good idea of what a ket-vector is - just an abstract vector waiting to be represented in some basis.

The "bra" vector... not the most intuitive concept at first, assuming you don't have much background in functional analysis. Mathematically, the previous answers discuss how the bra-vector is a linear functional that lives in the dual hilbert space... all gibberish to most people just starting to learn this material. The finite dimensional case is the easiest place to begin. Ket vectors are vertical $n\times 1$ matrices, where $n$ is the dimension of the space. Bra vectors are $1 \times n$ horizontal matrices. We "identify" the ket vector $|a\rangle = (1,2,3)^T$ with the bra vector $\langle a| = (1,2,3),$ although they are not strictly speaking "the same vector," one does correspond to the other in an obvious way. Then, if we define $\langle a | a \rangle \equiv a \cdot a \in \mathbb{R}$ in the finite dimensional case, we see that $\langle a |$ acts on the ket vector $|a\rangle$ to produce a real (complex) number. This is exactly what we call a "linear functional". So we see that maybe it would be reasonable to define a whole new space of these horizontal vectors (call it the dual space), keeping in mind that each of these vectors in the dual space has the property that when it acts on a ket vector, it produces a real (complex) number via the dot product.

Finally, we are left with the infinite dimensional case. We now have the motivation to define the space of all bra-vectors $\langle \psi |$ as the space of all functions such that when you give another function as an input, it produces a real (complex) number. There are many beautiful theorems by Riesz and others that establish existence and uniqueness of this space of elements and their representation in a Hilbert space, but foregoing that discussion, the intuitive thing to do is to say that bra $\langle \phi |$ will be very loosely defined as the function $\phi^*$, and that when you give the input function $\psi(x),$ the symbol means $\langle \phi | \psi\rangle = \int \phi^*\psi \; dx \in \mathbb{R},$ hence $\phi$ is in the dual space, and it acts on a ket-vector in the Hilbert space to produce a real number. If anything needs clarification, just ask. Its a worthwhile notation to master, whether a mathematician or physicist.

I would like to extend Alex' answer, as well as answer your question in the comments: "Is the "bra" basically something like the dot product?"

If you have a vector space $V$ over a field $F$, which is, for now, finite dimensional, you can create another vector space, $V^*$, called the dual space of $V$, which consists of linear functionals defined on $V$. Linear functionals are essentially scalar valued linear maps.

So $$ V^*=\{\omega:V\rightarrow F\ |\ \omega \text{ is linear}\}. $$

As it turns out, if $V$ is $n$-dimensional, so is $V^*$, however you cannot really say anything more than that. BUT, if $V$ is real, and has a scalar product on it, then $V$ and $V^*$ are canonically isomorphic, in the sense that if you have a scalar product $\langle y,x\rangle$, then exists a unique $\omega_y\in V^*$ linear functional such that $$ \omega_y(x)=\langle y,x\rangle ,$$ and if you have a linear functional $\omega$, then there exists a unique $y_\omega\in V$ vector such that $$ \langle y_\omega,x\rangle=\omega(x). $$ In these cases, you can identify linear functionals with vectors and vice versa, and you can take a scalar product to be the action of a linear functional on the vector and vice versa.

What I said above is not true for infinite dimensional vector spaces in general, however, it is true for Hilbert-spaces, as long as you consider continuous linear functionals (in a finite dimensional space, all linear stuff are continous).

I will note that the above is also true for complex vector spaces with the caveat that since the scalar product is sesquilinear, rather than bilinear, the correspondance between the vector space and its dual is not an isomorphism but a conjugate-linear bijection.

With this in mind, in QM, a vector written as $|\psi\rangle$ is an element of a Hilbert space $\mathcal{H}$, a "bra" written as $\langle\psi|$ is an element of the continuous duals space of $\mathcal{H}$, the "bra-ket" written as $\langle\psi|\chi\rangle$ is both the action of $\langle\psi|$ on $|\chi\rangle$ and the scalar product of $|\psi\rangle$ and $|\chi\rangle$, where it is understood that $\langle\psi|$ is the "dual pair" of $|\psi\rangle$ through this conjugate linear bijection between $\mathcal{H}$ and $\mathcal{H}^*$.