How can I mix LaTeX in with Markdown?

Add the following code to the top of your Markdown files to get MathJax rendering support

<style TYPE="text/css">
code.has-jax {font: inherit; font-size: 100%; background: inherit; border: inherit;}
</style>
<script type="text/x-mathjax-config">
MathJax.Hub.Config({
    tex2jax: {
        inlineMath: [['$','$'], ['\\(','\\)']],
        skipTags: ['script', 'noscript', 'style', 'textarea', 'pre'] // removed 'code' entry
    }
});
MathJax.Hub.Queue(function() {
    var all = MathJax.Hub.getAllJax(), i;
    for(i = 0; i < all.length; i += 1) {
        all[i].SourceElement().parentNode.className += ' has-jax';
    }
});
</script>
<script type="text/javascript" src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.4/MathJax.js?config=TeX-AMS_HTML-full"></script>

and then `$x^2$` or `$$x^2$$` will render as expected :-)

You can always install a local version of MathJax if you don't want to use the online distribution, but you might need to host it through a local webserver.

UPDATE: these days I just use pandoc instead of canonical markdown, but the above is still useful.

Have you tried with Pandoc?

EDIT:

Although the documentation has become a bit complex, pandoc has supported inline LaTeX and LaTeX templates for 10 years.

Documents like the following one can be written in Markdown:

---
title: Just say hello!
author: My Friend
header-includes: |
    \usepackage{tikz,pgfplots}
    \usepackage{fancyhdr}
    \pagestyle{fancy}
    \fancyhead[CO,CE]{This is fancy}
    \fancyfoot[CO,CE]{So is this}
    \fancyfoot[LE,RO]{\thepage}
abstract: This is a pandoc test with Markdown + inline LaTeX
---

Just say hello!
===============

This could be a good example or inlined \LaTeX:

\begin{tikzpicture}
\begin{axis}
\addplot[color=red]{exp(x)};
\end{axis}
\end{tikzpicture}
%Here ends the furst plot
\hskip 5pt
%Here begins the 3d plot
\begin{tikzpicture}
\begin{axis}
\addplot3[
    surf,
]
{exp(-x^2-y^2)*x};
\end{axis}
\end{tikzpicture}

And now, just a few words to terminate:

> Goodbye folks!

Which can be converted to LaTeX using commands like this: pandoc -s -i Hello.md -o Hello.tex

Following is an image of the converted Hello.md to Hello.pdf file using MiKTeX as LaTeX processor with the command: pandoc -s -i Hello.md -o Hello.pdf

Finally, there are some open source LaTeX templates like this one: https://github.com/Wandmalfarbe/pandoc-latex-template, that can be used for better formatting.

As always, the reader should dig deeper if he has less trivial use cases than presented here.

Perhaps mathJAX is the ticket. It's built on jsMath, a 2004 vintage JavaScript library.

As of 5-Feb-2015 I'd switch to recommend KaTeX - most performant Javascript LaTeX library from Khan Academy.