How to use "`" in the same way it is used on the SE websites?

In addition to Werner's answer, the same is possible with the listings package. To undefine the shorthand use \lstDeleteShortInline`.

Be aware, that using ` as an escape character for inline code will most likely clash with language feature packages such as babel.

\documentclass{article}
\usepackage{listings}
\lstset{basicstyle=\ttfamily}
\begin{document}
\lstMakeShortInline`

Here is some inline code `\int f(x) \, \mathrm{d}x`

\end{document}

You can define your own verbatim shorthand using fancyvrb's \DefineShortVerb:

\documentclass{article}

\usepackage{fancyvrb}
\DefineShortVerb{\`}

\begin{document}

On any SE post one can use the quote in order to highlight a text.
For example, `this text is highlighted`.

\end{document}

To remove this functionality, use \UndefineShortVerb{\`}, as it may interfere with other uses of ` within your code.

It is more natural to use " or | rather than ` for this purpose, since ``foo'' is translated to “foo” in LaTeX.

There are several packages to define a short verbatim command for this. The basic one is shortvrb, which is part of LaTeX base package doc:

\documentclass{article}

\usepackage{shortvrb}
\MakeShortVerb|
% \DeleteShortVerb|

\begin{document}

Short verbatim: |foo#bar|

\end{document}

As Werner and Henri Menke said, you can also use fancyvrb, gmverb, newverbs for this functionality, or use listings with syntax highlight support.

Sometimes, however, you don't want to use verbatim text, only special fonts or color is needed. Then you can use some tricks to define a one-character command yourself:

\documentclass{article}

\newcommand\MakeShortHighlight[2][\texttt]{%
  \begingroup\lccode`\~=`#2\lowercase{\endgroup
    \def~##1~{#1{##1}}}%
  \catcode`#2=\active}
\newcommand\DeleteShortHightlight[1]{%
  \catcode`#1=12 }

\usepackage{xcolor}

\begin{document}

\MakeShortHighlight\"
Short highlighted: "typewriter text"


\MakeShortHighlight[\textcolor{red}]\|
Short highlighted: |red color|

\end{document}