What is XeTeX exactly and why should I use it?

If you have trouble with fonts and character sets, it may be for you: lots of people have the experience that things just work with Xetex which require fiddling with Pdftex. Two capabilities:

1. Xetex assumes its input is Unicode. To use Unicode with Pdftex, you need Tex code that manipulates encoding tables, like the {inputenc} Latex package;
2. Xetex treats many system fonts, like TTF and OTF, on a par with Tex fonts. Using them is generally no harder than giving the path to the fonts. Pdftex needs to have the font metrics described to it for system fonts. Luatex can figure out this from system fonts, but it's not a perfectly troublefree matter yet.

If these aren't issues for you, then there's not much reason to consider moving to Xetex. Pdftex has more sophisticated microtypography.

Look at Joseph Wright's answer to the Differences between LuaTeX, ConTeXt and XeTeX question: Luatex is intended to be the successor project to Pdftex, and he is informative about what is ahead.

I've switched to XeLaTeX a couple of months ago in order to use Open Type Fonts. Although I find the system stable and easy to use with the excellent fontspec package, there a few issues that should be mentioned:

• If you are using ligatures such as old style figures you cannot copy & paste text from a PDF (see Small-caps, old-style numbers, and some ligatures produce odd symbols in PDF copy text?). Instead that text will be represented by missing symbols. That is an absolute killer if you want your documents do be indexed or want that others can copy and paste easily.

• Someone (not me) mentioned on tex.sx that development of XeTeX has been progressing very slowly as the developer is busy with other things (no rant or criticism intended!). That means that it might take very long for a particular feature to be implemented or bugs to be fixed.

• As mentioned before, microtypography is currently an issue with XeTeX.

• It is recommended not to use babel with XeTeX (although it works if you are just using "Latin" languages).

These four issues (particularly the first one!) have convinced me to use LuaTeX instead of XeTeX as all of them are resolved in LuaTeX. I have yet to find any issues with it (apart from that it is a little slow with my fairly complicated templates). So if you want to change from pdfLaTeX you could also consider LuaTeX.

Here is a point that is implicit in the thorough comments and linked answers referenced above, but possibly not obvious.

Installing and managing fonts in traditional TeX systems can be daunting. For me, given that I want to focus on writing rather than software management, messing around with fonts was often not worth the trouble. XeTeX allows me to use lots of fonts easily while remaining ignorant (i.e. while allowing me to keep my time free for learning those things that are most important for me). On a Mac, at least, it's trivially easy to install fonts using a program like OS X's Font Book, and then XeTeX lets me pull those fonts into my LaTeX documents. Sounds like something similar is true for other popular operating systems.