Mathematica 11 with High DPI 4k screen

The best thing to do, IME, is to set the default notebook zoom level to something higher. This way, the text is rendered crisply and at a reasonable size.

You can do that as follows:

1. Go to Preferences -> Advanced -> Open Option Inspector
2. Set Show option values to Global Preferences
3. Go to Notebook Options
4. Go to Display Options
5. Change magnification to whatever works for you.

Now, "100%" (default mag) will be rendered at whatever magnification you chose here.

Mathematica 11.2 seems to use Qt 5.6 framework which supports DPI scaling override. At least on Linux, it's possible to get bearable results by starting Mathematica as follows:

QT_SCALE_FACTOR=1.5 Mathematica

QT_SCALE_FACTOR is an environment variable that defines UI scaling. It seems to work quite well except for the splash screen.

With Windows 10 Build 15002 and above, you can use the following settings to obtain a higher resolution of MMA than magnification for 200%, for example.