When to name a variable with $

Look at the system context to see what the convention is. I can find 257 built-in symbols in 11.3,

symbols = Names["System`$*"];
Length @ symbols
(* 257 *)

In the system context there seems to be two main uses of the $ naming convention. It is used for static symbols without value, that can indicate e.g. evaluation status, like $Aborted, $Failed, $Canceled, etc. The other is "constants" that may have values defined at runtime, like $BaseDirectory or $FrontEnd (but not necessarily having a value at startup, e.g. $Pre or $Post by default have no values). Since their values are often settable even by the user, they are not true constants.

You don't know what fonts are available on the user's system, but you can write your code to reference $FontFamilies. Perhaps $MinPrecision is system dependent, so using that variable means you don't have to hardcode a value in your functions.

You certainly don't know $SystemID or $OperatingSystem when writing a function, but you can depend on them being defined when the functions are run.

You can look at the values for all your $ constants in a dataset,

Dataset@AssociationThread[symbols -> (Symbol /@ symbols)]

While $__ variables are not special in terms of evaluation they are special in terms of language/package design. They alert a developer/user to the fact that $__ variables are environmental variables. Sometimes they have values extracted from an overarching process/environment and hence are constant-like (e.g.$OperatingSystem, $FrontEnd, $Version, $InputFileName, $CloudBase,$MemoryInUse etc) which can be useful for enacting relevant code bases dependent on these settings.

At other times $__ variables are designed to be user-modified by affecting the action of a suite of corresponding functions. Sometimes these settings should be done sparingly given their role in well-designed idioms (for example, $ContextPath,$Contexts,$Packages affecting and being affected by corresponding functions Get, Needs) while at other times their modification seems more routine-like (e.g.$PerformanceGoal,$MaxPrecision,$DisplayFunction,$IterationLimit).

In this latter case where they are designed to be modified, it is similar to globally setting an option value which can be convenient if this setting is likely to recur through a session/package. So, for example, the following pairs are equivalent

Get[file, Path -> $Path]
Get[file]

and

Predict[training,PerformanceGoal -> $PerformanceGoal]
Predict[training]

So when the values of $Path and $PerformanceGoal are likely to be consistent in a given environment they can be set once initially with all subsequent applications of Get[file] and Predict[training] not needing the specific option setting.

Semantically then, $__ variables also alert the user/developer to the fact that they are being treated as global variables even without any formal interpretation (n.b. the default value of $Context is "Global`"). Generally it is considered bad practice to cavalierly introduce global variables since they tend to work against developing modularity and independence in your code base. Hence the use of $ is useful for flagging justified exceptions and hence one takeaway might be:

Don't use global variables but if you must, try and restrict to the above scenarios and always prefix with $.

(Speculating but maybe this convention is a legacy of the environmental variables of operating systems like \$Home and \$Path (n.b. $HomeDirectory and $Path) where the syntax of shell scripts specifies a leading \$ to access a named variable's value along with pedagogical treatments conveniently using pre-defined environmental variables as first examples).