How to stop a PowerShell script on the first error?

$ErrorActionPreference = "Stop" will get you part of the way there (i.e. this works great for cmdlets).

However for EXEs you're going to need to check $LastExitCode yourself after every exe invocation and determine whether that failed or not. Unfortunately I don't think PowerShell can help here because on Windows, EXEs aren't terribly consistent on what constitutes a "success" or "failure" exit code. Most follow the UNIX standard of 0 indicating success but not all do. Check out the CheckLastExitCode function in this blog post. You might find it useful.

You should be able to accomplish this by using the statement $ErrorActionPreference = "Stop" at the beginning of your scripts.

The default setting of $ErrorActionPreference is Continue, which is why you are seeing your scripts keep going after errors occur.