Build/Release flow on VSTS with MSI deployed desktop app using configuration transforms
We had exactly the same problem building MSIs from a Visual Studio solution that contained a WiX Installer project, using config transforms on the app.config to replace the configuration.
As you suggested, we originally went down the route of running an Azure DevOps build pipeline with multiple builds for every configuration in the solution, but this quickly became inelegant and wasteful as not only did we require builds for (dev/stage/qa/live) but also had configurations that applied to multiple customers, which ended up in 12 + configurations in the solution and really long build times.
Replace config within the MSI
The solution we ended up with, as alluded to in a previous answer, was to build the MSI only once in a build pipeline, copy the MSI along with all our replacement app.config files to the drop folder, and then run a custom application within the release pipelines to forcibly replace the Application.exe.config inside the MSI. Unfortunately, this isn't as simple as just 'unzipping the MSI', replacing the config and then 're-zipping' within a release task because the MSI uses a custom file format and maintains an internal database that needs to be modified properly.
We ended up creating a custom C# .NET console application using the method posted in this stack overflow answer, which we then hosted on our on-premises build agent so that we could run a simple powershell task within our release pipeline that called our custom console application with some relevant parameters:
"C:\BuildTools\msi_replace_file.exe" -workingfolder "$(System.DefaultWorkingDirectory)/_BuildOutput/drop/Application.Installer/bin/Release/" -msi "Application.Installer.msi" -config "Application.exe.config"
We then had a release pipeline stage for each 'configuration' that performed these basic steps:
There are various other methods for replacing a file in an MSI, as described in this question, but we chose to create a C# application using the utilities within the Microsoft.Deployment.* namespace that are provided as part of the WiX Toolset. This would guarantee compatibility with the version of WiX we were using to build our installer in the first place and gave us full control of the process. However, I appreciate that this approach is quite brittle (which I'm not happy about) and not especially scalable as its relying on a custom tool to be hosted on our on-premises build agent. I intend to improve this in the future.
You should also be aware that hacking the MSI in this way could cause problems in the future, especially if you change your tool-chain or upgrade to a later version of WiX.
Building the MSI from the release pipeline
I do not personally like the idea of copying the required dlls/assets to the drop location and then 'building' the MSIs within the release pipeline, because for us the act of building the WiX project was very much part of our 'build process' and was integrated into our visual studio solution, so it felt like moving the creation of the MSI to the release pipelines was counter intuitive and would also potentially require us to create custom tasks on the build agents to run the WiX CLI tools (heat.exe, light.exe, candle.exe) against a version of our WXS file or have build steps that just built the wixproj file instead of the whole solution. However, I can see how this alternative approach may be suitable for others and I think is equally valid depending on your circumstances.