Windows installer MSI format
The binary format is not open, I wouldn't even go as far as calling it a standard.
However I assume it's not the file format you're concerned with (which is undocumented), but rather the tables, actions, sequences, etc. that are visible when interfacing with the MSI via the Windows Installer SDK, API, Orca, etc.
Everything is covered in the SDK documentation, however for some reason search engines don't rank the documentation very highly. Also you can no longer download the Windows Installer SDK separately, it is now bundled as part of the Windows SDK.
- Windows Installer SDK Documentation
- Windows SDK Download
MSI files are COM structured storage, which is described here: Structured Storage (Windows).