Visual Studio 2017 is not detected by extensions for installation
From the log file it looks like an older vsixinstaller.exe tried to install your extension, not the one from VS 2017. It should be in a path like "C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\Common7\IDE\VSIXInstaller.exe"
Try to explicitly run it with Roslynator.VisualStudio.1.2.51.vsix as a parameter.
As Hans Passant commented, huge changes have been made by Microsoft on the installation, so previous approaches of detecting VS won't work any more. For example, to detect VS installation folder, now a few extra tools or scripts are required,
https://github.com/Microsoft/vswhere
So guides such as How to: Migrate Extensibility Projects to Visual Studio 2017 are critical and you can follow the tips.
Press Win+R and write the path to correct version VSIXIntaller. for example.
"E:\Microsoft Visual Studio\2017\Enterprise\Common7\IDE\VSIXInstaller.exe" "C:\Users\jdonoso.IMLG\Downloads\Ankh.VS15Package.vsix"
I have 3 versions of Visual Studio, but the correct installer is VS2017 since the package is for VS2017. Take this into consideration.
That's why I had the same mistake. Solve it using the correct version.