How to register "custom tool" with Visual Studio 2017 or Visual Studio 2019 to make it work?

You may have to follow different approach here by creating a Visual Studio extension (VSIX), below I have explained it in detail, hope it helps.

How to create a Custom Tool or Single File Generator in Visual Studio 2017:

Prior to VS2017 creating a custom tool required implementing Interface IVsSingleFileGenerator and code to register and unregister the custom tool in the system registry, but in VS2017, Microsoft has changed entire registry structure. The change is, VS will make registry entries to a private registry so that system registry is not messed up. While previously the registry entries were made in the system registry, now they are made to

C:\Users\xyz\AppData\Local\Microsoft\VisualStudio\15.0_xx\privateregistry.bin

Visual studio 2017 also supports testing your tool directly by running it from the visual studio itself (F5), which starts another instance of Visual Studio called Visual Studio Experimental Instance and your tool can be tested in it since it makes registry entries to

C:\Users\xyz\AppData\Local\Microsoft\VisualStudio\15.0_xxExp\privateregistry.bin

Follow below steps to create a Custom Tool in VS2017:

1. We need to create a VSIX extension
2. Add new Visual Studio Package
3. Implement IVsSingleFileGenerator
4. Add the registry entry code
5. Compile and test the tool by running it in VS2017
6. Install the tool by double clicking the generated .VSIX file

We will create an extension/custom tool as an example named "CountLines" which will read a file (having Custom Tool property set to CountLines) and generate an XML file containing the number of lines in the file. e.g. <LineCount>1050</LineCount>

1. Create a VSIX extension In order to create an extension you must have installed Visual Studio Extensibility Tools which is included as an optional feature in Visual Studio setup. If it is not installed you can also install it by modifying VS 2017 setup. Create new VSIX (Visual Studio Extension) project by selecting

New Project -> Extensibility -> VSIX Project

give it some name like "CountLinesVSIX".

2. Add new Visual Studio Package Once VSIX project is created, add new Visual Studio Package to it by selecting

Add -> New Item -> Extensibility -> Visual Studio Package

give it name "CountLines.cs". In CountLines.cs we need to delete existing code and replace it with our code for IVsSingleFileGenerator implementation

3. Implement IVsSingleFileGenerator Write your custom implementation for interface IVsSingleFileGenerator, our example code is as below

using System;
using System.Runtime.InteropServices;
using Microsoft.VisualStudio;
using Microsoft.VisualStudio.Shell;
using Microsoft.VisualStudio.Shell.Interop;
using System.Text;

namespace CountLinesVSIX
    {
    [PackageRegistration(UseManagedResourcesOnly = true)]
    [InstalledProductRegistration( "CountLines", "Generate XML with line count", "1.0")] 
    [Guid("202E7E8B-557E-46CB-8A1D-3024AD68F44A")]
    [ComVisible(true)]
    [ProvideObject(typeof(CountLines))]
    [CodeGeneratorRegistration(typeof(CountLines), "CountLines", "{FAE04EC1-301F-11D3-BF4B-00C04F79EFBC}", GeneratesDesignTimeSource = true)]
    public sealed class CountLines : IVsSingleFileGenerator
    {

        #region IVsSingleFileGenerator Members

        public int DefaultExtension(out string pbstrDefaultExtension)
        {
            pbstrDefaultExtension = ".xml";
            return pbstrDefaultExtension.Length;
        }

        public int Generate(string wszInputFilePath, string bstrInputFileContents,
          string wszDefaultNamespace, IntPtr[] rgbOutputFileContents,
          out uint pcbOutput, IVsGeneratorProgress pGenerateProgress)
        {
            try
            {
                int lineCount = bstrInputFileContents.Split('\n').Length;
                byte[] bytes = Encoding.UTF8.GetBytes("<LineCount>" + lineCount.ToString() + "</LineCount>" );
                int length = bytes.Length;
                rgbOutputFileContents[0] = Marshal.AllocCoTaskMem(length);
                Marshal.Copy(bytes, 0, rgbOutputFileContents[0], length);
                pcbOutput = (uint)length;
            }
            catch (Exception ex)
            {
                pcbOutput = 0;
            }
            return VSConstants.S_OK;
        }

        #endregion
    }
}

We need to provide an unique GUID for our extension such as one in above code [Guid("202E7E8B-557E-46CB-8A1D-3024AD68F44A")]. GUID can be created from VS2017 by selecting "Tools -> Create GUID". Select GUID format as Registry format. Note that GUID mentioned above code is without curly braces.

[ComVisible(true)] is required for COM Interops

[CodeGeneratorRegistration(typeof(CountLines), "CountLines", "{FAE04EC1-301F-11D3-BF4B-00C04F79EFBC}", GeneratesDesignTimeSource = true)] is a class attribute with code to register the tool. Params being GeneratorType, GeneratorName, and C# language GUID

You can also derive from "TemplatedCodeGenerator" which supports custom TextTemplate formatting, which may require some additional code implementation.

4. Add the registry entry code Create new class file with below code, name it CodeGeneratorRegistrationAttribute.cs

using System;
using System.Globalization;
using Microsoft.VisualStudio.Shell;

namespace CountLinesVSIX
{

    [AttributeUsage(AttributeTargets.Class, AllowMultiple = true, Inherited = true)]
    public sealed class CodeGeneratorRegistrationAttribute : RegistrationAttribute
    {
        private string _contextGuid;
        private Type _generatorType;
        private Guid _generatorGuid;
        private string _generatorName;
        private string _generatorRegKeyName;
        private bool _generatesDesignTimeSource = false;
        private bool _generatesSharedDesignTimeSource = false;

        public CodeGeneratorRegistrationAttribute(Type generatorType, string generatorName, string contextGuid)
        {
            if (generatorType == null)
                throw new ArgumentNullException("generatorType");
            if (generatorName == null)
                throw new ArgumentNullException("generatorName");
            if (contextGuid == null)
                throw new ArgumentNullException("contextGuid");

            _contextGuid = contextGuid;
            _generatorType = generatorType;
            _generatorName = generatorName;
            _generatorRegKeyName = generatorType.Name;
            _generatorGuid = generatorType.GUID;
        }

        /// <summary> 
        /// Get the generator Type 
        /// </summary> 
        public Type GeneratorType
        {
            get { return _generatorType; }
        }

        /// <summary> 
        /// Get the Guid representing the project type 
        /// </summary> 
        public string ContextGuid
        {
            get { return _contextGuid; }
        }

        /// <summary> 
        /// Get the Guid representing the generator type 
        /// </summary> 
        public Guid GeneratorGuid
        {
            get { return _generatorGuid; }
        }

        /// <summary> 
        /// Get or Set the GeneratesDesignTimeSource value 
        /// </summary> 
        public bool GeneratesDesignTimeSource
        {
            get { return _generatesDesignTimeSource; }
            set { _generatesDesignTimeSource = value; }
        }

        /// <summary> 
        /// Get or Set the GeneratesSharedDesignTimeSource value 
        /// </summary> 
        public bool GeneratesSharedDesignTimeSource
        {
            get { return _generatesSharedDesignTimeSource; }
            set { _generatesSharedDesignTimeSource = value; }
        }


        /// <summary> 
        /// Gets the Generator name  
        /// </summary> 
        public string GeneratorName
        {
            get { return _generatorName; }
        }

        /// <summary> 
        /// Gets the Generator reg key name under  
        /// </summary> 
        public string GeneratorRegKeyName
        {
            get { return _generatorRegKeyName; }
            set { _generatorRegKeyName = value; }
        }

        /// <summary> 
        /// Property that gets the generator base key name 
        /// </summary> 
        private string GeneratorRegKey
        {
            get { return string.Format(CultureInfo.InvariantCulture, @"Generators\{0}\{1}", ContextGuid, GeneratorRegKeyName); }
        }
        /// <summary> 
        ///     Called to register this attribute with the given context.  The context 
        ///     contains the location where the registration inforomation should be placed. 
        ///     It also contains other information such as the type being registered and path information. 
        /// </summary> 
        public override void Register(RegistrationContext context)
        {
            using (Key childKey = context.CreateKey(GeneratorRegKey))
            {
                childKey.SetValue(string.Empty, GeneratorName);
                childKey.SetValue("CLSID", GeneratorGuid.ToString("B"));

                if (GeneratesDesignTimeSource)
                    childKey.SetValue("GeneratesDesignTimeSource", 1);

                if (GeneratesSharedDesignTimeSource)
                    childKey.SetValue("GeneratesSharedDesignTimeSource", 1);

            }
        }

        /// <summary> 
        /// Unregister this file extension. 
        /// </summary> 
        /// <param name="context"></param> 
        public override void Unregister(RegistrationContext context)
        {
            context.RemoveKey(GeneratorRegKey);
        }
    }
}

Above code will make sure your entries are made to VS private registry

5. Compile and test the tool by running it in VS2017 You may add "Install targets" in "source.extension.vsixmanifest" to ensure different VS2017 editions are supported by your extesion. Run your tool in VS 2017 to test if it is working as expected. Once you Run the VSIX, the Visual Studio Experimental Instance will install the extension and register it in registry "C:\Users\xyz\AppData\Local\Microsoft\VisualStudio\15.0_xxExp\privateregistry.bin". You can see the installed extension by selecting "Tools -> Extensions and updates". To test the tool we will have to open a dummy project, select a file in solution explorer, go to its properties and update Custom Tool property to "CountLines". Once this is done VS will run the tool in background and generate the output, in our example it will generate a xml file under the selected file. Alternatively, once Custom Tool property is set, your can right click the file and select "Run Custom Tool"

6. Install the tool by double clicking the generated .VSIX file Once tested successfully, try installing the VSIX which can be found at location "projectName/bin/debug". Install the VSIX by double clicking the file, follow the installation steps. Now your tool will be available for use in VS2017. Using tool is similar, right click the file on which you want to run the custom tool and select "Run Custom Tool"

In case you want to uninstall the extention, go to "Tools -> Extensions and updates -> select your extension" and click uninstall. Note that tool will not get uninstalled until VS is closed. Once closeed you will get a popup window to uninstall, select "Modify" to uninstall.

Well, during research, I got the answer of this problem.

Solution:

1. We have to load the the .bin file (via load hiv).
2. Make the changes or edit the bin to register your tool.
3. Unload hive.

Step#1: Load Hive.

a) Open registry (regedit). Select node HKEY_LOCAL_MACHINE.

b) Go to | File -> Load Hive

c) Select the bin file which is available at -> %LocalAppData%\ Microsoft\ VisualStudio\ 15.0_'instance_id'\privateregistry.bin.

d) Provide a key name. This will create a new key entry in HKEY_LOCAL_MACHINE with your key name.

e) You can check the .bin file at HKEY_LOCAL_MACHINE\YourKeyName\Software\Microsoft\VisualStudio\

Step#2: Edit the bin: You can now register your custom tool by following the same way as you were doing for other VS versions. Actually the only problem was to get the VS2017 keys in to the global registry and that is solved using Step#1 above.

Step#3: Unload Hive.

a) select your key under HKEY_LOCAL_MACHINE.

b) Go to | File menu

c) Unload Hive.