Unable to open another excel file (when one Excel is opened by .net)

Use the IgnoreRemoteRequests property of the Excel application object:

xlApp.IgnoreRemoteRequests = True

This is the equivalent of checking the Excel UI option at
File | Options | Advanced | General | Ignore other applications that use Dynamic Data Exchange (DDE).
(See this related answer on SuperUser.)

I couldn't conveniently reproduce your scenario with a .NET application, but ran some tests by late binding an Excel.Application object from Word VBA and it worked as intended. I created a hidden Excel application, and was able to perform actions on it before and after opening files by double-clicking in File Explorer.

In my tests, the setting was not still toggled on the next time I opened Excel normally, but you might want to capture its value and restore it before quitting your application object, in case that behavior isn't universal.

Edit: This behavior has been around since at least Excel 2003, and I verified using Excel 2016 (32-bit).

In Excel 2013 or later, Excel switched to a single document interface: each workbook opens in its own window.

At least through 2016, the Visual Basic Editor has remained a multiple document interface, and you can easily see which files are open in an application session by looking at the Project Explorer pane in the VBE.

Looks like your problem is caused by the same instance of xlApp. What I would do is simple: I would initialize new instance of xlApp and then quite that instance of the application. This way it won't interfere with any other open instance of Excel. Here is how I use it in C#:

using Excel = Microsoft.Office.Interop.Excel;

Excel.Application xlApp = new Excel.Application();  // by default this is invisible

do whatever you have to do with the new instance of xlApp and then quite the application. Once you quite the application it won't be in your task manager.

xlApp.Application.Quit();

I have designed several applications where I had to process the same Excel files that most likely would've been open by those same users. This approach (opening new instance of excel in invisible mode and then quitting the application) never caused any problems for me.