Unloading the Assembly loaded with Assembly.LoadFrom()

You can't unload assembly from the current AppDomain. But you can create new AppDomain, load assemblies into it, execute some code inside new AppDomain and then unload it. Check the following link: MSDN

Unfortunately you can not unload an assembly once it is loaded. But you can unload an AppDomain. What you can do is to create a new AppDomain (AppDomain.CreateDomain(...) ), load the assembly into this appdomain to work with it, and then unload the AppDomain when needed. When unloading the AppDomain, all assemblies that have been loaded will be unloaded. (See reference)

To call the garbage collector, you can use

GC.Collect(); // collects all unused memory
GC.WaitForPendingFinalizers(); // wait until GC has finished its work
GC.Collect();

GC calls the finalizers in a background thread, that's why you have to wait and call Collect() again to make sure you deleted everything.

Can you use another AppDomain?

AppDomain dom = AppDomain.CreateDomain("some");     
AssemblyName assemblyName = new AssemblyName();
assemblyName.CodeBase = pathToAssembly;
Assembly assembly = dom.Load(assemblyName);
Type [] types = assembly.GetTypes();
AppDomain.Unload(dom);

Instead of using LoadFrom() or LoadFile() you can use Load with File.ReadAllBytes(). With this it does not use the assembly file but will read it and use read data.

Your code will then look like

Assembly assem = Assembly.Load(File.ReadAllBytes(filePath));
sw = Stopwatch.StartNew();
var types1 = assem.GetTypes();
sw.Stop();
double time1 = sw.Elapsed.TotalMilliseconds;

From here We cannot unload the file unless all the domains contained by it are unloaded.

Hope this helps.:)