Is there any implementation to Remove by Key and get the Value at the same time?

The ConcurrentDictionary has a TryRemove method that does this. It works just like TryGet but it also removes the element.

The University of Copenehagen's Generic Collection Library has a Dictionary.Remove() method that appears to do what you want:

bool Remove(K k, out V v)

Returns true if the dictionary contains an entry whose key equals k and if so removes that entry and assigns the associated value to v; otherwise returns false and assigns the default value for T to v.

I've not used this library myself, but I've seen it recommended a few times here on Stack Overflow. It's free to use commercially, subject to this MIT-style license.

Dictionary<TKey, TValue>.TryGetValue and Dictionary<TKey, TValue>.Remove methods are both O(1) operations, so I don't think you should be concerned about performance here.