Remove-Item errors when run at end of TFS build

You should be able to delete everything if you go from the bottom up. Sort the results of Get-ChildItem by their full name in descending order before deleting the items:

Get-ChildItem -Path $directory -Force -Recurse |
  Sort-Object -Property FullName -Descending |
  Remove-Item -Recurse -Force