What is the impact of deleting a TFS build agent working directory?

You can do it if your builds are not incremental (incremental get). If your builds always get all source files for every build it is OK to delete working directories. Build history and labels are not affected. Your build logs are in the drop location and it shouldn't be a problem either.

It is a problem for incremental (at least from incremental get) builds. These builds are getting only the latest changes from the TFS source control before every build, not the whole workspace as defined in the build definition's workspace.

Check your build definition's Process → CleanWorkspace settings. If it is set to All, it should not be a problem to delete the build directory.