How to clean up the function closure (environment) when returning and saving it?

One way to fix the problem is to remove the large variable from the environment before returning.

computation <- function() 
{
    big_matrix <- matrix(rnorm(2000*2000), nrow = 2000, ncol = 2000)

    exp.value <- 4.5
    prior <- function (x) rep(exp.value, nrow(x))

    rm(big_matrix) ## remove variable

    list(
        some_info = 5.18,
        prior = prior
    )
}

The problem with your list2env method is that by default it points to the current environment as the parent environment for the new environment so you are capturing everything inside the function anyway. You can instead specify the global environment as the base environment

computation <- function() 
{
  big_matrix <- matrix(rnorm(2000*2000), nrow = 2000, ncol = 2000)

  exp.value <- 4.5
  prior <- function (x) rep(exp.value, nrow(x))
                                                              # explicit parent
  environment(prior) <- list2env(list(exp.value = exp.value), parent=globalenv()) 

  list(
    some_info = 5.18,
    prior = prior
  )
}

(If you specify emptyenv() then you won't be able to find built in functions like rep())