Create polygons representing bounding boxes for subgroups using sf

One option is to use a nested dataframe using tidyr::nest and then purrr::map. I also used a wrapper function to simplify the map call

library(tidyverse)
box_sf <- pts %>% 
  group_by(group) %>%
  nest()

bbox_wrap <- function(x) st_as_sfc(st_bbox(x))
box_sf <- box_sf %>% 
  mutate(bbox = map(data, bbox_wrap))

This will give you a list of bounding boxes as a column of a dataframe. If you want to convert back to an sf object you can do this:

box_sf %>% 
  mutate(geometry = st_as_sfc(do.call(rbind, bbox))) %>% 
  select(-data, -bbox) %>% 
  st_as_sf()

Seems a little roundabout, I'm hoping to see a solution using group_by as you originally intended