Converting points to polygons by group

You need to summarise after your group_by statement then your approach works perfectly fine. Directly from POINTS --> POLYGON, and keeping the crs (if there is one).

set.seed(999)
xy = data.frame(x=runif(24), y=runif(24))
xy$ID = rep(1:6, each = 4)
head(xy)

xys = st_as_sf(xy, coords=c("x","y"))

polys = xys %>% 
  dplyr::group_by(ID) %>% 
  dplyr::summarise() %>%
  st_cast("POLYGON")

plot(polys)

If you want the outer polygon you can add st_convex_hull()

polys = polys %>% 
  st_convex_hull()

plot(polys)

Make a reproducible example:

> set.seed(999)
> xy = data.frame(x=runif(24), y=runif(24))
> xy$ID = rep(c(1:6), rep(4,6))
> head(xy)
           x         y ID
1 0.38907138 0.8260703  1
2 0.58306072 0.8195141  1
3 0.09466569 0.5684927  1
4 0.85263123 0.6196068  1
5 0.78674676 0.8308805  2
6 0.11934226 0.4588336  2

Make the data frame into an sf data frame:

> xys = st_as_sf(xy, coords=c("x","y"))

Then aggregate by ID, combine the points and cast to POLYGON, turn the whole thing into an sf data frame. In a one-liner:

> polys = st_sf(
  aggregate(
    xys$geometry,
    list(xys$ID),
    function(g){
       st_cast(st_combine(g),"POLYGON")
    }
   ))

> polys
Simple feature collection with 6 features and 1 field
geometry type:  POLYGON
dimension:      XY
bbox:           xmin: 0.03014558 ymin: 0.01902308 xmax: 0.9875201 ymax: 0.8690149
epsg (SRID):    NA
proj4string:    NA
  Group.1                       geometry
1       1 POLYGON ((0.3890714 0.82607...
2       2 POLYGON ((0.7867468 0.83088...
3       3 POLYGON ((0.3907724 0.52724...
4       4 POLYGON ((0.03014558 0.8162...
5       5 POLYGON ((0.1665847 0.10961...
6       6 POLYGON ((0.9074913 0.35951...

Looks pretty awful but that's random data for you...