Can I adjust the lower limit of scale_color_brewer?

You cannot just set a lower limit. But you can use a palette with more colors than needed and map the brightest colors to unused levels. Below is an example with 9 levels:

ggplot(data.frame(x=1:6, y=10:15, w=letters[1:6]), aes(x, y, color=w)) + 
  geom_point() + theme_bw() + 
  scale_color_brewer(type="seq", palette=1, 
                     limits=c(LETTERS[1:3], letters[1:6]), 
                     breaks=letters[1:6])