Is there a package for Bayesian Linear Regression?
There is a lot around:
See Bayesian Statistics and Econometrics using Mathematica
http://library.wolfram.com/search/?query=Bayesian&collection=library&x=0&y=0
Google mathematica Bayesian
I am a bit late, but here is a short paper of mine with Wolfram Language code. It covers Bayesian regression as well as model selection. Take notice of the books in the references. Excellent for self study. http://djafari.free.fr/MaxEnt2014/papers/04_paper.pdf
For some reason this question didn't come up when I tried searching for it before, so I ended up creating a new question for it and answering it myself.
How to fit a linear model in the Bayesian way in Mathematica?
In this answer I provide a link to my Bayesian inference repository which has a function that's the Bayesian equivalent of Mathematica's LinearModelFit
.