Is is possible to have an automatic width or height on a photo in an image formula?

Doesn't look like it, using the IMAGE formula directly. A few options I would suggest.

• as part of your normal business process, re-scale the images upon upload. Obviously depending on the process and/or personnel involved, or if they are self-service uploads, this may not be feasible.
• do a nightly / hourly / whatever rescale of all new images. This would be a pretty easy task via an API integration and the excellent ImageMagick library.
• if changing your upload process to introduce a scaling step, or a nightly rescale, is undesirable, you could introduce a middle-man server to do the scaling on the fly. So a URL like 'http://posse.force.com/photos/servlet/servlet.FileDownload?file='+LEFT(SYSTEM_Photo_Id__c,15) becomes something more like 'http://myimagescaleserver/scale?sfid='+LEFT(SYSTEM_Photo_Id__c,15)+'&width=150'

For option 3, there is a discussion here with some interesting options, including the little tidbit that Google Docs supports dynamic image resizing which I was not aware of. Image Resizer also seems like a pretty good server option, and it's free.

EDIT: realized I left out a fourth option, which is a hack but it seems like a pretty common approach to this sort of issue. You use a sidebar component with some JavaScript (usually backed by jQuery) which looks at your page, finds the profile image (probably by combination of table location and URL), and then dynamically resizes it. It's a hack because Salesforce doesn't recommend DOM manipulation and doesn't guarantee that anything will remain the same from release to release, etc. but I've seen similar things done at several clients in the past. Here's an example of what this sort of page-manipulation component looks like.

EDIT 2: just found this thread which has what look to be pretty viable middle-man options that are already around and look pretty easy to use (and mostly free!), if you don't mind a third party processing your images.

Well actually I've found a way doing it!
Using -1 as the value for the value you want to be set equivalently to css "auto".
e.g:
IMAGE(url,"alt text", 100, -1)
It will actually make the browser ignore this css setting and give it the default "auto" value.