Combining a class selector and an attribute selector with jQuery

I think you just need to remove the space. i.e.

$(".myclass[reference=12345]").css('border', '#000 solid 1px');

There is a fiddle here http://jsfiddle.net/xXEHY/

Combine them. Literally combine them; attach them together without any punctuation.

$('.myclass[reference="12345"]')

Your first selector looks for elements with the attribute value, contained in elements with the class.
The space is being interpreted as the descendant selector.

Your second selector, like you said, looks for elements with either the attribute value, or the class, or both.
The comma is being interpreted as the multiple selector operator — whatever that means (CSS selectors don't have a notion of "operators"; the comma is probably more accurately known as a delimiter).