Is it OK to use unknown HTML tags?

You should always approach HTML as it is defined in its respective specification. "Defining" new tags is a bit of an extreme approach. It might pass a browser check because it implements various failsafes, but there is no guarantee of this. You're entering the land of Undefined Behaviour, at best. Not to mention you will fail validation tests, but you seem to be aware of that.

If you wish to be more semantically expressive in your markup, you can use HTML5 which defines quite a bit of more descriptive tags for describing the structure of your page instead of generic divs which need to be appended ids or classes.

In the end, a short answer: No, it's bad practice, you shouldn't do it and there could be unforeseen problems later on in your development.

user1309389 had a very good answer, and I agree with the appeal to the spec. But I disagree with their conclusion, and I think they're wrong about made-up elements leading to "undefined behaviour". I want to propose an alternative way of thinking about it, rooted in how the spec and browsers actually handle made-up elements.

It's 2015, we're on the verge of the CustomElement spec being widely adopted, and polyfills are readily available. Now is a great time to be wondering about "making up your own elements". In the near future, you'll be able to create new elements with your own choice of tag and behaviour in a fully standard and supported way that everyone can love. But until this lands in all browsers, and until the majority of people are using supporting browsers, you can take advantage of the hard work of the Polymer or X-Tags projects to check out the future of custom elements in a nearly-standard and mostly-supported way that quite a few people can love. This is probably the "right thing" to do. But it doesn't answer your question, and frankly I find "just use X" or "don't do X" to be less helpful than "here's how the spec covers this, and here's what browsers do". So, here's what I love.

Against the heartfelt recommendation (and sometimes screaming) of much of the web dev community, I've been working with "made-up" elements for the past year, in all of my production projects, without a polyfill, and I've had no unsolvable issues (yet), and no complaints. How? By relying on the standard behaviour of HTMLUnknownElement, the part of the W3C spec that covers the case of "made-up" elements. If a browser encounters an unrecognized HTML element, there is a well-defined and unambiguous way that it should be handled, and HTMLUnknownElement defines that behaviour, and you can build on top of that. HTMLUnknownElement also has to be powerful and correct enough to "not break the web" when encountering all the tags that are now obsolete, like the <blink> tag. It's not recommended that you use HTMLUnknownElement, but in theory and in practice, there's absolutely no harm in doing so, if you know what you're doing.

So how does HTMLUnknownElement work? It is just an extension of the HTMLElement interface, which is the standard interface underlying all HTML elements. Unlike most other elements however, HTMLUnknownElement doesn't add any special behaviour — you get a raw element, unadorned with any special behaviour nor constraining rules about use. The HTMLDivElement interface works almost exactly the same way, extending HTMLElement and adding almost no additional behaviour. Put simply, making up your own element is almost identical to using a div or span.

What I like about "making-up" elements is the change of mindset. You should use or invent HTML elements based on several factors, ranging from how clear it makes the markup to read, to how the browser and screen readers and search engines parse your code, to how likely your code is to be "correct" by some objective measure. I sparingly use made-up elements, but I use in exactly the way Richard described, to make things more meaningful for the author of the HTML, not just meaningful to a computer service that extracts metadata. When used in a consistent way across a team, there can be a big benefit since made-up elements can concisely express what they're for.

I particularly like using made-up elements to indicate when I will be using JS to define extra behaviour for an element. For instance, if I have an element that will have children added/removed by JS, I will use a made-up element as a clue that this element is subject to special behaviour. By the same token, I don't use a made-up element when a standard element will suffice. You will see <dynamic-list> live happily next to <div> in my code.

Now, about those pesky validators. Yes, using made-up elements isn't "valid" in the sense that it won't pass a "validator". But many commonly used features, patterns, and systems of modern HTML and JS development fail all the W3C validators. The validators aren't the law — the spec is. And the law isn't binding — the implementations in all the browsers are. The utility of validators has been dimishing for years as the flexability of HTML has been increasing, and as browsers have shifted in their relationship to the spec. Validators are great for people who aren't comfortable with HTML and need guidance. But if you're comfortable taking your guidance from the spec and from browser implementations, there's no reason to worry about being flunked by the validator. Certainly, if you follow many of the guidelines offered by Google, Apple, Microsoft, etc, for implementing any experimental features, you'll be working outside the bounds of the validator. This is absolutely an okay thing to do, so long as you're doing it deliberately and you know enough about what you're doing.

Therefore, if you're going to make up your own elements and rely on HTMLUnknownElement, don't just treat it like a wild west. You need to follow a few simple rules.

1. You have to use a hyphen in the name of your tag. If you do, you are guaranteed to never collide with a future edition of the HTML spec. So never say <wrong>, always say <quite-right>.

2. Made-up elements can't be self-closing — you have to close them with a closing tag. You can't just say <wrong> or <still-wrong />, you have to say <totally-good></totally-good>.

3. You have to define a display property for your element in CSS, otherwise the rendering behaviour is undefined.

That's about it. If you do these things, you should be good to use made-up elements in IE9 and up, relying on the safety net of the HTMLUnknownElement. For me, the benefits far, far outweigh the costs, so I've been using this pattern heavily. I run a SaaS site catering to major industrial corporations, and I've had no trouble or complaints thus far. If you have to support older versions of IE, it's wise to stay far away from any "2015" technology or their crude approximations, and stay safely within the well-trodden parts of the spec.

So, in summary, the answer to your question is "yes, if you know what you're doing".

No. You will fail validation, you will get random issues cross browser and you WILL be eaten by said dinosaurs. CSS is the answer if you want your page to behave predictably.