Why do people publish on arXiv instead of other places?
The major use case of arXiv is for disseminating manuscripts that you also publish in a journal or conference. By posting a preprint on arXiv, people can find your research, build on it, cite it, and give you feedback on it immediately, while at the same time the same work goes through the (sometimes slow) peer review process. Some of these papers will fall out of the peer review pipeline at some point, and only appear on arXiv, but that doesn't necessarily mean that they are less useful, important, or sound.
To give a specific example, in the last year or so I have been working in an area so new that most of the relevant research is still only available on arXiv. I expect these papers will eventually appear in journals too, but the slow peer review process means that the latest journal issues do not represent the state of the art for this particular topic. Conferences have a quicker peer review cycle and are more current, but most only publish relatively short papers.
arXiv is also useful for work that is in a format not suited for a conference or journal (e.g. a thesis), or for extended versions of papers that are published somewhere else.
I would caution against rules like "I should not cite materials on arXiv because they are not peer reviewed." Peer review does not guarantee sound, high quality research, nor is the inverse true. You should critically evaluate each paper, peer reviewed or not, on its individual merits. (Also see this related question on Math Overflow.)
Other reasons why authors post their papers on arXiv before or around the same time as submitting to a journal are:
To claim scientific priority: Delays at the journal end might lead to your research getting scooped. Publishing on arXiv lets the world know that you were the first one to come up with these results.
To get comments from peers: ArXiv allows others to comment on articles that are posted. This acts as an open peer review, where others in your field give feedback and suggest improvements that will help improve your paper and thereby increase your chances of acceptance by a journal. You can revise your paper based on these comments before submitting to a journal.
To give readers access to their article: Some articles are not widely read because readers don't have subscription to the article. Also, authors sometimes cannot afford the charges for publishing their article open access. In such cases, posting on arXiv ensures a wider readership and could even lead to more citations.
I've written an article about preprints recently which you might find interesting: The role of preprints in reasearch dissemination
Part of your question seems to be "Why do some researchers submit to arXiv while others don't"?
A point not yet mentioned: arXiv only covers a few specific subject areas. It's the very first line on the home page:
Open access to 1,175,314 e-prints in Physics, Mathematics, Computer Science, Quantitative Biology, Quantitative Finance and Statistics
Since engineering isn't on that list, that would be an excellent reason for most engineering researchers not to post their work to arXiv.
The "notable researchers in your field" who you mention as being prolific arXiv posters are, in some sense, not really in your field. Strogatz is a mathematician and Baez is a mathematical physicist. Both those fields are covered by arXiv, so it makes sense that they would use it.
Now it's entirely possible that although you think of your field as engineering, some of your papers might contain "enough" physics or mathematics to be on-topic for the corresponding sections of arXiv. But you might have to think about it first.
It should also be strongly emphasized that these people are not submitting to arXiv instead of peer-reviewed journals, but in addition to. This is perfectly acceptable by the standards of most journals in mathematics and physics. There are plenty of peer-reviewed outlets for their work and they're using them. Just look at their CVs.