Is there a high-concept explanation for why "simplicial" leads to "homotopy-theoretic"?
I don't think I have a compelling answer to this question, but maybe some bits and pieces that will be helpful. One point is that all of the examples that you bring up are related to the first: simplicial sets can be used as a model for the homotopy theory of spaces. Pretty much any homotopy theory can be "described" in terms of the homotopy theory of spaces, just like any category can be "described" in terms of the category of sets (via the Yoneda embedding, for example). So if you've decided that "space" means simplicial set, then it's pretty natural to start thinking about presheaves of simplicial sets when you want to think about the homotopy theory of (pre)sheaves of spaces, as in motivic homotopy theory.
But that just brings us to the question "why use simplicial sets as a model for the homotopy theory of spaces"? It's certainly not the only model, and some alternatives have been listed in the other responses. Another alternative is more classical: the category of topological spaces can be used as a model for the homotopy theory of spaces. So, you might ask, why not develop the theory of the cotangent complex using topological commutative rings instead of simplicial commutative rings? There's no reason one couldn't do this; it's just less convenient than the alternative.
There are several things that make simplicial sets very convenient to work with.
1) The category of simplicial sets is very simple: it is described by presheaves on a category with not too many objects and not too many morphisms, so the data of a simplicial set is reasonably concrete and combinatorial. The category of topological spaces (say) is more complicated in comparison, due in part to pathologies in point-set topology which aren't really relevant to the study of homotopy theory.
2) The category of simplices is (op)-sifted. This is related to the concrete observation that the formation of geometric realizations of simplicial sets (or simplicial spaces) commutes with finite products. More generally it guarantees a nice connection between the homotopy theory of simplicial sets and the homotopy theory of bisimplicial sets, which is frequently very useful.
3) The Dold-Kan correspondence tells you that studying simplicial objects in an abelian category is equivalent to studying chain complexes in that abelian category (satisfying certain boundedness conditions). So if you're already convinced that chain complexes are a good way to do homological algebra, it's a short leap to deciding that simplicial objects are a good way to do homological algebra in nonabelian settings. This also tells you that when you "abelianize" a simplicial construction, you're going to get a chain complex (as in the story of the cotangent complex: Kahler differentials applied to a simplicial commutative ring yields a chain complex of abelian groups).
4) Simplicial objects arise very naturally in many situations. For example, if U is a comonad on a category C (arising, say, from a pair of adjoint functors), then applying iterates of U to an object of C gives a simplicial object of C. This sort of thing comes up often when you want to study resolutions. For example, let C be the category of abelian groups, and let U be the comonad U(G) = free group generated by the elements of G (associated to the adjunction {Groups} <-> {Sets} given by the forgetful functor,free functor). Then the simplicial object I just mentioned is the canonical resolution of any group by free groups. Since "resolutions" play an important role in homotopy theory, it's convenient to work with a model that plays nicely with the combinatorics of the category of simplices. (For example, if we apply the above procedure to a simplicial group, we would get a resolution which was a bisimplicial free group. We can then obtain a simplicial free group by passing to the diagonal (which is a reasonable thing to do by virtue of (2) )).
5) Simplicial sets are related to category theory: the nerve construction gives a fully faithful embedding from the category of small categories to the category of simplicial sets. Suppose you're interested in higher category theory, and you adopt the position that "space" = "higher-groupoid" = "higher category in which all morphisms are invertible". If you decide that you're going to model this notion of "space" via Kan complexes, then working with arbitrary simplicial sets gives you a setting where categories (via their nerves) and higher groupoids (as Kan complexes) both sit naturally. This observation is the starting point for the theory of quasi-categories.
All these arguments really say is that simplicial objects are nice/convenient things to work with. They don't really prove that there couldn't be something nicer/more convenient. For this I'd just offer a sociological argument. The definition of a simplicial set is pretty simple (see (1)), and if there was a simpler definition that worked as well, I suspect that we would be using it already.
There are several people here much more qualified to speak about that, so I shall just give you some pointers now. One of the questions Grothendieck tried to answer when writing "Pursuing Stacks" was — I don't know how he put it, though — "what are the properties of the simplicial category which make it so useful in homotopy theory?" That is where the theory of test categories stems from. As Georges Maltsiniotis puts it: "Le slogan de Grothendieck est que toute catégorie test est aussi “bonne” que celle des ensembles simpliciaux pour “faire de l’homotopie”." Which means "Grothendieck's motto is that any test category is as "good" as the category of simplicial sets to "make homotopy theory"." The theory was further developed by Denis-Charles Cisinski. The two books to read on this subject are:
Maltsiniotis's "La Théorie de l'homotopie de Grothendieck" ("Grothendieck's Homotopy Theory"), the introduction of which is remarkably well-written:
http://www.math.jussieu.fr/~maltsin/ps/prstnew.pdf
and
Cisinski's (augmented version of his) thesis "Les Préfaisceaux comme modèles des types d'homotopie" ("Presheaves as Models for Homotopy Types"): http://www.math.univ-toulouse.fr/~dcisinsk/ast.pdf
Both are available in SMF's collection Astérisque.
I shall give you more details if nobody else shows up to explain the yoga (I myself have but a smattering of it).
EDIT: Well, here are some details. You are asking: "What is so wonderful about Δ that allows a model structure (and one, moreover, Quillen equivalent to topological spaces) appear?" The shortest answer would be: "$\Delta$ is a test category". Let's try to see what it means. (I am feeling a bit guilty, for what follows is essentially a rephrasing, with the same notations, of some parts of Maltsiniotis's crystal-clear introduction to his book. I hope it will at least benefit those who cannot read French. Please note that Maltsiniotis's book is based on material written by Grothendieck in "Pursuing Stacks" almost thirty years ago.)
The starting point of the theory of test categorie is similar to your question. Namely, Grothendieck seeks to find all the couples $(M, W)$ where $M$ is a category and $W \subseteq Ar(M)$ such that the localized category $W^{-1}M$ be equivalent to the homotopy category $Hot$, and such that $W$ is natural in some sense (with respect to the structure of the underlying category). Given the difficulty to answer such general a question, Grothendieck then requires of $M$ to be a presheaf category on a small category $A$. Adding another slight condition on the small category $A$ (requiring that the "nerve functor" $i_{A}^{*} : Cat \to \widehat{A}$, $C \to (a \mapsto Hom_{Cat}(A/a, C))$, send weak equivalences to weak equivalences, where weak equivalences of $Cat$ are those functors the classical nerve of which are simplicial weak equivalences, and weak equivalences in the presheaf category $\widehat{A}$ are those morphisms sent to weak equivalences of $Cat$ by the functor $A/?$), he is lead to define the notion of weak test category. One of the properties of such a category $A$ is that the localization of its presheaf category by weak equivalences is equivalent to the homotopy category $Hot$. Of course, the simplicial category is a test category. But it is even better that that. It is a strict test category, which implies (by definition) for instance that cartesian product reflects the product of homotopy types. This theory shows, by the way, that the cubical category differs from the simplicial category in this respect: indeed, the cubical category is not a strict test category (but it is a test category, which of course lies somewhere between being weak test and being strict test). You might think that, since the cubical category is not a strict test category, strict test categories ought to be pretty scarce. In fact, there are plenty of them. For instance, every full subcategory of $Cat$ the objects of which are non-empty, and which is stable under finite products, and one object of which has at least two objects (possibly isomorphic) is a strict test category. There are results allowing one to check that a given category is a (weak, local, strict…) test category, which I will not state here. Just one example: Joyal's category $\Theta$ (related to infinity stuff) is a test category (this was proved by Cisinski/Maltsiniotis and Ara).
Actually, there is more than that in the theory. You can ask what are the formal properties of weak equivalences of $Cat$ that make the theory works so well. That is what Grothendieck answered by defining basic localizers. Indeed, what you need is just a class $W$ of functors between small categories such that: $W$ is weakly saturated (which means it contains identities, it satisfies a two out of three axiom, and if $i$ has a retraction such that $ir$ is in $W$, then $i$ (and thus $r$) is in $W$) ; if $A$ is a small category which has a terminal object, then $A \to e$ is in $W$ ($e$ stands for the point category) ; and $W$ satisfies the relative version of Quillen's Theorem A. That is all you need to develop the theory of test categories. Grothendieck then proceeds to rewrite all the theory with respect to an arbitrary basic localizer replacing $\mathcal{W}_{\infty}$, the classical weak equivalences of $Cat$.Therefore, for every basic localizer $W$, there are notions of $W$-weak test category, $W$-local test category, $W$-test category, $W$-strict test category and so on. Truncated homotopy types provide instances of basic localizers $\mathcal{W}_{n}$ for every $n \geq 0$, but there are many others.
And here is a theorem: for every basic localizer $W$, for every $W$-test category $A$, there is a closed model category structure on the category of presheaves on $A$, the weak equivalences of which are those defined above (so that, in particular, the localized category is equivalent to the localized category $W^{-1}Cat$) and the cofibrations of which are the monomorphisms. In fact, you have to make a slight set theoretic assumption for this result to hold (namely, that the basic localizer is accessible, that is, it is the smallest one containing some set of arrows). It was conjectured by Grothendieck and proved by Cisinski.
OK, now it might still be unclear as to what are the advantages of this theory. One of them is that you can work with other basic localizers than the classical one (the $W_{\infty}$ of above). Classical weak equivalences are related to Artin-Mazur equivalences in slice presheaves toposes, and these can be replaced, for instance, by any other topos morphisms defined by cohomological properties. (See the first paragraph of page 12 of Maltsiniotis's book, for instance.)
There are much more stuff in Grothendieck's homotopy theory, but I shall limit myself to that now.
By the way, there has been a very nice expository talk (in French) by Maltsiniotis on Grothendieck's 1980's work at IHES two years ago:
http://www.dailymotion.com/video/x8jsnw_colloque-grothendieck-georges-malts_tech.
EDIT: I just added some details and thought I could elaborate on two points of Jacob Lurie's answer as well in the language of Grothendieck's homotopy theory (which I of course do not claim to be better). When he states that the (op)-siftedness of the simplicial category guarantees "a nice connection between the homotopy theory of simplicial sets and the homotopy theory of bisimplicial sets", I guess the key result he is alluding to is the classical "bisimplicial lemma", which states that, if $f : X \to Y$ is a bisimplicial morphism such that $f_{n,.} : X_{n,.} \to Y_{n,.}$ is a simplicial weak equivalence for every $n \geq 0$, then $\delta^{\ast}(f):\delta^{\ast}X \to \delta^{\ast}Y$ is a simplicial weak equivalence. Here, $\delta : \Delta \to \Delta \times \Delta$ stands for the diagonal functor, and $\delta^{\ast}$ for the induced functor which send a bisimplicial set $X$ to the simplicial set $n \mapsto X_{n,n}$. I would like to point out that a similar result holds for every totally aspherical category, that is, a small category $A$ such that the functor $A \to e$ is a weak equivalence (which means that it belongs to the basic localizer we are considering) and such that (one among many equivalent properties) the diagonal functor $A \to A \times A$ is aspherical (which means that for every $(a_{1}, a_{2}) \in A \times A$, the comma category $\delta \downarrow (a_{1}, a_{2})$ is aspherical). For such a category $A$, whenever $f$ is a morphism in the category of presheaves $\widehat{A \times A}$ such that $f_{a,.}$ is a weak equivalence for all $a \in A$, then $\delta^{\ast}f$ is a weak equivalence (in the category of presheaves, see above). The simplicial category $\Delta$ is $W_{\infty}$-totally aspherical, a (non-trivial) fact from which one can deduce the "bisimplicial lemma". The siftedness has to do with the $W_{0}$-total asphericity, therefore I was puzzled as to how to deduce the "bisimplicial lemma" from it (one needs $W_{\infty}$ as basic localizer). It seems Jacob Lurie is tacitly taking the $(\infty,1)$-categorical viewpoint, which makes the two properties equivalent. (Thanks to Georges Maltsiniotis for poiting that to me.)
As to Dold-Kan correspondence, I asked Maltsiniotis if a similar result holds with other Grothendieck test categories, and the answer is that there is no such result in general, but there is already a conjecture in "Pursuing Stacks" regarding an analogous correspondence for any strict test category.
I am not sure many people wanted to read all that but I thought I would share what I knew since this stuff is not written down in any currently available text.
Dear Akhil,
I am not an expert in simplicial methods by any means, but I thought it might help to give an answer at a much lower level than the other answers and comments. What will come just reflects my own (somewhat meager!) attempts to understand some simplicial constructions. My point here will not primarily be to explain why simplicial constructions beat cubical or other constructions, but just to give some examples of how you can use them and what they mean. (Also, this answer is not very "high concept"; rather it is very very low concept! But hopefully it might still be useful.)
Firstly, you can think of a simplicial set as just a big bag of simplices with instrutions on how to glue them: you have a set of points, a set of intervals, a set of $2$-simplices, etc., and the boundary maps tell you how to glue. If you actually glue them according to the boundary maps, you get a space. So at first blush it is reasonable to think of simplicial sets as just a technical improvement on the pretty simple idea of simplicial complexes. Since reasonable spaces (from the point of view of algebraic topology, algebraic geometry, or smooth manifolds) can be triangulated, it is then not so surprising that one can capture a lot about topological spaces in this way.
Now let's suppose you have something a little more sophisticated instead, like a simplicial scheme: now you have a scheme of points, a scheme of 1-simplices, etc.
You can think of the scheme of points as just the basic scheme underlying the simplicial scheme; call it $X_0$. Now the scheme $X_1$ of $1$-simplices has boundary maps to the scheme of points. So you can think of $X_1$ as a kind of correspondence on $X_0$. For simplicity, imagine that the two boundary maps into $X_0$ are closed embeddings, so that you have two copies of $X_1$ sitting inside $X_0$. The fact that this is the scheme of $1$-simplices tells you that you are supposd to join all matching points in the two copies of $X_1$ by 1-simplices, and that you should think of these 1-simplices as varying continuously along the two copies of $X_1$. Now you glue in a family of 2-simplices indexed by $X_2$ in the same way, etc.
How do these arise: well a good example (taken from Deligne's Hodge III paper) is given by considering the resolution of singularities $\tilde{X}$ of a singular projective variety $X$. You can make the simplicial scheme $X_n:= \tilde{X}\times_X \cdots \times_X \tilde{X}$ ($n+1$ copies) with boundary maps given by projections and degeneracies given by partial diagonals. (Side note: This construction does show one advantage of simplicial constructions over various alternatives, namely, you can produce simplicial objects simply by taking iterated products; this provides a very convenient bridge between practice and theory, which might well be harder in other --- say cubical --- models.)
In particular $X_0$ is just $\tilde{X}$, so this simplicial scheme is $\tilde{X}$ with a bunch of simplices attached. If you think about how they are attached, you'll see that the $1$-simplices join all the points that lie in a single fibre under the projection $\tilde{X} \to X$. And then every triangle made up of $1$-simplices bounds a $2$-simplex, and so on.
So this simplicial scheme is a model for $X$ in which the parameterizing schemes are smooth (edit: as Bhargav notes in a comment below, to actually get smooth schemes beyond $X_0$, one typically has to do more, but let me suppress this here), but one has glued in $1$-simplices explaining how points should be identified in order to get back down to $\tilde{X}$ (and the higher simplices are added just to ensure that no extra topological strucure is being created by the $1$-simplices you have glued in).
This indicates that working simplicially, one has flexibility in making certian constructions, e.g. rather than forming a quotient directly (like actually passing from $\tilde{X}$ to $X$), we can instead form the quotient by gluing in paths between the points that are to be identified (and then adding higher order simplices as needed to kill of the loops, etc., that are accidentally introduced in the process of adding these paths).
Of course, one can then go further to make constructions that would not actually be possible in the non-simplicial world. E.g. suppose that you want to define relative etale cohomlogy $H^i(X,Y; \mathbb Q_{\ell})$, for a closed subscheme $Y$ of $X$. Topologically, this is the same as the (reduced) cohomology of the space obtained from $X$ by collapsing $Y$ to a point. You can't usually do this in the world of schemes, but you can do it simplicially, using some variant of the construction described above.
So, to get an answer to the question "why does "simplicial" make everything work so well?", I would suggest that you not only think about the formalism (model categories and so on), but also that you play around with various constructions of the type I've described, and related ones (e.g. the constructions of $BG$ and $EG$ for an algebraic group as simplicial schemes), and try to picture them physically as schemes with simplices being glued in. Try to think of other constructions from topology and see if you can figure out how you would make them in the world of schemes using simplicial schemes. Of course, your explicit constructions will match with the general formalism, but they should also help to illuminate it, and to provide intuition.