Homotopy function complex for quasi-categories

Yes, you can compute the mapping spaces in ∞-categories by taking the biggest Kan subcomplex of the internal hom.

The trick is not to use the Joyal model structure, but instead the model structure on marked simplicial sets defined in Higher Topos Theory, proposition 3.1.3.7 (in the case $S=\Delta^0$). By proposition 3.1.4.1 the fibrant objects are exactly the quasicategories with the equivalences marked, by corollary 3.1.4.4 it is a simplicial model category and its mapping spaces are defined exactly as you describe and by theorem 3.1.5.1 this is Quillen equivalent to the Joyal model structure in a way that does the obvious thing on fibrant objects.

You certainly can take the maximal Kan complex of the quasicategory of maps between quasicategories (note that the mapping quasicategory already has all of the information you want -- you're just forgetting the non-invertible 1-morphisms in order to get at the underlying space).

The standard way of replacing $s\mathcal{S}et_{\mathrm{Joyal}}$ with a Quillen equivalent model category enriched over $s\mathcal{S}et_\mathrm{Quillen}$ (i.e. a simplicial model category) is to use "marked simplicial sets," described e.g. in Riehl's Categorical Homotopy Theory.

A marked simplicial set is a simplicial set along with a distinguished collection of 1-simplices (always including the degenerate simplices) which are regarded as "equivalences". We turn quasicategories into marked simplicial sets by marking the 1-simplices which were already equivalences in the quasicategory.

The reason why marked simplicial sets form a simplicial model category essentially follows from the fact that marked simplicial sets are a subcategory of the category of presheaves on a certain indexing category $\Delta^+$, which looks just like $\Delta$ except we factor $s_0:[1]\to[0]$ through an intermediary object $e$. 1-simplices in the image of $X_e\to X_1$ are understood to be the "equivalences".

Note that in this case, our cosimplicial frame is the standard one in simplicial sets, with each 1-simplex marked.

As you're probably aware, substituting the Joyal model structure for a Quillen equivalent simplicial model category gives us access to a number of technical tools and constructions; for example, we can use weighted limits to calculate homotopy limits as Riehl describes in her book.

I've left a lot of details hazy. If you have further questions, I'll try to clear them up, but you may just want to consult the book I've mentioned, which is available here: http://www.math.jhu.edu/~eriehl/cathtpy.pdf