Topology of matrices
Compactness
(a) This is true. First, the set of these matrices is bounded by $\sqrt{n}$, that is $||A|| \leq \sqrt{n}$. One way to see this is to first show that for orthogonal matrices, $||A||_2 = 1$, and then show that $||A|| \leq \sqrt{n}||A||_2$. To show that it is closed is not that difficult either. Let $\{A^k\}_{k\in\mathbb{N}}$ be a sequence of orthogonal matrices with limit $A$. Due to the nature of the norm $A$ is the component-wise limit, that is $A_{i,j} = \lim_{k\to\infty} A^k_{i,j}$. It should now be easy to see that $A$ is also orthogonal. Another option for showing that the set is closed is to observe (or show) that the map $A\to A^T A$ is continuous and the set of orthogonal matrices is the preimage of $I$.
edit: Actually, showing that $||A||\leq \sqrt{n}$ is even easier, namely, almost by definition we have, $||A|| = \sqrt{\mathrm{tr}\left(A^T A\right)}$, but $A^T A = I$, so $||A|| = \sqrt{\mathrm{tr}\left(I\right)} = \sqrt{n}$.
(b) This is false as this set is unbounded. For example, take the matrix with $a_{1,1} = M$, $a_{2,2} = \frac{1}{M}$, $a_{k,k}=1$ and zeros of of diagonal. The norm of this matrix is greater than $M$, so we get a family of matrices whose determinant is $1$ and whose norm cannot be bounded.
(c) The same argument as the previous case.
- Connectedness
(a) This set is not connected. For example, the set of matrices with determinant $1$ is a closed set as is the set of matrices with determinant $-1$ and they are disjoint and they cover the set of orthogonal matrices.
(b) If $A$ and $B$ are matrices with trace $1$, then $f(t) = t A + (1-t) B$ is a path from $A$ to $B$ and it is very easy to see that for each $t$, $f(t)$ is a matrix with trace $1$. The set is thus path connected and therefore connected.
(c) Similar argument as for the previous case works. For $A$ and $B$ positive definite, for each $t\in [0,1]$, $f(t)$ is a symmetric positive definite matrix, which is easy to see straight from the definition. So this space is also path connected.
Let $E:=\Bbb R^{n^2}$.
(a) We have $O_n:=F^{-1}(\{0\})$ where $F\colon E\to E$ is defined as $F(A):=A^tA-I$. We have to show that $F$ is continuous, but it follows from the fact that $M\colon E^2\to E$, $M(A,B)=A\cdot B$ and $Ad\colon E\to E$, $Ad(M):=M^t$ are continuous. We have $\lVert Ax\rVert=\lVert x\rVert$ for all $x$ where $\lVert\cdot \rVert$ is the Euclidian norm, which proves boundedness.
(b) The map $\det\colon E\to \Bbb R$ is continuous, which gives closeness. But it's not bounded, as $\pmatrix{a&0&0\\0&a^{—1}&0\\ 0&0&I_{n-2}}$ shows.
(c) The sequence $\{\frac 1k I_n\}$ of invertible matrices converges to the null matrix, which is not invertible.
(a) We can write $$O_n=(O_n\cap \det^{—1}(-\infty,0))\sqcup (O_n\cap \det^{—1}(0,+\infty)),$$ hence as a disjoint union of two open sets for the induced topology.
(b) The set of such matrices is arcwise connected.
(c) If $A$ and $B$ are symmetric positive definite, then so is $tA+(1-t)B$ for $0\leq t\leq 1$.