Proving maximum of dot product using derivatives

Firstly, it should be noted that, since the dot product is linear and not constantly zero (for non-zero $b$), it has no maximum. However, it does if we fix it to a sphere, and then it represents how large the dot-product can be without making $a$ larger. So, we optimize

Where are the extremes of $a\cdot b$ where $b$ is fixed and $a$ is on the unit sphere.

This can be done without reference to coordinate systems, which is always a plus. In particular, we can define $$f(a)=a\cdot b.$$ We define a sort of directional derivative on this- in particular, define the function $$a(t)=a + ta'$$ which we will use to consider an infinitesimal change in $a$ - that is, we are reducing this to a single variable problem where $a$ is changing with time. Basically, we want vectors $a$ such that $f(a(t))$ has a critical point at $0$, regardless of choice of $a'$ - this represents that, no matter how we perturb $a$ by another vector $a'$, the value of $f$ will be locally optimal.

However, since $a$ is supposed to lie on a sphere, we can't simply let $a'$ vary in any direction. Rather, it must travel on a tangent to the sphere, since otherwise we are increasing or decreasing the magnitude of $a$ to change its dot-product with $b$, which is cheating. This means, owing to the nature of spheres, that $a\cdot a'=0$. So, the question is, how can we make $t=0$ a critical point of $$f(a(t))=a\cdot b + ta'\cdot b$$ be a critical point for every $a'$?

Well, the derivative of the above with respect to $t$ is $$a'\cdot b$$ and we wish for this to be 0 for a critical point. Thus, $a'$ and $b$ must be perpendicular. So, we can ask, simply, "For what vectors $a$ does it hold that, for any $a'$ perpendicular to $a$, the vector $a'$ is also perpendicular to $b$?" - Or, in algebraic terms, "For what vectors $a$ does it hold that any $a\cdot a' = 0\Rightarrow b\cdot a' = 0$?"

Intuitively, we know that this can only hold if $a$ and $b$ are parallel, meaning that the only critical points of the function happen when $a$ and $b$ are parallel. If we want to do this algebraically, note that if $a$ and $b$ are parallel, then if $a'\cdot a=0$, so must $a'\cdot b = 0$, since $a$ and $b$ are multiples of one another. Otherwise, we could choose any tangent vector on the plane spanned by $a$ and $b$; explicitly, we could choose the component of $b$ perpendicular to $a$: $$a'=b-\frac{a\cdot b}{a\cdot a}a.$$ Clearly, $a\cdot a'=a\cdot b-\frac{a\cdot b}{a\cdot a}a\cdot a = 0$. However, we also need that $a'\cdot b=b\cdot b - \frac{(a\cdot b)^2}{a\cdot a}=0$ if $a$ is a critical point, which holds only when $$(a\cdot a)(b\cdot b)=(a\cdot b)^2$$ and noting that $b=ca+a'$ for scalar $c=\frac{a\cdot b}{a\cdot a}$, we can substitute: $$(a\cdot a)((ca+a')\cdot (ca+a'))=(a\cdot (ca+a'))^2$$ and expanding, noting that $a\cdot a'=0$: $$c^2(a\cdot a)^2 + (a\cdot a)(a'\cdot a')=c^2(a\cdot a)^2$$ and canceling terms and dividing out $(a\cdot a)$, which is non-zero, yields $$a'\cdot a' = 0$$ and because the dot product is positive-definite, this means $$a'=\overrightarrow{0}.$$ This only happens when $a$ and $b$ are parallel, since, in this case, $b=ca+a'=ca$. In every other case, therefore, there is some choice of $a'$ with non-zero derivative. So, thus, we verify that only parallel vectors $a$ to $b$ are critical points.

The alternate definition of dot product is $\textbf{a}\cdot\textbf b = ||\textbf a||\, ||\textbf b||\, \cos \theta$, where $\theta$ is the (smaller) angle between two vectors $\textbf a$ and $\textbf b$.

Now, $||\textbf a||,\, ||\textbf b||$ are constants, so maximum of dot product occurs when $\cos \theta$ is maximum, conditioned to $0 \le \theta \le \pi$.

Now, you can prove that maximum value of $\cos \theta$ occurs at $\theta = 0$ using simple trigonometric argument or using derivatives and maximum.