$(\infty,2)$-Categorical Analogue of the Local Nature of Equivalences
TL DR: That is not enough. If you let $\psi_i:G(i)\to F(i)$ be the left adjoint of $\phi_i$ you also need the condition that for every $f:i\to j$ the canonical morphism $$\psi_jG(f)\to F(f)\psi_i$$ adjoint to $G(f)\to G(f)\phi_i\psi_i\cong \phi_jF(f)\psi_i$, is an equivalence.
The key word to remember here is relative adjunctions. If we unstraighten your functors $F$ and $G$, we end up with two cartesian fibrations $E\to I^{op}$ and $E'\to I^{op}$ and a map of cartesian fibrations $\phi:E\to E'$ such that for each $i\in I$ the functor $\phi_i:E_i\to E'_i$ is a right adjoint. Then, by theorem 7.3.2.6 in Higher Algebra, there is a relative left adjoint $\psi$. The result you are after would follow if and only if $\psi$ were a map of cartesian fibrations (i.e. iff it sends cartesian edges to cartesian edges). Unwrapping the various definitions, this is exactly the condition I wrote above.
To see a counterexample when the condition is not satisfied, pick two categories $C_0,C_1$ with an initial object and a functor $f:C_0\to C_1$ that does not preserve the initial object (e.g. let $C_0$ be finite pointed sets, $C_1$ be finite sets and $f$ the functor that forgets the pointing). This assembles to a functor $F:\Delta^1\to \mathrm{Cat}_\infty$.
Now let $G:\Delta^1\to \mathrm{Cat}_\infty$ be the constant functor at $*$. There is an obvious natural transformation $\phi:F\to G$ and $\phi_0$ and $\phi_1$ have both left adjoints $\psi_0$ and $\psi_1$ (the inclusion of the initial object). However these do not assemble into a natural transformation $\psi:G\to F$ because $\psi_1$ is not naturally equivalent to $f\psi_0$.
FWIW, there is nothing "truly $\infty$" about this question; the same question can be asked for 1-categories and the answer is the same. In that case it fits into two abstract frameworks:
1: doctrinal adjunction. For a 2-monad $T$ on a 2-category $K$ and a pseudo $T$-algebra map $(g,\bar{g}):A\to B$ whose underlying morphism $g:A\to B$ in $K$ has a left adjoint $f:B\to A$, there is a canonical induced structure of an oplax $T$-algebra map on $f$ (the mate of the pseudo $T$-morphism structure on $g$), and the whole adjunction lifts to the 2-category of $T$-algebras (and pseudo morphisms) if and only if this oplax structure is in fact a pseudo structure.
Now there is a 2-monad $T$ on $\mathrm{Cat}^{\mathrm{ob}(I)}$ whose algebras are functors $I\to \mathrm{Cat}$. The pseudo $T$-morphisms are pseudo natural transformations, and the lax/oplax $T$-morphisms are lax/oplax natural transformations. When doctrinal adjunction is unraveled in the case of this 2-monad, it amounts to exactly the condition mentioned by Denis.
2: property-like structure. You mentioned that the space of adjoints to a given morphism is contractible (if nonempty), i.e. that "having an adjoint" appears to be a mere property of a morphism (rather than structure on it). From this perspective it may be surprising that the adjoints don't fit together. In fact, though, having an adjoint is something in between a "property" and "structure" called a property-like structure: a structure that is unique on objects when it exists, but is not necessarily preserved by morphisms.
One of the simplest examples of property-like structure is "having an identity element" for a semigroup: a semigroup can have at most one identity element, but a semigroup homomorphism need not preserve identities. A more well-known example is "having colimits" for a category: they are unique (up to unique isomorphism) when they exist, but not every functor preserves them (even up to isomorphism). The latter is an example of a special kind of property-like structure called lax-idempotent, in that it is automatically preserved laxly by every morphism (in this case, the comparison map $\mathrm{colim} \circ F \to F \circ \mathrm{colim}$). More precisely, a 2-monad $T$ on a 2-category $K$ is lax-idempotent if every $K$-morphism between $T$-algebras has a unique structure of lax $T$-morphism.
Now there is a 2-monad $T$ on the 2-category $\mathrm{Cat}^{\mathbf{2}}$ whose algebras are functors equipped with a left adjoint, and this 2-monad is lax-idempotent: the unique lax $T$-morphism structure on a commutative square is, again, its mate under the adjunctions. Now your given natural transformation is a functor $I\to \mathrm{Cat}^{\mathbf{2}}$, and the fact that it has adjunctions "pointwise" means that this functor lifts "objectwise" to $T$-algebras. Lax-idempotence of $T$ therefore implies that the functor lifts to the 2-category of $T$-algebras and lax $T$-morphisms; hence it lifts to pseudo $T$-morphisms if and only if all these mates are isomorphisms.
As far as know, neither of these abstract contexts has yet been worked out in the $\infty$ case. But someone should!