Why is electric field inside a cavity of a non-conducting sphere not zero?

Gauss's Law only gives results for the integral over a whole, closed surface. This does not mean the electric field is zero. It simply means that all field lines entering the volume also exit it at some other point.

In the video the electric field lines are as shown in blue in the diagram below.

Let the edge of the cavity be the Gaussian surface which has no charge within it.

Consider small areas $\Delta A$ on either side of the cavity as shown in the diagram in red. Because of the symmetrical nature of the situation you can imagine that the electric flux entering the cavity through area $\Delta A$ is the same as the electric flux leaving the area $\Delta A$ on the right hand side.
This means that the net flux through those two surfaces is zero.

Doing the same for the whole Gaussian surface leads to the result that the net flux though the surface is zero commensurate with the fact that there is no charge within the surface.

Just to add to what Noah said, the law says that the total flux through the surface is zero. To make this point clear, imagine a region of uniform electric field $\vec{E} = E_0\hat{x}$ along the x axis. Consider a cube of unit area faces with two of its opposite faces normal to the field. From the RHS of Gauss law we know that the flux has to be zero as there are no charges enclosed by the cube. The LHS says

$\sum_{n=1}^6 \vec{E}\cdot \vec{A}_{n} = E_0\hat{x}\cdot\hat{x} + E_0\hat{x}\cdot-\hat{x} = 0$

Thus the flux is zero by explicit calculation as well. Remember that the electric field and area are vectors. So the relative directions are very important. If flux is zero all you can be sure of is that the extent of field lines entering is equal to that of exiting.