What is meant with EPSG:4326 projection

You've got a good grasp of it. EPSG 4326 (i.e. WGS 84) is not a projection. But if you don't associate a projection to this geographic coordinate system, and naively render the coordinates as x/y coordinates on a grid, you do get something that is sort of a projection: the pseudo plate carée (equirectangular) projection. (This is not the same as an actual plate carée projection with a WGS 84 GCS, as this would have units in metres, not degrees). This is a rather lazy habit that everyone is in, but it's also quite convenient. When people refer to rendering data "in EPSG 4326" without further information, this is 99% of the time what they mean (and 70% of the time that is the limit of their conceptual understanding of the matter, anyway... coordinate systems and projections are hard).

You can be lazy like this if you just want to draw things on a map and look at them, or for passing data between applications, but it is not particularly useful for lots of important things (accurate representations of shapes, measuring distances, etc.), at which point you would need to start working with non-pseudo projections.

This openlayers example is a bit confusing.

Most webmaps display in "Web Mercator" ie. 'Google Maps-style' projections - EPSG / SRID: 3857

OpenStreetMap data is collected, stored, and distributed in 'unprojected lat/lon' aka WGS84 - EPSG / SRID: 4326.

However, local data for administrative organizations are projected to State Plane zones or UTM zones (traditionally by "GIS" practices, though we store our data in WGS84 and project using PostGIS st_transform() when we need units in feet).

Openlayers has the ability to project your webmap to any coordinate system, and I believe this example is showing how to do that given any of the data storage coordinate system issues noted above.