Gnome 3 (Gnome Shell) JavaScript/CSS engine

For JavaScript the engine GJS (based on SpiderMonkey). For theming, it is used a sort of superset of CSS2. You might be interested in reading the documentation of GtkCssProvider and the migration information.

On the other hand, there are some CSS3-like operations available, such as radial and linear gradients. You might want to take a look the article Styling GTK+ with CSS to get some ideas. The article was written by the developer who wrote GtkCssProvider.

The JavaScript engine is GJS. The CSS is, I believe, internal - in GTK 3, all theming is done using CSS.