How to use pseudo selectors in material-ui?

The solution that worked for me is the following

export const useStyles = makeStyles(theme=>({
        header:{
            position: "relative!important",
            background: "linear-gradient(150deg,#7795f8 15%,#6772e5 70%,#555abf 94%)",
            margin: -50,
            padding: -50,
            height: 500,
        },

        span: props => ({
            padding:50,
            background: "rgba(255, 255, 255, .1)",
            borderRadius: "50%",
            position: "absolute",
            "&:nth-child(1)": {
                left: "-4%",
                bottom: "auto",
                background: "rgba(255, 255, 255, .1)"
            },
            "&:nth-child(2)":{
                right: "4%",
                top: "10%",
                background: "rgba(255, 255, 255, .1)"
            },
            "&:nth-child(3)":{
                top: 280,
                right: "5.66666%",
                background: "rgba(255, 255, 255, .3)"
            },
            "&:nth-child(4)":{
                top: 320,
                right: "7%",
                background: "rgba(255, 255, 255, .15)"
            },
            "&:nth-child(5)":{
                top: "38%",
                left: "1%",
                right: "auto",
                background: "rgba(255, 255, 255, .05)"
            },
            "&:nth-child(6)": {
                width: 200,
                height: 200,
                top: "44%",
                left: "10%",
                right: "auto",
                background: "rgba(255, 255, 255, .15)"
            },
            "&:nth-child(7)": {
                bottom: "50%",
                right: "36%",
                background: "rgba(255, 255, 255, .04)"
            },
            "&:nth-child(8)": {
                bottom: 70,
                right: "2%",
                background: "rgba(255, 255, 255, .2)"
            },
            "&:nth-child(9)": {
                bottom: "1%",
                right: "2%",
                background: "rgba(255, 255, 255, .1)"
            },
            "&:nth-child(10)": {
                bottom: "1%",
                left: "1%",
                right: "auto",
                background: "rgba(255, 255, 255, .05)"
            }

        }),

Just adding with Jorge Santos Neil, you don't necessarily need to use props. I am adding one of the example that is tested for a different scenario. A note is that, this needs to be included in "makeStyles" and then used as a "className". Simply putting it inside "styles={{}}" does not work.

Example:

const useStyles = makeStyles((theme) => ({
  paragraphWithWarningDiv: {
    margin: "32px 0px 24px",
    "& :nth-child(1)": {
      marginBottom: "100px"
    }
  }
}));

After a while fighting to have your code up and running I found what is wrong with your code.

Everything seems to be fine, the selector for rootListItem works right out of the box, the problem is that you can not use the pseudo-selector :hover on an element that has display: none. Instead you should be using opacity: 0 and opacity: 1, it will hide your ListItemSecondaryAction but at the same time it will allow you to hover. So, elements with display: none, doesn't technically display and thereby, you cannot hover them.

About your pseudo selector in global, you just wrote it wrongly. Using colon instead of dot after div and changing backgroundColor to 'yellow' instead of "'yellow',"

'li > div:nth-of-type(1)': {
        display: 'block !important',
        backgroundColor: 'yellow',
    },

I didn't know how does your TreeMenu look like as a component, so I just created a list with ul / li / div nodes.

const styles = {
root: {
    backgroundColor: 'white',
    '&:hover': {
        backgroundColor: '#99f',
    },
},
hoverEle: {
    visibility: 'hidden',
    '&:hover': {
        visibility: 'inherit',
    },
},
rootListItem: {
    backgroundColor: 'white',
    opacity: 0,
    '&:hover': {
        opacity: 1,
        backgroundColor: '#99f',
    },
},
'@global': {
    'li > div:nth-of-type(1)': {
        display: 'block !important',
        backgroundColor: "yellow",
    },
},
};

And:

<div>
    {treeNode.map(node => (
        <ListItem
            key={`${node.Type}|${node.NodeID}`}
            id={`${node.Type}|${node.NodeID}`}
            className={classes.root}
            button
            divider
            disableGutters={false}
            dense
            onClick={() => {}}
            title={''}
            onMouseOver={() => {}}
        >
            <ListItemText primary={node.NodeName} />
            <ListItemSecondaryAction classes={{ root: classes.rootListItem }}>
                <ul><li><div>Elem 1</div></li><li><div>Elem 2</div></li></ul>
            </ListItemSecondaryAction>
            <div className={classes.hoverEle}>
                <ul><li><div>Elem 1</div></li><li><div>Elem 2</div></li></ul>
            </div>
        </ListItem>
    ))}
</div>

*I am using treeNode that is an array for me and I removed the rest of the functions and TreeMenu.