CREATE DATABASE or ALTER DATABASE failed because the resulting cumulative database size would exceed your licensed limit of 10240 MB per database

The instance name for SQL Server Express is by default SQLEXPRESS - but it can be anything you choose during installation. If you install SQL Server Express as the default (un-named) instance, then you get MSSQLSERVER as the pseudo instance name for SQL Server Express.

Hence, you really cannot rely on the instance name to judge whether your SQL Server is the Express edition or not. You need to use

SELECT @@Version

to get that information.

I had the same error in my Express Edition as the official documentatios says, to fix it without shrink the DB I upgraded my version, from Express to Developer edition. Go to SQL Server Installation Center->Maintenance->Edition upgrade.