SSRS Detail members can only contain static inner members
If these are being returned from the same SP, then you should only have one Detail group (and one tablix).
The detail group (on your outer tablix) will be repeated once for each row returned by your dataset. A tablix contained within that row for the same dataset doesn't make sense: what would it repeat?
It sounds like you need to combine your two tablixes into one, not nest one in the other.
The easiest approach is probably to add a group to the parent tablix which will take the place of the former detail group. It will probably group by whatever field you joined to your old subreport dataset (or something similar.)
Then put your old subreport fields in the detail rows.