Change nested list of symbols to nested list of numbers

symbols = {e4, {e200, {e1, {e11, e23}}}};
Map[FromDigits@StringDrop[SymbolName[#], 1] &, symbols, {-1}]

{4, {200, {1, {11, 23}}}}

Map[FromDigits @ StringCases[SymbolName[#], NumberString] &, symbols, {-1}]

{4, {200, {1, {11, 23}}}}

Alternative versions, as suggested by Kuba,

Function[, FromDigits@StringDrop[SymbolName[#], 1], Listable]@symbols
Function[, FromDigits@StringCases[SymbolName[#], NumberString], Listable]@symbols

And, inspired by Jack LaVigne's comment, Replace and ReplaceAll:

Replace[symbols, s_Symbol:>   FromDigits[StringDrop[SymbolName[s], 1]],∞ ]
symbols /. Except[List, s_Symbol] :> FromDigits@StringDrop[SymbolName[s], 1] 

Finally, per J.M.'s comment, a variation on Coolwater's answer with combination of ToExpression and StringDelete:

 ToExpression @ StringDelete[LetterCharacter]@ToString@symbols

Alternatively:

symbols = {e4, {e200, {e1, {e11, e23}}}};
ToExpression[StringReplace[ToString[symbols], "e" -> ""]]

{4, {200, {1, {11, 23}}}}