Sort a nested list

Python 2, 114 101 78 73 62 bytes

k=lambda t:t*(t<[])or t.sort(key=k)or sum(z for z in t if[]>z)

I knew there was a better way to filter lists out.

Sorts a python list (and its sublists) in-place.

https://eval.in/540457 thanks @tac for letting me know in-place solutions are acceptable, and @xnor + @feersum for further optimizations!

Jelly, 13 bytes

fFSµ€Ụị߀µ¹<?

Try it online! or verify all test cases.

How it works

fFSµ€Ụị߀µ¹<?  Main link. Input: A (list)

   µ€          Apply the chain to the left to each item B in A.
 F             Flatten B.
f              Filter; intersect B with flattened B, yielding a list.
               This returns the numbers in B if B is a list, [B] if B is a number.
  S            Compute the sum of the resulting list.
     Ụ         Sort the indices of A according to the computed sums.
       ߀      Recursively apply the main link to each B in A.
      ị        Retrieve the items of the list (right) at those indices (left).
         µ     Convert the preceding chain into a single link.
            ?  If:
           <     A compared with itself is truthy:
                   Execute the link to the left.
          ¹      Else, apply the identity function to A.

Comparing (<) a number with itself yields 0 (falsy), but comparing a non-empty list with itself yields a list of 0's (truthy), so < can be used to distinguish numbers from lists.

Lua, 172 bytes

function p(a)if type(a)~="table"then return a end s(a)local t=0 for i,v in next,a do t=t+p(v)end return t end
function s(t)table.sort(t,function(a,b)return p(a)<p(b)end)end

The function s sorts a Lua table (a data structure that serves as a list among other things in Lua) in place according to the rules.