A simple number system

Haskell, 357 306 277 251 228 224 188 185 180 bytes

A token-based parser with an explicit stack. (%) takes a token stack and a character and either pushes (opcode,defaultnumber) or (0,number) for ({[<, or pops topmost numbers and one opcode and pushes the answer for )}]>. Opcodes are encoded by a ascii enumeration hack.

Kudos to @ChristianSievers for his great answer that I borrowed some ideas from.

One-liner!

s%c|elem c"([<{",g<-div(fromEnum c)25=(g,[0,0,1,-1,1]!!g):s|(a,(o,n):b)<-span((==0).fst)s=(0,[foldr1(flip$[(+),quot,(-),(*)]!!(o-1))$snd<$>a,n]!!(0^length a)):b
snd.head.foldl(%)[]

Now with less error-handling! Usage:

*Main> map (snd.head.foldl(%)[]) ["()","(()())","([][])","({}<>)","({}[])","[]","[[][]]","[()<>]","{()}","{([]<>)}"]
[0,0,-2,2,0,-1,0,-1,0,0]

Thanks @ChristianSievers for saving 14+3 bytes!

Thanks @Zgarb for saving some+4 bytes!

PEG.js (ES6), 132 bytes

x=a:[([{<]b:x*[)\]}>]{var x='([<'.indexOf(a)
b.length^1||b.push(0)
return~~eval(b.length?b.join(('+-/'[x]||'*')+' '):~-('10'[x]||2))}

Should be fixed now.

Explanation

More readable:

x=a:[([{<]
  b:x*
  [)\]}>]
{
  var x='([<'.indexOf(a)
  b.length^1||b.push(0)
  return ~~eval(
    b.length?
      b.join(('+-/'[x]||'*')+' ')
    :~-('10'[x]||2))
  )
}

PEG.js is an extended version of Javascript specifically made for parsing. It's VERY strict, which is why I had to use var. In addition, there seems to be a bug with brackets inside strings, which bloated the code significantly.

To start, we define a rule x that matches any bracket a that may or may not contain multiple expressions that matches rule x.

For each match to rule x, we push a 0 to the array of the inner match b if b's length is 1.

If b's length > 0, then we find the index of a in ([< and get a character from +-/ using that index. If the result is undefined (meaning that a was {), then we turn the result into *. Finally, we tack on a space and join b with the result.

If b's length = 0, then we find the index of a in ([< and get a character from 10 using that index. If the result is undefined (meaning that a was { or <), then we turn the result into 2. Finally, we simply decrement.

At the end, we can just evaluate the expression and floor the result.

Python 2, 292 265 248 235 223 206 204 bytes

r=reduce
s=input()
for n in')]}>':s=s.replace(n,'),')
for a in'(*x:sum(x)','[a=-1,*x:a-sum(x)','{*x:r(int.__mul__,x,1)','<a=1,*x:r(int.__div__,x,a)':s=s.replace(a[0],'(lambda %s)('%a[1:])
print eval(s)[0]

Replaces all brackets with a lambda that does what the bracket does, then evaluates the resulting Python code. Requires its input surrounded by quotes, like '[<><>([]{})]'.

This program stores the type of bracket as the first character in each string in the for, and everything after the keyword lambda as the rest. It then uses the first character to substitute; the rest of it is combined into a lambda like (lambda*x:sum(x))().

Try it on Ideone!