Molecules to Atoms

CJam, 59 57 bytes

q{:Ci32/")("C#-"[ ] aC~* Ca C+"S/=~}%`La`-S%$e`{~": "@N}/

Try it online in the CJam interpreter.

How it works

q             e# Read all input from STDIN.
{             e# For each character:
  :Ci         e#   Save it in C and cast to integer.
  32/         e#   Divide the code point by 32. This pushes
              e#   2 for uppercase, 3 for lowercase and 1 for non-letters.
  ")("C#      e#   Find the index of C in that string. (-1 if not found.)
  -           e#   Subtract. This pushes 0 for (, 1 for ), 2 for digits,
              e#   3 for uppercase letters and 4 for lowercase letters.

 "[ ] aC~* Ca C+"

 S/           e#   Split it at spaces into ["[" "]" "aC~*" "Ca" "C+"].
 =~           e#   Select and evaluate the corresponding chunk.
              e#     (   : [    : Begin an array.
              e#     )   : ]    : End an array.
              e#     0-9 : aC~* : Wrap the top of the stack into an array
              e#                  and repeat that array eval(C) times.
              e#     A-Z : Ca   : Push "C".
              e#     a-z : C+   : Append C to the string on top of the stack.
}%            e#
`             e# Push a string representation of the resulting array.
              e# For input (Au(CH)2)2, this pushes the string
              e# [[["Au" [["C" "H"] ["C" "H"]]] ["Au" [["C" "H"].["C" "H"]]]]]
La`           e# Push the string [""].
-             e# Remove square brackets and double quotes from the first string.
S%            e# Split the result at runs of spaces.
$e`           e# Sort and perform run-length encoding.
{             e# For each pair [run-length string]:
  ~           e#   Dump both on the stack.
  ": "        e#   Push that string.
  @N          e#   Rotate the run-length on top and push a linefeed.
}/            e#

Python3, 157 154 bytes

import re
s=re.sub
f=s("[()',]",'',str(eval(s(',?(\d+)',r'*\1,',s('([A-Z][a-z]*)',r'("\1",),',input()))))).split()
for c in set(f):print(c+":",f.count(c))

Only supports input using regular brackets.

Before creating the golfed solution using eval above I created this reference solution, which I found very elegant:

import re, collections

parts = filter(bool, re.split('([A-Z][a-z]*|\(|\))', input()))
stack = [[]]
for part in parts:
    if part == '(':
        stack.append([])
    elif part == ')':
        stack[-2].append(stack.pop())
    elif part.isdigit():
        stack[-1].append(int(part) * stack[-1].pop())
    else:
        stack[-1].append([part])

count = collections.Counter()
while stack:
    if isinstance(stack[-1], list):
        stack.extend(stack.pop())
    else:
        count[stack.pop()] += 1

for e, i in count.items():
    print("{}: {}".format(e, i))

Pyth, 66 65 bytes

VrSc-`v::z"([A-Z][a-z]*)""('\\1',),"",?(\d+)""*\\1,"`(k))8j": "_N

Port of my Python answer. Only supports input using regular brackets.