Type signature in a where clause

You probably can't give go a type signature as-is.

The reason for this is that it makes use of polymorphic arguments bound by test. This means that, inside go, the identifier f has type (ao -> ai) for some specific, but unknown types ao and ai.

Type variables are generally only in scope for the single type signature where they're introduced, so when you give go its own type signature, the ao and ai there are new, polymorphic types, which of course causes a type error when trying to combine them with the similarly named, but fixed (and unknowable) types from test's signature.

The end result is that you can't write the type of go explicitly, which is not very satisfying. To solve this, GHC offers the ScopedTypeVariables extension, which allows bringing variables introduced in a type signature in scope inside the where clause of the function, among other things.

Note that if you only use the where clause to create an internal scope for definitions, and don't make use of identifiers bound by arguments to the outer function, you can write type signatures in the where clause just like you can for top level bindings. If you don't want to use GHC extensions, you can simply pass the parameters in redundantly. Something like this should work in that case:

test :: Monad m => (ao -> ai) -> Iteratee ai m b -> Iteratee ao m b
test f iter = go f $$ iter
  where go :: Monad m => (ao -> ai) -> Step ai m b -> Iteratee ao m b
        go f (Continue k) = continue $
             \stream -> go f $$ k (fmap f stream)
        go _ (Yield res _) = yield res EOF

You probably want that type, but with the ScopedTypeVariables extension enabled and with the variables from test's type signature in scope:

{-# LANGUAGE ScopedTypeVariables #-}
import Data.Enumerator
test :: forall m ai ao b. Monad m => (ao -> ai) -> Iteratee ai m b -> Iteratee ao m b
test f iter = go $$ iter
   where go :: Step ai m b -> Iteratee ao m b
         go (Continue k) = continue $
            \stream -> go $$ k (fmap f stream)
         go (Yield res _) = yield res EOF

See the GHC documentation for more information.