How do I create a Rust HashMap where the value can be one of multiple types?

You should use an enum type as value in your HashMap. That enum needs to have a variant for each possible type (boolean, number, string, list, map...) and an associated value of appropriate type for each variant:

enum JsonValue<'a> {
    String(&'a str),
    VecOfString(Vec<String>),
    AnotherHashMap(HashMap<&'a str, u32>),
}

Fortunately, there already is an implementation of a JSON value type, part of the serde_json crate which is built on the serde crate.

Here is how your code would look if you used the serde_json crate:

extern crate serde_json;

use serde_json::{Value, Map, Number};

fn main() {
    let mut inner_map = Map::new();
    inner_map.insert("x".to_string(), Value::Number(Number::from(10u64)));
    inner_map.insert("y".to_string(), Value::Number(Number::from(20u64)));

    let mut map = Map::new();
    map.insert("key1".to_string(), Value::String("test".to_string()));
    map.insert(
        "key2".to_string(),
        Value::Array(vec![
            Value::String("a".to_string()),
            Value::String("b".to_string()),
        ]),
    );
    map.insert("key3".to_string(), Value::Object(inner_map));

    println!("{}", serde_json::to_string(&map).unwrap());
    // => {"key1":"test","key2":["a","b"],"key3":{"x":10,"y":20}}
}

Here is another approach that may be more palatable to you. The serde_json crate provides a way to construct serde_json::Value objects from JSON literals. Your example would look like this:

#[macro_use]
extern crate serde_json;

fn main() {
    let item = json!({
        "key1": "value",
        "key2": ["val", "val", "val"],
        "key3": { "keyX": 12 }
    });

    let response = serde_json::to_string(&item).unwrap();
}