Create variable key/value pairs with argparse (python)

The first thing I'd try is use parse_known_args to handle other arguments, and handle the list of extras with my on routine. Adding the '--conf-' handling to argparse would be more work.

argv = '--conf-key-1 value1 --conf-key-2 value2'.split()
p = argparse.ArgumentParser()
args, extras = p.parse_known_args(argv)

def foo(astr):
    if astr.startswith('--conf-'):
        astr = astr[7:]
    astr = astr.replace('-','_')
    return astr

d = {foo(k):v for k,v in zip(extras[::2],extras[1::2])}
# {'key_1': 'value1', 'key_2': 'value2'}

The extras parsing could be more robust - making sure that there are proper pairs, rejecting badly formed keys, handling =.

Another approach would be to scan sys.argv for --conf- strings, and use those to construct add_argument statements.

keys = [k for k in argv if k.startswith('--conf-')]
p = argparse.ArgumentParser()
for k in keys:
    p.add_argument(k, dest=foo(k))
print vars(p.parse_args(argv))

---

If you would accept '--conf key1 value1 --conf key2 value2 ...' as the input, you could define

parser.add_argument('--conf', nargs=2, action='append')

which would produce:

namespace('conf': [['key1','value1'],['key2','value2']])

which could easily be turned into a dictionary. Or a custom Action could use setattr(namespace, values[0], values[1]) to enter the key/value pairs directly into the namespace.

I believe there have been SO question(s) about accepting '"key1:value" "key2:value2"' inputs.

I had a similar issue and found a very workable pattern that works well with argparse (here three key-pairs: foo, bar and baz:

mycommand par1 --set foo=hello bar="hello world" baz=5

1. Defining the optional, multivalued argument

The set argument must be defined so:

import argparse
parser = argparse.ArgumentParser(description="...")
...
parser.add_argument("--set",
                        metavar="KEY=VALUE",
                        nargs='+',
                        help="Set a number of key-value pairs "
                             "(do not put spaces before or after the = sign). "
                             "If a value contains spaces, you should define "
                             "it with double quotes: "
                             'foo="this is a sentence". Note that '
                             "values are always treated as strings.")
args = parser.parse_args()

The argument is optional and multivalued, with a minimum of one occurrence (nargs='+').

The result is a list of strings e.g. ["foo=hello", "bar=hello world", "baz=5"] in args.set, which we now need to parse (note how the shell has processed and removed the quotes!).

2. Parsing the result

For this we need 2 helper functions:

def parse_var(s):
    """
    Parse a key, value pair, separated by '='
    That's the reverse of ShellArgs.

    On the command line (argparse) a declaration will typically look like:
        foo=hello
    or
        foo="hello world"
    """
    items = s.split('=')
    key = items[0].strip() # we remove blanks around keys, as is logical
    if len(items) > 1:
        # rejoin the rest:
        value = '='.join(items[1:])
    return (key, value)


def parse_vars(items):
    """
    Parse a series of key-value pairs and return a dictionary
    """
    d = {}

    if items:
        for item in items:
            key, value = parse_var(item)
            d[key] = value
    return d

At this point it is very simple:

# parse the key-value pairs
values = parse_vars(args.set)

You now have a dictionary:

values = {'foo':'hello', 'bar':'hello world', 'baz':'5'}

Note how the values are always returned as strings.

This method is also documented as a git gist.