Python: Accessing YAML values using "dot notation"

I ended up using python-box. This package provides multiple ways to read config files (yaml, csv, json, ...). And not only that, it allows you to pass dict or strings directly:

from box import Box
import yaml # Only required for different loaders

# Pass dict directly
movie_box = Box({ "Robin Hood: Men in Tights": { "imdb stars": 6.7, "length": 104 } })

# Load from yaml file
# Here it is also possible to use PyYAML arguments, 
# for example to specify different loaders e.g. SafeLoader or FullLoader
conf = Box.from_yaml(filename="./config.yaml", Loader=yaml.FullLoader) 

conf.mysql.user.pass

A lot more examples, are available in the Wiki.

The Simple

You could use reduce to extract the value from the config:

In [41]: config = {'asdf': {'asdf': {'qwer': 1}}}

In [42]: from functools import reduce
    ...: 
    ...: def get_config_value(key, cfg):
    ...:     return reduce(lambda c, k: c[k], key.split('.'), cfg)
    ...: 

In [43]: get_config_value('asdf.asdf.qwer', config)
Out[43]: 1

This solution is easy to maintain and has very few new edge cases, if your YAML uses a very limited subset of the language.

The Correct

Use a proper YAML parser and tools, such as in this answer.

---

The Convoluted

On a lighter note (not to be taken too seriously), you could create a wrapper that allows using attribute access:

In [47]: class DotConfig:
    ...:     
    ...:     def __init__(self, cfg):
    ...:         self._cfg = cfg
    ...:     def __getattr__(self, k):
    ...:         v = self._cfg[k]
    ...:         if isinstance(v, dict):
    ...:             return DotConfig(v)
    ...:         return v
    ...:     

In [48]: DotConfig(config).asdf.asdf.qwer
Out[48]: 1

Do note that this fails for keywords, such as "as", "pass", "if" and the like.

Finally, you could get really crazy (read: probably not a good idea) and customize dict to handle dotted string and tuple keys as a special case, with attribute access to items thrown in the mix (with its limitations):

In [58]: class DotDict(dict):
    ...:     
    ...:     # update, __setitem__ etc. omitted, but required if
    ...:     # one tries to set items using dot notation. Essentially
    ...:     # this is a read-only view.
    ...:
    ...:     def __getattr__(self, k):
    ...:         try:
    ...:             v = self[k]
    ...:         except KeyError:
    ...:             return super().__getattr__(k)
    ...:         if isinstance(v, dict):
    ...:             return DotDict(v)
    ...:         return v
    ...:
    ...:     def __getitem__(self, k):
    ...:         if isinstance(k, str) and '.' in k:
    ...:             k = k.split('.')
    ...:         if isinstance(k, (list, tuple)):
    ...:             return reduce(lambda d, kk: d[kk], k, self)
    ...:         return super().__getitem__(k)
    ...:
    ...:     def get(self, k, default=None):
    ...:         if isinstance(k, str) and '.' in k:
    ...:             try:
    ...:                 return self[k]
    ...:             except KeyError:
    ...:                 return default
    ...:         return super().get(k, default=default)
    ...:     

In [59]: dotconf = DotDict(config)

In [60]: dotconf['asdf.asdf.qwer']
Out[60]: 1

In [61]: dotconf['asdf', 'asdf', 'qwer']
Out[61]: 1

In [62]: dotconf.asdf.asdf.qwer
Out[62]: 1

In [63]: dotconf.get('asdf.asdf.qwer')
Out[63]: 1

In [64]: dotconf.get('asdf.asdf.asdf')

In [65]: dotconf.get('asdf.asdf.asdf', 'Nope')
Out[65]: 'Nope'

On the one hand your example takes the right approach by using get_config_value('mysql.user.pass', config) instead of solving the dotted access with attributes. I am not sure if you realised that on purpose you were not trying to do the more intuitive:

print(config.mysql.user.pass)

which you can't get to work, even when overloading __getattr__, as pass is a Python language element.

However your example describes only a very restricted subset of YAML files as it doesn't involve any sequence collections, nor any complex keys.

If you want to cover more than the tiny subset you can e.g. extend the powerful round-trip capable objects of ruamel.yaml:¹

import ruamel.yaml

def mapping_string_access(self, s, delimiter=None, key_delim=None):
    def p(v):
        try:
            v = int(v)
        except:
            pass
        return v
       # possible extend for primitives like float, datetime, booleans, etc.

    if delimiter is None:
        delimiter = '.'
    if key_delim is None:
        key_delim = ','
    try:
        key, rest = s.split(delimiter, 1)
    except ValueError:
        key, rest = s, None
    if key_delim in key:
        key = tuple((p(key) for key in key.split(key_delim)))
    else:
        key = p(key)
    if rest is None:
        return self[key]
    return self[key].string_access(rest, delimiter, key_delim)

ruamel.yaml.comments.CommentedMap.string_access = mapping_string_access


def sequence_string_access(self, s, delimiter=None, key_delim=None):
    if delimiter is None:
        delimiter = '.'
    try:
        key, rest = s.split(delimiter, 1)
    except ValueError:
        key, rest = s, None
    key = int(key)
    if rest is None:
        return self[key]
    return self[key].string_access(rest, delimiter, key_delim)

ruamel.yaml.comments.CommentedSeq.string_access = sequence_string_access

Once that is set up you are can run the following:

yaml_str = """\
mysql:
    user:
        pass: secret
    list: [a: 1, b: 2, c: 3]
    [2016, 9, 14]: some date
    42: some answer
"""

yaml = ruamel.yaml.YAML()
config = yaml.load(yaml_str)

def get_config_value(path, data, **kw):
    return data.string_access(path, **kw)

print(get_config_value('mysql.user.pass', config))
print(get_config_value('mysql:user:pass', config, delimiter=":"))
print(get_config_value('mysql.list.1.b', config))
print(get_config_value('mysql.2016,9,14', config))
print(config.string_access('mysql.42'))

giving:

secret
secret
2
some date
some answer

showing that with a bit more forethought and very little extra work you can have flexible dotted access to many to a vast range of YAML files, and not just those consisting of recursive mappings with string scalars as keys.

1. As shown you can directly call config.string_access(mysql.user.pass) instead of defining and using get_config_value()
2. this works with strings and integers as mapping keys, but can be easily extended to support other key types (boolean, date, date-time).

---

¹ _{This was done using ruamel.yaml a YAML 1.2 parser, of which I am the author.}

It's quite old question, but I came here hunting for the answer, but looking for more simpler solution. Finally, came up with my own solution using easydict library; installed using pip install easydict

  def yaml_load(fileName):
    import yaml
    from easydict import EasyDict as edict
    fc = None
    with open(fileName, 'r') as f:
      fc = edict(yaml.load(f))
      ## or use safe_load
      ## fc = edict(yaml.safe_load(f))

    return fc

Now, simply call yaml_load with the valid yaml filename:

config = yaml_load('./config.yml')

## assuming: config["mysql"]["user"]["pass"] is a valid key in config.yml
print("{}".format(config.mysql.user.pass))