What is the function of square brackets around table keys in lua?

You cannot omit the brackets

> x = { 'a' = 1 }
stdin:1: '}' expected near '='

the correct code is

> x = { ['a'] = 1 }
> print(x['a'])
1

or

> x = { a = 1 }
> print(x['a'])
1

However, the second one has its limitations. What if you want to have a key called "-"?

> x = { - = 1 }
stdin:1: unexpected symbol near '='
> x = { '-' = 1 }
stdin:1: '}' expected near '='

again the correct way is to use brackets

> x = { ['-'] = 1 }
> print(x['-'])
1

Or you want to create a field of name which is contained in a variable called a?

> a = 'cat'
> x = { [a] = 1 } 
> print(x['cat'])
1

Brackets are used as a general form of key creation, they give you ability to put any hashable object as a key - not only strings.

It's simply the long form of specifying keys in a table. You can put any value between the [] (except nil. And floating-point NaNs). Whereas without them, you can only use identifiers.

For example:

tbl =
{
  key name = 5,
}

That's a compile error, since "key name" isn't an identifier (due to the space). This works:

tbl =
{
  ["key name"] = 5,
}

And this:

tbl =
{
  "key name" = 5,
}

Is also a compile error. If Lua sees a naked value like this, it thinks you're trying to add to the array part of the table. That is, it confuses it with:

tbl =
{
  "key name",
}

Which creates a 1-element array, with tbl[1] equal to "key name". By using [], the compiler can easily tell that you meant for something to be a key rather than the value of an array element.

The long form also lets you distinguish between:

local name = "a name";

tbl =
{
  ["name"] = 5,
  [name] = 7,
}

The second part means to evaluate the expression name, the result of which will be the key. So this table has the keys "name" and "a name".