How to rename a BTRFS subvolume?

Just mv it. That's the recommended way in the Ubuntu community documentation.

And to further clarify why that is the right way to do it, here is a quote from the btrfs sysadmin guide:

Snapshots

A snapshot is simply a subvolume that shares its data (and metadata) with some other subvolume, using btrfs's COW capabilities. Once a [writable] snapshot is made, there is no difference in status between the original subvolume, and the new snapshot subvolume. To roll back to a snapshot, unmount the modified original subvolume, and mount the snapshot in its place. At this point, the original subvolume may be deleted if wished. Since a snapshot is a subvolume, snapshots of snapshots are also possible.

there are few important things to note. The namings:

btrfs Subvolume - independent data container inside file system. It is represented as directory of the existing FS. If you create new subvolume, it will be empty, ready to use logical data block inside filesystem. Very convenient to use where data should be logically separated e.g. different VM's or different clients on different subvolumes. This allows very fast removal of all logical data block with just subvolume delete command.

btrfs Snapshot - a copy of existing subvolume with all its data at the moment os snapshot done. Can be used as operational backup for reverting settings or changes, e.g. make subvolume snapshot, make changes (VM or data), test if everything is ok, after some grace period remove snapshot. Important to note: snapshots can be read only (-r switch) and thus can be used as increment blocks of FS changes and possibly transferred to absolutely another BTRFS!

Current (2016-12-30) BTRFS limitations:

Copying or moving data between subvolumes, e.g. mv dir1/dataset1 dir_subvolume1/ produces all real io to copy data to another subvolume and in case of move, removing from original one. And very efficient copying of tons of data just by making references and thus using COW feature of BTRFS:

cp -a --reflink=always dir1/dataset1 dir_subvolume1/

and if needed:

rm -rf dir1/dataset1

Read only Subvolume snapshot can be renamed (moved with mv) at the existing directory level, but cannot be renamed/moved to different subdirectory level. e.g. mv /btrfs/subvol_snap1 /btrfs/.snaphots is not possible, produces not much explained error: mv: cannot move 'subvol_snap1' to '.snapshots/subvol_snap1': Read-only file system. To be able to move such a snapshot, you need to create new read-only snapshot of existing read-only snapshot to the new preferred location, and then remove old one:

btrfs sub snap -r /btrfs/subvol_snap1 /btrfs/.snaphots/subvol_snap1
btrfs sub del /btrfs/subvol_snap1

Just for more easy live:

btrfs sub list /btrfs

I hope this will save tons of time to all new btrfs fans :)

If you wanted to rename the root volume to a nested subvolume, you would need to snapshot it and then do a find $ROOT_VOL -xdev -delete to remove the previous contents of the root volume. The reverse manipulation (renaming a subvolume to the root volume) doesn't seem possible.