When to use /dev and /sys for userspace-kernel communication?
Very roughly:
/dev contains device nodes, which in earlier Unix systems was the only way to interact with the kernel. There are two types of these, block devices and character devices. The corresponding API is geared to something that will allow block-based I/O (some kind of disk) or character based I/O (e.g. a serial port).
/sys (and /proc) were added later, possibly inspired by the Plan 9 OS. They provide complete directory subtrees, and the file entries in these subtrees contain text that describes the internal state of the kernel module when read, or, when written, set the internal state.
So a typical application would be:
You want to write a kernel driver for some kind of storage device? Use a /dev node to access the device itself, and /sys (or /proc) entries to fine-tune how the storage gets accessed.
The coverage of /sys is in Chapter 14, "The Linux Device Model". It provides some more example code to play with. But I guess the book is a more code-driven approach, and it's useful to ask what the design principles look like.
When to use /dev and /sys for userspace-kernel communication?
The very first answer, is that you do not choose yourself. When you write a driver for a type of device, the kernel already has many examples of the same type of device. You use equivalent code patterns to the existing devices. (The most up to date documentation is the code itself. Many kernel interfaces do not have complete or up-to-date documentation, sorry!)
This is a major reason for writing device drivers. You are providing consistent interfaces that programs can use, without having to code different details for every different device.
This overrides any more general advice. If a subsystem of Linux (i.e. a type of device) uses a method that seems "wrong", but does so consistently, you should also be consistent and "wrong" when you write a driver for that subsystem.
/dev
/dev/ should be used for the data path. Except for network devices, which are covered in a different section of the book.
/dev/ special files should be used for standardized unix operations: read(), write(), poll()/select(), mmap(). It is also desirable when ioctl()s have been effectively standardized in unix or linux. These few system calls (and a few derived variants) are what is used in almost all cases. Start getting familiar with them :). ioctl() is an escape hatch here, that can be used to let your driver define any number of other operations.
/sys
sysfs writes should be used in a much smaller number of cases, for configuration parameters. They must be in plain text format, and should contain a single value only.[*] You won't want to have too many different sysfs files. You can start to see their limitations quite quickly.
We can also say that sysfs files should basically read a variable (which may or may not also be writable). I think reading them should not cause any hardware operation. I expect you were already thinking along these lines though.
An advantage of sysfs files is that they are very convenient to experiment with. You can easily use shell commands to list, read, and write them. This is also a danger: you must take care not to publish sysfs files in an experimental state. Once other people start relying on your sysfs file, kernel maintainers will be extremely unhappy if you want to change things in some way which breaks users scripts.
Traversing sysfs - the directory hierarchy and symlinks - can also be useful to see how devices are organized.
Also, conceptually watching for changes in sysfs is how programs are able to detect new devices (e.g. when plugged in). Technically these events are not delivered through the filesystem itself, but each event points to a specific sysfs directory.
The udev daemon listens for these events; normal programs should tend to be rely on udev/libudev if they need to. For example udev rules may read or write some files in the sysfs directory when the event is received, e.g. to change settings on newly discovered devices.
Looking at examples
Looking at existing examples is an extremely good idea. Although you probably shouldn't look at just one example, and assume it works the same everywhere else :-).
As dirkt says, access to block storage devices like /dev/sda is a very standardized use of /dev. A few parameters such as maximum physical IO size being exposed in /sys/class/block/sda/ - have a look in the queue subdirectory.
Many sysfs files are documented in the kernel tree, Documentation/ABI/*/sysfs-*. For example: https://www.kernel.org/doc/Documentation/ABI/testing/sysfs-block
Please note that "character device" isn't very specific. It's used for basically any device which is not either a block device, or a network device :-). If you ever had to implement some entirely new class of device, you would likely have to define an entirely new type of character device, and have the terrifying job of defining some new set of ioctl() operations.
You might start looking in /sys/class/, to see look at some other types of devices that are defined on your own system.
/sys/ also includes a list of /dev/ devices, but it is listed by device number instead of name. See ls -l /sys/dev/char/ and ls -l /sys/dev/block/. This helps explains how udev is able to manage /dev: every device that appears in /dev is listed as an object in /sys.[**]
[*] sysfs uevent files include multiple values, and are actually a core feature. but the uevent file cannot be used to change the values inside. So this is just to say: don't look at uevent and think my advice is wrong; you should not define a file like this yourself. The device driver can add lines to its uevent file; I think a good example would be if you have some very useful identifying properties, which a udev rule wants to test.
[**] Except /dev/pts/0 and so on are not listed in /sys, because /dev/pts/0 is actually implemented by a separate filesystem "devpts". Please ignore /dev/pts/0 as a very, very special case. There's an answer where I concluded this, but I really don't think it adds anything to what I just said. It's here: Does TTY always get used when we open any terminal?.