What is a subnet mask, and the difference between a subnet mask of 255.255.255.0 and 255.0.0.0?

The zero in the subnet mask will correspond to the xxx of your IP address. If you need more than 255 different addresses, you'll have to change the DHCP IP's to 10.0.xxx.xxx (broadcast IP of 10.0.255.255) and the subnet mask to 255.255.0.0.

Theoretically, 255.0.0.0 is a valid subnet mask for 10.0.0.0 to 10.255.255.255 addresses. This wikipedia article shows the valid addresses for private networks.

But in your case (10.0.0.xxx), you should use 255.255.255.0.

Most probably, you are after a subnet mask of 255.255.255.0.

A subnet mask refers to the practice of subnetting, which is, from my understanding of it, a way of dividing one network into smaller logical networks. A subnet mask is used to tell the network devices (whether it be the computer or any routers, modems, etc.) what addresses are local (belongs to this network), and what addresses are remote (belongs to that network).

So, if a computer's IP address is 192.168.1.104, and its subnet mask is 255.255.255.0, then the computer (and every other device attached to the same network) will assume that every IP on that computer's local network will be in the format 192.168.1.xxx, with xxx being the only part that would vary. Likewise, if the subnet mask is 255.255.0.0, then the computer will assume that every IP address in its local network will be in the format of 192.168.xxx.xxx.

Subnetting a network is not really useful in a home network, such as yours. It is mainly used on larger networks (with 255+ computers) to reduce unnecessary network activity. MarkM provided this in the comments:

One of the biggest advantages of subnetting in a large network is to reduce the broadcast traffic. If you have thousands of hosts on a single subnet, your switches will be choking on ARP, DHCP, and other broadcasts. In a home network usually there isn't much of a need unless you want something like a guest wireless LAN that doesn't have any routing available to your main home network.

_{This explanation is very rough and sketchy, so please forgive me if I have made a mistake or two.}

The netmask is used to identify which portion of the IP address represents the network address and which portion represents the machine address^*.

Consider the class A network 10.0.0.0 to 10.255.255.255 (this is the designated "private" class A block i.e. packets destined for these addresses will not be routed). In binary the addresses are: 00001010.00000000.00000000.00000000 through to 00001010.11111111.11111111.11111111. The netmask is generated by assigning a 1 for each bit in these addresses that does not change IE:

00001010.00000000.00000000.00000000
00001010.11111111.11111111.11111111
-----------------------------------
11111111.00000000.00000000.00000000

Which converts to 255.0.0.0, the classic class A netmask.

^* Actually, to get the network address given an IP address, you just do a logical AND. For example, for the netmask 255.0.0.0 and the IP address 10.0.0.1:

00001010.00000000.00000000.00000001
11111111.00000000.00000000.00000000
-----------------------------------
00001010.00000000.00000000.00000000

And 00001010.00000000.00000000.00000000 translates to 10.0.0.0 which is indeed the network address.

Note that you usually don't need a class A network for a home network (do you need more than 255 addresses?) and can thus use 255.255.255.0 as netmask and/or use a class C network (e.g. 192.168.1.0).