Why does the government/military not use special operating systems?

There are a number of reasons why building their "own OS" is not a viable option.

1. Research Cost

To built a new OS, from ground up without the use of any existing code would require significant research. Even today, there are only four or five popularly used kernels like Unix , Linux kernel, BSD, XNU and Windows NT.

2. Security through obscurity

It's a proven concept that security through obscurity rarely helps. Yes, it's a new OS so no "hacker" knows how it works, but it is a fact that overtime information about it will be revealed through ex or disgruntled employees. Maybe even through the researchers themselves.

Being a "custom" OS, it will have security issues of their own and no one apart from the original researchers would be able to identify or fix them.

3. COST, COST and COST

Even if such a OS was made, to patch various issues a dedicated maintenance team would have to be kept. You'd additionally need to tailor defensive software, etc to work perfectly on that machine. Any vulnerabilities in that software if emulated would just pass on. In order to make custom software, the OS specifications would have be disclosed, thus we'd need to make custom office software, mail clients, etc.

Ultimately, it's just not viable to make an OS system and use it solely for defense. As I've said earlier security through obstructed rarely helps security, it just makes it more "time-consuming-ly" difficult, but at the end of the day, the cost outweighs the benefits largely.

GreenHills Software is a company that specializes on creating OSes for military embedded systems. Their INTEGRITY system uses a microkernel (less code with less bugs) that has been formally verified (even less bugs).

Such special-purpose domains, where only the most basic software components can be re-used, are probably the only area where it is manageable to deploy a custom OS. Virtually everywhere else, the requirements for interfacing with the outside world, with regular users and with new hardware every few years will completely eat up all your development resources.

Moreover, to my knowledge it is still not well understood how to manage very large software projects. If you have the money and the people to build a new OS in a few years, chances are it will be much more expensive, take much longer, and completely borked.

Finally, there are not a lot of things where a new custom development could improve. After accommodating all the user's and policy enforcer's requirements, you will probably notice that your specialized OS becomes pretty darn similar to existing solutions. For example, its cool to imagine your minimal OS with a few thousand lines of code. Lets say you have that secured. Now you want a web browser with SSL/X509. The most simple implementation of those will likely be multiple times the size of your kernel. And if you don't use existing libraries it will be riddled with compatibility and security bugs.

The main area where OS design could be improved today is - maybe - to move towards a microkernel OS. A design as it was proposed in the Perseus and Nizza architecture could allow you to run security-critical applications with high isolation assurance next to your regular commodity OS, and let the commodity applications refer to the secure applications for tasks like signing documents, establishing session keys etc. More recent examples of this are Genode, TrustVisor and Qubes OS. However, to make such design scalable to many applications in many highly isolated compartments, you need a modern microkernel. So Qubes and TrustVisor are already out the door.

Addon: Just noticed that everyone is focusing on US and making kernels, so maybe some side note: The US, the German, the French and probably also most other governments have been and still are looking into hardened OS for certain purposes. The German government is using the SINA box, which is a hardened Linux plus smartcard that implements a VPN gateway for not-quite-so-critical tasks. Makes you wonder what they use for critical VPNs. They have funded research in alternate OS design. Today they are funding virtualization for Android, so you can run a isolated compartment on an otherwise mostly standard Android phone. SELinux by the US is well-known and the french government has a similar system. The NSA is now also trying to get SELinux into Android. The fact that several millions are spend this way bascially confirms aforementioned problems.

A few data points. The large cost of deploying and using a secure OS comes much more from implementation and continuing assurance than from research. Many research projects had good ideas about security and advanced the state of the art since Multics, but few of these systems did more than demonstrate a proof of concept. There is a secure OS available. Google "AESEC." This verifiably secure A1-class operating system is a descendant of Multics that runs on Intel architecture.

Multics was not a custom system: it was a standard product from Honeywell, sold to government, industry, and education; it failed to succeed because not enough customers would pay extra for security.
Multics provided a rich feature set for its day, but it depended on hardware features that were not provided when microcomputers became popular in the 1980s. Customers chose to buy a lot of little insecure computers rather than a single big secure system. Multics security did not depend on obscurity, but rather on architecture, design, disciplined implementation, and continued assurance. When Multics was being sold, it was "modern" and had features comparable or better than other contemporary systems. Multics pioneered virtual memory, multi-processor support, encryption, graphics APIs, and networking. (It didn't have "thousands of drivers," but device drivers in Multics couldn't undo security.)

(How I know this: I worked on Multics for 16 years, and OS security for industry and government for many more than that.)