Why is the OS obfuscation defense against "It's a Unix system!" not widely implemented?
Before I tear your idea apart, let me say that it's a really interesting idea and it was super fun to think about.
Please continue to think outside the box and ask interesting questions!
Alright, let's do this!
Let's take a step back and ask why that baby monitor is running Linux in the first place? What if there was no operating system and the application was written in bare microcontroller code (think arduino code)? Then there would be no sudo or ls or even a shell for the attacker to use, right?
I'm not an expert here, but I expect that we, as an industry, have gravitated towards putting Linux on anything big enough to run it largely for developer convenience:
- Reducing dev time: When building your WiFi-and-bluetooth-capable web-administered cloud-syncing self-patching whizpopper with companion Android and iOS apps, Linux comes with all the libraries, utilities, and drivers you need to do that.
- Increasing testability: If the device is running bash or busybox with an SSH port, then it's super easy to connect in and figure out what went wrong during your product testing phase.
For your obfuscation idea to work, you'd need to obfuscate not only the names of command-line utilities like sudo and ls, but also every Linux kernel API to prevent the attacker from dropping in their own compiled binary that calls the kernel directly. So let's take another look at your idea:
Implementation would be fairly easy for compilers. Take the simplest case of "rename this function and all calls to it." You could give an OS compiler and an application compiler the same randomized names and they'd be able to talk to each other.
You'll need to do this randomized compilation yourself; otherwise someone could look up the mappings on google.
So, you'll need to build the kernel from source with your "obfuscating compiler" so that only you know the mappings of the obfuscated kernel APIs. (ever built the linux kernel from source? It's certainly more of a chore than docker pull alpine, which is the direction that dev culture seems to be going).
But an operating system is more than just the kernel. You want drivers for the Broadcom BCM2837 wifi chip that comes on that mini-pc device? You'll need to build that driver against your obfuscated kernel with your compiler, if Broadcom will even give you the source code. Then you'll need to build the entire GNU wifi and networking software stacks. How many other things will you need to find source for and add to your build pipeline before you have a functioning OS?
Oh, and if the upstream repos of any of those things issues a patch, you're now responsible for re-building it (assuming you saved the compiler obfuscation mapping files that match your kernel binary) and pushing it out to your devices because - by design - your devices can not use patch binaries produced by the vendor.
Oh, and in order to foil hackers, there'll be none of this "Here's the binary files for Whizpopper 1.4.7", oh no, you'll need to build a uniquely obfuscated version of everything from the kernel up per device that you ship.
So to your questions:
- Is OS obfuscation as described used widely and I just haven't encountered it?
- If not used widely, what are the practical or technical barriers to usage?
I think the answer is that what you're describing pretty much completely defeats the purpose of using pre-existing software components if you need to find and build literally everything from source. It might actually be less effort to ditch the operating system entirely, pretend it's 1960, and write your application directly in CPU microcode.
I like security more than most developers, but like f* that.
Mike's answer says basically everything I have to offer about why this is a bad idea from a development perspective (and, as Ghedipunk's comment says, an unusable security feature provides no security). So instead, I'm going to talk about why from a security perspective, you would never do this.
The answer is actually surprisingly simple: it's a waste of time, and there are strictly better options. Every stupid IoT doodad (remember, the "s" in "IoT" stands for Secure) that doesn't bother to implement such features sure as hell wouldn't go with an approach like you suggest.
- The whole idea just won't work for restricting system calls. An attacker can just set a few registers and invoke an opcode and boom, they're in the kernel executing the syscall of their choice; who cares what the symbolic name of it is? Sure, you can tamper with the syscall table to complicate this (if you don't mind needing to recompile everything and making debugging your custom kernel a form of utter hell) but it's like obfuscating the OS in use; why bother when there are so few candidates, anyhow? Even if the attacker doesn't want to reverse engineer existing code on the system, brute-forcing should be possible unless the search space of usable call indices is wider than I've ever seen on an embedded system.
- Why obfuscate command names when you can just make them entirely inaccessible? A
chrootwon't work for shell built-ins if you're running a shell, but it works fine for everything else and really, why would your custom-built single-purpose app-in-a-box be running a shell? I mean, dev units would have one installed, for test purposes, and maybe that doesn't get removed in your retail image because you're lazy or think you'll need it again. But an attacker wouldn't be able to run it from the context that its app runs as. A simplechroot(or a more complicated sandbox/jail/container) can make the program unable to run - or even access - any files beyond those required for its job. - Why obfuscate the kernel APIs when you can just remove access to them? There are a number of sandboxing systems that can restrict what calls a process (or its descendants... if it's even allowed to create any) can make. See https://stackoverflow.com/questions/2146059/limiting-syscall-access-for-a-linux-application
If your objective is to deprive an attacker of ls and cat, there's an even better alternative to obfuscation: just don't install those utilities.
While I wouldn't say this is a widely implement approach, it is at least implement. For example consider distroless, a collection of docker images with pretty much nothing in them. Some of them (like the one for go) literally have nothing in them. An attack on a system running in such a container can't get shell access because there isn't any shell to run.
The only way then to get shell access by attacking an application in such a container is then to circumvent the container runtime, which is designed to prevent precisely that.
While I gave an example of a docker image, the same concept can be applied to operating systems generally. For example, ls and cat are part of the coreutils package in Debian. You could run apt-get remove coreutils and be confident an attacker won't be able to use ls or cat as part of an attack. Of course this means you can't use them either, and there's probably a lot of other stuff that depends on coreutils that will also have to be removed, but for an embedded device or a server that does just one thing that may be OK.
The general principle is reducing "attack surface": the more "stuff" a target has, the easier it is to compromise. The stuff could be open network ports, lines of code, or binaries installed. If increasing security is the objective, then removing all unnecessary "stuff" is a good start.