Linux: productive sysadmins without root (securing intellectual property)?
Solution 1:
Everything said so far here is good stuff but there is one 'easy' non technical way that helps negates a rogue sys admin - the four eyes principle which basically requires that two sysadmins be present for any elevated access.
EDIT: The two biggest items that I've seen in comments are discussing cost and the possibility of collusion. One of the biggest ways that I've considered to avoid both of those issues is with the use of a managed service company used only for verification of actions taken. Done properly the techs wouldn't know each other. Assuming the technical prowess that a MSP should have it would be easy enough to have a sign off of actions taken.. maybe even as simple as a yes/no to anything nefarious.
Solution 2:
If people truly need admin access to a system then there is little you can do to restrict their activities on that box.
What the majority of organisations do is trust, but verify - you might give people access to parts of the system but you use named admin accounts (e.g. you don't give them direct access to root) and then audit their activities to a log they cannot interfere with.
There's a balancing act here; you might need to protect your systems but you do need to trust people to do their jobs too. If the company was formerly "bitten" by an unscrupulous employee then this might suggest that the companies hiring practices are poor in some way, and those practices were presumably created by the "top managers". Trust begins at home; what are they doing to fix their hiring choices?
Solution 3:
What you are talking about is known as the "Evil Sysadmin" risk. The long and short of it is:
- A sysadmin is someone who has elevated privileges
- Technically adept, to a level that would make them a good 'hacker'.
- Interacting with systems in anomalous scenarios.
The combination of these things makes it essentially impossible to stop malicious action. Even auditing becomes hard, because you have no 'normal' to compare with. (And frankly - a broken system may well have broken auditing too).
There are bunch of mitigative steps:
- Privilege separation - you can't stop a guy with root from doing anything on the system. But you can make one team responsible for networking, and another team responsible for 'operating systems' (or Unix/Windows separately).
- Limit physical access to kit to a different team, who don't get admin accounts... but take care of all the 'hands' work.
- Separate out 'desktop' and 'server' responsibility. Configure the desktop to inhibit removal of data. The desktop admins have no ability to access the sensitive, the server admins can steal it, but have to jump through hoops to get it out the building.
- Auditing to a restricted access system -
syslogand event level auditing, to a relatively tamper-resistant system that they don't have privileged access to. But collecting it isn't enough, you need to monitor it - and frankly, there's a bunch of ways to 'steal' information that might not show up on an audit radar. (Poacher vs. gamekeepers) - apply 'at rest' encryption, so data isn't stored 'in the clear', and requires a live system to access. This means that people with physical access can't access a system that's not actively being monitored, and that in an 'anomalous' scenario where a sysadmin is working on it, the data is less exposed. (e.g. if the database isn't working, the data probably isn't readable)
- Two man rule - if you're ok with your productivity being crippled, and your morale likewise. (Seriously - I've seen it done, and the persistent state of working and being watched makes it extremely difficult working conditions).
- Vet your sysadmins - various records checks may exist depending on country. (Criminal records check, you might even find you can apply for a security clearance in some cases, which will trigger vetting)
- Look after your sysadmins - the absolute last thing you want to do is tell a "trusted" person that you don't trust them. And you certainly don't want to damage morale, because that increases the chance of malicious behaviour (or 'not-quite-negligence, but a slipping in vigilance'). But pay according to responsibility as well as skill set. And consider 'perks' - that are cheaper than salary, but probably valued more. Like free coffee, or pizza once a week.
- and you can also try and apply contract conditions to inhibit it, but be wary of the above.
But pretty fundamentally - you need to accept that this is a trust thing, not a technical thing. Your sysadmins will always be potentially very dangerous to you, as a result of this perfect storm.
Solution 4:
Without putting yourself into an insane technical mind twist to try and come up with a way to give a sysadmin power without giving them power(its likely doable, but would ultimately be flawed in some way).
From a business practice standpoint there is a set of simple solutions. Not cheap solution's, but simple.
You mentioned that the pieces of IP you are concerned about are divided and only people at the top have the power to see them. This is essentially your answer. You should have multiple admins, and NONE of them should be an admin on enough systems to put together the complete picture. You of course would need at least 2 or 3 admins for each piece, in case an admin is sick or in a car accident or something. Maybe even stagger them. say you have 4 admins, and 8 pieces of information. admin 1 can access systems that have piece 1 and 2, admin 2 can get to pieces 2 and 3, admin 3 can get to 3 and 4, and admin 4 can get to 4 and 1. Each system has a backup admin, but no admin is able to compromise the complete picture.
One technique the military uses as well is the limitation of moving data. In a sensitive area there may only be a single system that is capable of burning a disk, or using a USB flash drive, all other systems are restricted. And the ability to use that system is extremely limited and requires specific documented approval by higher ups before anyone is allowed to put any data on anything that could lead to information spillage. Along the same token, you ensure that network traffic between different systems is limited by hardware firewalls. Your network admins who control the fire walls have no access to the systems that they are routing, so they cant specifically get access to information, and your server/workstation admins ensure that all data to and from a system is configured to be encrypted, so that your network admins cant tap the network and gain access to the data.
All laptops/workstations should have encrypted hard drives, and each employee should have a personal locker they are required to lock the drives/laptops in at the end of the night to ensure that no one comes in early/leaves late and gains access to something they aren't supposed to.
Each server should in the very least be in its own locked rack, if not its own locked room, so that only the admins responsible for each server have access to it, since at the end of the day physical access trumps all.
Next there is a practice that can either hurt/help. Limited contracts. If you think you can pay enough to keep attracting new talent, the option of only keeping an each admin for a pre-determined set of time (IE 6 months, 1 year, 2 years) would allow you to limit how long someone would have to attempt to put together all the pieces of your IP.
My personal design would be something along the lines of... Split your data into however many pieces, lets say for the sake of having a number 8, you have 8 git servers, each with their own set of redundant hardware, each administrated by a different set of admins.
Encrypted hardrives for all workstations that will touch the IP. with a specific "project" directory on the drive that is the only directory users are allowed to put their projects in. At the end of each night they are required to sanatize their project directories with a secure deletion tool, then hard drives are removed and locked up(just to be safe).
Each bit of the project has a different admin assigned to it, so a user would only interact with the workstation admin they are assigned to, if their project assignment changes, their data is wiped, they are assigned a new admin. Their systems should not have burning capabilities and should be using a security program to prevent the use of USB flash drives to transfer data without authorization.
take from this what you will.
Solution 5:
It would be similar to the challenge of hiring a janitor for a building. The janitor gets to have all the keys, can open any door, but the reason is that the janitor needs them to do the job. Same with system admins. Symmetrically one can think of this age old problem and look at ways trust is granted historically.
Although there's no clean-cut technical solution, the fact that there's none shouldn't be a reason that we don't try any, an aggregation of imperfect solutions can give somewhat great results.
A model where trust is earned:
- Give fewer permissions to begin with
- Gradually increase permissions
- Put a honeypot and monitor what happens in the coming days
- If the sysadmin reports it instead of trying to abuse it, that's a good start
Implement several levels of administrative powers:
- Level 1: Can modify lower tier of configuration files
- Level 2: Can modify slightly higher tier of configuration files
- Level 3: Can modify slightly higher tier of configuration files and OS settings
Always create an environment where total access by one person is not possible:
- Split systems in clusters
- Give cluster admin powers to different groups
- Minimum 2 groups
Use the Two-man rule when doing high-level core changes:
- https://en.wikipedia.org/wiki/Two-man_rule
- Require an admin from cluster1 and cluster2 for core changes
Trust and verify:
- Log everything
- Log monitoring and alerting
- Ensure all actions are distinguishable
Paperwork:
- Have them sign paperwork to have the legal system be able to help you by suing them if they hurt you gives more incentive not to do so