In this post I shall try to demonstrate why big databases of private data are vulnerable to attack by determined criminals or unsavourary elements.
What I say will apply in the case that the government, a business or an NGO is holding a large amount of data in a database where there are a significant number of people with access to the whole database (we've found that HMRC has many more people with access to their databases).
It is practically impossible (with encryption technology currently available) to administer a large database without allowing a large number of people to have access to the whole database.
Furthermore there will always be an even larger number of people who, although they may not technically have access to the database, have inside information that makes it possible to obtain access.
A list of possible attacks that could be mounted:
1) Someone with access to the database could be bribed to steal the information.
A possible countermeasure to the stealing of data would be to watermark any copy of the database depending on who downloaded it. When person X copies it they get a copy with a few unique dummy pieces a,b and c of data added in. When criminals attempt to use the data a,b or c you know who leaked the information.
This countermeasure is unrealistic as anyone with maintenance access to the database would be able to obtain a copy without the watermark. Alternatively someone with access to the database would probably be able to access it using another persons account. Furthermore as it will only become clear that the information has been stolen when statistically improbable amounts of identify theft (or other related crime) start occurring and any sensible criminal insider would have long since quit their job and flown off to a country with no extradition treaty with the UK (or otherwise hidden themselves).
2) Threaten a family member of someone with access to the database if they do not provide the information/figure out that there is an issue.
A possible countermeasure is to observe your staff carefully for signs of stress or to provide police protection for staff's loved ones 24-7.
These measures would become very expensive and leaky the more staff they need to be applied to.
Threatening people isn't a great criminal strategy as it dramatically increases the chances of being detected and the database is less valuable if people know you've stolen it (as they can take countermeasures). Never the less if there's no other method available this is pretty likely to work.
3) Bribe a minor official to provide inside information with which to hack into the computer system.
This risk is very hard to counter as in any modern system such people number in the thousands.
4) Pay lots of people to attempt to hack into the system until one succeeds (best used with 3).
A possible counter would be to disconnect your computer network from the internet.
Experienced hackers might smuggle a Trojan program into the network by disguising it as upgrade software (would help to bribe someone at Microsoft say). The program could output sensitive info in encrypted format to devices attached to the network. The hackers then just need to arrange for a laptop to be stolen and they have the info.
Alternatively its possible to recover the image on a monitor from the reflection off that monitor onto a nearby reflective surface (even frosted glass). A Trojan could alter the monitors image in subtle ways to transmit the information through a reflection to the outside world. There are many other ways of smuggling information out of an organization of this nature. This technology may sound fanciful but it has already been demonstrated in the lab and we can't assume that criminals won't get their hands on a similarly useful technology.
5) 4 can be done by compromising the integrity of the network itself by altering a computer that the company bought.
A possible counter measure might be for the computers sent to the organization to have detailed diagnostics to ensure they behave in the expected manner.
This might work or might not depending on the way the computer has been modified.
The list was not meant to be exhaustive but just to give an idea of the range of different types of attacks that are possible and the sophisticated methods that might be employed. The point here is that the attacks have an essentially fixed cost. That is it won't matter much what the system you're trying to hack into is (or how it is set up) the money it will take is roughly the same (assuming the system isn't totally open). Although quite expensive the cost becomes affordable once the value of the data you're trying to steal passes this fixed cost. A good technology is one that where additional levels of security could be achieved by spending a little extra cash on the system. Unfortunately we don't have such a technology as yet.
It is certainly the case that big databases like the HMRC database are worth the criminals while to break into. I'm afraid I therefore suspect that such databases are or soon will be hacked into. Just one more thing: I'm not sure there is no way in principle to aggregate large amounts of data like this. It just seems that any system remotely like those organizations are using today are not sufficient to protect large collections of private data. There are some promising developments in creating secure virtual machines running on networks of computers. Until these new ideas are proven though I think we should stay away from these types of database.