I may just be re-discovering an idea that's widely known, but I realized recently why cryptography is so important to computer security. When there's no there there, the only physical constraint is time, and cryptography is a way to leverage that physical constraint into a security constraint.
Arguably, the fundamental problem of security (analogous to the fundamental problem of economics being scarcity) is constraining bad actors.
Good actors function in a society where their constraints are moral, ethical and legal. We find that we can accomplish great things when we rely on these constraints, but the trade off is that we depend on them as universal constraints. We can have a powerful dynamic economy, so long as no one commits fraud, or insider trading, etc. Realistically, we can tolerate a certain amount of violation, but everyone needs to know the rules and at least make some attempt to play by them.
The weakness of these constraints is that they are volitional: we are only constrained by them by conscious acceptance. As an upside, they're cheap to implement. (Enforcement being a completely separate matter.)
Enter the bad actor. Mallory (as he is sometimes known) recognizes that, if he's willing to disregard the damage done by violating the volitional constraints that everyone else agrees to, he can make huge gains for himself.
Recognizing that Mallory and his ilk will not accept volitional constraints, we remove the choice in the constrain - Mallory cannot help being a part of the physical universal and bound to laws of physics. So, we enforce constraints by their application. We take advantage of the fact that solid objects cannot co-locate (well before we understood why this would be so) by building walls and vaults and fences and keeps. We use the fact that matter has continued existence to make receipts and licenses and ledgers. We use the fact that matter cannot be created from nothing (and therefore cannot be perfectly duplicated) with signatures, seals, coinage, keys...
But on a computer, none of those things are true. There is only data (not even a separate 'code' as much as it's attempted), which is only a pattern, and it exists in the province of a machine designed to manipulate patterns. The duplication and removal of a pattern is trivial, and to move it from one place to another not much more difficult.
The most remaining physical constraints (that I can think of) are time, connection, and the properties of the hardware the computer system runs on (e.g. power consumption, waste heat, MTBF of components.) We use connection all the time: firewalls and airgaps, black and white machines, etc. The trouble is that it is cumbersome to manage connection perfectly, and it tends to be poorly scoped to the action under constraint. The properties of the hardware are difficult to leverage, since it is often feasible to circumvent them as a constraint - Mallory moves processing to his hardware and throws money at it.
But time, if properly exploited, can be a very effective constraint. We all have the same amount of time, and cryptographic algorithms can leverage a few seconds of my weak machine's time into more time on a super computer than is likely to be left in the universe.
Something that's interesting about all this is that we haven't removed the volitional aspect from the constraints entirely - merely moved it around, to a smaller and more visible place maybe. Castles are conquered with the aid of a traitor, accounting systems foiled by colluding participants, and users give their passwords away for chocolate. As our security systems become more complex and removed from the things they protect, it becomes harder to associate the new volitional constraints they require with the moral, ethical or legal constraints that good actors voluntarily accept.