Reversing: Secrets of Reverse Engineering

Reverse engineering is the process of extracting the
knowledge or design blueprints from anything man-made.

Two categories of reverse engineering applications
: security-related and software development–related.

Security-Related Reversing
For some people the connection between security and
reversing might not be immediately clear. Reversing is
related to several different aspects of computer security.
For example, reversing has been employed in encryption
research—a researcher reverses an encryption product
and evaluates the level of security it provides. Reversing
is also heavily used in connection with malicious software,
on both ends of the fence: it is used by both Malware
developers and those developing the antidotes. Finally,
reversing is very popular with crackers who use it to
analyze and eventually defeat various copy protection
schemes.

Malicious Software
Reversing is used extensively in both ends of the
malicious software chain. Developers of malicious
software often use reversing to locate vulnerabilities
in operating systems and other software. Such
vulnerabilities can be used to penetrate the system’s
defense layers and allow infection—usually over the
Internet. Beyond infection, culprits sometimes employ
reversing techniques to locate software vulnerabilities
that allow a malicious program to gain access to
sensitive information or even take full control of the system.
At the other end of the chain, developers of antivirus
software dissect and analyze every malicious program
that falls into their hands. They use reversing techniques
to trace every step the program takes and assess the damage
it could cause, the expected rate of infection, how it could be
removed from infected systems, and whether infection can
be avoided altogether.

Reversing Cryptographic Algorithms
Cryptographic algorithms can be roughly divided into two
groups: restricted algorithms and key-based algorithms.

Restricted algorithms are the kind some kids play with;
writing a letter to a friend with each letter shifted several
letters up or down. The secret in restricted algorithms is the
algorithm itself. Once the algorithm is exposed, it is no longer
secure. Restricted algorithms provide very poor security
because reversing makes it very difficult to maintain the
secrecy of the algorithm. Once reversers get their hands
on the encrypting or decrypting program, it is only a matter
of time before the algorithm is exposed. Because the
algorithm is the secret, reversing can be seen as a way
to break the algorithm.

On the other hand, in key-based algorithms, the secret
is a key, some numeric value that is used by the algorithm
to encrypt and decrypt the message. In key-based algorithms
users encrypt messages using keys that are kept private.
The algorithms are usually made public, and the keys are
kept private (and sometimes divulged to the legitimate
recipient, depending on the algorithm). This almost makes
reversing pointless because the algorithm is already known.
In order to decipher a message encrypted with a key-based
cipher, you would have to either:
■■ Obtain the key
■■ Try all possible combinations until you get to the key
■■ Look for a flaw in the algorithm that can be employed to
extract the key or the original message