Reversing Custom Cryptography - A Practical Example

Fingerprint Icon


When carrying out internal pentests, it’s common to find configuration files that contain sensitive information such as database credentials. These can sometimes be found by exploring the filesystems of the servers and workstations, but can also be found in open network shares.

Although the passwords are often stored in clear text, it’s also common for them to be stored in some kind of encrypted form. While there are some common approaches to this (such as the aspnet_regiis tool for IIS), many applications implement their own custom encryption methods. The phrase “custom encryption” should set alarm bells ringing, as it’s often poorly implemented and exploitable.

As pentesters we are usually working under tight time constraints, which limit how much time we can reasonably invest in looking at one application on one server, when there could be dozens or even hundreds of systems in scope. While it would be lovely to have a handy vulnerability researcher to pass the application over to for proper analysis, that’s usually not an option. Therefore, we need a quick approach to trying to break this custom encryption, and decrypting the passwords, so that we can attempt to leverage them elsewhere in the environment.

The following sections give an example of the process that was used on a recent internal pentest to obtain the clear text SQL Server passwords from an encrypted configuration file.

Encrypted Config File

While reviewing the accessible file shares on a network using ShareAudit, we found a configuration file containing a connection string with a strange looking password:

<connection name="ExampleSQL" type="JDBC" serv="jdbc:sqlserver://SQL01:1433;DatabaseName=Example_Live" user="Example_Live" pass="14580732486776034978970709774584097120768759222066388930840242023818533832017" driver=""/>

A quick test confirmed that this wasn’t the actual password, and the unusual format of 77 numeric digits (which corresponds to 256 bits) suggested that that the encryption/obfuscation was something custom. The first step was to have a quick google to see if anyone had published a decrypter (or at least some details about how to do it), but as the application was fairly niche there was nothing useful online.

As well as the config file, we also had a copy of the application itself - so the next step was to dig into it and try and find out how the encryption works. The application was written in Java which is nice and easy to decompile, so that makes our lives a lot easier. However, a quick look at the application folder shows that there were ~3200 “.class” files (plus a load of third-party libraries), so we need to try and narrow down where to look.

Initial Analysis

As a very quick (and crude) first step, we grepped for “decrypt” across all of the “.class” files, which returned about half a dozen results. The first of these that sounded promising was example/serv/DBSQLServerConnection.class, so we opened that up in JD-GUI to have a look.

Happily, the code wasn’t obfuscated, so we found a function that read the encrypted password from the configuration file:

package com.example.serv;
public class DBSQLServerConnection {
  public DBSQLServerConnection(String dataSource) {
    XMLElement connection Config.getElement("connection");dataSource);
    this.encryptedPassword = connection.getAttrValue("pass");

This encrypted password was then passed to the StringCrypt.decrypt() method:

import com.example.util.StringCrypto;

public String getPassword() {
  return StringCrypto.decrypt(this.encryptedPassword);

Opening up this file, we found a couple of hard-coded encryption keys at the top, various other helper methods, and then the decrypt() method:

package com.example.util;
public class StringCrypto
  private static BigInteger encrKey = new BigInteger("3[...]9");
  private static BigInteger decrKey = new BigInteger("4[...]1");
  private static BigInteger sharKey = new BigInteger("4[...]1");

  public static String decrypt(String s) {
    BigInteger q = new BigInteger(s);
    BigInteger d = q.modPow(decrKey, sharKey);
    byte[] dat = d.toByteArray();
    return new String(dat);

Different Decryption Approaches

At this point we could see that the encryption was just using a hard-coded key - so we had everything that we need to decrypt the password. There are several different approaches that we can take to accomplish this.

Firstly, we could write our own decryption script in a language of our choice (such as Python). This means that we can use whatever language we’re most comfortable with and makes it easy to create a small portable script that can be re-used. However, this means that we need to carefully read and understand all the code, and then re-implement it in another language. Doing so is often a lot more complicated that it initially appears, because of all the small differences between the languages, and the functions that are available. Little things like padding, character sets and data types can often cause problems and waste a lot of time.

In order to avoid the complexities of translating between languages, we could just copy the Java code out, modify what we need to and then compile it. This can simplify the process, but depending on the code, this may require a lot of little changes to be made, especially if the code pulls in lots of other classes and methods from elsewhere in the application - so it’s rarely as easy as just copy/pasting the code.

A simpler solution is just to import the com.example.StringCrypto class, and then we can call the decrypt() method directly from it. This means that we need a copy of the .class file (so it’s not a very portable solution), but if we’re just using it as a one-off then it’s far faster than the alternative approaches.

Note that we can use the same approach with other languages, although the syntax will change slightly. For example, in C# we need to add the relevant DLL into the project as a dependency, and then we can just import with required class with using.

Decrypting the Password

With this latter approach, we only needed a few lines of code for our decrypter:

import com.example.util.StringCrypto;

class Decrypter {
    public static void main(String args[]) {
        String crypted = args[0];

We then compiled it and pointed to the folder containing all the existing .class files in our classpath:

$ javac -classpath ~/webroot

And then we were able to decrypt the string we found earlier:

$ java Decrypter 14580732486776034978970709774584097120768759222066388930840242023818533832017

It turned out that not only did the account have a very weak password, but it also had the sysadmin role on the SQL Server instance. This meant that we could read and modify all of the information in the various databases in the instance, and also that we could execute operating system commands with the xp_cmdshell stored procedure. This meant that we could pivot through this server to target other systems on the network, and that we were able to identify a couple of serious issues that we might not otherwise been able to find.

Final Thoughts

The purpose of this article isn’t to highlight the focus on the issues within this specific application, but rather to show the process that was followed to identify and decrypt the passwords, and the minimal amount of programming ability or cryptographic knowledge that can be needed.

Of course, not every application is going to be this straightforward - but when you do find encrypted credentials it’s always worth investing a little bit of time to see if there’s a quick and easy way to decrypt them.