AES Encryption done in OpenSSL is decrypted successfully but fails when encrypted in Java

Sreeram Nair :

I have used the below OpenSSL code to do an AES encryption which is decrypting successfully in the Tax website

openssl rand 48 > 48byterandomvalue.bin
hexdump /bare 48byterandomvalue.bin > 48byterandomvalue.txt

set /a counter=0
for /f "tokens=* delims= " %%i in (48byterandomvalue.txt) do (
set /a counter=!counter!+1
set var=%%i
if "!counter!"=="1" (set aes1=%%i)
if "!counter!"=="2" (set aes2=%%i)
if "!counter!"=="3" (set iv=%%i)
)

set result1=%aes1:~0,50%
set result1=%result1: =%
set result2=%aes2:~0,50%
set result2=%result2: =%
set aeskey=%result1%%result2%
set initvector=%iv:~0,50%
set initvector=%initvector: =%

openssl aes-256-cbc -e -in PAYLOAD.zip -out PAYLOAD -K %aeskey% -iv %initvector%

openssl rsautl -encrypt -certin -inkey test_public.cer -in 
48byterandomvalue.bin -out 000000.00000.TA.840_Key

But I wanted to do the same this in Java as part of migration, so i used the javax.crypto and java.security libraries but the decryption is failing when I upload the file on the Tax website

//creating the random AES-256 secret key
SecureRandom srandom = new SecureRandom(); 
KeyGenerator keyGen = KeyGenerator.getInstance("AES");
keyGen.init(256);
SecretKey secretKey = keyGen.generateKey();
byte[] aesKeyb = secretKey.getEncoded();

//creating the initialization vector
byte[] iv = new byte[128/8];
srandom.nextBytes(iv);
IvParameterSpec ivspec = new IvParameterSpec(iv);

byte[] encoded = Files.readAllBytes(Paths.get(filePath));
str = new String(encoded, StandardCharsets.US_ASCII);

//fetching the Public Key from certificate
FileInputStream fin = new FileInputStream("test_public.cer");
CertificateFactory f = CertificateFactory.getInstance("X.509");
X509Certificate certificate = (X509Certificate)f.generateCertificate(fin);
PublicKey pk = certificate.getPublicKey();

//encrypting the AES Key with Public Key
Cipher RSACipher = Cipher.getInstance("RSA/ECB/PKCS1Padding");
RSACipher.init(Cipher.ENCRYPT_MODE, pk);
byte[] RSAEncrypted = RSACipher.doFinal(aesKeyb);

FileOutputStream out = new FileOutputStream("000000.00000.TA.840_Key");
out.write(RSAEncrypted);
out.write(iv);
out.close();

Also, the AES key generated in java is different from the one generated via openssl. Can you guys please help.

EDIT 1: Below is the code for AES Encrpytion used:

Cipher AESCipher = Cipher.getInstance("AES/CBC/PKCS5Padding");
AESCipher.init(Cipher.ENCRYPT_MODE, secretKey, ivspec);
byte[] AESEncrypted = AESCipher.doFinal(str.getBytes("UTF-8"));
String encryptedStr = new String(AESEncrypted);

Topaco :

The data that script and Java-code encrypt with RSA differ:

The script generates a random 48-bytes-sequence and stores it in the file 48byterandomvalue.bin. The first 32 bytes are used as AES key, the last 16 bytes as IV. Key and IV are used to encrypt the file PAYLOAD.zip with AES-256 in CBC-mode and store it as file PAYLOAD. The file 48byterandomvalue.bin is encrypted with RSA and stored as file 000000.00000.TA.840_Key.

In the Java-code, a random 32-bytes AES key and a random 16-bytes IV are generated. Both are used to perform the encryption with AES-256 in CBC-mode. The AES key is encrypted with RSA, concatenated with the unencrypted IV and the result is stored in the file 000000.00000.TA.840_Key.

The content of the file 000000.00000.TA.840_Key is different for script and Java-code. For the Java-code to generate the file 000000.00000.TA.840_Key with the script-logic, the unencrypted AES key must be concatenated with the unencrypted IV and this result must be encrypted with RSA:
```
...
//byte[] aesKeyb byte-array with random 32-bytes key
//byte[] iv      byte-array with random 16-bytes iv
byte[] key_iv = new byte[aesKeyb.length + iv.length];
System.arraycopy(aesKeyb, 0, key_iv, 0, aesKeyb.length);
System.arraycopy(iv, 0, key_iv, aesKeyb.length, iv.length);
...
byte[] RSAEncrypted = RSACipher.doFinal(key_iv);
FileOutputStream out = new FileOutputStream("000000.00000.TA.840_Key");
out.write(RSAEncrypted);
out.close();
...
```
Note: The IV doesn't have to be secret and therefore doesn't need to be encrypted. The encryption is only necessary to generate the result of the script in the Java-code.

Another problem concerns the conversion of arbitrary binary data into strings. This generally leads to corrupted data if the encoding is unsuitable (e.g. ASCII or UTF8). Therefore

...
byte[] encoded = Files.readAllBytes(Paths.get(filePath));
str = new String(encoded, StandardCharsets.US_ASCII);           // Doesn't work: ASCII (7-bit) unsuitable for arbitrary bytes, *        
...
byte[] AESEncrypted = AESCipher.doFinal(str.getBytes("UTF-8")); // Doesn't work: UTF-8 unsuitable for arbitrary bytes and additionally different from * 
String encryptedStr = new String(AESEncrypted);                 // Doesn't work: UTF-8 unsuitable for arbitrary bytes
...

should be replaced by

...
byte[] encoded = Files.readAllBytes(Paths.get(filePath));
...
byte[] AESEncrypted = AESCipher.doFinal(encoded);
FileOutputStream out = new FileOutputStream("PAYLOAD");
out.write(AESEncrypted);
out.close();
...

A suitable encoding to store arbitrary data in a string is e.g. Base64, but this isn't necessary in this case, because Base64-encoding isn't used in the script either.

Try these changes. If other issues occur, it would be best to test AES encryption, RSA encryption, and key_iv-generation separately. This makes it easier to isolate bugs.