One day in a Hadoop cluster, opening hive-site.xml revealed the metastore password stored in clear text. The file was readable by the entire hadoop group, multiple services, and dozens of engineers with cluster access. While formally a violation of basic security norms, this practice remains widespread.
Removing the password proved straightforward thanks to Hadoop’s built-in mechanism, yet it raised a deeper question: once the password disappeared from the config, where did it go, and did security actually improve?
Egor Ivshin, security analyst at Arenadata, is responsible for the security of the ADH product. His role involves identifying vulnerabilities through SCA, SAST, and manual testing, then working with development teams on remediation.
Origin of Secrets in Hadoop Configs
Components such as Hive, Spark, and HBase commonly store sensitive values including database passwords, encryption keys, and S3 tokens directly in XML files. Typical examples include javax.jdo.option.ConnectionPassword in hive-site.xml and AWS credentials in core-site.xml.
Historical tutorials, the need for rapid deployment, and simple inertia explain why clear-text storage persists. The result is uncontrolled proliferation of secrets across backups, developer laptops, and public repositories.
How Credential Provider API Works
The Credential Provider API lets applications request passwords from a provider rather than reading them from XML. When hadoop.security.credential.provider.path points to a JCEKS location, conf.getPassword() checks the keystore first. If the alias is absent and hadoop.security.credential.clear-text-fallback is false, the application fails instead of falling back to the XML value.
Not every component calls getPassword(); those using plain get() require either code changes or continued XML storage.
JCEKS Structure and Limitations
JCEKS is a Java keystore format that stores encrypted aliases. Hadoop supports jceks://file, localjceks://file, and jceks://hdfs URIs. HDFS storage simplifies synchronization and rotation for most services.
The format is locked to the PBEWithMD5AndTripleDES algorithm. Although NIST deprecated 3DES in 2018 and Sweet32 demonstrated practical collision attacks after 32 GB of data, the small size of keystore files keeps real-world risk low. Stronger algorithms require alternative formats or external secret managers such as OpenBao.
Protecting the Master Password
Security ultimately depends on the master password used to unlock the JCEKS file. Common approaches include environment variables, a dedicated password file with strict permissions (chmod 400), or relying on HDFS Transparent Data Encryption so no additional password is needed.
Environment variables are visible via /proc and core dumps. Password files can be placed on tmpfs but still require distribution and rotation across nodes.
Rotation and Automation in ADCM
Manual rotation involves creating a new JCEKS, updating permissions, and restarting services. Arenadata automates this via the “Manage credential encryption” action in ADCM. After execution, JCEKS files receive 640 permissions for local storage and 755 for HDFS paths, balancing security with operational requirements.
Alias names must exactly match the property names referenced in XML. The hadoop credential list command confirms stored aliases without revealing values.
