public

SSL/TLS

2020/01/07

What is the goal

  1. Need to access the console with https//SSL/TLS (port 7183 vs 7180)
  2. Need to encrypt the traffic between agent and server with TLS/SSL. But any cert would be ok
  3. Need to authenticate server and agent so only right certs can be used between them
  4. TLS/SSL for cloudera clusters (services)

How it works

A listening (server) component need to provide a public cert to client component, it also need a key so it can decrypt the client message encrypted using the cert. Client need to trust the cert that server component provided.

So we are talking about 3 things. Public Cert, Private Key, and Truststore. Private key and truststore need to be protected by password. so there are 2 or more passwords. Public/Private key can be provided in 2 formats, jks and pem for different components.

In cloudera manager, only agent uses pem format of pub/pri keys. other components need to use jks which stores both pub and pri certs.

How to make a cert trusted? for most components jssecacert are used as truststore. For cm agent, a trusted cert is specified in /etc/cloudera-cm-agent/config.ini, either to specify the self-signed cert, or to specify a root cert, root + intimediate certs.

All the agent configuration is in config.ini. All the other configurations are in scm database.

what we need

Config CM

  /cm/config (Administration -> Settings -> security):
    KEYSTORE_PATH (cm step1)
    KEYSTORE_PASSWORD (cm step1)
    WEB_TSL (cm step1) # once this switch flips the cm server will try to start using TLS
    TRUSTORE_PATH (agent step4)
    TRUSTORE_PASSWORD (agent step4)

  /cm/service/config (Services -> Configuration -> Scope (Service-Wide) -> Security )
    ssl_client_truststore_location (cm step2)
    ssl_client_truststore_password (cm step2)
    Note: the display name for above 2 parameters were wrong on document. 

  /cm/config (Administration -> Settings -> security):
    AGENT_TLS (agent step1) # once this set cm start use TLS to communicate with agent
    NEED_AGENT_VALIDATION (agent step4) # once this set cm verify if agent is using right cert (in trust store and if the cert actually is valid for the server (DN or SAN))
  
  I've not verified Whether following is needed 
    ssl_server_keystore_password
    ssl_server_keystore_location
    in
    /cm/service/roleConfigGroups/mgmt-ACTIVITYMONITOR-BASE/config
    /cm/service/roleConfigGroups/mgmt-HOSTMONITOR-BASE/config
    /cm/service/roleConfigGroups/mgmt-SERVICEMONITOR-BASE/config
    /cm/service/roleConfigGroups/mgmt-NAVIGATORMETASERVER-BASE/config

Config agent

Reverse configuration

You can update the CONFIGS table in scm database to reverse the web_tls or agent_tls in case things do not work out as expected, other settings such as the passwords, locations of keystore and trust store, does not matter.

select * from CONFIGS where ATTR=’agent_tls’; select * from CONFIGS where ATTR=’web_tls’;

Cert - Key convertion

I have experience 2 scinarious need to convert between the format

  1. start from self-signed cert
openssl pkcs12 -in agentkey.p12 -passin pass:password! -nocerts -out agent.key -passout pass:password!
  1. start from CA signed cert
    • i received cert/key in .pfx format (exported from windows cert store), .pfx is a type of pkcs12
    • i used openssl to retrieve all certs, key, client cert, cacerts respectively in pem format 
      winpty openssl pkcs12 -in abc.pfx -out abc.all.pem -nodes
winpty openssl pkcs12 -in abc.pfx -out abc.pri.pem -nocerts -nodes
winpty openssl pkcs12 -in abc.pfx -out abc.pub.pem -nokeys -clcerts -nodes
winpty openssl pkcs12 -in abc.pfx -out abc.ca.pem -nokeys -cacerts -nodes
    • i used keytool to import client (pub) cert, cacerts, i manually split cacert to root and int, to jssecacert 
      keytool -importcert -alias root_ca -keystore jssecacerts -file ca.root.pem -noprompt -storepass changeit
keytool -importcert -alias int1 -keystore jssecacerts -file ca.int1.pem -noprompt -storepass changeit
keytool -importcert -alias int2 -keystore jssecacerts -file ca.int2.pem -noprompt -storepass changeit
keytool -importcert -alias client_cert -keystore jssecacerts -file abc.pub.pem -noprompt -storepass changeit
    • i used openssl to convert all.pem to p12 (for some reason the .pfx does not work as p12 in some situations) 
      openssl pkcs12 -export -in abc.all.pem -name abc-all -passout pass:actual_password > abc.all.p12
    • use keytool to import the p12 to a jks 
      keytool -importkeystore  -deststorepass actual_password -destkeystore abc-keystore.jks -srckeystore abc.all.p12  -srcstoretype PKCS12 -srcstorepass actual_password
    • use keytool to import the p12 to a jks

Self-signed certs

The basic idea of using self-signed certs in Cloudera is:

https://www.cloudera.com/documentation/enterprise/5-14-x/topics/sg_self_signed_tls.html

 keytool -export -alias cmhost -keystore example.jks -rfc -file selfsigned.pem

keytool -importcert -alias xxxx -keystore jssecacerts -file selfsigned.pem -storepass changeit

in /etc/cloudera-scm-agent/config.ini

client_key_file=/opt/cloudera/security/x509/agent.key
client_keypw_file=/opt/cloudera/security/x509/agentkey.pw 
client_cert_file=/opt/cloudera/security/x509/keyforall.pem

to get private key from keystore:

keytool -importkeystore -srckeystore key-keystore.jks \
-srcstorepass password -srckeypass password \
-destkeystore agentkey.p12 \
-deststoretype PKCS12 -srcalias keyforall -deststorepass \
password -destkeypass password

openssl pkcs12 -in agentkey.p12  -passin pass:password -nocerts -out  agent.key -passout pass:password

Root and Intermediate CA

https://www.cloudera.com/documentation/enterprise/5-14-x/topics/sg_add_root_ca_explicit_trust.html

Kerberos

Use AD user other than current login user to operate HDFS

Query with AD users

Query for AD User for LDAP

Linux

ldapsearch -v -H ldap://adserver.abc.com -D'abc@abc.com' -w'abcpassword' -b'OU=team,OU=Office,OU=users,dc=abc,dc=com'
# the above command query everything under the OU

ldapsearch -v -H ldap://adserver.abc.com -D'abc@abc.com' -w'abcpassword' -b'CN=group_name,OU=groups,OU=team,OU=Office,OU=users,dc=abc,dc=com'
# the above command query everything in the CN group

Windows

Get-ADUser -Filter 'SamAccountName -like "*admin"'
Get-ADUser -Filter 'Name -like "*admin"'
Get-ADGroup -Filter 'Name -like "*admin*"'

API

I spent half day trying to get the cm_api to work (need python2, then caused problem with VS Code, Git Bash, etc, etc), only to find out there is a newer better version available: https://cloudera.github.io/cm_api/docs/python-client-swagger/ or https://archive.cloudera.com/cm6/6.2.0/generic/jar/cm_api/swagger-html-sdk-docs/python/docs/ServicesResourceApi.html#read_services

and this ultimate doc: https://archive.cloudera.com/cm6/6.2.0/generic/jar/cm_api/swagger-html-sdk-docs/python/README.html It worked. And I also download/save the self-signed cert in DER (.cer) format using Chrome, then convert to .PEM format, then the client is able to use it so there’s not a warning or error when dealing with this particular self-signed cert.

Here is a very clear version of API doc https://archive.cloudera.com/cm6/6.0.0/generic/jar/cm_api/apidocs/index.html

LDAP

DN = Distinguished name, the full path to the object CN = common name OU = folder name, from children to parents, for example, ou=child_folder, ou=parent_folder dc = domain, dc=abc,dc=com