PKI realms structure
The concept of PKI realms is designed to provide a standardized way for various applications to access X.509 certificates and private keys. The management of the keys and certificates is moved outside of the application into a fixed directory and file structure, which other applications can access.
PKI realm overview
A "PKI realm" is a placeholder name for a bundle of the private key, X.509 certificate and Root CA certificate. This bundle has a certain directory structure, rules for naming files and what symlinks are present. It is designed so that from the outside of the debops.pki role other Ansible roles, or services they manage, have a standardized, uniform location where they can find X.509 certificates and private keys.
In different guides that describe setting up TLS for different services like
webservers, mail servers, databases, etc. the private keys and X.509
certificates are usually put in different directories - for example
/etc/nginx/ssl/
, /etc/postfix/certs/
, /etc/ssl/certs/
,
and so on. The debops.pki role turns this around by setting up an
uniform set of directories split into "PKI realms", so that a host can have
multiple sets of certificates, each for different purposes. Then, various
services can be configured to get the private key and certificate files from
those specific directories, including privileged access to the private keys
when needed.
PKI realms have a concept of multiple certificate authorities - there's one set of private keys which can be signed by different CAs - internal CA, external CA, ACME CA and self-signed when everything else is disabled. There can be an "example.org" PKI realm which has certificates signed by both internal CA and the Let's Encrypt CA (via ACME), and the pki-realm script used to manage the realms on the remote hosts will automatically switch between them after checking the validity of their X.509 certificates.
The application view
Different applications have different requirements for X.509 certificates and private keys. Some of them support keys and certificates in separate files, others require them combined in a single file. The PKI realm is designed to support both schemes at once.
The realms are located in the /etc/pki/realms/
directory. Each realm is
contained in it's own subdirectory. By default a domain
realm is configured
by the role, and it's simplified directory structure looks like this:
/etc/pki/realms/
└── domain/
├── CA.crt
├── default.crt
├── default.key
├── default.pem
└── trusted.crt
Each of these symlinks point to a file contained in another subdirectory (see the next section). The contents of these files are:
CA.crt
This is the "trust anchor" or Root Certificate Authority certificate used by the application to check the validity of client certificates. This file is publicly readable.
CA.crt
may contain a CA certificate different than the one that issued the server certificate. In this case, this can be used to have separate client and server Certificate Authorities.default.crt
This is the server certificate with optionally bundled Intermediate Certificate Authorities and Diffie-Hellman parameters. It is sent to the clients during connection establishment by the application. This file is publicly readable. Note that not all software appreciates embedded DH parameters in this file. Some Java-based applications, or at least Graylog 4.1 and other software using recent versions of the Bouncy Castle library, throw exceptions when trying to parse this file. Consider using
public/cert_intermediate.pem
when that happens.default.key
This is the server private key. It's readable only by the
root
account and bypki_private_group
– this can be used to limit access to different private keys by different UNIX accounts.default.pem
This file contains the private key, server certificate and Intermediate CA certificate(s). It has the same restrictions as the private key – can be read only by the
root
account and bypki_private_group
.trusted.crt
This is the complete trust chain of intermediate and root CA certificates, without the server certificate, similar to
CA.crt
. It is used for automatic OCSP stapling verification by the server, and works with the primary CA in case the alternative Certificate Authority is used for client certificates.
All of the above filenames are static, which means that the only thing you need to change to select a different PKI realm is the realm directory name.
Example nginx configuration
To use the domain
realm in your nginx configuration, you can add
something similar to this example in your configuration file:
server {
listen [::]:443 ssl;
# HTTPS support
ssl_certificate /etc/pki/realms/domain/default.crt;
ssl_certificate_key /etc/pki/realms/domain/default.key;
# OCSP Stapling support
ssl_stapling on;
ssl_stapling_verify on;
ssl_trusted_certificate /etc/pki/realms/domain/trusted.crt;
# X.509 Client certificate support
ssl_verify_client optional;
ssl_verify_depth 2;
ssl_trusted_certificate /etc/pki/realms/domain/CA.crt;
}
This configuration explains where each certificate is used, but this is not sufficient to enable HTTPS for the webserver. Refer to the nginx documentation for the rest of the required configuration options.
If you use the debops.nginx
Ansible role provided with the project, it has
extensive integration with the debops.pki
role and can configure the
webserver automatically. Usually all you need to do is to make sure the default
realm matches the one you would like to use for each server configuration.
The PKI realm directory structure
This is an example domain
realm directory, created on each remote host
managed by debops.pki
. The current set of certificates active in this realm
is provided by the internal debops.pki
Certificate Authority:
/etc/pki/realms/
└── domain/
├── acme/
├── config/
│ ├── environment
│ └── realm.conf
├── external/
├── internal/
│ ├── alt_intermediate.pem
│ ├── alt_root.pem
│ ├── cert.pem
│ ├── gnutls.conf
│ ├── intermediate.pem
│ ├── request.pem
│ └── root.pem
├── private/
│ ├── key_chain_dhparam.pem
│ ├── key_chain.pem
│ ├── key.pem
│ └── realm_key.pem
├── public/
│ ├── alt_intermediate.pem -> ../internal/alt_intermediate.pem
│ ├── alt_intermediate_root.pem
│ ├── alt_root.pem -> ../internal/alt_root.pem
│ ├── alt_trusted.pem -> alt_intermediate_root.pem
│ ├── cert_intermediate_dhparam.pem
│ ├── cert_intermediate.pem
│ ├── cert.pem -> ../internal/cert.pem
│ ├── cert.pem.sig
│ ├── chain.pem -> cert_intermediate_dhparam.pem
│ ├── full.pem
│ ├── intermediate_root.pem
│ ├── root.pem -> ../internal/root.pem
│ └── trusted.pem -> intermediate_root.pem
├── selfsigned/ (optional)
│ ├── cert.pem
│ ├── gnutls.conf
│ ├── request.pem
│ └── root.pem -> cert.pem
├── CA.crt -> public/alt_trusted.pem
├── default.crt -> public/chain.pem
├── default.key -> private/key.pem
├── default.pem -> private/key_chain_dhparam.pem
└── trusted.crt -> public/trusted.pem
On the Ansible Controller, there's a corresponding directory structure located
in the secret/
directory maintained by the debops.secret Ansible role:
secret/pki/
├── realms/
│ ├── by-group/
│ │ └── all/
│ │ └── domain/
│ │ ├── external/
│ │ └── private/
│ └── by-host/
│ └── hostname.example.com/
│ └── domain/
│ ├── external/
│ ├── internal/
│ │ ├── alt_intermediate.pem
│ │ ├── alt_root.pem
│ │ ├── cert.pem
│ │ ├── intermediate.pem
│ │ └── root.pem
│ └── private/
└── requests/
└── domain/
└── hostname.example.com/
└── domain/
└── request.pem
Your version might not contain all of the shown files and symlinks, for example
the alt_*.pem
versions of intermediate and root CA certificates are only
present if an alternative CA is configured.
Both directories are maintained and kept in sync using two Bash scripts provided by the role, pki-realm and pki-authority. Ansible tasks are used to copy files to and from Ansible Controller to remote hosts.
How a PKI realm is created
Each PKI realm starts with a simple directory structure created on the Ansible
Controller in the secret/
directory:
secret/pki/
└── realms/
├── by-group/
│ └── all/
│ └── domain/
│ ├── external/
│ └── private/
└── by-host/
└── hostname.example.com/
└── domain/
├── external/
├── internal/
└── private/
These directories are created at the beginning, so that Ansible can copy
private files before the actual PKI realm creation on remote hosts. This can be
used to provide a set of identical private RSA keys to multiple hosts at once
(using the directories in private/
subdirectories), custom scripts that
access external Certificate Authorities (using external/
subdirectories),
or DNS challenge API keys.
Next, PKI realm directories are created on the remote host:
/etc/pki/realms/
└── domain/
├── acme/
├── config/
│ ├── environment
│ └── realm.conf
├── external/
├── internal/
├── private/
└── public/
The config/realm.conf
file contains a set of Bash variables that define
different parameters of the PKI realm, for example the default DNS domain used
to generate the certificates, owner and group of various directories and files,
permissions applied to various directory and file types, and so on.
The config/environment
file contains per-realm environment variable
definitions which will be imported by the pki-realm script during
execution. These variables can be used to affect operation of external
commands, for example by specifying HTTP proxy to use on internal networks.
The acme/
, external/
and internal/
subdirectories hold
data files for different Certificate Authorities. Each CA is described in more
detail in a separate document, here is a brief overview:
acme/
This directory is for certificates issued using ACME (for example Let's Encrypt). It will be activated and used automatically when a host has a public IP address and the nginx webserver is installed and configured to support ACME Challenges (see the debops.nginx role for more details).
external/
This directory is used to manage certificates signed by an external Certificate Authority. To do this, you need to provide a special
script
file, which will be executed with a set of environment variables. This can be used to request a certificate from an external CA, like Active Directory or FreeIPA, or download a certificate from an external location.An alternative is to provide an already signed
cert.pem
file and optionally theintermediate.pem
androot.pem
files.internal/
This directory is used by the internal
debops.pki
Certificate Authority to transfer certificate requests as well as certificates.
If the internal CA is disabled either globally for a host, or for a particular
PKI realm, an alternative directory, selfsigned/
will be created. It
will hold a self-signed certificate, not trusted by anything else (not even the
host that has created it). This is done, so that services depending on the
existence of the private keys and certificates can function correctly at all
times.
The pki-realm script, located in /usr/local/lib/pki
on
remote hosts, checks which of these directories have valid
certificates in order of pki_authority_preference
, and the first
valid one is used as the "active" directory. Files from the active directory
are symlinked to the public/
directory.
The public/
directory holds the currently active certificates which are
symlinks to the real certificate files in one of the active directories mentioned above.
Some additional files are also created here by the pki-realm script, namely
the certificate chain (server certificate + intermediate certificates) and the
trusted chain (intermediate certificates + root certificate). The full
certificate contains server certificate + intermediate certificates + root
certificate, which might be required by some applications.
The private/
directory holds the private key of a given realm. Access to
this directory and files inside is restricted by UNIX permissions and only
a specific system group (usually ssl-cert
, but it can be configured) is
allowed to access the files inside.
The next step is the creation of all necessary files, like private/public keys,
certificate requests, etc. At this point, if Ansible was provided with a
private RSA key to use, it will copy it to the private/
directory.
After that, all necessary files are created by the pki-realm script
on the remote host. The directory structure changes a bit:
/etc/pki/realms/
└── domain/
├── acme/
├── config/
│ ├── environment
│ └── realm.conf
├── external/
├── internal/
│ ├── gnutls.conf
│ └── request.pem
├── private/
│ ├── key.pem
│ └── realm_key.pem
├── public/
├── CA.crt -> /etc/ssl/certs/ca-certificates.crt
└── default.key -> private/key.pem
As you can see, the configuration of a Certificate Request for an internal CA
has been created, and the internal/request.pem
file has been generated,
using the private/key.pem
RSA key. By default, if no root.pem
certificate is provided, the system CA certificate store is symlinked as
CA.crt
.
Afterwards, Ansible uploads the generated Certificate Signing Request (CSR) to
the Ansible Controller for the internal CA to sign (if it's enabled). The CSR is
uploaded to the secret/
directory:
secret/pki/
├── realms/
│ ├── by-group/
│ │ └── all/
│ │ └── domain/
│ │ ├── external/
│ │ └── private/
│ └── by-host/
│ └── hostname.example.com/
│ └── domain/
│ ├── external/
│ ├── internal/
│ └── private/
└── requests/
└── domain/
└── hostname.example.com/
└── domain/
└── request.pem
To avoid possible confusion, the secret/pki/requests/domain/
directory
points to the "domain" internal CA which is an intermediate CA located under
the "root" CA. The hostname.example.com/domain/
directory inside the
domain/
directory points to the "domain" realm on the
hostname.example.com
host.
When all of the requests from the remote hosts are uploaded to the Ansible
Controller, the pki-authority script inside the secret/lib
directory takes
over and performs certificate signing for all of the currently managed hosts.
The certificate named cert.pem
is placed in the internal/
directory of each host according to the realm the request came from.
In addition to the certificates, the CA intermediate and root certificates are
also symlinked to the internal/
directory, so that Ansible can
automatically copy their contents to the remote hosts. If a particular
Certificate Authority indicates that an alternative CA should be present, the
alt_*.pem
versions of intermediate and root certificates are also symlinked
there:
secret/pki/
├── realms/
│ ├── by-group/
│ │ └── all/
│ │ └── domain/
│ │ ├── external/
│ │ └── private/
│ └── by-host/
│ └── hostname.example.com/
│ └── domain/
│ ├── external/
│ ├── internal/
│ │ ├── alt_intermediate.pem
│ │ ├── alt_root.pem
│ │ ├── cert.pem
│ │ ├── intermediate.pem
│ │ └── root.pem
│ └── private/
└── requests/
└── domain/
└── hostname.example.com/
└── domain/
└── request.pem
When all of the requests have been processed, Ansible copies the content of the
directories to remote hosts. The content of the by-host/
directory is copied
first and overwrites all files that are present on remote hosts, the
by-group/
directory content is copied only when the corresponding files
are not present. This allows the administrator to provide the shared scripts or
private keys/certificates as needed, per host, per group or for all managed
hosts.
After certificates signed by the internal CA are downloaded to remote hosts, the directory structure might look similar to:
/etc/pki/realms/
└── domain/
├── acme/
├── config/
│ ├── environment
│ └── realm.conf
├── external/
├── internal/
│ ├── alt_intermediate.pem
│ ├── alt_root.pem
│ ├── cert.pem
│ ├── gnutls.conf
│ ├── intermediate.pem
│ ├── request.pem
│ └── root.pem
├── private/
│ ├── key.pem
│ └── realm_key.pem
├── public/
├── CA.crt -> /etc/ssl/certs/ca-certificates.crt
└── default.key -> private/key.pem
Other authority directories (acme/
and external/
) might also
contain various files.
After certificates are copied from the Ansible Controller, the
pki-realm script is executed again for each PKI realm configured on
a given host. It checks which authority directories have valid
certificates, picks the first viable one according to
pki_authority_preference
and activates them.
Certificate activation entails symlinking the certificate, intermediate and
root files to the public/
directory and generation of various chain files:
certificate + intermediate, intermediate + root and key + certificate
+ intermediate (which is stored securely in the private/
directory).
Some applications do not support a separate dhparam
file, and instead expect
that the DHE parameters are present after the X.509 certificate chain. If the
debops.dhparam role has been configured on a host and Diffie-Hellman
parameter support is enabled in a given PKI realm, DHE parameters will be
appended to the final certificate chains (both public and private). When
debops.dhparam regenerates the parameters, the pki-realm script will
automatically detect the new ones and update the certificate chains.
The end result is a fully configured PKI realm with a set of valid certificates available for other applications and services:
/etc/pki/realms/
└── domain/
├── acme/
├── config/
│ ├── environment
│ └── realm.conf
├── external/
├── internal/
│ ├── alt_intermediate.pem
│ ├── alt_root.pem
│ ├── cert.pem
│ ├── gnutls.conf
│ ├── intermediate.pem
│ ├── request.pem
│ └── root.pem
├── private/
│ ├── key_chain_dhparam.pem
│ ├── key_chain.pem
│ ├── key.pem
│ └── realm_key.pem
├── public/
│ ├── alt_intermediate.pem -> ../internal/alt_intermediate.pem
│ ├── alt_intermediate_root.pem
│ ├── alt_root.pem -> ../internal/alt_root.pem
│ ├── alt_trusted.pem -> alt_intermediate_root.pem
│ ├── cert_intermediate_dhparam.pem
│ ├── cert_intermediate.pem
│ ├── cert.pem -> ../internal/cert.pem
│ ├── cert.pem.sig
│ ├── chain.pem -> cert_intermediate_dhparam.pem
│ ├── full.pem
│ ├── intermediate_root.pem
│ ├── root.pem -> ../internal/root.pem
│ └── trusted.pem -> intermediate_root.pem
├── CA.crt -> public/alt_trusted.pem
├── default.crt -> public/chain.pem
├── default.key -> private/key.pem
├── default.pem -> private/key_chain_dhparam.pem
└── trusted.crt -> public/trusted.pem
During this process, at various stages special "hook" scripts might be run, which can react to events like realm creation, activation of new certificates and so on.