Unprivileged syslog and encrypted connections
The rsyslog
daemon can be used in a privileged or an unprivileged mode. In
a privileged mode the daemon is run on the root
account, has access to all
required files, sockets, etc. In the unprivileged mode, rsyslog
daemon is
started in a privileged mode first, opens required sockets/ports and then drops
all of its privileges and supplementary UNIX groups.
The debops.rsyslog
role allows you to select which mode is used by
configuring the rsyslog__unprivileged
boolean variable. By default, to
preserve original configuration, the role enables unprivileged mode on Ubuntu
hosts, leaving the configuration privileged on Debian hosts.
The unprivileged operation places certain restrictions on the system
configuration. In particular, the rsyslog
process only uses its primary
system group, dropping any additional groups the user is in. This means, that
using TLS with the default configuration maintained by debops.pki role
becomes problematic - unprivileged rsyslog
process uses only its own
primary group, so it cannot access private keys to allow encrypted connections.
There are multiple solutions to this problem, which you can use. Each one has pros and cons, and you should evaluate the selected method in a development environment before implementing it in production to avoid issues.
Run the daemon in privileged mode
This method is the default on Debian hosts. Ubuntu hosts use the unprivileged mode by default, and reverting to the privileged mode should work, but that hasn't been evaluated yet.
The daemon will be run with the root
permissions, and there shouldn't be
any issues with file access. Enabling TLS connections should work out of the
box. On the downside, an externally accessible service is running with root
permissions, so you should be careful what hosts have access to it, this is
controlled using the firewall.
To enable this mode, set the following in the Ansible inventory:
rsyslog__unprivileged: False
This will enforce the privileged operation.
Grant access to private keys by additional groups
The debops.pki role that maintains the DebOps X.509 infrastructure, allows you to specify additional system groups, which should have access to the private keys. This should be configured before the role creates the private keys, because the permissions are not enforced afterwards - this means that you will need to recreate the private keys and certificates, or update the permissions manually. Additional permissions are granted using the filesystem ACL support.
To enable rsyslog
to get access to the private keys in unprivileged mode by
the syslog
system groups, configure in the Ansible inventory:
# Ensure that needed system group is present
pki_private_groups_present:
- name: 'syslog'
system: True
# Add custom ACL groups to private files and directories for all PKI realms
pki_private_dir_acl_groups: [ 'syslog' ]
pki_private_file_acl_groups: [ 'syslog' ]
# Or, add custom ACL groups to private files only in default PKI realm
pki_default_realms:
- name: 'domain'
acme: False
private_dir_acl_groups: [ 'syslog' ]
private_file_acl_groups: [ 'syslog' ]
After the PKI realm is recreated, you can check the result using command:
root@logs:~# getfacl /etc/pki/realms/domain/private
root@logs:~# getfacl /etc/pki/realms/domain/private/key.pem
You should see the syslog
entry on the list of groups that can access the
respective files and directories. When the rsyslog
process is restarted, it
should be able to access the private keys without issues. To enable the
unprivileged mode on Debian hosts, you might want to enforce it through the
Ansible inventory. Here it is, with example log forwarding to remote host with
TCP over TLS:
# Enable unprivileged operation
rsyslog__unprivileged: True
# Enable TLS support
rsyslog__capabilities: [ 'tls' ]
# Forward logs over encrypted TCP connection
rsyslog__forward: [ '*.* @@logs.{{ ansible_domain }}:6514' ]
Create custom PKI realm for syslog
The debops.pki role allows you to create multiple PKI realms with different purposes and configuration. If you don't want to modify and existing infrastructure in place, creating a separate internal realm just for syslog might be an easy alternative.
To create new PKI realm, add this to the Ansible inventory for all involved hosts:
# Ensure that needed system group is present
pki_private_groups_present:
- name: 'syslog'
system: True
# Create custom realm for syslog
pki_realms:
- name: 'syslog'
acme: False
private_dir_group: 'syslog'
private_file_group: 'syslog'
When the new PKI realm is created, the private directory and files inside
should be owned by the syslog
group. This should ensure that the
rsyslog
daemon in the unprivileged mode, running as syslog
user, should
have access to them. The certificates should be signed by existing
debops.pki Certificate Authority, so they should be trusted by all hosts in
the cluster.
When the new PKI realm is ready, you can tell debops.rsyslog
role to use it:
# Enable unprivileged operation
rsyslog__unprivileged: True
# Enable TLS support
rsyslog__capabilities: [ 'tls' ]
# Change the default PKI realm used by rsyslog
rsyslog__pki_realm: 'syslog'
# Forward logs over encrypted TCP connection
rsyslog__forward: [ '*.* @@logs.{{ ansible_domain }}:6514' ]
When the new configuration is applied, you should see in the rsyslog
configuration files that the daemon is using the correct private key and
certificate.
Testing encrypted connections
To make sure that the logs are sent over an encrypted connection, you can check
the traffic using the tshark
command. On the receiving server, run the
command:
root@logs:~# tshark -i eth0 -f "dst port 514 or dst port 6514" \
-d tcp.port==514,syslog -d tcp.port==6514,syslog
This will output packets that are sent to TCP ports 514 (plaintext traffic) and 6514 (TLS traffic). Afterwards, on remote hosts try sending some test log messages:
user@host:~$ logger Test log message, please ignore
If the connection is not encrypted, you should see something similar to this (notice the unencrypted contents of the packet):
9 132.751792 192.0.2.2 -> 192.0.2.1 Syslog 133 USER.NOTICE: May 16 14:06:05 host user: Test log message, please ignore\n
If the connection is encrypted, output should look similar to this:
9 132.751792 192.0.2.2 -> 192.0.2.1 Syslog 164 \027\003\003\000]\000\000\000\000\000\000\000\037\257\301,\030\365\311\324\023qR9\b\352\203\256\306\260T\023\022\016g\271\220\325\031\250\326\323\0045\3549\270\277>\205\301\256\325\234\246\tzt\333\255\002\006K"\254\334\021wB1\353\f\356,u\344\220\207d\024o\305\234\b\201\003Js[\2533\261\207\231?k\230J
Of course, the contents of the logs should appear normally in the log files,
for example in /var/log/remote/hosts/host/syslog
you should see:
May 16 14:06:05 host user: Test log message, please ignore