etcd Encryption and Authentication

Note!
It is strongly recommended to back up the configuration files under /etc/matrixdb6 to another directory before performing any operations, or contact YMatrix technical support for assistance.

In upgrade scenarios, after reinstalling the YMatrix RPM package, restart the supervisor service on all nodes using:

sudo systemctl restart matrixdb6.supervisor.service

Quick Start

Basic Workflow

  1. Shut down the database cluster:
    mxstop -af
  2. Enable TLS certificates for etcd:
    supervisorctl enable-etcd-credential
  3. Start the database cluster:
    mxstart -a

Verifying Current Status

Run etcdctl with certificates specified:

  • Command:

    etcdctl --endpoints="$ETCD_ENDPOINTS" \
    --cacert "$MXCONFDIR/etcd_credential/ca.crt" \
    --cert   "$MXCONFDIR/etcd_credential/client.crt" \
    --key    "$MXCONFDIR/etcd_credential/client.key" \
    --write-out=table endpoint status
  • Example output:

    [mxadmin@ip-172-31-35-209 ~]$ etcdctl --endpoints="https://172.31.35.209:4679,https://172.31.32.54:4679,https://172.31.47.1:4679" \
    >     --cacert "$MXCONFDIR/etcd_credential/ca.crt" \
    >     --cert   "$MXCONFDIR/etcd_credential/client.crt" \
    >     --key    "$MXCONFDIR/etcd_credential/client.key" \
    >     --write-out=table endpoint status
    +----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
    |          ENDPOINT          |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
    +----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
    | https://172.31.35.209:4679 | 67e0d780ecd95b3b |  3.5.20 |  164 kB |     false |      false |         4 |        473 |                473 |        |
    |  https://172.31.32.54:4679 | 9c2d17a0b36aed50 |  3.5.20 |  164 kB |      true |      false |         4 |        473 |                473 |        |
    |   https://172.31.47.1:4679 | 5fa5b306916e2d84 |  3.5.20 |  164 kB |     false |      false |         4 |        473 |                473 |        |
    +----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+

The default configuration directory is /etc/matrixdbX.

  • Check etcd endpoints (HTTP/HTTPS):

    grep -n "etcd_endpoints" "/etc/matrixdb6/physical_cluster.toml"

    Example:

    [mxadmin@ip-127-1-1-1 ~]$ grep -n "etcd_endpoints" "/etc/matrixdb6/physical_cluster.toml"
    4:etcd_endpoints = ['https://127.1.1.1:4679', 'https://127.1.1.1:4679', 'https://127.1.1.1:4679']
  • Verify whether TLS credentials exist:

    ls -la "/etc/matrixdb6/etcd_credential"

    Example:

    [mxadmin@ip-127-1-1-1 ~]$ ls -la "/etc/matrixdb6/etcd_credential"
    total 28
    drwx------. 2 mxadmin mxadmin  123 Jan 22 03:26 .
    drwxr-xr-x. 6 root    root     288 Jan 22 01:52 ..
    -rw-------. 1 mxadmin mxadmin 1846 Jan 22 02:47 ca.crt
    -rw-------. 1 mxadmin mxadmin 3272 Jan 22 02:47 ca.key
    -rw-------. 1 mxadmin mxadmin 1513 Jan 22 02:47 client.crt
    -rw-------. 1 mxadmin mxadmin 1708 Jan 22 02:47 client.key
    -rw-------. 1 mxadmin mxadmin  407 Jan 22 01:52 meta.json
    -rw-------. 1 mxadmin mxadmin 1558 Jan 22 02:47 server.crt
    -rw-------. 1 mxadmin mxadmin 1704 Jan 22 02:47 server.key

How to determine if TLS is enabled:
TLS enabled: endpoints in physical_cluster.toml use https://..., and the files ca.crt, ca.key, server.crt, server.key, client.crt, and client.key exist under /etcd_credential.
TLS disabled: endpoints use http://..., and the /etcd_credential directory does not exist.

Enabling etcd TLS (Certificate Generation)

supervisorctl enable-etcd-credential

Rolling Back to HTTP (Disabling Certificates)

supervisorctl enable-etcd-credential --rollback

Command Details

Enabling etcd TLS

  • Command:

    supervisorctl enable-etcd-credential
  • Effects:

    • Stops all services except supervisord.
    • Generates certificate files under /etcd_credential.
    • Updates /service/etcd.conf to change the client URL to https:// and enables --client-cert-auth.
    • Updates physical_cluster.toml to replace endpoints with https://....
    • Restarts all services uniformly.
  • Example Output:

    [mxadmin@ip-127-1-1-1 ~]$ supervisorctl enable-etcd-credential
    EnableEtcdCredential succeeded.

Rolling Back to HTTP

  • Command:

    supervisorctl enable-etcd-credential --rollback
  • Use Cases:

    1. TLS enablement failed and automatic rollback did not succeed; full HTTP recovery is required.
    2. TLS is already enabled but needs to be temporarily reverted to certificate-free HTTP.
  • Effects:

    1. Reverts /service/etcd.conf: removes TLS-related parameters and changes the client URL from https:// back to http://.
    2. Updates physical_cluster.toml: reverts endpoints to http://....
    3. Cleans up local credentials: deletes the /etcd_credential directory.
  • Example Output:

    [mxadmin@ip-127-1-1-1 ~]$ supervisorctl enable-etcd-credential --rollback
    EnableEtcdCredential succeeded.