This document describes how to deploy, verify, monitor, switch, and recover the YMatrix 6.8 disaster recovery feature.
Note!
The disaster recovery deployment method in YMatrix 6.8 is different from earlier versions and is not backward compatible. Use this document as the source of truth for deployment and operations. Do not continue using the legacy model where only a Subscriber is deployed on the primary-cluster side and only a Publisher is deployed on the backup-cluster side.
Disaster Recovery (DR) is the ability to restore IT infrastructure access and functionality after a disaster event.
YMatrix disaster recovery means that when the YMatrix database cluster that serves applications, or the primary cluster, suffers a disaster, a backup YMatrix database cluster takes over database services so that database data and service availability can continue.
YMatrix 6.8 disaster recovery is a database-cluster-level DR solution with the following characteristics:
A cluster with complete database capabilities, including high availability components and database components.
The YMatrix database cluster that stores data protected by DR and provides database services under normal conditions.
The YMatrix database cluster that backs up data from the primary cluster and can recover and provide database services after a disaster.
The backup cluster is a fully functional YMatrix database cluster. When it runs in the backup role, it backs up all data from the primary cluster and can execute some read-only operations. After disaster recovery switch, the backup cluster becomes a normal service-role YMatrix database cluster.
Note!
Data consistency on the backup cluster is valid only at global consistency points. During backup, read-only queries are supported, but WAL replay synchronization across shards is controlled only at global consistency points. Query results at arbitrary points in time are not guaranteed to be correct.
Components used for data backup and service switching, mainly Subscriber and Publisher.
A DR component used to connect to the primary cluster. It receives backup requests from the Publisher on the backup-cluster side and sends them to the primary cluster. It also receives backup data from the primary cluster and sends that data to the Publisher on the backup-cluster side.
A DR component used to connect to the backup cluster. It receives backup requests from the backup-cluster side and sends them to the Subscriber on the primary-cluster side. It also receives backup data sent from the primary-cluster Subscriber and sends that data to the backup cluster.
In YMatrix 6.8, both the primary-cluster side and backup-cluster side must deploy Subscriber and Publisher so that the two clusters can transmit data in both directions. Subscriber and Publisher on the same cluster side can be deployed on the same pubsub host. In general, only one of the two components on the same cluster side works at a given time.
Before deployment, confirm that:
Note!
The per-shard memory estimates for Subscriber and Publisher are based on release testing and may change in later versions. Reserve resources based on the actual shard count and workload.
When deploying DR, the primary-cluster GUC synchronous_commit supports only the following values:
offlocalremote_writeon, recommendedDo not set it to remote_apply.
On the primary cluster master, run the following command as mxadmin:
gpconfig -c synchronous_commit -v <one of the supported values>Reload the configuration:
mxstop -umatrixmgrBefore deployment, confirm on the primary-cluster side that the matrixmgr database exists and that the matrixmgr extension is loaded.
If the delivery environment already provides a matrixmgr check script or standard SQL, run the check according to the delivery standard. Do not continue deployment until the database and extension status have been confirmed.
The following instructions mainly apply when the backup cluster is deployed in a different data center from the primary cluster. For other scenarios, adjust according to the actual network plan.
Prepare the primary-cluster side:
Prepare the backup-cluster side:
Confirm connectivity between the two sides:
The placeholders <db_cluster_id>, <COUNT>, <shard_id>, and {{...}} in this document must be replaced with actual values from the target cluster before execution.
Install the same YMatrix software as the primary cluster on the primary-cluster pubsub host.
On all primary-cluster nodes, configure /etc/hosts as root and add the pubsub hostname using the primary-cluster-side IP address.
On the primary-cluster master, create a configuration directory as root:
mkdir ~/pubsubOn the primary-cluster master, run the following commands as root to add the pubsub host to the primary physical cluster:
# Initialize expand
/opt/ymatrix/matrixdb6/bin/mxctl expand init > ~/pubsub/expand.init
# Add the pubsub host
cat ~/pubsub/expand.init | /opt/ymatrix/matrixdb6/bin/mxctl expand add --host {{pubsub hostname or primary-cluster-side IP address}} > ~/pubsub/expand.add
# Run network connectivity checks
cat ~/pubsub/expand.add | /opt/ymatrix/matrixdb6/bin/mxctl expand netcheck > ~/pubsub/expand.nc
# Generate the plan
cat ~/pubsub/expand.nc | /opt/ymatrix/matrixdb6/bin/mxbox deployer expand --physical-cluster-only > ~/pubsub/expand.plan
# Execute the plan
cat ~/pubsub/expand.plan | /opt/ymatrix/matrixdb6/bin/mxbox deployer execEdit pg_hba.conf on all database nodes in the primary cluster, including master, standby, primary, and mirror. Add the following rule before #user access rules, preferably before the first non-trust rule:
host all,replication mxadmin {{primary-cluster pubsub internal IP address}}/32 trustOn the primary-cluster master, reload the configuration as mxadmin:
/opt/ymatrix/matrixdb6/bin/mxstop -uOn the primary-cluster pubsub host, create the configuration directory as root:
mkdir -p /etc/matrixdb6/pubsubOn the primary-cluster pubsub host, create /etc/matrixdb6/pubsub/subscriber.conf as root:
# /etc/matrixdb6/pubsub/subscriber.conf
module = 'subscriber'
perf_port = 5587 # Performance diagnostics HTTP port. pprof is disabled by default. See Operations Notes.
verbose = true # Adjust the log level as needed.
debug = false # Adjust the log level as needed.
[[receiver]]
type = 'db'
cluster_id = '' # Primary cluster_id
host = '' # Hostname or IP of the primary-cluster master
port = 4617 # Supervisor listen port of the primary-cluster master
# Optional parameters. System defaults are used if omitted.
# dbname = 'template1'
# username = 'mxadmin'
# restore_point_creation_interval = '60s' # Data consistency is guaranteed through restore points. Default: 60s.
[[sender]]
type = 'socket'
port = 9320 # Subscriber listen portOn the primary-cluster pubsub host, create /etc/matrixdb6/service/subscriber_<db_cluster_id>.conf as root:
# /etc/matrixdb6/service/subscriber_<db_cluster_id>.conf
[program:subscriber_<db_cluster_id>]
process_name=subscriber_<db_cluster_id>
command=%(ENV_MXHOME)s/bin/mxdr -s <COUNT> -c /etc/matrixdb6/pubsub/subscriber.conf
directory=%(ENV_MXHOME)s/bin
autostart=true
autorestart=true
stopsignal=TERM
stdout_logfile=%(ENV_MXLOGDIR)s/subscriber_<db_cluster_id>.log
stdout_logfile_maxbytes=50MB
stdout_logfile_backups=10
redirect_stderr=true
user=mxadmin<COUNT> is the number of shards in the primary cluster. Run the following SQL on the primary-cluster master to obtain it:
SELECT count(1) FROM gp_segment_configuration WHERE role='p';On the primary-cluster pubsub host, run as root:
/opt/ymatrix/matrixdb6/bin/supervisorctl update
/opt/ymatrix/matrixdb6/bin/supervisorctl start subscriber_<db_cluster_id>Check Subscriber status:
/opt/ymatrix/matrixdb6/bin/supervisorctl statusCheck that:
subscriber_<db_cluster_id> exists.On the primary-cluster pubsub host, create /etc/matrixdb6/pubsub/publisher.conf as root:
# /etc/matrixdb6/pubsub/publisher.conf
module = 'publisher'
perf_port = 6698 # Performance diagnostics HTTP port. pprof is disabled by default. See Operations Notes.
verbose = true # Adjust the log level as needed.
debug = false # Adjust the log level as needed.
[[receiver]]
type = 'socket'
host = '<required>' # Corresponding Subscriber IP address, that is, the backup-cluster pubsub external IP address
port = 9320 # Corresponding Subscriber listen port
[[sender]]
type = 'db'
host = '<required>' # Publisher listen host, that is, the primary-cluster pubsub internal IP address
listen = 6432 # Publisher base listen port. The port range changes with shard count.
# Optional parameters. System defaults are used if omitted.
# username = 'mxadmin'
# dbname = 'template1'On the primary-cluster pubsub host, create /etc/matrixdb6/service/publisher_<db_cluster_id>.conf as root:
# /etc/matrixdb6/service/publisher_<db_cluster_id>.conf
[program:publisher_<db_cluster_id>]
process_name=publisher_<db_cluster_id>
command=%(ENV_MXHOME)s/bin/mxdr -s <COUNT> -c /etc/matrixdb6/pubsub/publisher.conf
directory=%(ENV_MXHOME)s/bin
autostart=true
autorestart=true
stopsignal=TERM
stdout_logfile=%(ENV_MXLOGDIR)s/publisher_<db_cluster_id>.log
stdout_logfile_maxbytes=50MB
stdout_logfile_backups=10
redirect_stderr=true
user=mxadminOn the primary-cluster pubsub host, run as root:
/opt/ymatrix/matrixdb6/bin/supervisorctl update
/opt/ymatrix/matrixdb6/bin/supervisorctl start publisher_<db_cluster_id>Check Publisher status:
/opt/ymatrix/matrixdb6/bin/supervisorctl statusCheck that the publisher_<db_cluster_id> process exists, is Running, and has no abnormal logs.
Install the same YMatrix software as the primary cluster on the backup-cluster pubsub host and backup-cluster machines, and install required dependencies. Preparation is the same as primary-cluster deployment.
On the backup-cluster pubsub host, create the pubsub configuration directory as root:
mkdir -p /etc/matrixdb6/pubsubOn the backup-cluster pubsub host, create a working directory as root:
mkdir ~/pubsubThe mapping configuration defines how shards in the primary cluster map to primaries in the backup cluster.
On the primary-cluster master, run the following SQL as mxadmin to generate /tmp/mapping.tmp as a template for the backup-cluster shard mapping file:
COPY (
SELECT content, hostname, port, datadir
FROM gp_segment_configuration
WHERE role='p'
ORDER BY content
) TO '/tmp/mapping.tmp' DELIMITER '|';Modify the generated file:
Run the following command to view configuration instructions:
mxbox deployer dr pub configAfter modification, move the file to the ~/pubsub directory of the root user on the backup-cluster Publisher host and use it as mapping.conf.
On the backup-cluster pubsub host, create /etc/matrixdb6/pubsub/publisher.conf as root:
# /etc/matrixdb6/pubsub/publisher.conf
module = 'publisher'
perf_port = 6698 # Performance diagnostics HTTP port. pprof is disabled by default. See Operations Notes.
verbose = true # Adjust the log level as needed.
debug = false # Adjust the log level as needed.
[[receiver]]
type = 'socket'
host = '<required>' # Corresponding Subscriber IP address, that is, the primary-cluster pubsub external IP address
port = 9320 # Corresponding Subscriber listen port
[[sender]]
type = 'db'
host = '<required>' # Publisher listen host, that is, the backup-cluster pubsub internal IP address
listen = 6432 # Publisher base listen port. The port range changes with shard count.
# Optional parameters. System defaults are used if omitted.
# username = 'mxadmin'
# dbname = 'template1'On the backup-cluster pubsub host, create /etc/matrixdb6/pubsub/subscriber.conf as root:
# /etc/matrixdb6/pubsub/subscriber.conf
module = 'subscriber'
perf_port = 5587 # Performance diagnostics HTTP port. pprof is disabled by default. See Operations Notes.
verbose = true # Adjust the log level as needed.
debug = false # Adjust the log level as needed.
[[receiver]]
type = 'db'
cluster_id = '' # Backup cluster_id. This field can be left empty here.
host = '<required>' # Hostname or IP of the backup-cluster master
port = 4617 # Supervisor listen port of the backup-cluster master
# Optional parameters. System defaults are used if omitted.
# dbname = 'template1'
# username = 'mxadmin'
# restore_point_creation_interval = '60s' # Data consistency is guaranteed through restore points. Default: 60s.
[[sender]]
type = 'socket'
port = 9320 # Subscriber listen portRun the following commands on the backup-cluster pubsub host as root.
Collect information from pubsub, master, and segment machines:
# Collect pubsub host information
/opt/ymatrix/matrixdb6/bin/mxctl setup collect --host localhost > ~/pubsub/collect.0
# Collect master host information
cat ~/pubsub/collect.0 | /opt/ymatrix/matrixdb6/bin/mxctl setup collect --host {{backup-cluster master host IP}} > ~/pubsub/collect.1
# Collect other segment host information
cat ~/pubsub/collect.1 | /opt/ymatrix/matrixdb6/bin/mxctl setup collect --host {{backup-cluster other host IP}} > ~/pubsub/collect.2
...
# Collect other segment host information
cat ~/pubsub/collect.n-1 | /opt/ymatrix/matrixdb6/bin/mxctl setup collect --host {{backup-cluster other host IP}} > ~/pubsub/collect.nRun network connectivity checks:
cat ~/pubsub/collect.n | /opt/ymatrix/matrixdb6/bin/mxctl setup netcheck > ~/pubsub/plan.ncGenerate the deployment plan:
/opt/ymatrix/matrixdb6/bin/mxbox deployer dr pub plan --wait --collect-file ~/pubsub/plan.nc --mapping-file ~/pubsub/mapping.conf --publisher-file /etc/matrixdb6/pubsub/publisher.conf --subscriber-file /etc/matrixdb6/pubsub/subscriber.conf > ~/pubsub/planIf additional mirror replicas need to be deployed on the backup-cluster side, add --with-mirror to the plan command. With this option, the deployed backup cluster includes standby and mirror.
Execute the deployment plan:
/opt/ymatrix/matrixdb6/bin/mxbox deployer dr pub setup --plan-file ~/pubsub/planThis step deploys Subscriber, Publisher, and the backup cluster, connects Publisher, and starts backup.
On each primary host in the primary cluster, run as root:
ps aux | grep postgres | grep walsenderA process connected to Subscriber should exist and be in streaming state.
On the primary-cluster pubsub host, run as mxadmin:
/opt/ymatrix/matrixdb6/bin/supervisorctl statusThe subscriber_<db_cluster_id> and publisher_<db_cluster_id> processes should exist and be Running. Check logs to confirm that there are no abnormal logs.
On the backup-cluster pubsub host, run as mxadmin:
/opt/ymatrix/matrixdb6/bin/supervisorctl statusThe subscriber_<db_cluster_id> and publisher_<db_cluster_id> processes should exist and be Running. Check logs to confirm that there are no abnormal logs.
On each machine in the backup cluster, run as root:
ps aux | grep postgres | grep walreceiverProcesses connected to Subscriber should exist and be in streaming state. The total number of walreceiver processes connected to Publisher in the backup cluster should be the same as the shard count of the original cluster.
If standby and mirror were deployed in the backup cluster, master-to-standby and primary-to-mirror replication in the backup cluster also use walsender and walreceiver.
On each machine in the backup cluster, run as root:
ps aux | grep postgres | grep walsender
ps aux | grep postgres | grep walreceiverConfirm that corresponding walsender and walreceiver processes exist, and check the master-to-standby and primary-to-mirror replication relationships against the deployment plan.
Connect to the masters of the primary cluster and backup cluster respectively, and run the following SQL as a user with query permission:
SELECT * FROM gp_segment_configuration ORDER BY content, dbid;Check whether the results are consistent.
Connect to the primary-cluster master and create a test database:
CREATE DATABASE test_dr;Connect to the test_dr database on the primary cluster and create a test table:
CREATE TABLE test_1(c int) DISTRIBUTED BY (c);Insert data into the test table:
INSERT INTO test_1 SELECT * FROM generate_series(0,999);Connect to the test_dr database on the primary cluster and backup cluster respectively, and run:
SELECT gp_segment_id, count(1) FROM test_1 GROUP BY gp_segment_id;Check whether the results are consistent.
restore_point_creation_interval controls the interval for creating restore points. The default is 60s. When observing logs, determine whether the restore point interval meets expectations based on the explicit configuration or the default value. You can continuously insert data during observation.
The log path is the stdout_logfile configured in /etc/matrixdb6/service/subscriber_<db_cluster_id>.conf. Run:
grep "Created global restore point" <subscriber_log_file>The command should return output, and the log interval should match the configured restore point interval.
Check restore point creation records in the log directory of each primary data directory in the primary cluster:
grep "restore point" *.csv | grep "created at"There should be output, indicating that the primary cluster periodically creates restore points under Subscriber control.
Check the records showing that WAL replay reached restore points in the log directory of the backup-cluster master data directory:
grep "cluster wide WAL replayed to global restore point" *.csvThere should be output showing that the backup cluster has replayed WAL to a restore point.
Check WAL replay progress on the backup cluster. Connect to the backup-cluster master and each primary in utility mode with PGOPTIONS='-c gp_role=utility', and run on each corresponding instance:
SHOW mx_redo_stop_pit;
SELECT pg_last_wal_receive_lsn(), pg_last_wal_replay_lsn();Expected results:
pg_last_wal_replay_lsn is always less than or equal to mx_redo_stop_pit.pg_last_wal_receive_lsn is greater than or equal to mx_redo_stop_pit.These results indicate that the backup cluster has read the restore point generated by the primary cluster through Subscriber and Publisher, and the restore point has taken effect.
Primary-cluster-related system catalogs include:
pg_catalog.gp_replication_slotspg_catalog.gp_stat_replicationThe replication slot used by DR is:
internal_disaster_recovery_rep_slotOn the primary cluster, query the replication slot as a user with required permissions, such as mxadmin:
SELECT *
FROM pg_catalog.gp_replication_slots
WHERE slot_name = 'internal_disaster_recovery_rep_slot'
ORDER BY gp_segment_id;Query replication status:
SELECT *
FROM
pg_catalog.gp_replication_slots s
LEFT JOIN
pg_catalog.gp_stat_replication r
ON
s.gp_segment_id = r.gp_segment_id AND s.active_pid = r.pid
WHERE s.slot_name = 'internal_disaster_recovery_rep_slot'
ORDER BY s.gp_segment_id;The backup-cluster-related system catalog is:
pg_catalog.gp_stat_wal_receiverOn the backup cluster, query walreceiver status as a user with required permissions, such as mxadmin:
SELECT *
FROM pg_catalog.gp_stat_wal_receiver
WHERE slot_name = 'internal_disaster_recovery_rep_slot'
ORDER BY gp_segment_id;Subscriber and Publisher expose a performance diagnostics port through perf_port, but pprof routes are disabled by default.
On the host where Subscriber or Publisher runs, send SIGUSR2 to the target process to toggle /debug/pprof:
kill -USR2 <pid>Toggle rules:
SIGUSR2: enable /debug/pprof.SIGUSR2: disable /debug/pprof.Access examples:
http://<host>:5587/debug/pprof/http://<host>:6698/debug/pprof/Note!
Enable pprof only temporarily during troubleshooting. Avoid exposing diagnostic endpoints for a long time.
This section describes how to handle internal replica failures in the backup cluster while the backup cluster is still running as a DR cluster. The goal is to keep receiving and replaying WAL from the primary cluster.
Example commands in this section are run on the backup-cluster master as mxadmin. If the delivery standard has different requirements, follow the delivery standard.
| Command | Purpose |
|---|---|
mxdr failover |
Promotes the standby or mirror in the same shard to the new DR master or DR primary when the current DR master or DR primary data directory is unrecoverable and cannot continue backup work in a short time. |
mxdr rebuild |
Rebuilds a target standby or mirror from the current healthy replica when a standby or mirror data directory is damaged, or when the old master or primary has been repaired after failover. |
mxdr rebalance |
Restores role distribution to the deployment plan after failover and rebuild when the running roles differ from preferred roles and cluster status is OK_UNBALANCED. |
A DR primary or DR master exception affects WAL receiving and replay for the corresponding shard and can also cause WAL accumulation on the primary-cluster side. Restore it first. A DR mirror or DR standby exception usually does not affect current synchronization, but reduces internal high availability in the backup cluster.
If only the DR primary, DR mirror, DR master, or DR standby process exits abnormally and the data directory is still available, restart with mxstop and mxstart first:
# Stop the cluster
mxstop -a -f
# Start the cluster
mxstart -aIf disk files of a DR mirror or DR standby are damaged and cannot be recovered by restart, while the current DR primary or DR master is healthy, run mxdr rebuild to rebuild the replica from the corresponding healthy primary or master:
# Run mxdr rebuild to repair the damaged mirror or standby
mxdr rebuild -s <shard_id>If the DR primary data directory is damaged and can no longer receive and replay WAL, stop the backup cluster, run mxdr failover so that the mirror in the same shard temporarily takes over synchronization, repair the failed primary, run mxdr rebuild, and then run mxdr rebalance if the cluster enters OK_UNBALANCED. The same process applies when the DR master data directory is unrecoverable.
# Stop the cluster
mxstop -a
# Run mxdr failover so that the standby temporarily takes over
mxdr failover -s <shard_id>
# Start the cluster
mxstart -a
# Run mxdr rebuild to repair the failed standby
mxdr rebuild -s <shard_id>
# Run mxdr rebalance to restore cluster roles
mxdr rebalanceThis section applies when all machines on the primary-cluster side are physically unavailable. In this scenario, operations can only be performed on the backup-cluster side.
Primary-backup switch promotes the backup cluster to a normal service cluster. Before switching, confirm that database services need to be taken over by the backup cluster.
mxdr switchOn the backup-cluster master, run as mxadmin:
mxdr switchThe command asks for confirmation:
Continue promoting cluster for disaster recovery to normal cluster? Yy|Nn (default=N):Add -a to skip confirmation:
mxdr switch -aAfter switching, on the Publisher host, run as mxadmin:
/opt/ymatrix/matrixdb6/bin/supervisorctl statusGet the Publisher name managed by supervisor, for example publisher_<db_cluster_id>.
Stop Publisher:
/opt/ymatrix/matrixdb6/bin/supervisorctl stop publisher_<db_cluster_id>Check status again:
/opt/ymatrix/matrixdb6/bin/supervisorctl statusConfirm that the Publisher status becomes exited.
Subscriber can also be stopped optionally in the same way.
mxdr recover Use CaseAfter the backup cluster is promoted to a normal service cluster by mxdr switch, if the original primary-cluster environment recovers and you want the original primary cluster to rejoin the DR relationship as a cluster synchronized from the new primary cluster, use mxdr recover.
When using mxdr recover, data on the original primary cluster should still be available, and the cluster status should still be recoverable to a fully normal state.
When the original primary cluster is downgraded to a cluster synchronized from another cluster, the pg_hba.conf rules on each database node are reset to the state after cluster initialization. If users manually configured pg_hba.conf rules on the master, reconfigure them after downgrade completes.
Command syntax:
mxdr recover [-a]Parameter:
-a, --no-prompt: Skips the confirmation prompt. If omitted, the command asks whether to convert a normal cluster to synchronize with its DR cluster.Before execution, confirm that:
mxstop -a -f first.Procedure:
mxdr recover requirements.mxdr recover on the original primary cluster.To skip confirmation, run:
mxdr recover -aAfter disaster recovery switch, the backup-cluster YMatrix database cluster changes to the normal service role. Its usage is the same as a normal YMatrix database cluster. Applications using database services can switch connections to this cluster to continue serving.
mxdrmxdr is used to run Subscriber and Publisher and to control backup database cluster switching.
Tools for MatrixDB stream replication.
Usage:
mxdr [flags]
mxdr [command]
Available Commands:
completion Generate the autocompletion script for the specified shell
failover Fail over within disaster recovery cluster
help Help about any command
rebalance Rebalance disaster recovery cluster
rebuild Rebuild old primary or master after disaster recovery failover
recover Convert a normal cluster to synchronize with its DR cluster
switch Promote disaster recovery cluster
Flags:
-c, --config-file string path of the configuration file to start up
-h, --help help for mxdr
-s, --shard-cnt int number of shards of the source database cluster
-v, --version version for mxdr
Use "mxdr [command] --help" for more information about a command.View complete parameters for a subcommand with:
mxdr <command> --helpmxbox deployer drmxbox deployer dr is used to deploy disaster recovery components and the backup cluster.
Disaster recovery deployer
Usage:
mxbox deployer dr [command]
Available Commands:
pub Publisher deployer for dr
Flags:
-h, --help help for dr
Use "mxbox deployer dr [command] --help" for more information about a command.mxbox deployer dr pub deploys Publisher, Subscriber, and the backup cluster.
Publisher deployer for dr
Usage:
mxbox deployer dr pub [command]
Available Commands:
config Generate a config template file
plan Generate plan for migrating host.
setup Execute steps to deploy physical publisher.
Flags:
-h, --help help for pub
Use "mxbox deployer dr pub [command] --help" for more information about a command.View complete help with:
mxbox deployer dr --help
mxbox deployer dr pub --help
mxbox deployer dr pub <command> --helpDR does not currently support database cluster topology changes, so cluster expansion by adding shards is not supported.
Do not use tablespace in clusters that need DR enabled.
Data consistency on the backup cluster is valid only at global consistency points.
When the backup cluster performs primary-backup switch after a disaster, it switches to a global consistency point to guarantee final consistency of backup data.
During data backup, the backup cluster supports read-only queries. However, because WAL replay synchronization across shards is controlled only at global consistency points, query results at arbitrary points in time cannot be guaranteed to be correct.