Influxdb2 full restore error: failed to restore TSM snapshot for shard XXX: 500 Internal Server Error

[tomklapka]

Steps to reproduce:

1. influx backup --host <HOST_URL> --token <TOKEN> --org '<ORG>' /path/to/backup

2. influx restore --host <HOST_URL> --full --token <TOKEN> /path/to/backup

Expected behaviour:
Successful restore of db

Actual behaviour:
InfluxCLI fails on Error: failed to restore TSM snapshot for shard 10954: 500 Internal Server Error

Environment info:

K3s=1.31.3, Official influxdb:2.7.5 docker image.
Influx CLI dev (git: a79a2a1b825867421d320428538f76a4c90aa34c) build_date: 2024-04-16T14:34:32Z

Config:
DOCKER_INFLUXDB_INIT_MODE=setup
DOCKER_INFLUXDB_INIT_ORG=default
DOCKER_INFLUXDB_INIT_BUCKET=default
DOCKER_INFLUXDB_INIT_RETENTION=48h
INFLUXD_BOLT_PATH=/var/lib/influxdb2/influxd.bolt
INFLUXD_ENGINE_PATH=/var/lib/influxdb2
INFLUXD_SESSION_LENGTH=1440
INFLUXD_REPORTING_DISABLED=true
DOCKER_INFLUXDB_INIT_USERNAME=XXX
DOCKER_INFLUXDB_INIT_PASSWORD=XXX
DOCKER_INFLUXDB_INIT_ADMIN_TOKEN=XXX

Logs:

influxdb2-restore-error.log

[tomklapka]

Possible duplicates:
#23212
#15323

[davidby-influx]

It looks like the shard being restored is outside of its retention period, so has been reaped.

ts=2025-01-27T19:48:16.524579Z lvl=info msg="Attempting deletion of shard from store" log_id=0uHxwn20000 service=retention op_name=retention_delete_check db_instance=5ad8ce2b635727a6 db_shard_id=10954 db_rp=autogen
ts=2025-01-27T19:48:16.532477Z lvl=info msg="Deleted shard" log_id=0uHxwn20000 service=retention op_name=retention_delete_check db_instance=5ad8ce2b635727a6 db_shard_id=10954 db_rp=autogen

ts=2025-01-27T20:00:59.402699Z lvl=warn msg="internal error not returned to client" log_id=0uHxwn20000 handler=error_logger error="shard 10954 doesn't exist on this server"

[tomklapka]

Hi, @davidby-influx bucket id 5ad8ce2b635727a6 has retention policy of 30 days. What exactly can cause shards to be saved outside this period ? For sure I queried data minus 30d and it returns 0 points. On the other hand - in the restored instance, this seems to have happened naturally in all buckets with an explicit retention policy set when the backup is several days or weeks old. In this case the retention cleaning tasks should run after importing all the data, not during.

[davidby-influx]

The retention enforcement process runs continuously, and is not blocked by the restoration process.

Perhaps change your retention period to longer before restoring the expired data; that might work, but I'm not sure.

Let me add @gwossum to the conversation, who knows more about backup and restore, to see if he has a suggestion.

[gwossum]

@tomklapka Try creating a new bucket with a longer retention period. Then, use the --new-bucket option with influx restore to restore the shard to the new bucket.

[tomklapka]

@gwossum thanks, this could help, but what if I need the rest of entities like tasks, tokens, dashboards etc. ? Influxcli can make full backup or just buckets.