lightning: reformat TiDB Lightning config #19167

Oreoxmt · 2024-11-28T09:12:57Z

What is changed, added or deleted? (Required)

lightning: reformat TiDB Lightning config
ref: pingcap/docs#19523

Which TiDB version(s) do your changes apply to? (Required)

Tips for choosing the affected version(s):

By default, CHOOSE MASTER ONLY so your changes will be applied to the next TiDB major or minor releases. If your PR involves a product feature behavior change or a compatibility change, CHOOSE THE AFFECTED RELEASE BRANCH(ES) AND MASTER.

For details, see tips for choosing the affected versions (in Chinese).

What is the related PR or file link(s)?

This PR is translated from:
Other reference link(s):

Do your changes match any of the following descriptions?

Delete files
Change aliases
Need modification after applied to another branch
Might cause conflicts after applied to another branch

ti-chi-bot · 2024-11-28T09:13:00Z

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

ti-chi-bot · 2024-11-28T09:13:07Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please ask for approval from oreoxmt, ensuring that each of them provides their approval before proceeding. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Oreoxmt · 2024-11-28T09:14:29Z

/hold

…ommit pingcap/docs-cn@6058dbf

hfxsd · 2024-12-25T14:40:43Z

tidb-lightning/tidb-lightning-configuration.md

+- 默认值为 `"false"`，表示通过 TiDB Lightning 下发 `ADMIN CHECKSUM TABLE <table>` 命令给 TiKV 执行。
+- 建议将该值设为 `"true"`，以便在 checksum 失败时更容易定位问题。当该值为 `"true"` 时，如果需要调整并发，请在 TiDB 中设置 [`tidb_checksum_table_concurrency`](/system-variables.md#tidb_checksum_table_concurrency) 变量。
+- 默认值：`"false"`


Suggested change

- 默认值为 `"false"`，表示通过 TiDB Lightning 下发 `ADMIN CHECKSUM TABLE <table>` 命令给 TiKV 执行。

- 建议将该值设为 `"true"`，以便在 checksum 失败时更容易定位问题。当该值为 `"true"` 时，如果需要调整并发，请在 TiDB 中设置 [`tidb_checksum_table_concurrency`](/system-variables.md#tidb_checksum_table_concurrency) 变量。

- 默认值：`"false"`

- 默认值：`"false"`

- 可选值：

- `"false"`：表示通过 TiDB Lightning 下发 `ADMIN CHECKSUM TABLE <table>` 命令给 TiKV 执行。

- `"true"`：当该值为 `"true"` 时，如果要调整并发，需要在 TiDB 中设置 [`tidb_checksum_table_concurrency`](/system-variables.md#tidb_checksum_table_concurrency) 系统变量。

- 建议将该值设为 `"true"`，以便在执行 CHECKSUM 失败时更容易定位问题。

hfxsd · 2024-12-25T14:41:22Z

tidb-lightning/tidb-lightning-configuration.md

+#### `check-disk-quota`
+
+- 使用物理导入模式时，检查本地磁盘配额的时间间隔。
+- 默认值：`"60s"`


取值范围？

tidb-lightning/tidb-lightning-configuration.md

hfxsd · 2024-12-25T23:54:37Z

tidb-lightning/tidb-lightning-configuration.md

+
+- 物理导入模式下，向 TiKV 发送数据时一次请求中最大 KV 数量。
+
+<!-- 示例值：32768 -->


默认值？

tidb-lightning/tidb-lightning-configuration.md

hfxsd · 2024-12-26T00:02:39Z

tidb-lightning/tidb-lightning-configuration.md

+- 如果“导入”更快，区块大小的差异就会更小；比值为 0 时则说明区块大小一致。
+- 取值范围：`[0, 1)`
+
+<!-- 示例值：`0.75` -->


默认值？

tidb-lightning/tidb-lightning-configuration.md

hfxsd · 2024-12-26T00:16:46Z

tidb-lightning/tidb-lightning-configuration.md

+- 设置 TiDB 会话变量，提升 Checksum 和 Analyze 的速度。详情参考[控制 `ANALYZE` 并发度](/statistics.md#控制-analyze-并发度)。
+- 如果将 [`checksum-via-sql`](#checksum-via-sql) 设置为 `"true"`，则会通过 TiDB 执行 `ADMIN CHECKSUM TABLE <table>` SQL 语句来进行 Checksum 操作。在这种情况下，`distsql-scan-concurrency` 参数设置不会生效
+
+<!-- 示例值：`15` -->


默认值？

hfxsd · 2024-12-26T00:17:48Z

tidb-lightning/tidb-lightning-configuration.md

+
+- 设置 TiDB 会话变量，提升 Checksum 和 Analyze 的速度。详情参考[控制 `ANALYZE` 并发度](/statistics.md#控制-analyze-并发度)。
+
+<!-- 示例值：`20` -->


系统变量 tidb_build_stats_concurrency 的默认值是 2，是否需要和这个配置项的默认值相同？

hfxsd · 2024-12-26T00:23:15Z

tidb-lightning/tidb-lightning-configuration.md

+
+设置其他 TiDB 会话变量。
+
+#### `tidb_enable_clustered_index`


这是个系统变量，好像应该删掉？可能在以前的文档里只是举例。

hfxsd · 2024-12-26T00:23:56Z

tidb-lightning/tidb-lightning-configuration.md

+
+- 解析和执行 SQL 语句的默认 SQL 模式。
+
+<!-- 示例值：`"ONLY_FULL_GROUP_BY,NO_AUTO_CREATE_USER"` -->


默认值和可选值？

hfxsd · 2024-12-26T00:24:31Z

tidb-lightning/tidb-lightning-configuration.md

+- 设置数据库连接允许的最大数据包大小，对应于系统参数中的 `max_allowed_packet`。
+- 如果设置为 `0`，会使用下游数据库 global 级别的 `max_allowed_packet`。
+
+<!-- 示例值：`67_108_864` -->


取值范围和默认值？

hfxsd · 2024-12-26T00:27:01Z

tidb-lightning/tidb-lightning-configuration.md

+- 可选值：
+ - `"required"`：在导入完成后执行 CHECKSUM 检查，如果 CHECKSUM 检查失败，则会报错退出
+ - `"optional"`：在导入完成后执行 CHECKSUM 检查，如果报错，会输出一条 WARN 日志并忽略错误
+ - `"off"`：导入结束后不执行 CHECKSUM 检查


Suggested change

- 可选值：

- `"required"`：在导入完成后执行 CHECKSUM 检查，如果 CHECKSUM 检查失败，则会报错退出

- `"optional"`：在导入完成后执行 CHECKSUM 检查，如果报错，会输出一条 WARN 日志并忽略错误

- `"off"`：导入结束后不执行 CHECKSUM 检查

- 可选值：

- `"required"`：在导入完成后执行 CHECKSUM 检查，如果 CHECKSUM 检查失败，则会报错退出

- `"optional"`：在导入完成后执行 CHECKSUM 检查，如果报错，会输出一条 WARN 日志并忽略错误

- `"off"`：导入结束后不执行 CHECKSUM 检查

hfxsd · 2024-12-26T00:27:59Z

tidb-lightning/tidb-lightning-configuration.md

+
+- TiDB Lightning 自动刷新导入模式状态的持续时间，该值应小于 TiKV 对应的设定值。
+
+<!-- 示例值：`"5m"` -->


默认值和取值范围？

hfxsd · 2024-12-26T00:28:24Z

tidb-lightning/tidb-lightning-configuration.md

+
+- 在日志中打印导入进度的持续时间。
+
+<!-- 示例值：`"5m"` -->


默认值和取值范围？

tidb-lightning/tidb-lightning-configuration.md

Co-authored-by: xixirangrang <[email protected]>

ti-chi-bot · 2025-01-22T04:36:30Z

@Oreoxmt: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
pull-verify	`e126de8`	link	true	`/test pull-verify`

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

hfxsd · 2025-01-22T07:33:20Z

tidb-lightning/tidb-lightning-configuration.md

+
+#### `table-concurrency`
+
+- 数据引擎的最大并行数。每张表被切分成一个用于存储索引的“索引引擎”和若干存储行数据的“数据引擎”。`index-concurrency` 和 `table-concurrency` 这两项设置控制两种引擎文件的最大并发数。通常情况下，你可以使用默认值。


Suggested change

- 数据引擎的最大并行数。每张表被切分成一个用于存储索引的“索引引擎”和若干存储行数据的“数据引擎”。`index-concurrency` 和 `table-concurrency` 这两项设置控制两种引擎文件的最大并发数。通常情况下，你可以使用默认值。

- 数据引擎的最大并行数。每张表被切分成一个用于存储索引的“索引引擎”和若干存储行数据的“数据引擎”。`index-concurrency` 和 `table-concurrency` 这两项设置控制两种引擎文件的最大并发数。通常情况下使用默认值。

hfxsd · 2025-01-24T01:55:01Z

tidb-lightning/tidb-lightning-configuration.md

+
+#### `parallel-import`
+
+- 是否允许启动多个 TiDB Lightning 实例（物理导入模式）并行导入数据到一个或多个目标表。该参数仅限目标表为空的场景使用。


Suggested change

- 是否允许启动多个 TiDB Lightning 实例（物理导入模式）并行导入数据到一个或多个目标表。该参数仅限目标表为空的场景使用。

- 是否允许启动多个 TiDB Lightning 实例（物理导入模式）[并行导入数据](/tidb-lightning/tidb-lightning-distributed-import.md)到一个或多个目标表。该参数仅限目标表为空的场景使用。

hfxsd · 2025-01-24T06:25:12Z

tidb-lightning/tidb-lightning-configuration.md

+- 该参数用于设置在完全并发下，“导入”和“写入”过程的持续时间比。该值可以通过计算 1 GiB 大小的单张表的（导入时长/写入时长）得到。你可以在日志文件中查看精确的时间。
+- 如果“导入“更快，区块大小的差异就会更小。比值为 `0` 表示区块大小相同。


Suggested change

- 该参数用于设置在完全并发下，“导入”和“写入”过程的持续时间比。该值可以通过计算 1 GiB 大小的单张表的（导入时长/写入时长）得到。你可以在日志文件中查看精确的时间。

- 如果“导入“更快，区块大小的差异就会更小。比值为 `0` 表示区块大小相同。

- 该参数用于设置在完全并发下，导入和写入过程的持续时间比。该值可以通过计算 1 GiB 大小的单张表的（导入时长/写入时长）得到。你可以在日志文件中查看精确的时间。

- 如果导入更快，区块大小的差异就会更小。比值为 `0` 表示区块大小相同。

hfxsd · 2025-01-24T06:47:23Z

tidb-lightning/tidb-lightning-configuration.md

+<!-- 示例值：`67_108_864` -->
+
+#### `tls`
+


Suggested change

hfxsd · 2025-01-24T06:50:25Z

tidb-lightning/tidb-lightning-configuration.md

+    - `"required"`：在导入完成后执行 CHECKSUM 检查，如果 CHECKSUM 检查失败，则会报错退出
+    - `"optional"`：在导入完成后执行 CHECKSUM 检查，如果报错，会输出一条 WARN 日志并忽略错误
+    - `"off"`：导入结束后不执行 CHECKSUM 检查


Suggested change

- `"required"`：在导入完成后执行 CHECKSUM 检查，如果 CHECKSUM 检查失败，则会报错退出

- `"optional"`：在导入完成后执行 CHECKSUM 检查，如果报错，会输出一条 WARN 日志并忽略错误

- `"off"`：导入结束后不执行 CHECKSUM 检查

- `"required"`：在导入完成后执行 Checksum 检查，如果 Checksum 检查失败，则会报错退出

- `"optional"`：在导入完成后执行 Checksum 检查，如果报错，会输出一条 WARN 日志并忽略错误

- `"off"`：导入结束后不执行 Checksum 检查

hfxsd · 2025-01-24T06:52:50Z

tidb-lightning/tidb-lightning-configuration.md

+- 可选值：
+    - `"false"`：表示通过 TiDB Lightning 下发 `ADMIN CHECKSUM TABLE <table>` 命令给 TiKV 执行。
+    - `"true"`：当该值为 `"true"` 时，如果要调整并发，需要在 TiDB 中设置 [`tidb_checksum_table_concurrency`](/system-variables.md#tidb_checksum_table_concurrency) 系统变量。
+- 建议将该值设为 `"true"`，以便在执行 CHECKSUM 失败时更容易定位问题。


Suggested change

- 建议将该值设为 `"true"`，以便在执行 CHECKSUM 失败时更容易定位问题。

- 建议将该值设为 `"true"`，以便在执行 Checksum 失败时更容易定位问题。

hfxsd · 2025-01-24T06:52:58Z

tidb-lightning/tidb-lightning-configuration.md

+
+#### `analyze`
+
+- 配置是否在 CHECKSUM 结束后对所有表逐个执行 `ANALYZE TABLE <table>` 操作。


Suggested change

- 配置是否在 CHECKSUM 结束后对所有表逐个执行 `ANALYZE TABLE <table>` 操作。

- 配置是否在 Checksum 结束后对所有表逐个执行 `ANALYZE TABLE <table>` 操作。

ti-chi-bot bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Nov 28, 2024

ti-chi-bot bot added missing-translation-status This PR does not have translation status info. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Nov 28, 2024

Oreoxmt added type/enhancement The issue or PR belongs to an enhancement. translation/doing This PR’s assignee is translating this PR. labels Nov 28, 2024

ti-chi-bot bot removed the missing-translation-status This PR does not have translation status info. label Nov 28, 2024

ti-chi-bot bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Nov 28, 2024

Oreoxmt added 14 commits December 25, 2024 16:49

lightning: reformat global configuration

75b10d6

lightning: reformat task configuration > lightning

9d27d46

lightning: reformat task configuration > security

718c124

lightning: reformat task configuration > checkpoint

7eff543

lightning: reformat task configuration > conflict

127c9b9

lightning: reformat task configuration > tikv-importer

ebeeb70

lightning: reformat task configuration > cron

a87abc6

lightning: reformat task configuration > post-restore

0b21e5c

lightning: reformat task configuration > tidb.security

bf01826

lightning: reformat task configuration > tidb.session-vars

c71ebe1

lightning: reformat task configuration > tidb

950d22b

lightning: reformat task configuration > mydumper.files

c83e48b

lightning: reformat task configuration > mydumper.csv

d565ef5

lightning: reformat task configuration > mydumper

5241a65

Oreoxmt force-pushed the reformat-config-tidblightning branch from 08f842c to 5241a65 Compare December 25, 2024 10:17

Oreoxmt added 2 commits December 25, 2024 18:21

lightning: reformat strcture

19eb986

fix ci

6058dbf

Oreoxmt added the needs-cherry-pick-release-8.5 Should cherry pick this PR to release-8.5 branch. label Dec 25, 2024

github-actions bot pushed a commit to Oreoxmt/pingcap-docsite-preview that referenced this pull request Dec 25, 2024

Preview PR pingcap/docs-cn#19167 and this preview is triggered from c…

b83b659

…ommit pingcap/docs-cn@6058dbf

Oreoxmt marked this pull request as ready for review December 25, 2024 10:32

hfxsd reviewed Dec 25, 2024

View reviewed changes

tidb-lightning/tidb-lightning-configuration.md Outdated Show resolved Hide resolved

hfxsd reviewed Dec 25, 2024

View reviewed changes

tidb-lightning/tidb-lightning-configuration.md Outdated Show resolved Hide resolved

hfxsd reviewed Dec 25, 2024

View reviewed changes

tidb-lightning/tidb-lightning-configuration.md Outdated Show resolved Hide resolved

hfxsd reviewed Dec 25, 2024

View reviewed changes

tidb-lightning/tidb-lightning-configuration.md Outdated Show resolved Hide resolved

hfxsd reviewed Dec 25, 2024

View reviewed changes

tidb-lightning/tidb-lightning-configuration.md Outdated Show resolved Hide resolved

hfxsd reviewed Dec 25, 2024

View reviewed changes

tidb-lightning/tidb-lightning-configuration.md Outdated Show resolved Hide resolved

hfxsd reviewed Dec 26, 2024

View reviewed changes

tidb-lightning/tidb-lightning-configuration.md Outdated Show resolved Hide resolved

hfxsd reviewed Dec 26, 2024

View reviewed changes

tidb-lightning/tidb-lightning-configuration.md Outdated Show resolved Hide resolved

hfxsd reviewed Dec 26, 2024

View reviewed changes

tidb-lightning/tidb-lightning-configuration.md Outdated Show resolved Hide resolved

Apply suggestions from code review

e126de8

Co-authored-by: xixirangrang <[email protected]>

hfxsd reviewed Jan 22, 2025

View reviewed changes

hfxsd reviewed Jan 24, 2025

View reviewed changes

tidb-lightning/tidb-lightning-configuration.md



#### `tls`

Copy link

Collaborator

hfxsd Jan 24, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change

hfxsd reviewed Jan 24, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

lightning: reformat TiDB Lightning config #19167

lightning: reformat TiDB Lightning config #19167

Oreoxmt commented Nov 28, 2024 •

edited

Loading

ti-chi-bot bot commented Nov 28, 2024

ti-chi-bot bot commented Nov 28, 2024

Oreoxmt commented Nov 28, 2024

hfxsd Dec 25, 2024 •

edited by Oreoxmt

Loading

hfxsd Dec 25, 2024

hfxsd Dec 25, 2024

hfxsd Dec 26, 2024

hfxsd Dec 26, 2024

hfxsd Dec 26, 2024 •

edited

Loading

hfxsd Dec 26, 2024

hfxsd Dec 26, 2024

hfxsd Dec 26, 2024

hfxsd Dec 26, 2024

hfxsd Dec 26, 2024

hfxsd Dec 26, 2024

ti-chi-bot bot commented Jan 22, 2025

hfxsd Jan 22, 2025

hfxsd Jan 24, 2025

hfxsd Jan 24, 2025

hfxsd Jan 24, 2025

hfxsd Jan 24, 2025

hfxsd Jan 24, 2025

hfxsd Jan 24, 2025


		- 物理导入模式下，向 TiKV 发送数据时一次请求中最大 KV 数量。

		<!-- 示例值：32768 -->


		- 设置 TiDB 会话变量，提升 Checksum 和 Analyze 的速度。详情参考[控制 `ANALYZE` 并发度](/statistics.md#控制-analyze-并发度)。

		<!-- 示例值：`20` -->


		设置其他 TiDB 会话变量。

		#### `tidb_enable_clustered_index`


		- 解析和执行 SQL 语句的默认 SQL 模式。

		<!-- 示例值：`"ONLY_FULL_GROUP_BY,NO_AUTO_CREATE_USER"` -->


		- TiDB Lightning 自动刷新导入模式状态的持续时间，该值应小于 TiKV 对应的设定值。

		<!-- 示例值：`"5m"` -->


		- 在日志中打印导入进度的持续时间。

		<!-- 示例值：`"5m"` -->


		#### `table-concurrency`

		- 数据引擎的最大并行数。每张表被切分成一个用于存储索引的“索引引擎”和若干存储行数据的“数据引擎”。`index-concurrency` 和 `table-concurrency` 这两项设置控制两种引擎文件的最大并发数。通常情况下，你可以使用默认值。


		#### `parallel-import`

		- 是否允许启动多个 TiDB Lightning 实例（物理导入模式）并行导入数据到一个或多个目标表。该参数仅限目标表为空的场景使用。

	- 是否允许启动多个 TiDB Lightning 实例（物理导入模式）并行导入数据到一个或多个目标表。该参数仅限目标表为空的场景使用。
	- 是否允许启动多个 TiDB Lightning 实例（物理导入模式）[并行导入数据](/tidb-lightning/tidb-lightning-distributed-import.md)到一个或多个目标表。该参数仅限目标表为空的场景使用。

		- 该参数用于设置在完全并发下，“导入”和“写入”过程的持续时间比。该值可以通过计算 1 GiB 大小的单张表的（导入时长/写入时长）得到。你可以在日志文件中查看精确的时间。
		- 如果“导入“更快，区块大小的差异就会更小。比值为 `0` 表示区块大小相同。

	- 建议将该值设为 `"true"`，以便在执行 CHECKSUM 失败时更容易定位问题。
	- 建议将该值设为 `"true"`，以便在执行 Checksum 失败时更容易定位问题。


		#### `analyze`

		- 配置是否在 CHECKSUM 结束后对所有表逐个执行 `ANALYZE TABLE <table>` 操作。

lightning: reformat TiDB Lightning config #19167

Are you sure you want to change the base?

lightning: reformat TiDB Lightning config #19167

Conversation

Oreoxmt commented Nov 28, 2024 • edited Loading

What is changed, added or deleted? (Required)

Which TiDB version(s) do your changes apply to? (Required)

What is the related PR or file link(s)?

Do your changes match any of the following descriptions?

ti-chi-bot bot commented Nov 28, 2024

ti-chi-bot bot commented Nov 28, 2024

Oreoxmt commented Nov 28, 2024

hfxsd Dec 25, 2024 • edited by Oreoxmt Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hfxsd Dec 26, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ti-chi-bot bot commented Jan 22, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Oreoxmt commented Nov 28, 2024 •

edited

Loading

hfxsd Dec 25, 2024 •

edited by Oreoxmt

Loading

hfxsd Dec 26, 2024 •

edited

Loading