Skip to content

Commit

Permalink
chore: update README
Browse files Browse the repository at this point in the history
  • Loading branch information
@hhyasdf
hhyasdf committed Jul 21, 2023
1 parent ba74a4a commit 5057d6e
Show file tree
Hide file tree
Showing 2 changed files with 60 additions and 101 deletions.
6 changes: 2 additions & 4 deletions README-zh_CN.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@

### 手动编译

```
```bash
go get github.com/AliyunContainerService/image-syncer
cd $GOPATH/github.com/AliyunContainerService/image-syncer

Expand All @@ -44,7 +44,7 @@ make

### 命令用例

```shell
```bash
# 获得帮助信息
./image-syncer -h

Expand Down Expand Up @@ -125,8 +125,6 @@ quay.io/coreos/kube-rbac-proxy@sha256:14b267eb38aa85fd12d0e168fffa2d8a6187ac53a1

--proc 并发数，进行镜像同步的并发goroutine数量，默认为5

--records 指定传输过程中保存已传输完成镜像信息（blob）的文件输出/读取路径，默认输出到当前工作目录，一个records记录了对应目标仓库的已迁移信息，可以用来进行连续的多次迁移（会节约大量时间，但不要把之前自己没执行过的records文件拿来用），如果有unknown blob之类的错误，可以删除该文件重新尝试，image-syncer 在 >= v1.1.0 版本中移除了对于records文件的依赖

--retries 失败同步任务的重试次数，默认为2，重试会在所有任务都被执行一遍之后开始，并且也会重新尝试对应次数生成失败任务的生成。一些偶尔出现的网络错误比如io timeout、TLS handshake timeout，都可以通过设置重试次数来减少失败的任务数量

--os 用来过滤源 tag 的 os 列表，为空则没有任何过滤要求，只对非 docker v2 schema1 media 类型的镜像格式有效
Expand Down
155 changes: 58 additions & 97 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -16,25 +16,25 @@ English | [简体中文](./README-zh_CN.md)

- Support for many-to-many registry synchronization
- Supports docker registry services based on Docker Registry V2 (e.g., Alibaba Cloud Container Registry Service, Docker Hub, Quay.io, Harbor, etc.)
- Network & Memory Only, don't rely on large disk storage, fast synchronization
- Incremental Synchronization, use a disk file to record the synchronized image blobs' information
- Concurrent Synchronization, adjustable goroutine numbers
- Automatic Retries of Failed Sync Tasks, to resolve the network problems while synchronizing
- Network & Memory Only, doesn't rely on any large disk storage, fast synchronization
- Incremental Synchronization, ignore unchanged images automatically
- BloB-Level Concurrent Synchronization, adjustable goroutine numbers
- Automatic Retries of Failed Sync Tasks, to resolve the network problems (rate limit, etc.) while synchronizing
- Doesn't rely on Docker daemon or other programs

## Usage

### GitHub Action

You can use [image-sync-action](https://github.com/marketplace/actions/image-sync-action) to try image-syncer online without relying on any physical resources.
You can use [image-sync-action](https://github.com/marketplace/actions/image-sync-action) to try image-syncer online without paying for any machine resources.

### Install image-syncer

You can download the latest binary release [here](https://github.com/AliyunContainerService/image-syncer/releases)

### Compile Manually

```
```bash
go get github.com/AliyunContainerService/image-syncer
cd $GOPATH/github.com/AliyunContainerService/image-syncer

Expand All @@ -44,135 +44,96 @@ make

### Example

```shell
```bash
# Get usage information
./image-syncer -h

# With this command, configure file path is "./config.json", default target registry is "registry.cn-beijing.aliyuncs.com",
# default target namespace is "ruohe", 6 of goroutine numbers, every failed task will be retried 3 times.
./image-syncer --proc=6 --auth=./auth.json --images=./images.json --namespace=ruohe \
--registry=registry.cn-beijing.aliyuncs.com --retries=3
./image-syncer --proc=6 --auth=./auth.json --images=./images.json --registry=registry.cn-beijing.aliyuncs.com --retries=3
```

### Configure Files

After v1.2.0, image-syncer supports both YAML and JSON format, and origin config file can be split into "auth" and "images" file. A full list of examples can be found under [example](./example), meanwhile the older version of configuration file is still supported via --config flag.
Image-syncer supports `--auth` and `--images` flag for passing authentication file and image sync configuration file, both of which supports YAML and JSON format. Seperate authentication information is more flexible to reuse it in different sync missions.

> The older version (< v1.2.0) of configuration file is still supported via `--config` flag, you can find the example in [config.yaml](./example/config.yaml) and [config.json](./example/config.json).
#### Authentication file

Authentication file holds all the authentication information for each registry, the following is an example of `auth.json`

```java
{
// Authentication fields, each object has a URL as key and a username/password pair as value,
// if authentication object is not provided for a registry, access to the registry will be anonymous.

"quay.io": { // This "registry" or "registry/namespace" string should be the same as registry or registry/namespace used below in "images" field.
// The format of "registry/namespace" will be more prior matched than "registry"
"username": "xxx", // Optional, if the value is a string of "${env}" or "$env", image-syncer will try to find the value in environment variables, after v1.3.1
"password": "xxxxxxxxx", // Optional, if the value is a string of "${env}" or "$env", image-syncer will try to find the value in environment variables, after v1.3.1
"insecure": true // "insecure" field needs to be true if this registry is a http service, default value is false, version of image-syncer need to be later than v1.0.1 to support this field
},
"registry.cn-beijing.aliyuncs.com": {
"username": "xxx",
"password": "xxxxxxxxx"
},
"docker.io": {
"username": "xxx",
"password": "xxxxxxxxxx"
},
"quay.io/coreos": { // "registry/namespace" format is supported after v1.0.3 of image-syncer
"username": "abc",
"password": "xxxxxxxxx",
"insecure": true
}
}
Authentication file holds all the authentication information for each registry. For each registry (or namespace), there is a object which contains username and password. For each images sync rule in image sync configuration file, image-syncer will try to find a match in all the authentication information and use the best(longest) fit one. Access will be anonymous if no authentication information is found.

You can find the example in [auth.yaml](./example/auth.yaml) and [auth.json](./example/auth.json), here we use [auth.yaml](./example/auth.yaml) for explaination:

```yaml
quay.io: # This "registry" or "registry/namespace" string should be the same as registry or registry/namespace used below in image sync rules. And if an url match multiple objects, the "registry/namespace" string will actually be used.
username: xxx
password: xxxxxxxxx
insecure: true # Optional, "insecure" field needs to be true if this registry is a http service, default value is false.
registry.cn-beijing.aliyuncs.com:
username: xxx # Optional, if the value string is a format of "${env}" or "$env", use the "env" environment variables as username.
password: xxxxxxxxx # Optional, if the value string is a format of "${env}" or "$env", use the "env" environment variables as password.
docker.io:
username: "${env}"
password: "$env"
quay.io/coreos:
username: abc
password: xxxxxxxxx
insecure: true
```
#### Image sync configuration file
Image sync configuration file defines all the image synchronization rules, the following is an example of `images.json`

```java
{
// Rules of image synchronization, each rule is a kv pair of source(key) and destination(value).

// The source of each rule should not be empty string.

// If you need to synchronize images from one source to multi destinations, add more rules.

// Both source and destination are docker image url (registry/namespace/repository:tag),
// with or without tags.

// For both source and destination, if destination is not an empty string, "registry/namespace/repository"
// is needed at least.

// You cannot synchronize a whole namespace or a registry but a repository for one rule at most.

// The repository name and tag of destination can be deferent from source, which works like
// "docker pull + docker tag + docker push"
Image sync configuration file defines all the image synch rules. Each rule is a key/value pair, of which the key refers to "the source images url" and the value refers to "the destination images url". The source/destination images url is mostly the same with the url we use
in `docker pull/push` commands, but still something different in the "tags and digest" part:

"quay.io/coreos/kube-rbac-proxy": "quay.io/ruohe/kube-rbac-proxy",
"xxxx":"xxxxx",
"xxx/xxx/xx:tag1,tag2,tag3":"xxx/xxx/xx"
1. The source images url cannot be empty.
2. If the source images url contains no tags or digest, all the tags of source repository will be synced.
3. The source images url can have more than one tags, which should be seperated by comma, only the specified tags will be synced.
4. The source images url can at most one digest, and the destination url should only have no digest or the same digest at the same time.
5. If the destination url has no digest or tags, it means the source images will keep the same tags or digest after being synced.
6. The destination url can have more than one tags, the number of which must be the same with the tags in the source images url, then all the source images' tags will be changed to a new one (correspond from left to right).
7. If the destination url is empty, all the source images will be synced to the "default registry" (by command line parameter) and in the same repository with the source images url.

// If a source doesn't include tags, it means all the tags of this repository need to be synchronized,
// destination should not include tags at this moment.

// Each source can include more than one tags, which is split by comma (e.g., "a/b/c:1", "a/b/c:1,2,3").
You can find the example in [images.yaml](./example/images.yaml) and [images.json](./example/images.json), here we use [images.yaml](./example/images.yaml) for explaination:

// If a source includes just one tag (e.g., "a/b/c:1"), it means only one tag need to be synchronized;
// at this moment, if the destination doesn't include a tag, synchronized image will keep the same tag.

// When a source includes more than one tag (e.g., "a/b/c:1,2,3"), at this moment,
// the destination should not include tags, synchronized images will keep the original tags.
// e.g., "a/b/c:1,2,3":"x/y/z".

// When a destination is an empty string, source will be synchronized to "default-registry/default-namespace"
// with the same repository name and tags, default-registry and default-namespace can be set by both parameters
// and environment variable.
}
```yaml
quay.io/coreos/kube-rbac-proxy: quay.io/ruohe/kube-rbac-proxy
quay.io/coreos/kube-rbac-proxy:v1.0: quay.io/ruohe/kube-rbac-proxy
quay.io/coreos/kube-rbac-proxy:v1.0,v2.0: quay.io/ruohe/kube-rbac-proxy
quay.io/coreos/kube-rbac-proxy@sha256:14b267eb38aa85fd12d0e168fffa2d8a6187ac53a14a0212b0d4fce8d729598c: quay.io/ruohe/kube-rbac-proxy
```

### Parameters

```
-h --help usage information
-h --help Usage information

--config set the path of config file, this file need to be created before starting synchronization, default
--config Set the path of config file, this file need to be created before starting synchronization, default
config file is at "current/working/directory/config.json". (This flag can be replaced with flag --auth
and --images which for better orgnization.)

--auth set the path of authentication file, this file need to be created before starting synchronization, default
--auth Set the path of authentication file, this file need to be created before starting synchronization, default
config file is at "current/working/directory/auth.json". This flag need to be pair used with --images.

--images set the path of image rules file, this file need to be created before starting synchronization, default
--images Set the path of image rules file, this file need to be created before starting synchronization, default
config file is at "current/working/directory/images.json". This flag need to be pair used with --auth.

--log set the path of log file, logs will be printed to Stderr by default
--log Set the path of log file, logs will be printed to Stderr by default

--namespace set default-namespace, default-namespace can also be set by environment variable "DEFAULT_NAMESPACE",
if they are both set at the same time, "DEFAULT_NAMESPACE" will not work at this synchronization,
default-namespace will work only if default-registry is not empty.
--registry Set default-registry, default-registry can also be set by environment variable "DEFAULT_REGISTRY",
if they are both set at the same time, "DEFAULT_REGISTRY" will not work at this synchronization.

--registry set default-registry, default-registry can also be set by environment variable "DEFAULT_REGISTRY",
if they are both set at the same time, "DEFAULT_REGISTRY" will not work at this synchronization,
default-registry will work only if default-namespace is not empty.
--proc Number of goroutines, default value is 5

--proc number of goroutines, default value is 5
--retries Times to retry failed tasks, default value is 2, the retries of failed tasks will start after all the tasks
are executed once, this can resolve most occasional network problems during synchronization

--records image-syncer will record the information of synchronized image blobs to a disk file, this parameter will
set the path of the records file, default path is "current/working/directory/records", a records file can be
reused to make incremental synchronization if it is really generated by yourself. image-syncer remove the
dependence of records file after v1.1.0
--os OS list to filter source tags, not works for docker v2 schema1 media, takes no effect if empty

--retries number of retries, default value is 2, the retries of failed sync tasks will start after all sync tasks
are executed once, reties of failed sync tasks will resolve most occasional network problems during
synchronization
--arch Architecture list to filter source tags, takes no effect if empty

--os os list to filter source tags, not works for docker v2 schema1 media, takes no effect if empty
--arch architecture list to filter source tags, takes no effect if empty
--force Force update manifest whether the destination manifest exists
```
### FAQs
Expand Down

0 comments on commit 5057d6e

Please sign in to comment.