3.10

Highlights

• Global (cluster) config meta-version 2
• Bucket metadata (BMD) meta-version 2
• Python SDK
• Build: Go 1.18; upgrade all packages
• MessagePack archiving

Global Config v2 & Bucket Metadata v2

• bump up cluster config's meta-version; handle backward compatibility !5122, !5123, !5124
• redefine and extend feature flags !5132
• capacity and space watermarks (cleanup, low/high, OOS) !5143, !5144
• BMD meta-v2 with backward-compatible logic to load v1 !5145, !5150
• BMD meta-v2: simplify mirroring configuration !5151
• add transport section !5153
• support loading previous meta-version !5154
• readonly knobs; backward compatibility !5155
• introduce cos.size type and "kb", "mb", etc. suffixes !5156
• cos.size vs backward compatibility !5157
• add memsys section and tunables !5158
• log flush time interval !5159
• join-cluster-at-startup timeout !5161
• add log-stats time, add validators !5171
• continued refactoring !5172
• add tcb (transform-and-copy-bucket) section; stream bundle multiplier; compression !5173
• revise docs/configuration.md !5180

msgpack archiving; list, create, and read msgpack formatted shards

• msgp (generic) shard !5100, !5101
• fix preambles in the msgp generated files !5086
• alternative msgp; sgl as io.ByteScanner; Python unpack to test !5103
• assorted text fixes; more msgpack !5104
• archiving formats: add msgpack (in addition to tar, tgz, zip) !5107, !5108
• add integration tests !5100, !5101

Python SDK

• basic bucket and object operations !5109
• introduce pydantic; refactor list-buckets !5120
• CI: add Python SDK test !5112
• improve list-objects API !5163
• refactor API calls for usability: !5128
• move direct requests call out of list objects, create and delete bucket, cluster map !5130
• put-object & head-bucket API !5141
• improve list object API !5163
• fix return type in _request_raw method !5164
• add head-object method !5166
• add setUp and tearDown in tests !5167
• improve creating temporary file !5168
• add evict-bucket and delete-object; add props argument for list-objects !5175
• get-object returns a stream instead of the entire content !5184
• update README, docs !5178, !5190
• fix naming; add read-all method for streams !5196
• list_object to support pagination !5198
• improve object streaming; rename msg.py -> types.py !5203
• Python SDK tutorial !5218

CLI

• show cluster config feature flags in human-readable format !5133
• fix setting backend bucket props !5152
• add ais object rm --all to remove all in one shot !5195
• verbose and non-verbose multi-object operations !5197

Tests

• re-enable smoke-test for Cloud buckets !5098
• msgp (generic) shard !5100, !5101
• add missing wait for put-copies; BMD string !5204
• failback; devtools: wait-cluster; transport !5209
• dummy keepalive tracker !5211

Docs

• website: tech blog on promoting local and shared files !5082
• explain msgp usage; add msgp-update to Makefile !5085
• update msgp generator usage !5094
• msgpack !5110
• revise and amend configuration.md !5180
• performance tuning !5115, !5116
• development, first-time usage, production deployments (summary) !5127
• getting started (recommendations) !5129
• local playground, tracking access time (more edits) !5139
• fix references, add a reference !5140
• erasure coding (space utilization ratio, IO performance) !5142

Build

• upgrade fasthttp package !5092
• upgrade direct dependencies !5096, !5097
• transition to Go 1.18 !5106
• upgrade all dependencies except 1.18 !5113
• upgrade all dependencies except 1.18; v3.10-rc2 !5181
• upgrade all dependencies including 1.18; v3.10-rc3 !5186
• upgrade all minors !5200

Bug fixes and performance improvements; continuous refactoring

• skip bucket name, etc. validation when initializing lom !5083
• follow-up !5084
• unsafe string: inline and use it consistently !5087
• write-msg-pack (ref) !5088
• control-plane broadcast & unicast: result value factories and decoders !5089, !5090, !5091, !5093, !5095
• global rebalance: dynamic reg/unreg streams; housekeeper name suffix !5099
• fix sending http request body on redirect: recompute content-length !5102
• not restoring atime when !5105
• multi-object operations !5111
• local from-scratch deployment: add user-friendly tips !5114
• follow-up !5117, !5118
• write-policy: both metadata and data; bump up config's meta-version; backward compatibility (major) !5121
• deploy: fix Dockerfiles and related scripts !5126
• assorted usability tweaks !5134
• local playground: print ports AIS listens on !5135
• remove bench/soaktest; rewrite devtools/readme !5136
• config: write policy for data must be 'immediate' !5137
• docker + LVM deployment: no disks fix !5146
• move mock-bmd => cluster/mock (refactoring) !5147
• docker + LVM deployment: no disks !5148
• local playground: pgrep to check; quiet run-cmd !5160, !5162
• fix deploy.sh script for older bash versions !5165
• target startup: skip docker union mounts when enumerating drives !5169, !5170
• dedup fix HEAD(obj) system attributes !5174
• get-object lock/unlock (refactoring) !5176
• URI parsing, config validations, and assorted ref !5177, !5179
• making "rlock" exception to forcefully remove corrupted !5182
• dont-lookup-remote-bucket = true: proceed silently, log-wise !5183, !5185
• remove +build directive (obsolete) !5188, !5189
• get random proxy (ref) !5191
• cluster map (ref); node equality !5192
• PUT object vs remote backend returning status=503 !5193
• multi-object operation, range template (ref) !5194
• aisloader works !5199
• when shutting down warn with details !5201
• api: xaction wait-IC min waiting time !5202
• compute-checksum and clone-md (ref) !5205
• memsys: mem-free to include buff/cache !5206
• memsys: revise sys/mem (major) !5207
• lif to lom conversion: free on error; EC error handling !5208
• revise xs/rename-bucket !5210
• deferred unlock via lightweight lif (ref) !5212
• lom as a reader, lif as unlocker !5213
• get-from-neighbor (gfn): reg hk housekeepr under lock; abort to self-expire !5215
• lom copy: allocate with name !5216
• lom xattr (ref) !5217