3.8

Highlights

• ETL: !4621, !4624, !4633, !4649, !4681, !4702, !4780, !4790, !4802, !4831, !4833
• storage cleanup: !4632, !4741, !4748, !4753, !4754
• custom user-defined object metadata, system Cloud-specific metadata: !4655, !4657, !4659, !4661, !4663, !4667, !4668, !4670, !4673
• reinforce and protect ais volume and associated metadata: !4683, !4684, !4698, !4699, !4701, !4703, !4813, !4814, !4834, !4837
• safely add and remove mountpaths (disks) at runtime !4721, !4722, !4728, !4729, !4735, !4736, !4740, !4744, !4789, !4818
• "easy URL": support gs/bucket/object, s3/bucket/object, and similar (easy) URLs across all backends !4711
• introduce node standby mode !4688, !4689, !4691
• performance monitoring with a scope node | batch job | cluster !4792, !4793, !4794, !4798, !4800, !4810, !4812
• support ais targets with no disks (feature) !4825
• Kubernetes Operator v0.9

CLI

• fix file name and message when extracting a file from an archive !4602
• templates: circumvent index wrap around for aliases !4603
• multi-object transform and copy: continue-on-err; add similar keywords !4605
• show object (and its variations): improve usability and fix minors !4674
• show config improvements; follow-up !4676
• add command to immediately unregister node from cluster map !4693
• new command ais storage cleanup !4704
• attach mountpath and detach mountpath (ref) !4705
• show storage subcommands !4713
• --wait option for storage cleanup command !4773
• report human-readable stats in show job xaction !4788
• add commands to manage AIS CLI configuration !4791
• improve alias command and docs !4795
• follow-up: introduce xaction snapshot (aka "snap") into CLI templates !4806
• add ais cluster show stats subcommand !4807
• refresh for stats !4811
• improve job xaction show command; minor refactoring !4815, !4816
• node IDs are very special completions (provide visual cue) !4823
• fix archive ls; set and show config usability; log_level !4824
• remove glog's vmodule (obsolete) !4826
• improve ais cluster rebalance command, allow ais archive ls to list entire bucket (of tarballs, etc.) !4828
• do not print hint if JSON format is on !4839
• introduce unknown-value and not-set-value, and use consistently; fix stop-maintenance logic !4842
• downloader: fix error handling, improve usability, !4844

Integration and unit testing; CI/CD

• add Go 1.17 shuffling for short tests !4615
• restore original primary in TestForwardCP !4619
• use new t.Setenv instead of os.Setenv and defer os.Unsetenv !4620
• CI: fix lint errors on MacOS !4622
• wait for xaction to finish when enabling mirroring !4628
• tests/cleanup: ignore "server closed idle connection" !4631
• fix failing TestGetAfterReregisterWithMissedBucketUpdate !4640
• dev-tools/tassert: timestamp failure message, customize print-stack !4647
• dev-tools and tests: continued refactoring and cleanup !4654, !4656
• CI: add minimal cluster job that runs with 1 proxy and 1 target !4643
• save/check cluster state; log: memory reporting !4677
• CI: skipping tests in short mode !4776
• test cluster config persistence and mountpath add/remove across restarts !4821
• massive read operation vs detach/attach mountpath (non-redundent bucket) !4822
• wait for mirroring, re-enable mountpaths !4835

Documentation & blog; website aiatscale.org

• transition to Go 1.17 (add a note) !4623
• remove empty reference to Python client !4636
• remote AIS cluster: examples and references !4665
• remote buckets (naming notation, CLI examples, cross-refs) !4669
• bucket property inheritance, LRU note; E2E tests: set-custom !4673
• configuration.md; transport/Rx: PDU header validation !4679
• website: AIStore & ETL: Using AIS/PyTorch connector to transform ImageNet (post #2) !4694
• remote ais cluster, global namespace !4697
• cleanup http_api docs and add missing json for list buckets operation !4715
• website: AIStore & ETL: Introduction (post #1) !4716
• update command-line usage !4727
• ais cluster show stats !4809
• update ais show job CLI !4817
• website: copying existing file datasets in two easy steps !4829
• update main v3.8 README !4830
• touch cluster.md; bump version to 3.8 !4838
• What's new in AIS v3.8

Build and toolchain; dependencies

• transition to Go 1.17 and use new Go features !4627
• make: add run and restart make targets !4707
• reinitialize go-modules and upgrade all minor versions !4724, !4726

Bug fixes and improvements

• multi-object copy/transform: handle errors; ref common (part two) !4601
• multi-object archive/copy/transform: aborting logic and cleanup !4606
• general: use Go 1.17 enhancements !4607
• health-checker: more efficient method to read directory entries !4608
• bucket copy/transform: aborting logic, cleanup, and test !4609
• transport/streams: fix broadcast in a single-target cluster !4610
• revise err-aborted, refactor common errors, downloader !4611
• cluster config vs bucket props vs LRU !4612
• erasure-coding: fix bucket-encoding xaction hang on errors and upon abort !4613
• deployment: add registry URL in docker build and docker push commands !4614
• add atomic counted err value !4616, !4617
• on-demand xaction: mutex to protect compound state !4618
• etl: add more details when K8s pod times out on being ready; add debug log !4621, !4624
• intra-cluster notifications: minor refactoring !4625
• xaction: add immutable (original) bucket !4626
• revise xaction aborting logic !4629
• lru pkg: trash non-existing buckets; health check: docs and follow-up ref-s !4630
• storage cleanup: remove artifacts of erasure-coding (unfinished slices, redundant replicas and metafiles) !4632
• etl: fix waiting for a ready condition when starting K8s pod !4633
• fix global-rebalance can-start & must-run helpers !4634, !4635
• log: add backend module; AWS: ignore "unknown region" most of the time !4637
• single-target cluster: support bucket renaming !4638
• node name vs 1) nodes joining the cluster and 2) early startup !4639
• xaction pkg: consistent naming and refactoring !4641
• refine health check: add 'ready to rebalance'; fix test to wait for cluster state !4642
• default bucket props (ref) !4644
• glog: set-node, reduce header; transport streams: add destination ID !4645
• copy configuration values for default bucket props !4646
• retriable connection errors; transport: do retry !4648
• etl: remove single-pod limitation (i.e., the capability to run multiple ETLs) !4649
• retriable connection errors: use the same condition consistently throughout !4650
• single-target cluster shutdown and assorted fixes !4651
• global rebalance: refine preemption logic, abort associated streams (major ref) !4652, !4653
• custom object metadata: set/replace vs add/update !4655
• consistent custom versioning, checksumming, ETag (major refactoring) !4657
• default buffer sizes (TCP, HTTP) !4658
• custom metadata: unify downloader; unify version comparison !4659, !4661
• downloader: make object comparison much more rigorous !4662
• object metadata: to/from header converters !4663
• unify the code that checks local/remote equality, change backend.HeadObj() API, simplify cold-GET !4664
• erasure-coding config and bucket props: add batch size back for compatibility !4666
• object metadata: system attributes (as opposed to user-defined) !4667
• object metadata: unset custom keys on PUT, resolve bucket copy vs inc-version dichotomy !4668
• unify object props and system object attributes (major revision) !4670
• get-bucket-props: remove redundant code, simplify !4671, !4672
• configuration: add transport-idle-teardown !4675
• memsys: tracking memory stats and responding to OOM (major rev) !4678
• xmeta (tool): add VMD, refactor !4680
• etl: add new (non-HTTP) communicator and stop relying on the K8s API Server !4681
• SizeBytes() and AtimeUnix() methods for CT !4682
• volume: revise, refactor, and reinforce operations on metadata !4683, !4684
• downloader: refactoring !4685
• cluster membership: shutdown, decommission, maintenance !4686
• target standby mode !4688, !4689, !4691
• dev scripts: deploy remote ais cluster with the same build tags !4690
• refactor and simplify admin-join/auto-join logic !4692
• follow-up: self-join, admin-join, immediate removal from the cluster map !4695
• phase out "unregister" & further consistent renames !4696
• add volume package (ref) !4698
• volume: consolidate loading and initialization; bootstrap in two passes !4699
• cmn/context => cmn/cos/context; rename "attach/detach remote ais" (minor ref) !4700
• volume: keep it sync with local config !4701
• etl: add more Python runtimes !4702
• volume: keep it sync with local configuration !4703
• api to attach/detach/enable/disable mountpaths: pass node ref consistently !4706
• make: add run and restart targets !4707
• error formatting (ref) !4708
• api package (major ref) !4709, !4710
• "easy URL": accommodate list-objects with no json msg !4711
• deploy script: fix getting the last parameter !4712
• intra-cluster reverse proxying vs content-length !4714
• improve error formatting and content !4717, !4718
• cluster/hrw and cmn/err: continued refactoring !4719
• graceful mountpath (disk) removal - major revision !4721, !4722
• fs: more validation and naming consistency !4725
• api to attach mountpath with force !4728
• transactional (begin -- commit) to remove mountpath, with committing after resilvering !4729
• lint and refactoring, housekeeping !4730
• follow-up: add wait-for-resilver, sort mountpaths; increase CI timeout !4732
• add/remove disks: make it transactional with resilvering in-between !4735, !4736, !4740, !4744
• refactor and split reb pkg, add res (resilver) pkg !4737
• revise GFN (state): simplify global, move/rewrite local !4738
• follow-up: move global GFN state to reb pkg !4739
• separate storage cleanup xaction from LRU one !4741
• xaction registry vs housekeeper: tweak initializations !4742
• housekeeper: run first & reinforce; x-factories: ordered initialization !4743
• cos pkg: make ExitLogf print file and line of the caller !4745
• housekeeper: add wait for housekeeper readiness when registering !4746
• fs: get and get-available; test-init; refactoring !4747
• storage cleanup (feature, major upgrade) !4748
• rename API action constants for consistency; tweak OOS logic !4749
• dsort: fix panic when creating shard !4750
• new pkg to combine both cleanup and eviction logic !4751
• memsys: reduce free-mem requirement for tests & minor fixes !4752
• storage cleanup: repackage, rewrite, and refactor !4753, !4754
• memsys for tests (follow-up) !4755
• parse request query only once (core) !4757
• create and lookup buckets on the fly (major rev) !4758
• memsys and housekeeper: further clarification !4759
• follow-up: do not lookup remote bucket; stopping hk !4760
• revise and refactor logic to handle deleted (formerly trash) !4761
• memsys vs mock target: more orderly initialization !4762
• move mock target out of production code !4763
• memsys: tune-up low-watermark !4764
• delete/undelete: retain bucket names when deleting !4766
• memsys: two distinct pairs of slab allocators !4767
• general: rename mountpoint to mountpath for consistency !4768
• memsys: tune-up for OOM !4769
• memsys: reintroduce max-depth !4770
• reduce glog and tweak panic-OOM condition !4771
• memsys: fix swapping state !4772, !4774
• fs: make mountpath flags atomic !4775
• xreg pkg: remove obsolete RebalanceArgs struct !4777
• ais pkg: split etl.go into tgtetl.go and prxetl.go for consistency !4778
• cmn pkg: sort actions alphabetically !4779
• etl: integrate etl communicator with xaction !4780
• set-name when joining nodes; keepalive logging; test tweaks !4781
• not aborting xactions anymore when adding/removing mountpaths !4782
• fs: refactor fqn parsing !4783
• fs: refactor fqn parsing, update unit tests and docs !4784, !4785
• log rotation: uniformity of the headers !4786
• fs: minor fixes; tests !4787
• add an option to remove mountpaths without resilvering !4789
• etl: manage etl metadata !4790
• refactor xaction stats: remove redundant interfaces, rename !4792
• xactions: introduce snapshot, add in and out counters !4793, !4794
• integrate: streams and target stats, rebalance and xaction snapshots !4796
• dsort: wait for all in-flight stream requests !4797
• data mover to support xaction stats; in, out, and locally processed !4798
• transport: complete transition to using opcodes !4799
• stats pkg: add get-daemon-stats API, remove redundant code and refactor !4800
• etl: fix panic in unit test !4801
• etl: change signature of InitSpec function to require options !4802
• memsys: tweak init-once logic yet again !4803
• consolidate all mock objects in cluster/mock !4804
• xactions: complete transition to snaps (ref) !4805
• dsort: fix placement of aborted check !4808
• fix streams received-bytes count; bump CLI and aisloader versions !4810
• remove logic to compute min/max k-alive latency; simplify & refactor !4812
• capability to recover lost or corrupted node ID at startup !4813, !4814
• detach/disable mountpath: actually wait for resilvering !4818
• uuid/gid: consolidate and refactor generation and validation !4819, !4820
• aistore with no disks (feature) !4825
• http provider: ETag check and prefetch prefix !4827
• etl metadata: minor fixes and ref !4831
• rebalance: simplify-out Tx semaphore !4832
• always re-encode etl metadata; improvements and ref !4833
• volume: when creating new volume (and associated VMD), make sure to have a record of configured fspaths !4834
• cmn pkg: add is-err-not-found !4836
• option to force starting up with a lost or missing mountpath !4837
• change one api constant to support rolling upgrade !4840
• fix ais advanced remove-from-smap command !4841
• minikube: RBAC v1 !4843