Merge pull request #390 from rusq/doc-2

Main doc update

README.md

@@ -7,6 +7,7 @@ files and emojis.  Generate Slack Export without admin privileges.
 
 **Quick links**:
 
+- [Installation And Quickstart](#installation-and-quickstart)
 - [Join the discussion in Telegram](https://t.me/slackdump).
 - [Buy me a cup of tea](https://ko-fi.com/rusq_), or use **Github Sponsors**
   button on the top of the page.
@@ -32,24 +33,19 @@ Typical use scenarios:
 * create a Slack Export archive without admin access, or
 * save your favourite emojis.
 
-There are four modes of operation (more on this in [User Guide][ug]):
+There are several modes of operation
 
 1. List users/channels
 1. Dumping messages and threads
 1. Creating a Slack Export in Mattermost or Standard modes.
+1. Creating an Archive
 1. Emoji download mode.
+1. Viewing export, dump or archive files or directories (displays images).
+1. Search mode (messages and files).
 
-Slackdump accepts two types of input (see [Dumping
-Conversations][usage-channels] section):
+Run `slackdump help` to see all available options:
 
-1. the URL/link of the channel or thread, OR
-1. the ID of the channel.
-
-[ug]: doc/README.rst
-[usage-channels]: doc/usage-channels.rst
-
-Quick Start
-===========
+## Installation and Quickstart
 
 On macOS, you can install Slackdump with Homebrew::
 
@@ -64,12 +60,10 @@ On other Operating Systems, please follow these steps:
 1. Run the `./slackdump` or `slackdump.exe` executable (see note below).
 1. You know the drill:  use arrow keys to select the menu item, and Enter (or
    Return) to confirm.
+1. Follow these [qickstart instructions][man-quickstart].
 
 [releases]: https://github.com/rusq/slackdump/releases/
 
-By default, Slackdump uses the EZ-Login 3000 automatic login, and interactive
-mode.
-
 > [!NOTE] 
 > On Windows and macOS you may be presented with "Unknown developer" window,
 > this is fine.  Reason for this is that the executable hasn't been signed by
@@ -85,6 +79,18 @@ mode.
     "Unknown developer" window, then go to System Preferences -> Security and
     Privacy -> General, and press "Open Anyway" button.
 
+### Getting Help
+
+- Quickstart guide: `slackdump help quickstart`, read [online][man-quickstart].
+- Generic command overview: `man ./slackdump.1`
+- [Ez-Login 3000](https://github.com/rusq/slackdump/wiki/EZ-Login-3000) Guide.
+- V2 to V3 migration notes: `slackdump help v2migrate`, read [online][man-v2migrate].
+- What's new in V3: `slackdump help whatsnew`, read [online][man-changelog].
+
+[man-quickstart]: https://github.com/rusq/slackdump/blob/master/cmd/slackdump/internal/man/assets/quickstart.md
+[man-v2migrate]: https://github.com/rusq/slackdump/blob/master/cmd/slackdump/internal/man/assets/v2migr.md
+[man-changelog]: https://github.com/rusq/slackdump/blob/master/cmd/slackdump/internal/man/assets/changelog.md
+
 
 ## Slackord2: Migrating to Discord
 
@@ -94,7 +100,8 @@ nice GUI, that is compatible with the export files generated by Slackdump.
 
 ## User Guide
 
-For more advanced features and instructions, please see the [User Guide][ug].
+For more advanced features and instructions, please see the [User Guide][ug],
+and read `slackdump help` pages.
 
 # Previewing Results
 
@@ -183,15 +190,9 @@ No, you don't.  Just run the application and EZ-Login 3000 will take
 care of the authentication or, alternatively, grab that token and
 cookie from the browser Slack session.  See [User's Guide][ug].
 
-
-
 #### I'm getting "invalid_auth" error
 
-Go get the new Cookie from the browser and Token as well.
-
-#### Slackdump takes a very long time to cache users
-
-Disable the user cache with `-no-user-cache` flag.
+Run `slackdump workspace new <name or url>` to reauthenticate.
 
 #### How to read the export file?
 

cmd/slackdump/internal/apiconfig/check.go

@@ -9,6 +9,7 @@ import (
 	"github.com/charmbracelet/bubbles/viewport"
 	tea "github.com/charmbracelet/bubbletea"
 	"github.com/charmbracelet/lipgloss"
+
 	"github.com/rusq/slackdump/v3/cmd/slackdump/internal/golang/base"
 	"github.com/rusq/slackdump/v3/cmd/slackdump/internal/ui"
 	"github.com/rusq/slackdump/v3/cmd/slackdump/internal/ui/bubbles/filemgr"
@@ -18,16 +19,14 @@ var CmdConfigCheck = &base.Command{
 	UsageLine: "slackdump config check",
 	Short:     "validate the existing config for errors",
 	Long: `
-# Config Check Command
+# Check Command
 
-Allows to check the config for errors and invalid values.
+Validates the configuration file for errors and invalid values.
 
 Example:
 
     slackdump config check myconfig.toml
 
-It will check for duplicate and unknown keys, and also ensure that values are
-within the allowed boundaries.
 `,
 }
 

cmd/slackdump/internal/apiconfig/new.go

@@ -18,22 +18,21 @@ var CmdConfigNew = &base.Command{
 	UsageLine: "slackdump config new",
 	Short:     "creates a new API config with the default values",
 	Long: `
-# Config New Command
+# "New" Command
 
 Creates a new API configuration file containing default values. You will need
 to specify the filename, for example:
 
     slackdump config new myconfig.toml
 
-If the extension is omitted, ".toml" is automatically appended to the filename.
+If the extension is omitted, ".toml" is automatically appended to the name of
+the file.
 `,
 	FlagMask:   cfg.OmitAll,
 	PrintFlags: true,
 }
 
-var (
-	fNewOverride = CmdConfigNew.Flag.Bool("y", false, "confirm the overwrite of the existing config")
-)
+var fNewOverride = CmdConfigNew.Flag.Bool("y", false, "confirm the overwrite of the existing config")
 
 func init() {
 	CmdConfigNew.Run = runConfigNew

cmd/slackdump/internal/archive/assets/archive.md

@@ -1,10 +1,21 @@
-# Command: archive
+# Archive Command
 
 The `archive` command saves your Slack workspace as a directory of files. By
 default, it archives the entire workspace that your user can access. You can
 customize the archive to include specific channels, groups, or direct messages
 by providing their URLs or IDs.
 
+The benefits of using "Archive" over "Export" and "Dump" are:
+- it is well documented (see `slackdump help chunk`);
+- it can be converted to other formats, including the native Slack Export
+  format (see `slackdump help convert`);
+- it is easier to parse with tools like `jq` or `grep`;
+- it is more convenient to build your own tools around it;
+- it is used internally by Slackdump to generate Slack Export and dump files,
+  so it can be seen as "master" format for the data;
+- It is natively supported by `slackdump view` command, everything else uses
+  an adapter.
+
 ## Features
 
 ### Default Behaviour
@@ -29,6 +40,8 @@ output includes:
 - **`CXXXXXXX.json.gz`**: Messages from a channel or group, where `XXXXXXX` is
   the channel ID.
 - **`DXXXXXXX.json.gz`**: Direct messages, where `XXXXXXX` is the user ID.
+- **`__uploads/`**: A directory containing files attached to messages that were
+  downloaded, if the file download is enabled.
 
 ### File Format
 - Files are saved as **JSONL** (newline-delimited JSON) and compressed with

cmd/slackdump/internal/archive/assets/search.md

@@ -1,4 +1,4 @@
-# Command: search
+# Search Command
 
 The `search` command allows you to search and export messages or files from a
 Slack workspace based on specified query terms. This command supports searching

cmd/slackdump/internal/convertcmd/assets/convert.md

@@ -0,0 +1,10 @@
+# Convert Command
+
+Convert Slackdump Archive format to other supported formats.
+
+By default it converts a directory with Slackdump archive to a ZIP-archive or
+directory in Slack Export format.
+
+Reference:
+- Slackdump archive "chunk" format: `slackdump help chunk`
+- `slackdump help archive`

cmd/slackdump/internal/convertcmd/convert.go

@@ -2,28 +2,28 @@ package convertcmd
 
 import (
 	"context"
+	_ "embed"
 	"errors"
 	"time"
 
 	"github.com/rusq/fsadapter"
+
 	"github.com/rusq/slackdump/v3/cmd/slackdump/internal/cfg"
 	"github.com/rusq/slackdump/v3/cmd/slackdump/internal/golang/base"
 	"github.com/rusq/slackdump/v3/internal/chunk"
 	"github.com/rusq/slackdump/v3/internal/chunk/transform/fileproc"
 	"github.com/rusq/slackdump/v3/internal/convert"
 )
 
+//go:embed assets/convert.md
+var convertMd string
+
 var CmdConvert = &base.Command{
 	Run:       runConvert,
 	UsageLine: "slackdump convert [flags] <source>",
 	Short:     "convert slackdump chunks to various formats",
 	Long: `
-# Convert Command
-
-Convert slackdump Chunks (output of "record") to various formats.
 
-By default it converts a directory with chunks to an archive or directory
-in Slack Export format.
 `,
 	CustomFlags: false,
 	FlagMask:    cfg.OmitAll & ^cfg.OmitDownloadFlag &^ cfg.OmitOutputFlag,

cmd/slackdump/internal/export/assets/export.md

@@ -1,4 +1,4 @@
-# Command: export
+# Export Command
 
 The `export` command saves your Slack workspace as a directory of files.
 By default, it exports the entire workspace that your user can access.
@@ -8,6 +8,9 @@ direct messages by providing their URLs or IDs.
 The ZIP file it generates is compatible with the Slack Export format with
 Slackdump specific extensions.
 
+__NOTE__: If you have an official Slack workspace export file, you can
+hydrate it with files by running `slackdump tools hydrate <export_file>`.
+
 The export file is understood by Slack Import feature with the following
 caveat:
 - files will not be imported, unless the `export` token is specified.

cmd/slackdump/internal/format/format.go

@@ -14,6 +14,7 @@ import (
 	"runtime/trace"
 
 	"github.com/rusq/slack"
+
 	"github.com/rusq/slackdump/v3/cmd/slackdump/internal/bootstrap"
 
 	"github.com/rusq/slackdump/v3/cmd/slackdump/internal/cfg"
@@ -23,17 +24,27 @@ import (
 	"github.com/rusq/slackdump/v3/types"
 )
 
+// TODO this is hacky in the following ways:
+// 1. User must extract the JSON file from the archive
+// 2. What about exports etc.?
+// 3. Getting users online is hacky, as it requires authentication to be present,
+//    but if the user doesn't need online users.  The login should happen locally.
+
 var CmdFormat = &base.Command{
 	Run:       runFormat,
 	UsageLine: "slackdump format [flags] <format> <file.json>",
-	Short:     "converts the slackdump files to a human readable format",
+	Short:     "converts the slackdump dump files to a human readable format",
 	Long: `
 # Format Command
+
+Format command formats the json files generated by "slackdump dump" command to
+a human readable format.  The command takes the format type and the file to
+convert as arguments.
 `, // TODO: add more info
 	CustomFlags: false,
 	FlagMask:    cfg.OmitAll & ^cfg.OmitWorkspaceFlag,
 	PrintFlags:  true,
-	RequireAuth: false,
+	RequireAuth: true,
 }
 
 //go:generate stringer -type dumptype -trimprefix=dt
@@ -155,7 +166,7 @@ type dump struct {
 // member variables populated.  If it fails to detect the type it will return
 // ErrUnknown and set the dump filetype to dtUnknown.
 func detectAndRead(rs io.ReadSeeker) (*dump, error) {
-	var d = new(dump)
+	d := new(dump)
 
 	if conv, err := unmarshal[types.Conversation](rs); err != nil && !isJSONTypeErr(err) {
 		return nil, err
@@ -221,7 +232,7 @@ func unmarshal[OUT any](rs io.ReadSeeker) (OUT, error) {
 
 func getUsers(ctx context.Context, dmp *dump, isOnline bool) ([]slack.User, error) {
 	if isOnline {
-		return getUsersOnline(ctx, cfg.CacheDir(), cfg.Workspace, cfg.LegacyBrowser)
+		return getUsersOnline(ctx)
 	}
 	rgn := trace.StartRegion(ctx, "userIDs")
 	ids := dmp.userIDs()
@@ -237,7 +248,7 @@ func getUsers(ctx context.Context, dmp *dump, isOnline bool) ([]slack.User, erro
 	return uu, nil
 }
 
-func getUsersOnline(ctx context.Context, cacheDir, wsp string, usePlaywright bool) ([]slack.User, error) {
+func getUsersOnline(ctx context.Context) ([]slack.User, error) {
 	sess, err := bootstrap.SlackdumpSession(ctx)
 	if err != nil {
 		return nil, err
@@ -282,7 +293,7 @@ func matchUsers(r io.Reader, ids []string) ([]slack.User, error) {
 		return nil, err
 	}
 	fileIDs := uu.IndexByID()
-	var matchingCnt = 0 // matching users count
+	matchingCnt := 0 // matching users count
 	for _, id := range ids {
 		if fileIDs[id] != nil {
 			matchingCnt++

cmd/slackdump/internal/man/assets/changelog.md

@@ -1,10 +1,36 @@
-# v2.3.0
+# v3.0.0
+
+Gist:
+- 2.6x dump speed improvement on channels with threads;
+- Support for enteprise workspaces;
+- json logging on demand;
+- new structured CLI;
+- improved TUI for the wizardry with bells and whistles;
+- multiple workspaces with ability to switch between them without `-auth-reset`;
+- api limits configuration files;
+- uninstallation tool;
+- slackdump system information tool;
+- pgp encryption for traces (under tools);
+- search results archival;
+
+## New Archive format
+
+Consider using the new `archive` command to save your workspace data.  You can read about
+it in the `slackdump help archive` command and the format it produces in the
+`slackdump help chunk` command.
+
+## Viewer
+
+Slackdump V3 introduces a viewer for exported data.  To view the exported data, run:
+```
+slackdump view <export file or directory>
+```
+
+NOTE: search results are not supported by the viewer yet.
+
 
 ## Breaking changes
 
-- legacy command line interface moved under "v1" command, so if you need to
-  use the old CLI, instead of `./slackdump`, run `./slackdump v1`. The legacy
-  CLI will be phased out:  deprecated in v2.4.0, and removed in v2.5.0;
 - `-download` flag renamed to `-files` and is set to "true" by default;
 - `-r` flag that allowed to generate text files was replaced by
   `slackdump convert` command.
@@ -21,7 +47,7 @@
       `workspace select` command;
     - The "_Current_" workspace can be overridden by providing the `-w <name>`
       flag.
-- Slackdump `record` mode allows to dump the entire workspace into a directory
+- Slackdump `archive` mode allows to dump the entire workspace into a directory
   of chunk files.
 - Slackdump `convert` mode allows to convert chunk files into other formats,
   such as Slack export format, or Slackdump format.