Investigate ignoring directories without valid perms on Walk

[bufdev]

I had a temporary .proto file in my home directory, and ran buf build, and got:

Failure: failed to enumerate module files: open .Trash: operation not permitted

I likely do not want to run buf build from my home directory, as this will do a large search across my whole computer, but in general, some directories may have sub-directories that the current user does not have permissions for. There's a product decision to be made here, but we may want to provide an option to storage.ReadBucket.Walk to ignore directories that we don't have permissions for.

I do not think this classifies as a breaking change - any command that would now be valid would have failed before, so we're not breaking previous functionality, and I do not think it is reasonable for someone to rely on buf erroring on a non-permissioned directory to error out a workflow.

This may have implications for the BSR, but I think unlikely (we need to investigate however).

[bufdev]

I chose to test out a quick fix for this, itself which would have massive implications that we would need to carefully think about:

diff --git a/private/pkg/storage/storageos/bucket.go b/private/pkg/storage/storageos/bucket.go
index c07d7446..0eeb88d2 100644
--- a/private/pkg/storage/storageos/bucket.go
+++ b/private/pkg/storage/storageos/bucket.go
@@ -17,6 +17,7 @@ package storageos
 import (
        "context"
        "errors"
+       "io/fs"
        "os"
        "path/filepath"
        "strings"
@@ -122,6 +123,11 @@ func (b *bucket) Walk(
                                if b.symlinks && os.IsNotExist(err) {
                                        return nil
                                }
+                               // we choose in buf applications to not return error if we attempt
+                               // to walk a directory that we cannot read, so we just continue
+                               if fileInfo.IsDir() && errors.Is(err, fs.ErrPermission) {
+                                       return nil
+                               }
                                return err
                        }
                        if err := walkChecker.Check(ctx); err != nil {

I then made a new file a.proto in my home directory, and attempted buf build a.proto. This opened up another issue: the actual vs expected performance of ProtoFileRefs is very different. We end up returning an enclosing bucket when we have a ProtoFileRef, and then walk it so that we find any files that the single .proto file imports, but this walk also walks over all non-.proto files in its search for .proto files. Somewhat unavoidable with the current architecture of buf, but obviously in a large directory such as a home directory, this means that buf will be walking for a long time (I ended up ctrl+c'ing my invocation).