Skip to content

Commit

Permalink
feat(core): support custom endpoint and addressing style for S3
Browse files Browse the repository at this point in the history
Add support for configurable S3 endpoint and addressing style (path-style/virtual-hosted)
to improve compatibility with S3-compatible storage services.

- Add forcePathStyle option to control URL addressing style
- Fix custom endpoint support implementation
- Improve URL generation logic for different configurations

Resolves: [#6920]
  • Loading branch information
aiden_lu committed Jan 7, 2025
1 parent 580ed25 commit bd58bc2
Show file tree
Hide file tree
Showing 4 changed files with 33 additions and 1 deletion.
5 changes: 5 additions & 0 deletions .changeset/afraid-otters-crash.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
"@logto/core": patch
---

support custom endpoint and addressing style for S3
3 changes: 2 additions & 1 deletion packages/core/src/utils/storage/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -19,14 +19,15 @@ export const buildUploadFile = (config: StorageProviderData): UploadFile | Uploa
return storage.uploadFile;
}

const { endpoint, bucket, accessKeyId, accessSecretKey, region } = config;
const { endpoint, bucket, accessKeyId, forcePathStyle, accessSecretKey, region } = config;

Check warning on line 22 in packages/core/src/utils/storage/index.ts

View check run for this annotation

Codecov / codecov/patch

packages/core/src/utils/storage/index.ts#L22

Added line #L22 was not covered by tests

const storage = buildS3Storage({
endpoint,
bucket,
accessKeyId,
secretAccessKey: accessSecretKey,
region,
forcePathStyle,

Check warning on line 30 in packages/core/src/utils/storage/index.ts

View check run for this annotation

Codecov / codecov/patch

packages/core/src/utils/storage/index.ts#L30

Added line #L30 was not covered by tests
});

return storage.uploadFile;
Expand Down
25 changes: 25 additions & 0 deletions packages/core/src/utils/storage/s3-storage.ts
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@ type BuildS3StorageParameters = {
secretAccessKey: string;
region?: string;
endpoint?: string;
forcePathStyle?: boolean;
};

export const buildS3Storage = ({
Expand All @@ -24,6 +25,7 @@ export const buildS3Storage = ({
secretAccessKey,
region,
endpoint,
forcePathStyle,

Check warning on line 28 in packages/core/src/utils/storage/s3-storage.ts

View check run for this annotation

Codecov / codecov/patch

packages/core/src/utils/storage/s3-storage.ts#L28

Added line #L28 was not covered by tests
}: BuildS3StorageParameters) => {
if (!region && !endpoint) {
throw new Error('Either region or endpoint must be provided');
Expand All @@ -35,6 +37,7 @@ export const buildS3Storage = ({
const client = new S3Client({
region: finalRegion,
endpoint,
forcePathStyle,

Check warning on line 40 in packages/core/src/utils/storage/s3-storage.ts

View check run for this annotation

Codecov / codecov/patch

packages/core/src/utils/storage/s3-storage.ts#L40

Added line #L40 was not covered by tests
credentials: {
accessKeyId,
secretAccessKey,
Expand All @@ -56,6 +59,28 @@ export const buildS3Storage = ({
return { url: `${publicUrl}/${objectKey}` };
}

if (endpoint) {
// Custom endpoint URL construction
if (forcePathStyle) {
// Path-style URL: https://endpoint/bucket/key
return {
url: `${endpoint}/${bucket}/${objectKey}`,
};
}
// Virtual-hosted style URL: https://bucket.endpoint/key
return {
url: `${endpoint.replace(/^(https?:\/\/)/, `$1${bucket}.`)}/${objectKey}`,
};
}

// AWS S3 standard URL construction
if (forcePathStyle) {
// Path-style URL: https://s3.region.amazonaws.com/bucket/key
return {
url: `https://s3.${finalRegion}.amazonaws.com/${bucket}/${objectKey}`,
};
}
// Virtual-hosted style URL: https://bucket.s3.region.amazonaws.com/key

Check warning on line 83 in packages/core/src/utils/storage/s3-storage.ts

View check run for this annotation

Codecov / codecov/patch

packages/core/src/utils/storage/s3-storage.ts#L62-L83

Added lines #L62 - L83 were not covered by tests
return {
url: `https://${bucket}.s3.${finalRegion}.amazonaws.com/${objectKey}`,
};
Expand Down
1 change: 1 addition & 0 deletions packages/schemas/src/types/system.ts
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,7 @@ export const storageProviderDataGuard = z.discriminatedUnion('provider', [
endpoint: z.string().optional(),
region: z.string().optional(),
bucket: z.string(),
forcePathStyle: z.boolean().optional(),
accessKeyId: z.string(),
accessSecretKey: z.string(),
...basicConfig,
Expand Down

0 comments on commit bd58bc2

Please sign in to comment.