Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

V3 export import #3

Open
wants to merge 13 commits into
base: main
Choose a base branch
from

Conversation

Moonlight63
Copy link

JSON v3 Format

The v3 format is a complete rewrite that focuses on properly handling draft/published content in Strapi 5:

Key Features

  • Draft/Published Support: Properly handles both draft and published versions of content
  • Locale Support (Experimental): Basic support for localized content - use with caution as this feature is still experimental
  • Smarter Relations: More efficient relation handling with configurable max depth
  • Document ID Tracking: Uses Strapi's document ID system for more reliable content tracking
  • Improved Media Handling: Better handling of media files with absolute URLs
  • Bulk Export: Export button appears in Strapi's bulk actions menu when entries are selected

Export Options

  • Export Relations: Include related content in the export
    • When enabled, shows additional options:
      • Deep Populate Relations: Include relations of related content
      • Deep Populate Component Relations: Include relations within components
      • Max Depth: Control how deep to traverse relations (1-20 levels)
  • Export All Locales (Experimental): Include all localized versions of content - not recommended for production use
  • Apply Filters and Sort: Use current view's filters and sorting in export

Import Options

  • Existing Entry Action: Choose how to handle existing entries
    • Warn: Show warning if the entry exists
    • Update: Update existing entries
    • Skip: Skip existing entries
  • Allow New Locales on Skip (Experimental): When skipping existing entries, still allow creation of new locales
  • Ignore Missing Relations: Continue import even if related entries are not found
  • Prevent Relation Changes: When skipping existing entries, prevent changes to their relations

ID Field Configuration

The v3 format uses a unique identifier field to track and match entries during import/export. This field must be both required and unique in your schema.

Setting Custom ID Field

You can configure which field to use as the identifier in your schema's plugin options:

// In your schema configuration:
{
  pluginOptions: {
    'strapi-import-export': {
      idField: 'customField'  // The field to use as identifier
    }
  },
  attributes: {
    customField: {
      type: 'string',
      required: true,
      unique: true
    }
    // ... other attributes
  }
}

Automatic ID Field Selection

If no idField is configured, the plugin will automatically look for fields in this order:

  1. uid - Typically a UUID from a custom field like Advanced-UUID
  2. name - If the content type is a single entry type, this is a good default
  3. title - Another common identifier field
  4. id - Falls back to Strapi's internal ID as last resort, but will not be exported with the data

Note: The selected field must be configured as both required: true and unique: true in your schema. The plugin will validate this and throw an error if the field is not properly configured.

How It Works

Bulk Actions

The plugin adds an "Export" button to Strapi's bulk actions menu. This appears when you select one or more entries in the content manager:

  • Select entries using the checkboxes in the list view
  • Click the "Export" button in the bulk actions menu
  • Choose your export options in the modal
  • Only the selected entries will be exported

Export Process

  1. Initial Export

    • Fetches all draft entries for the selected content type
    • For each draft, finds its corresponding published version
    • Groups content by locale (if localization is enabled)
    • Processes all relations, media, and components
  2. Relation Processing

    • When Export Relations is enabled:
      • Tracks all relations encountered during export
      • After initial export, processes each related content type
      • Continues until max depth is reached or no new relations found
    • Relations are stored using their identifier field value
  3. Media Handling

    • Media URLs are converted to absolute URLs using your server's public hostname (if the option is set in your plugin options).
    • Only the URL and metadata (name, alt text, caption, etc.) is exported, not the actual file

Import Process

  1. Validation

    • Checks that all content types exist
    • Validates identifier fields are properly configured
    • Ensures required fields are present
  2. Entry Processing

    • First processes published versions, then drafts
    • For each entry:
      1. Looks for existing entry using identifier field
      2. Handles according to Existing Entry Action setting
      3. Processes all relations, media, and components
  3. Media Import

    • For each media field:
      1. Existing Media Check:
        • Looks for existing media with the same hash or filename
        • If found, reuses the existing media entry
      2. New Media Import:
        • If no match found, downloads the file from the URL
        • Creates a new media entry with the downloaded file
        • Preserves metadata like alt text and captions
      3. Fallback Behavior:
        • If download fails, logs a warning but continues import
        • The field will be set to null if media cannot be resolved

Note: The import process is transactional per entry - if an entry fails to import, other entries will still be processed.

Troubleshooting

Common Issues

  • Media Import Fails: Check that your serverPublicHostname is configured correctly
  • Relations Not Found: Ensure related content is included in the export or already exists in target system
  • ID Field Errors: Verify your schema has properly configured unique identifier fields
  • Locale Issues: Make sure both source and target systems have the same locales configured

Import Validation Errors

The plugin performs several validations before import:

  • Schema existence and compatibility
  • Required fields presence
  • ID field configuration
  • Media file accessibility

Best Practices

  • Always test imports on a staging environment first
  • Use meaningful identifier fields (avoid relying on internal IDs)
  • Keep relation depth reasonable (3-5 levels max) to avoid performance issues
  • Back up your database before large imports
  • For large datasets, consider splitting the export into smaller chunks
  • Monitor the logs during import for any warnings or errors

…t controllers

- Improved error handling in ExportModal and error utility functions to better manage FetchError types.
- Updated Editor component to include TypeScript types for props.
- Removed deprecated models.js utility file and adjusted imports across services and controllers.
- Cleaned up error messages in export controllers for better clarity.
- Standardized import statements for consistency.
- Updated ExportModal and ImportModal components to include the exportAllLocales option.
- Modified export and import controllers to handle the new exportAllLocales flag.
- Enhanced data processing in export services to support exporting all locales.
- Improved handling of localizations in import services to align with the new exportAllLocales feature.
- Refactored related functions for better clarity and maintainability.
- Added error handling for import processes, including detailed error reporting in the ImportModal.
- Updated ImportModal component to display errors and failures using new UI elements like Accordion and Tabs.
- Enhanced server-side import logic to capture and return errors during data parsing and processing.
- Improved translation strings for error messages in the import process.
- Refactored import services to support structured error reporting and validation of file content.
…provements

- Added checkboxes for 'updateExisting' and 'ignoreMissingRelations' in the ImportEditor component to allow users to customize import behavior.
- Updated the ImportModal to reset upload state upon data source reset.
- Enhanced server-side import logic to handle new options and improve validation of imported data.
- Refactored validation functions to support new options and ensure better error reporting.
- Improved logging for debugging purposes during import processes.
…ed validation

- Added new export options in ExportModal for 'exportRelations', 'deepPopulateRelations', and 'deepPopulateComponentRelations'.
- Updated ImportEditor to include 'existingAction' options for handling existing entries during import.
- Enhanced server-side export and import logic to support new options and improve data processing.
- Refactored validation functions to accommodate new import options and ensure better error reporting.
- Improved handling of relations in export services to ensure accurate data processing and integrity.
…essing

- Introduced new import options: 'allowLocaleUpdates' and 'disallowNewRelations' to provide more control over import behavior.
- Refactored ImportContext and ImportProcessor classes to better manage processed records and improve error handling.
- Enhanced validation and processing logic to accommodate new options, ensuring robust data integrity during imports.
- Improved logging for better debugging and tracking of import processes.
- Updated interfaces for ImportFailure and ImportError to maintain consistency and clarity in error reporting.
…rovements

- Updated ExportModal and ImportModal components to include new options for handling relations and export formats.
- Refactored shouldShowOption logic to accommodate new conditions for displaying options based on selected formats.
- Enhanced ImportEditor to support versioning and improved error handling during data import.
- Improved user interface with hints and structured layouts for better user experience.
- Added new translation strings for enhanced clarity in export/import options.
- Cleaned up and optimized import/export processing logic for better performance and maintainability.
…on details

- Enhanced README.md with comprehensive plugin configuration instructions, key features of the v3 format, and detailed import/export processes.
- Added troubleshooting tips and best practices for using the plugin effectively.
- Updated About component to reflect changes in Strapi 5 versions and provide links to original repositories.
- Removed outdated references and improved clarity in documentation for better user guidance.
…ed error handling

- Added new parameters to export and import controllers, including 'documentIds', 'exportAllLocales', 'exportRelations', and 'deepPopulateRelations' for more granular control over data processing.
- Updated exportData and importData functions to accommodate new options and improve data handling.
- Enhanced error handling during data import to provide clearer feedback on failures and issues.
- Refactored validation logic to ensure robust processing and integrity of imported/exported data.
…ecide to do a pretty big refactor and commit and push without testing to make sure you didn't forget anything? Yeah, that was me. I think I fixed it now.
- Updated validation for ID fields to ensure 'uid' type attributes are handled correctly, requiring them to be optional.
- Refactored error messages for clarity, specifying requirements based on attribute types.
- Integrated the validateIdField function into the import validation process to centralize ID field checks and enhance consistency across services.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant