Export of components doesn't seem to work

[s-com-borchardtj]

🐛 Bug Report

The export for collection types seems to work fine with relations to other collection types but not for components.

Maybe it's because strapi 5 uses documentId instead of id for reference or this breaking change https://docs.strapi.io/dev-docs/migration/v4-to-v5/breaking-changes/no-shared-population-strategy-components-dynamic-zones ?

[GraemeFulton]

Thanks for that, do you know if this was the same behaviour on the previous v4 plugin?

Good point on the document id vs id. Not sure when I'll get chance to look at this

[s-com-borchardtj]

Thanks for your response! In another strapi 4 project the plugin worked just fine as far as I remember.

[khanhhood]

Hi @GraemeFulton Do you have any good news about the component supporting?

[GraemeFulton]

Hi I think I'll get chance to look at this just after christmas - if anyone else can have a go in the meantime, pull request is very welcome!

[Moonlight63]

I am currently also looking at the code because I cant seem to get relations to import. They export fine. But I have other high pri features I need to focus on before I can look at anything here.

[Moonlight63]

I started building a fork to fix some of the issues I was personally having. I am not ready for a pull request, but I wanted to post here for those waiting for an update. There are enough significant changes that I am putting everything under a new v3 export type, primarily because 1: I never liked the way that you had to set a deepness level for how you want relations to be exported, and 2: Why is the assumption that if you are exporting entities on one collection that you want to also export all relations with it?

So, list of some changes as of now:

Quality of life change: added an export button in the "bulk action" area, so you can choose to only export records that are selected in the list view.

Exports will now export both your published and draft version of a record. Draft will only be exported if it is different from the published version, and published will only be exported if one exists.

If you have localized versions of records, they can also optionally be exported (working on the UI for this now).
Limitation on this: because of the way things currently work, Localized records only get exported if there is a default version of the record. Additionally, if a default locale record exists in draft form only, and a published alternate locale exists, it still won't be exported because the default locale record is not published. I don't think this is a super huge deal as the times when record exists for an alternative locale but not the default should be relatively low, but it is still a limitation worth mentioning until I have the time to drastically change the logic of how records are gathered.

Exported entities are identified by the idField set in their schema, then fall back to a field labeled uid, then name, then title, and finally to standard id.
Rational: I don't know about anyone else, but I know that my primary use case for a plugin like this is first to mass import records to seed a new or even existing instance of an app, and also to transfer records from one instance to another (for example, export records from a production app to a beta staging app or local dev env). In cases like these, it is very rare that a record's ID in the backend strapi database would be the same on all instances, so I want the identifier to be something that I can actually control. My original plan was to let the default identifier be an auto-generated uuid that could be transferred between instances without affecting the strapi backend tracking, and this does work great (which is why uid is the next default I added), but I found out while working with the locale exports that the most popular and useful custom field for this very purpose, strapi-advanced-uuid, has a bug when you update alternate locales. (I might be rewriting that one next....)

Fixed alert styling.

Started converting more of the codebase to typescript.

On export, relations will only export the value of whatever your idField is, except for components, which are now not exported as relations, but rather as a sub array. I need to do more testing on dynamic zones though.
Interesting fact here about components. When importing new records, everything is fine, but components do get their own unique IDs just like regular records do as relations. That is why they don't work correctly in the v2 export... because they act like relations. The problem is that you can't query for "all" relations the way you can with other records, because strapi doesn't treat them like other records. It's not really a problem, but one of the side effects of this is that when you update a record by reimporting it, if you include the component array, it actually replaces the components with new IDs, instead of updating them. I would like to find out if this is able to be done differently for cleanliness of the backend, but functionally, it shouldn't matter.

Importing is smart enough to check if the value of idField changed between a published version and a draft version, but because I am not using the backend id, or documentId of a record, if you export a record, change the record's idField (name for example), then try to import the record from the old export, it will create a new record, because no record with that name currently exists.

TODO:
The import export of all collections from the plugin page is currently broken, but I will be fixing that.

I want to reimplement a feature to export relations like it was before, but slightly different.

Adding more options to control how things happen in the gui.

I am going to add an intermediate step to importing data, so when you import, it will scan the data, then report back what it wants to do (i.e.: Create , Update , This record has errors, here they are <err1, err2>, it will be skipped, This record has a required relation to , but that related object doesn't exist, skipping import, etc). Then the user can comb through, make sure everything looks good and continue, or make changes.

I didn't really intend on switch gears to working on something this big. I just need it to work. I have a different project I am trying to launch, and the only reason I dove into this was to mass import some data haha. If anyone wants to help me in this endeavor, by all means, pitch in.

Here is my fork

My goal is to get these changes into a good stable state where everything works and I'm not worried about data corruption (or warn the user if it is likely before they hit import) and then submit the pull request, but as you can see, there are still some things to work out.

[GraemeFulton]

@Moonlight63 thanks for this, I think your changes make a lot of improvements for me too:

1: I never liked the way that you had to set a deepness level for how you want relations to be exported, and 2: Why is the assumption that if you are exporting entities on one collection that you want to also export all relations with it?

I never understood the deepness level, and found if i set the deepness value too high, it just fails. The v3 export type sounds great for me then. Maybe we can get rid of this deepness level version and just use yours as v1

Exports will now export both your published and draft version of a record. Draft will only be exported if it is different from the published version, and published will only be exported if one exists.

That's great, it's a good upgrade to match Strapi 5 features

Exported entities are identified by the idField set in their schema, then fall back to a field labeled uid, then name, then title, and finally to standard id.

I think I had the same issues you explain in your rational for this - I just wanted to duplicate my production posts and keep relations in tact, but I could never do it with the current version

There are a lot of changes, I think it helps everyone as the current version has many issues that you address - I'll give it a test when you're ready. Thanks again, really appreciate your efforts

[Moonlight63]

For todays update... I think I have the import validation pretty well figured out. Here is a diagram for anyone to nitpick :cough: offer suggestions.

graph TD
    A[validateFileContent] --> B[Basic File Validation]
    A --> C[validateContentTypes]
    
    C --> D[For each content type]
    D --> E[validateModelConfiguration]
    D --> F[validateContentTypeEntries]
    
    F --> G[For each entry]
    G --> H[For each version]
    H --> I[For each locale]
    
    %% Structure Validation Branch
    I --> J[validateStructure]
    J --> J1[validateUnknownFields]
    J1 --> J1a[Check top-level fields]
    J1 --> J1b[Check component fields]
    J1 --> J1c[Check dynamic zone fields<br>ignore __component]
    
    J --> J2[validateMediaFields]
    J2 --> J2a[Check URLs/identifiers]
    
    J --> J3[validateComponentStructure]
    J3 --> J3a[Validate Components]
    J3 --> J3b[Validate Dynamic Zones]
    J3b --> J3c[Check array structure]
    J3b --> J3d[Validate __component exists]
    J3b --> J3e[Validate __component is allowed]
    
    %% Content Validation Branch
    I --> K[validateContent]
    K --> K1[Validate Required Fields]
    K --> K2[Validate Relations]
    K2 --> K2a[Check Database]
    K2 --> K2b[Check Import Data]
    
    K --> K3[validateComponentContent]
    K3 --> K3a[Validate Component Relations]
    K3 --> K3b[Validate Dynamic Zone Relations]
    
    %% Constraints Validation Branch
    I --> L[validateConstraints]
    L --> L1[validateUniqueFields]
    L1 --> L1a[Check Duplicates in Import]
    L1 --> L1b[Check Database<br>unless updateExisting]

    %% Styling
    classDef phase fill:#f9f,stroke:#333,stroke-width:2px
    class J,K,L phase

Here is a link to the code
I really want to make sure this is solid because its the only thing blocking the import of bad data. Corrupted data no fun. Good news though, it checks that relations exist, or will exist, on import, and gives you the option to ignore the warning (importing a dependency collection before the dependent for example).
As far as I can tell, dynamic zones are working in my testing. Strapi 5 changed the syntax i think according to my research. I never worked with Strapi 4. I think all I need to do now is restore the ability to export all related entitles and also restore the 'Whole DB Import Export' page.
I think I am going to do the related entities in 2 modes.
1: Export the entity with all its relations -> export relation with all its relations -> recursive output but keep track of what was already output so that we don't end up in an infinite loop. This would be a 'Deep relations export'.
2: Export the entity with all its relations -> export the relations but only include necessary relations to create the object. This would basically be a 1 level deep export.

I think I might also need to add a new mode for importing when the importer runs into something like a list or array to control erase and recreate all array elements or append to existing elements. This only matters for imports that update existing elements.

I have to admit, I didn't think there was going to be so much to do here. I am starting to see why this isn't just a default feature in Strapi, but it really still should be. If it was in the core, some of the hacky workarounds I am doing wouldn't be needed I feel.

[Moonlight63]

Lets call this a beta version: https://www.npmjs.com/package/strapi-import-export-v3?activeTab=readme
Have fun!

FYI: This is working pretty well for my needs, but I don't make heavy use of locales. I think the way that locales are being handled is not going to be sustainable in the long term. I smell a v4.

Happy New Year!

[GraemeFulton]

hey @Moonlight63 thanks for this - I was away the last month, so sorry for the delay! I'll give this a go and hope to merge the pull request too!