Exporting large number of Entries / Formie Submissions

[adrianjean]

I have 40,000++ entries (Formie Submissions and a Contact Database) that I need to export on a semi regular basis.

When I use the export feature for the entry type, the large amount of data times out and never finishes. I've gone so far as to setup a dev server with a crazy PHP max execution time and 6 CPU cores and 28GB ram — just to do the exports and I'm still unable to get an export to complete.

I have tried CSV, JSON and XML formats.

I raised the issue with the Formie folks (verbb/formie#2259 (comment)) but seems it's related to core CraftCMS functionality.

I've been noticing that managing large datasets in CraftCMS is a bit of a challenge, and I was wondering:

1. Am I wrong to keep using CraftCMS for these bigger data websites? I work on websites with sometimes, 100,000+ entries / records, and I always dread having to do any import export.
2. Are there are any tips you can provide to get this data out on a regular basis. Open CLI options as well.
3. If this process could be shunted to the Queue runner and parted out into chunks ... could it help?

Help!

[adrianjean]

Following up on this — I was using a hosted MySQL DB (DigitalOcean) and suspected that might be contributing to slowing things down because the web server CPU load was minimal during my export attempts (altho exporting did lock-up the site completely).

So I tried it with a local web and MySQL server and the same DB, and it actually completed.... It still took a long time to export the 40k+ submissions, but at least it completed. So seems the hosted DB has a massive impact on exporting this data.

Still would be great to queue these kinds of things to:

1. Prevent the site from locking up.
2. Let longer exports run until complete, and have some idea of when they might complete if we know how many records we are exporting.

[brandonkelly]

The built-in export functionality was not written with high scalability in mind. We’re discussing ways to improve it in the future, though.

[adrianjean]

Amazing — thanks so much for the update!