Scan for malformed CSV files to avoid SIS Upload Error

[jonespm]

Frequently a CSV files comes along that causes a "SIS Upload Error". One often reason for this the CSV files contain unescaped quotes. Something along the lines of

01234567,testuser,,"Nickname"","Firstname",[email protected],active

The problem is that these CSV files are contained within a zip file that we don't do any processing to. To fix this we'd have to uncompress the zip files, scan for malformed content and fix it and re-compress. The correct fix to escape double quotes like this is to put an extra double quote in front of it.

Currently fixing this is a manual process. It was believed that they'd try to fix this upstream (TDX 231033) but feels like this has been a problem for a few months so might be worth looking at, or at least doing processing for in the new process of the uploader if this was rewritten. The upstream fix looks like it was just to prevent these characters from being entered, so that still seems like we might see problems with the downstream data.