Possible corruption of large media files

After several days of investing strange media playback behavior, we have discovered a bug in our content storage system. The details are still not completely solid for which files could be affected, but as of right now we believe it to be larger (100mb+) files potentially getting corrupted as we push media into our storage array. It is not every file.. the bug occurs intermittently making it very hard to pin down. There have been sporadic reports of archived files getting corrupted and today some reports of new media files not playing properly once downloaded.

This issue is being addressed and we are working to detect the incomplete files and replace the corrupted or cut-off files from backup. Depending on the media type, sometimes the result of this bug will be a file that is just cut-off, or in the case of certain video codecs, the movie not playing at all. Quicktime reports "invalid public atom was found in the movie" in some cases, however in iTunes the file just doesn't show up.

While we are making a sweep of the terabytes of data we have stored, if you are experiencing "weird" issues like these please let us know immediately and we will investigate your issue specifically.

---

Update 10/8: The issue, unfortunately, seems to have extended beyond large files. About 5% of the files in our archives have been cut-off. We believe this has resulted from a script which ran to repair the mime-types of these files. The mime-type problem was causing files to not "stream" or progressive play download. The "fix" seems to have done more damage then good.

We are running some scripts now to recover as many of the files from backups as we can. Unfortunately there will likely be files which will need to be re-uploaded. We will contact the users directly who this pertains to via email.

There have been extra validation checks put into place to ensure this will never happen again.

---

Update 10/9: We have scoured through the backups that we do have for some of the media and have found about 50% of the potentiallycorrupted files. Users who have been affected will receive an email with a list of the files that we have detected a file size mismatch on. You will be asked to verify those files by downloading them and if they are cut-off or corrupted, please reupload those files at your earliest convenience. Once uploaded, they will be automatically pushed into the archives and you the storage will not be counted towards your monthly quota.

Stay tuned for more updates to come.