I was tasked with writing some code to pull all the research project data that we’d collected over the past 10-15 years into our new J2EE-based product, Kuali Coeus. The legacy system ran off SQL Server which is a lot more forgiving of character encodings and string data in general than the new system (which runs off MySQL).
It had taken me a while to figure out a way to map all the old data onto the new data structures, but I felt like I had done a pretty awesome job. The few batches I had tested it with all passed its tests with no problems. However when I unleashed it on a full dataset (some 6000 rows), about 60% (roughly 2 hours) of the way through, it crashed, and rolled the ENTIRE thing back.