Recently I accidentally sent out 1.6 million emails to customers. I was doing local development that integrated with a third party and didn’t realize my iterative testing was actually firing off emails in production.
The vendor doesn’t have a staging/sandbox environment so extra care is always needed when doing development.
It was a pretty embarrassing situation and I tried to take a few things away from it:
1. If you are integrating with a vendor and they only have production, find a way to mock their environment.
2. Always create safe data structures for testing. Create duplicates with the third party that mimic real data structures, then swap to live objects when going live.
3. Be careful! There should be a weighty, important feeling to doing work with integration. If you’re feeling very casual and loose, then something is wrong.
4. Put in safe guards. In this case I made a whitelist of objects that could be operated on remotely. I also put in place several business rules that would prevent mistakes in the future.
Big mistakes mean opportunities to learn big.