Oops x1.6 million

Recently I accidentally sent out 1.6 million emails to customers. I was doing local development that integrated with a third party and didn’t realize my iterative testing was actually firing off emails in production.

The vendor doesn’t have a staging/sandbox environment so extra care is always needed when doing development.

It was a pretty embarrassing situation and I tried to take a few things away from it:

1. If you are integrating with a vendor and they only have production, find a way to mock their environment.
2. Always create safe data structures for testing. Create duplicates with the third party that mimic real data structures, then swap to live objects when going live.
3. Be careful! There should be a weighty, important feeling to doing work with integration. If you’re feeling very casual and loose, then something is wrong.
4. Put in safe guards. In this case I made a whitelist of objects that could be operated on remotely. I also put in place several business rules that would prevent mistakes in the future.

Big mistakes mean opportunities to learn big.

Oops x1.6 million

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s