Pingdom Home

US + international: +1-212-796-6890

SE + international: +46-21-480-0920

Business hours 3 am-11:30 am EST (Mon-Fri).

Do you know if your website is up right now? We do! LEARN MORE

We made a mistake

Just about 2 hours ago, we started to send out our best wishes for the holiday season to our customers. Unfortunately a really embarrassing mistake was made, which affected a limited number of our customers.

Because of a loop in an email-script (not a part of the software that is the base of Pingdom services) gone bad, we managed to send out more than one email to 5,511 of our customers. By mistake, we also included a limited number of our other customer’s email addresses in the To: field of those emails. The exact number of emails that was sent before we stopped the script was 5,511. The person that received the most duplicate emails received 11 of them, and the number of email addresses that was included in the To: field was 5,511. That’s not all our 280,000 customers, but certainly more than enough.

The email-script in question sent out the emails in alphabetical order on the first part of the email address (before the “@”). When it reached 5,511 emails, everyone that had an email address that starts with “0…@” to “adem…@” had received an email. We want to be transparent about this so that you can easily tell if you were among those affected.

Since all of you that are affected are customers of ours, we hope you act as a good netizen and friendly neighbourly customer and delete these email(s) if you have received it. We know that you respect the importance of privacy and will do the right thing.

Lastly, we would like to say that we are sorry to all affected. We screwed up here, it’s as simple as that.

Our last hours before the holiday didn’t end up as we wished. But like we said in the email that gave birth to this post, we really hope that you will have a great holiday, and that you will understand and forgive us.

Those of you that are developers knows that even the greatest programmer make mistakes. This is why we have testing procedures in place. In this case the mistake was that our regular testing procedures were not followed as they should have been.

The script used in this case was not part of our regular Pingdom software development, testing, and deployment procedures since the script was just a temporary solution to send these emails out.

To summarize and to clarify:

  • This did not affect any of Pingdom’s monitoring services, alerts or other systems.
  • 5,511 of our customers have received email addresses of 5,511 other customers, all with an email address starting with “0…@” to “adem…@” (the part before the “@”).
  • No passwords, credit cards, or other personal information of our customers has been compromised.


No Comments

@pingdom Been there, done that. I know how it feels. Have a good Christmas guys! :)

@robclarkey Thanks, appreciate it, same to you :)

The best companies own up quickly to their mistakes, learn from them and move on. Happy Holidays!

@pingdom Honest, transparent and clear. You are doing a great job guys :) All the Best for 2013 !

@pingdom I’ve made similar mistakes myself – they happen to us all. Main thing is that you fessed up right away. Happy holidays :)

@pingdom Mistakes happen, you guys provide some awesome services and we’re happy to have been given a transparent update on this.
 
Happy 2013 guys!

Jovanni Hernandez

December 21st, 2012 at 4:31 pm


Thanks for the transparency, I dig the whole “no excuses” approach.

Frank O. Schäfer

December 21st, 2012 at 4:33 pm


great communication

@pingdom I got 5 emails.

It was not the end of the world…I still think you are doing a great job. These things happen.

Thanks for your honest statement! As a long time software engineer and system administrator I have committed my share of blunders, and I really appreciate when someone stands up and admits a mistake.
 
Keep up the good work, and happy holidays!

Gotta love Pingdom for being transparent.

Thanks guys :)

Martin Eichenberg

December 21st, 2012 at 5:14 pm


Hey … and don’t hit the intern (who built that script) too bad. ;)

It can happen to anyone, top services included. I think you can easily be forgiven :)
The only ones who don’t make mistakes like this are the ones who do nothing. Happy Holidays!

NicholasSimonCronwright

December 21st, 2012 at 11:32 pm


use thunderbird with mailmerge, better & safer….

@pingdom guys don’t worry. Worst things can happen. And the service you provide is superb! Happy holidays!

@ninetofive_me Thanks very much, appreciate it. Happy holidays.

@geerlingguy Thanks very much, appreciate it. Happy holidays.

@mneylon Thanks very much, appreciate it. Happy holidays.

@jannisg Thanks very much, appreciate it. Happy holidays.

mistakes happen! it’s not the fall that matters… it’s how you get back up. Pingdom and it’s staff are second to none. thanks for another amazing year of service.

@pingdom bummer. However, if that’s as bad as it gets right now, we’ll all be ok :)

@pingdom thumbs up for being transparent!

Mistakes happen. Glad you guys own up to it, and in all honesty, this is minor to mistakes that happen all the time that are not publicized. I’d rather have transparency than secrecy. Most others would not agree.