Pingdom Home

US + international: +1-212-796-6890

SE + international: +46-21-480-0920

Business hours 3 am-11:30 am EST (Mon-Fri).

Do you know if your website is up right now? We do! LEARN MORE

We made a mistake

Just about 2 hours ago, we started to send out our best wishes for the holiday season to our customers. Unfortunately a really embarrassing mistake was made, which affected a limited number of our customers.

Because of a loop in an email-script (not a part of the software that is the base of Pingdom services) gone bad, we managed to send out more than one email to 5,511 of our customers. By mistake, we also included a limited number of our other customer’s email addresses in the To: field of those emails. The exact number of emails that was sent before we stopped the script was 5,511. The person that received the most duplicate emails received 11 of them, and the number of email addresses that was included in the To: field was 5,511. That’s not all our 280,000 customers, but certainly more than enough.

The email-script in question sent out the emails in alphabetical order on the first part of the email address (before the “@”). When it reached 5,511 emails, everyone that had an email address that starts with “0…@” to “adem…@” had received an email. We want to be transparent about this so that you can easily tell if you were among those affected.

Since all of you that are affected are customers of ours, we hope you act as a good netizen and friendly neighbourly customer and delete these email(s) if you have received it. We know that you respect the importance of privacy and will do the right thing.

Lastly, we would like to say that we are sorry to all affected. We screwed up here, it’s as simple as that.

Our last hours before the holiday didn’t end up as we wished. But like we said in the email that gave birth to this post, we really hope that you will have a great holiday, and that you will understand and forgive us.

Those of you that are developers knows that even the greatest programmer make mistakes. This is why we have testing procedures in place. In this case the mistake was that our regular testing procedures were not followed as they should have been.

The script used in this case was not part of our regular Pingdom software development, testing, and deployment procedures since the script was just a temporary solution to send these emails out.

To summarize and to clarify:

  • This did not affect any of Pingdom’s monitoring services, alerts or other systems.
  • 5,511 of our customers have received email addresses of 5,511 other customers, all with an email address starting with “0…@” to “adem…@” (the part before the “@”).
  • No passwords, credit cards, or other personal information of our customers has been compromised.


50 comments
Mineral Water Plant
Mineral Water Plant

It's fantastic to be part of a group of blog writers. Your Blog Post Objectives are appropriate encouragement about the objective of weblogs and what I desire to as well. Keep composing a blog on blogs!

lucian303
lucian303

Mistakes happen. Glad you guys own up to it, and in all honesty, this is minor to mistakes that happen all the time that are not publicized. I'd rather have transparency than secrecy. Most others would not agree.

sexyprout
sexyprout

@pingdom thumbs up for being transparent!

sexyprout
sexyprout

@pingdom thumbs up for being transparent!

BillWeiss
BillWeiss

@pingdom bummer. However, if that's as bad as it gets right now, we'll all be ok :)

BillWeiss
BillWeiss

@pingdom bummer. However, if that's as bad as it gets right now, we'll all be ok :)

Mikel Bullis
Mikel Bullis

mistakes happen! it's not the fall that matters... it's how you get back up. Pingdom and it's staff are second to none. thanks for another amazing year of service.

Mikel Bullis
Mikel Bullis

mistakes happen! it's not the fall that matters... it's how you get back up. Pingdom and it's staff are second to none. thanks for another amazing year of service.

pingdom
pingdom

@jannisg Thanks very much, appreciate it. Happy holidays.

pingdom
pingdom

@jannisg Thanks very much, appreciate it. Happy holidays.

pingdom
pingdom

@mneylon Thanks very much, appreciate it. Happy holidays.

pingdom
pingdom

@mneylon Thanks very much, appreciate it. Happy holidays.

pingdom
pingdom

@geerlingguy Thanks very much, appreciate it. Happy holidays.

pingdom
pingdom

@geerlingguy Thanks very much, appreciate it. Happy holidays.

ninetofive_me
ninetofive_me

@pingdom guys don't worry. Worst things can happen. And the service you provide is superb! Happy holidays!

ninetofive_me
ninetofive_me

@pingdom guys don't worry. Worst things can happen. And the service you provide is superb! Happy holidays!

Andrea Carrer
Andrea Carrer

It can happen to anyone, top services included. I think you can easily be forgiven :) The only ones who don't make mistakes like this are the ones who do nothing. Happy Holidays!

Andrea Carrer
Andrea Carrer

It can happen to anyone, top services included. I think you can easily be forgiven :) The only ones who don't make mistakes like this are the ones who do nothing. Happy Holidays!

Martin Eichenberg
Martin Eichenberg

Hey ... and don't hit the intern (who built that script) too bad. ;)

Martin Eichenberg
Martin Eichenberg

Hey ... and don't hit the intern (who built that script) too bad. ;)

aisotton
aisotton

Thanks for your honest statement! As a long time software engineer and system administrator I have committed my share of blunders, and I really appreciate when someone stands up and admits a mistake.

 

Keep up the good work, and happy holidays!

aisotton
aisotton

Thanks for your honest statement! As a long time software engineer and system administrator I have committed my share of blunders, and I really appreciate when someone stands up and admits a mistake.   Keep up the good work, and happy holidays!

Stelios Sofos
Stelios Sofos

It was not the end of the world...I still think you are doing a great job. These things happen.

Stelios Sofos
Stelios Sofos

It was not the end of the world...I still think you are doing a great job. These things happen.

Jovanni Hernandez
Jovanni Hernandez

Thanks for the transparency, I dig the whole "no excuses" approach.

Jovanni Hernandez
Jovanni Hernandez

Thanks for the transparency, I dig the whole "no excuses" approach.

ViniciusFreitas
ViniciusFreitas

@pingdom Mistakes happen, you guys provide some awesome services and we're happy to have been given a transparent update on this.

 

Happy 2013 guys!

ViniciusFreitas
ViniciusFreitas

@pingdom Mistakes happen, you guys provide some awesome services and we're happy to have been given a transparent update on this.   Happy 2013 guys!

jlego
jlego

@pingdom I've made similar mistakes myself - they happen to us all. Main thing is that you fessed up right away. Happy holidays :)

jlego
jlego

@pingdom I've made similar mistakes myself - they happen to us all. Main thing is that you fessed up right away. Happy holidays :)

rmaunier
rmaunier

@pingdom Honest, transparent and clear. You are doing a great job guys :) All the Best for 2013 !

rmaunier
rmaunier

@pingdom Honest, transparent and clear. You are doing a great job guys :) All the Best for 2013 !

James Roberts
James Roberts

The best companies own up quickly to their mistakes, learn from them and move on. Happy Holidays!

James Roberts
James Roberts

The best companies own up quickly to their mistakes, learn from them and move on. Happy Holidays!

robclarkey
robclarkey

@pingdom Been there, done that. I know how it feels. Have a good Christmas guys! :)

robclarkey
robclarkey

@pingdom Been there, done that. I know how it feels. Have a good Christmas guys! :)

pingdom
pingdom

@ninetofive_me Thanks very much, appreciate it. Happy holidays.

pingdom
pingdom

@ninetofive_me Thanks very much, appreciate it. Happy holidays.

pingdom
pingdom

@robclarkey Thanks, appreciate it, same to you :)

pingdom
pingdom

@robclarkey Thanks, appreciate it, same to you :)