Data Loss :(

We passed two major milestones yesterday: our millionth SlimTimer hour was logged and we had our first ever widespread incident of data loss. Ugh!

What happened:

  • Due to what I’ll call ‘operator error’ we lost all data on any tags or tasks created after June 28th 04:00 GMT
  • We stopped the server to repair the database using nightly backups to fill in the gaps the best we could
  • We created a script that would create ‘recovery’ or dummy tasks to be placeholders for the tasks that we lost and give us something to attach your time entries to.

Who’s affected:

  • 360 People had tasks that were lost and now have placeholder tasks, with names like ‘[Recovery 190]’, in their places
  • More people may have lost tasks if they were created yesterday but not linked to any time entries
  • All tags added yesterday to both tasks and time entries were lost.

What you need to do:

  • If you have any placeholder tasks you need to rename those and add any tags, coworkers or reporters that may be missing
  • Check over your entries from yesterday and re-add any tags that may have been lost.

If you’re one of those 360 people you’ll be getting an email from me shortly, but anyone should feel free to contact me with any concerns they have.

We’re working on making sure this doesn’t happen again and increasing our database backup regularity.

a million apologies..

Rich


About this entry