# File recovery help needed + warning for windows users



## ANTIcarrot (Aug 14, 2009)

Windows has just deleted 70GB of data from a backup HDD. I rebooted the system and stepped out of the room. Windows can a 'consistancy check' and started deleting. So a warning. Plea for help at the end.
_
External discs are not safe. Anytime windows boots (for any reason) and you are not in the room to "Press X to not die" you can lose all your data. If your backup is plugged into the computer, you can lose it and your main copy at the same time._

I hindsight this must have been what caused a similar 20GB loss six years ago. At the time I thought it must have been a faulty RAID card. Apparentrtly it's just the biggest, nastiest, and most vicious windows bug in all of microsoft's history.

Can antyone offer any advice on file and/or directory recovery? Or at least how to prevent this kind of catastrophic data loss in the future?

And the last is actually a bit of an issue for me. Ironically, thought my most precious and irriplacable files seem to have been destroyed, the rest of my 'general' archive is probably intact - and I'd like it to stay that way. To that end ( a week before this all happened) I spent Â£300 on the hardware needed for a 4TB Raid-5 array - because beyond a certain disc size, there is no backup solution! I *really* don't want windows to fuck that up as well.


----------



## AshleyAshes (Aug 14, 2009)

I'm pretty certian that no version of Windows randomly deletes stuff on other drives just cause it felt like it.


----------



## Shino (Aug 14, 2009)

Yeah... no.

Whatever you did, it wasn't the OS. It sounds like either a virus, data or file allocation table corruption, or improperly configured backup program. Besides, there's a reason why physical external backups are reccommended on top of mirrored drives.

Windows doesn't go around randomly deleting large amounts of files. Not on it's own, anyways...


----------



## Kaamos (Aug 14, 2009)

Shino said:


> Windows doesn't go around randomly deleting large amounts of files. Not on it's own, anyways...



Windows has become self aware.


----------



## CAThulu (Aug 14, 2009)

Kaamos said:


> Windows has become self aware.



Good morning, Dave.


----------



## net-cat (Aug 14, 2009)

This, ladies and gentlemen, is why you always "safely remove" your external devices before unplugging them.


----------



## ToeClaws (Aug 14, 2009)

net-cat said:


> This, ladies and gentlemen, is why you always "safely remove" your external devices before unplugging them.



Yep - incorrectly unmounted drives may not be done writing data to their platters, and when Windows starts back up, sometimes it's aware of a badly "corrupted" drive table and attempts to correct it.  Usually, it's not a big deal, but sometimes the chkdsk method of repairing a very botched drive table is just to remove it.  Seen it happen on an Exchange server a few years ago, deleting gigs worth of e-mail accounts.

As Net-Cat points out, portable drives should always be unmounted properly before removing them (this includes USB drives).  This is also true of ALL operating systems, not just Windows.


----------



## Irreverent (Aug 14, 2009)

ANTIcarrot said:


> Can antyone offer any advice on file and/or directory recovery? Or at least how to prevent this kind of catastrophic data loss in the future?



If you're getting consistency checks on external devices, you need change your operating process (as alluded to by net-cat and TC) and verify your power supply stability.  If the power is glitching and causing the device to disconnect, that might be the source of the problem.



> To that end ( a week before this all happened) I spent Â£300 on the hardware needed for a 4TB Raid-5 array - because beyond a certain disc size, there is no backup solution!



Backing up 4TB data sets is trivial, if not tedious.  Its no excuse.  Backing up a 1 PB array is a challenge.

RAID-5 will give you protection from the failure of an individual disk in the striped set....it wont help you at all if the card takes a dump, or the enclosure is lost due to power, fire or flood issues.  So while RAID-5 is good, its not the end of the solution, its the start.  Don't be lulled into a sense of false confidence

If you don't want to invest in backup servers and software, the quick and dirty way to do it might be to build another 4TB array and mirror it to the first array.  You'd still have DR issues, but it would be a start.  A 4TB NAS at another location of the house might be an option too.


----------



## ZentratheFox (Aug 14, 2009)

Nice...


----------



## ANTIcarrot (Aug 17, 2009)

Shino said:


> Whatever you did, it wasn't the OS.



Well gosh darn it you must all be right. _And the data recovery company experts I've consulted must be wrong..._

I repeat, if windows finds something it doesn't like (such as a single bad sector) it can lop off arbitarily large parts of your file tree. (which can equate tens of gigabytes and MILLIONS of sectors) It does so without warning and without cause - because windows can and does function happily with these flaws present.

A single bad sector shouldn't would woudl not by itself have affect millions of others. That's a programming bug and a massive software flaw in the windows OS. The bad sector (if it existed at all, and wasn't just a false positive) wasn't the problem. It was windows taking it upon itself to try to 'fix' a problem that didn't exist; spreading the damage across an entire user folder.

I also repeat - external backup drives aren't secure. If it is conncted during boot (or during a crash reboot) and you're not there to say no, windows can start destroying data there as well. This can be especially agravating during an attempted backup proceedure - as was the case here.




			
				Irreverent said:
			
		

> If you don't want to invest in backup servers and software, the quick and dirty way to do it might be to build another 4TB array and mirror it to the first array.



So on top of the Â£300 I've spent all ready, that will be another Â£300 for another set of discs, and a Â£600 computer to run them on? I was hoping to spend a lot less than Â£1,200... And you know that coping (say) 3TB  over a typical home 40MBps 'gigabit' connection could take 20 hours? You're right. I should have said it's impossible to backup 4TB. But it can quickly become _impractical _for home users.

Which is why I only ever intended to backup the *important* parts of it, and leave the rest to chance.   At least before microsoft took it upon itself to fubar the file structure on the backup...


----------



## Runefox (Aug 17, 2009)

Well, if you haven't actually written any more data to the drive since, it should be trivial to recover your files using any file recovery program (Recuva, by the same guys who make CCleaner, is one that, while comparatively rudimentary, is free. Other options include EasyRecovery Pro and GetDataBack). Of course, the best course of action is to make a complete image of the drive and/or image it onto another drive before doing anything like that, to preserve the state of the drive.

What actually happens in a "consistency check" (which happens when Windows detects that the drive was not properly unmounted during bootup - The same happens with your other hard drives if necessary) is not usually permanent deletion - More than likely, the files still exist (or weren't completely finished writing when you removed the drive from the computer), whether they were simply marked as free space (the "normal" way to delete things), or moved to FILExxxx.CHK files (what CHKDSK does when it needs to delete file records from the master file table and can't recover the original path/filename due to the record being corrupt - Likely what's happened here). In the case of the .CHK files, take a look here.

Data recovery "experts" typically want you to spend hundreds in currency for things that simply aren't very advanced - Or to send your discs off to a white room, which is hit or miss and costs thousands. Technically, they are correct - A portion of the OS did "delete" the files, but this rarely means they're gone for good.

EDIT: I should stress that for the most part, as long as any filesystem is in use, you're going to run into this eventually. Linux has issues like this, and so does Mac OS X (though journalling filesystems are currently widespread with these OS'es, which results in a much lower chance for things to go south; FAT isn't a journalling filesystem, hence a high possibility that files will become corrupted during an improper shutdown, while NTFS does implement journalling). *Don't blame Windows for something it's not directly responsible for*. There are enough other things to blame Windows for to begin with, no need to pin extra stuff on it.

EDIT 2: RAID is silly. You're not getting a backup, and in the event of a power failure or sudden removal/shutoff, each drive has different data in its write cache and is writing something different (ultimately garbage) as it's dying. No filesystem can reconcile that, since it lies on a different level altogether - The RAID controller has to worry about it. In extreme cases, you'll lose your array; In other cases, you'll end up with this kind of deal. It's great for its intended purpose of keeping a production system going in the event of a hardware failure (hence "*Redundant* array of independent/inexpensive/intelligent/whatever-they're-using-now disks"), but it is *not ever* a replacement for backups in any situation. If you are dead-set on using RAID, use a UPS/Uninterruptable Power Supply with the system to ensure that you have time to properly shut the system down in the event of a power failure; If you're forcibly shutting the system down by hitting the power switch or holding the power button for 4 seconds, *stop it*.

EDIT 3: In the future, *back up your files* and *never rely on RAID*. RAID has likely been a major component in the loss of your data over the years, and in addition isn't failure-proof. Backups aren't, either, but if you keep your important data spread across multiple media (if you really want to, include RAID in that), the chances of actually losing data diminish to near zero (barring natural disasters, theft, or anything that destroys/moves the physical media).


----------



## Stratelier (Aug 17, 2009)

ANTIcarrot said:


> I repeat, if windows finds something it doesn't like (such as a single bad sector) it can lop off arbitarily large parts of your file tree.


No, that depends on _what the sector contains_.  It is by hell *not* arbitrary, but it _is_ a very low-level distinction.

For example, if the sector contains just a few files, well, the damage is limited to the affected files themselves.  But if that sector contains a directory, you can essentially kiss all the files inside it good-bye because if the OS can't read the directory, it can't locate the files.  Their data will still physically be there on the disk until overwritten (and can possibly be recovered using appropriate tools), but that's about all.

Heaven help you if the FAT sectors get damaged.  Remember the good old days where DOS wouldn't format your floppy because it said "track 0 bad - disk unusable" ?


----------



## Rel (Aug 18, 2009)

Runefox said:


> *Don't blame Windows for something it's not directly responsible for*.


Epic win right here ^

IMO, Raid is fine and all (it just slows saving down like hell), but its a waste of money, if you  have a perfectly fine eSATA External hard drive. (the files will be saved on there, and you can easily put them back on in case of the OS getting attacked, instead of having 2 hard drives that are corrupted). (then again, this is my opinion)


----------



## Runefox (Aug 18, 2009)

Rel said:


> IMO, Raid is fine and all (it just slows saving down like hell), but its a waste of money


I know I snipped it so that it's out of context, but RAID is mainly designed for rapid random read access (RAID 0, 5, and the Nested levels) and for redundancy in the case of hardware failure (RAID 1, 5 (limited), Nested levels). Its weaknesses lie in write performance, some extra difficulty in setup, extreme difficulty in recovery if an array is somehow broken, and the tendency for reliance on it as a bulletproof shield against data loss.

In particular, RAID 5 sounds good in theory, but can be very limiting performance-wise, and (while I hadn't heard of this prior), according to the Great Wikipedia of Infinite Knowledge (TM) as well as this article:



			
				The Great Wikipedia of Infinite Knowledge (TM) said:
			
		

> Worsening this issue has been a relatively stagnant unrecoverable read-error rate of disks for the last few years, which is typically on the order of one error in 1014 bits for SATA drives. As disk densities have gone up drastically (> 1 TB) in recent years, it actually becomes probable with a ~10 TB array that an unrecoverable read error will occur during a RAID-5 rebuild. Some of these potential errors can be avoided in RAID systems that automatically and periodically test their disks at times of low demand. Expensive enterprise-class disks with lower densities and better error rates of about 1 in 1015 bits can improve the odds slightly as well. But the general problem remains that, for modern drives with moving parts that use most of their capacity regularly, the disk capacity is now in the same order of magnitude as the (inverted) failure rate, unlike decades earlier when they were a safer two or more magnitudes apart. Furthermore, RAID rebuilding pushes a disk system to its maximum throughput, virtually guaranteeing a failure in the short time it runs.



All of this not mentioning that unless you have a hardware RAID controller (as in, a PCI/PCI-X/PCIe card), all RAID controller functions run through the processor and take up system bandwidth, which can further slow things down.


----------



## ArielMT (Aug 18, 2009)

I've had folders become unreadable on NTFS-formatted USB hard disks.  To Windows, they're just gone, and when mounted in Linux, they're unreadable and undeletable.

The cause every time points to the USB disk not being cleanly unmounted.  But I've also had Windows systems up and refuse to unmount USB disks, claiming they were still in use even when the only program still running that touched it is Windows Explorer, and all the relevant Explorer windows long since closed.  The only solution I've been able to find is to shut down and power off the machine completely before removing the disk, and hoping that the shutdown will be a clean one.  Awfully inconvenient at best.


----------



## Rel (Aug 18, 2009)

Runefox said:


> Its weaknesses lie in *write performance*, some extra difficulty in setup, *extreme difficulty in recovery if an array is somehow broken*, and the tendency for reliance on it as a bulletproof shield against data loss.


^ we have a winner.

An eSATA external HDD is alot more reliable and practical for saving files on a computer. RAID is more of a heavy duty, multiple system, extremely important data. Not really for personal use imo.

Whats the speed of eSATA again? like 1-2GBps? *is too lazy to use google*



ArielMT said:


> I've had folders become unreadable on NTFS-formatted USB hard disks. To Windows, they're just gone, and when mounted in Linux, they're unreadable and undeletable.
> 
> The cause every time points to the USB disk not being cleanly unmounted. But I've also had Windows systems up and refuse to unmount USB disks, claiming they were still in use even when the only program still running that touched it is Windows Explorer, and all the relevant Explorer windows long since closed. The only solution I've been able to find is to shut down and power off the machine completely before removing the disk, and hoping that the shutdown will be a clean one. Awfully inconvenient at best.



Ive never had a unmounting/saving problem with my 1TB WD USB external hard drive.


----------



## Runefox (Aug 18, 2009)

eSATA is precisely the same speed (1.5Gbps/3.0Gbps for "SATA I/II" respectively) as the rest of the SATA ports on the board; The only major difference is the shape and style of the connector. This is exemplified by the fact that cases using front eSATA ports typically connect a normal SATA cable to the motherboard's internal SATA ports.

When the SATA 6Gbps ("SATA III") standard is implemented, the speed will double to 6Gbps.

It's kind of funny, back when it was implemented, nothing in the consumer market could touch 1.5Gbps (at 150MB/s, slightly higher than the throughput of the highest-tier ATA standard, Ultra ATA-133/Ultra DMA Mode 6), much less 3.0Gbps, but SSD's are shattering those barriers now.


----------



## Rel (Aug 18, 2009)

Runefox said:


> eSATA is precisely the same speed (1.5Gbps/3.0Gbps for "SATA I/II" respectively) as the rest of the SATA ports on the board; The only difference is the shape and style of the connector. This is exemplified by the fact that cases using front eSATA ports typically connect a normal SATA cable to the motherboard's internal SATA ports.
> 
> When the SATA 6Gbps ("SATA III") standard is implemented, the speed will double to 6Gbps.


Lol i know what eSATA and SATA is, im not a complete moron lol. Never heard of SATA III though, but i guess they are right, 'You learn something new every day'.


----------



## Irreverent (Aug 18, 2009)

ANTIcarrot said:


> So on top of the Â£300 I've spent all ready, that will be another Â£300 for another set of discs, and a Â£600 computer to run them on? I was hoping to spend a lot less than Â£1,200...



For the record, I did say that backing up that kind of data is "tedious" not "inexpensive." 



> And you know that coping (say) 3TB  over a typical home 40MBps 'gigabit' connection could take 20 hours?



If you're only getting 40mb/s out of your GigE equipment, throw it out and buy new.  You should be getting at least 800mb/s.  At 1GigE speed, you can move 3TB in about 8.5 hours (allowing for overhead and couple of other factors).  And its only the initial full backup, you do incremental from that point on.



> I should have said it's impossible to backup 4TB. But it can quickly become _impractical _for home users.



Again, with some properly targeted investment, I'd say your wrong.  Whats impractical for home users is off-site storage of data sets that size.



Rel said:


> RAID is more of a heavy duty, multiple system, extremely important data. Not really for personal use imo.



I disagree.  Hardware raid and hardware raid-based NAS is well within the reach of the "average" user for personal use.


----------



## net-cat (Aug 18, 2009)

Irreverent said:


> If you're only getting 40MB/s out of your GigE equipment, throw it out and buy new.  You should be getting at least 800MB/s.  At 1GigE speed, you can move 3TB in about 8.5 hours (allowing for overhead and couple of other factors).  And its only the initial full backup, you do incrementals from that point on.


800 Mbyte/sec or 800 Mbit/sec? I'm thinking you meant the latter, as 800 Mbyte/sec over a 1000 Mbit/sec link is quite impressive.



Irreverent said:


> I disagree.  Hardware raid and hardware raid-based NAS is well within the reach of the "average" user for personal use.


While technically true, I'm going to argue that it's not worth it for the home user to use RAID, because most will opt to use the built-in RAID card on their $50 motherboard from some noname company. Which is just as likely (if not moreso) to fail as a modern hard drive. 

A more cost effective solution would be a second hard drive, rsync and a cron job. (Or robocopy and a scheduled task, for the Windows folks.)


----------



## LotsOfNothing (Aug 18, 2009)

This is what happens when you download your RAM from untrusted file sharing sites and programs, like Limewire.


----------



## Irreverent (Aug 18, 2009)

net-cat said:


> 800 Mbyte/sec or 800 Mbit/sec? I'm thinking you meant the latter, as 800 Mbyte/sec over a 1000 Mbit/sec link is quite impressive.



Cocktail-napkin mathematics.....   The fact that it was a Starbucks cocktail napkin is somewhat incriminatory.  I'll go fix it.  Right after I finish this Venti. 



> While technically true, I'm going to argue that it's not worth it for the home user to use RAID, because most will opt to use the built-in RAID card on their $50 motherboard from some noname company. Which is just as likely (if not moreso) to fail as a modern hard drive.



3rd party add on hardware raid cards are affordable, and I was thinking more about external appliances that support hardware raid.  The crowd that can afford $600 for a video card can aford to drop $100-300 on a hardware raid card. 



> A more cost effective solution would be a second hard drive, rsync and a cron job. (Or robocopy and a scheduled task, for the Windows folks.)



Simple and works too.


----------



## Runefox (Aug 18, 2009)

> 3rd party add on hardware raid cards are affordable, and I was thinking more about external appliances that support hardware raid. The crowd that can afford $600 for a video card can aford to drop $100-300 on a hardware raid card.


And for what? I disagree wholeheartedly that most people in these classes truly need RAID nor really understand what it's for beyond RAID 0. OP's mentality is case-in-point. RAID, no matter what level of redundancy, is not a backup solution, nor should it ever be treated as one. The RAID levels used for redundancy are exactly for that purpose - Redundancy, in the case that a running disc dies. That's great, but the extra cost versus the danger of relying on it to keep your data safe just isn't worth it. Use those extra discs for storage instead and perform regular backups of your important files.

With that said, implementing RAID into a proper backup schedule can negate the possibility that the running drive will die during a backup and decrease the risk of a read error because it's from multiple sources. In this case, it's a very Good Idea (TM). In OP's case, it was a very Bad Idea (TM).



> This is what happens when you download your RAM from untrusted file sharing sites and programs, like Limewire.


My head just exploded. XD


----------



## ArielMT (Aug 18, 2009)

LotsOfNothing said:


> This is what happens when you download your RAM from untrusted file sharing sites and programs, like Limewire.



Next, we'll be able to download entire computers from Limewire, then merchandise, then groceries.  Ew, yuck, I just tried downloading a peach from Limewire, and it just made a giant mess.

Seriously, though, there's sound advice elsewhere in this thread.


----------



## Irreverent (Aug 18, 2009)

Runefox said:


> And for what? I disagree wholeheartedly that most people in these classes truly need RAID nor really understand what it's for beyond RAID 0.



There does seem to be a strange dichotomy that people will over engineer every aspect of their systems (ostensibly for gaming) and then forget to looks at the fundamentals of dasd deployment.  Everybody can benefit from cheap and simple raid-1, even if its on an integral mobo chipset.  Disk just aint that expensive to NOT have it.


Hell, I have fire insurance, but I still keep fire extinguishers around.



> OP's mentality is case-in-point. RAID, no matter what level of redundancy, is not a backup solution, nor should it ever be treated as one. The RAID levels used for redundancy are exactly for that purpose - Redundancy, in the case that a running disc dies.



No argument here.



> That's great, but the extra cost versus the danger of relying on it to keep your data safe just isn't worth it. Use those extra discs for storage instead and perform regular backups of your important files.



Ok, now we quibble.   Life is too short to restore from backup. JBOD is a PITA.  Drop in a disk and raid1,5  will rebuild/synch itself.  Save the restore for when you've lost the entire array.  The price of disks makes this cheap enough.



> With that said, implementing RAID into a proper backup schedule can negate the possibility that the running drive will die during a backup and decrease the risk of a read error because it's from multiple sources. In this case, it's a very Good Idea (TM).



Yep. Home PC's and laptops use an agent for scheduled backup to the raid appliance, raid appliance uses a client to backup (off site) to a digital vault.  Its 2009, hell even my Mom does this.  Beyond the initial hardware investment (an extra disk per system) its $6.00 per month for the digital vault.


----------



## Runefox (Aug 18, 2009)

Irreverent said:


> Ok, now we quibble.   Life is too short to restore from backup. *JBOD is a PITA*.  Drop in a disk and raid1,5  will rebuild/synch itself.  Save the restore for when you've lost the entire array.  The price of disks makes this cheap enough.



Ooh, yeah, I wasn't meaning JBOD; I meant bypassing RAID altogether. Though you are correct; RAID arrays are (usually) easier to deal with in the event of a single disk failure than working with backups, but as with the articles I posted earlier, as capacity increases, so, too, do the chances of a disk failing _during the rebuild_ due to things like the chances of unrecoverable read errors occurring during such an intense operation increasing sharply with the capacity. Of course, the more disks you have in the array, the less likely this is to happen at all, and I'm not saying RAID isn't worth it to begin with - Just that it is absolutely not something to rely on alone.


----------



## Irreverent (Aug 18, 2009)

Runefox said:


> Ooh, yeah, I wasn't meaning JBOD; I meant bypassing RAID altogether.



Ah, got ya.  Agreed.


----------



## Carenath (Aug 19, 2009)

net-cat said:


> This, ladies and gentlemen, is why you always "safely remove" your external devices before unplugging them.


Which happens automatically when you properly shut-down the computer.



ToeClaws said:


> ... Seen it happen on an Exchange server a few years ago, deleting gigs worth of e-mail accounts.


Ouch. Sometimes I wish that would happen to our server at work, just to ram home how awful their setup is, relying on an incompetant third-party.



Runefox said:


> With that said, implementing RAID into a proper backup schedule can negate the possibility that the running drive will die during a backup and decrease the risk of a read error because it's from multiple sources. In this case, it's a very Good Idea (TM). In OP's case, it was a very Bad Idea (TM).
> 
> My head just exploded. XD


I use a RAID 5 setup, just for extra storage and less risk that I'll lose everything in one go if a disc dies. But I also backup everything to an external hard drive.


----------

