tag:blogger.com,1999:blog-7245518.post8962051372103732219..comments2022-03-26T08:53:39.054-07:00Comments on Max Bruning's weblog: ZFS Data RecoveryUnknownnoreply@blogger.comBlogger14125tag:blogger.com,1999:blog-7245518.post-85313235667359125612010-10-03T22:35:40.774-07:002010-10-03T22:35:40.774-07:00If you have deleted files, and the file system has...If you have deleted files, and the file system has not changed a lot since the deletions, you should be able to recover the deleted files. I have not been following zfs too closely as of late (though I am using it all the time). I thought there was a command now existing that allowed you to roll back to an earlier uberblock for recovery. Regardless, as I very seldom post to this blog (looks like almost 1 year now), your best bet if you want me to try to help is to send me email. max at bruningsystems dot com.Max Bruninghttps://www.blogger.com/profile/17610253175691041309noreply@blogger.comtag:blogger.com,1999:blog-7245518.post-89511874725432567802010-10-03T18:26:22.761-07:002010-10-03T18:26:22.761-07:00Hi Max,
What if you accidentally deleted some fil...Hi Max,<br /><br />What if you accidentally deleted some files on zfs is there a way to recover the files? Note there was no snapshot done and after the the files were delete nothing else was done on the system? Any help would be great!Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-7245518.post-72641063953941041582010-01-03T04:12:18.944-08:002010-01-03T04:12:18.944-08:00Hi Peter,
This would be much simpler to do by emai...Hi Peter,<br />This would be much simpler to do by email, but I don't have an email address for you.<br /><br />No, zdb will not work with exported pools in opensolaris.<br /><br />I have also noticed zfs has a problem if disks are moved around, but it has not been a big problem for me. There has been quite a bit of discussion about this lately on the zfs discussion list on opensolaris. You might want to try reading/posting there.<br /><br />As for ZFS losing your data, I doubt it. I suspect your data is there, but you need to find the right way to get at it.<br /><br />The tools I am using are: a few tools that deal directly with the raw disk, a modified mdb, and a modified zdb. I posted the modified mdb and zdb code quite a while ago. I have kept them up to date (i.e., the tools work with the latest opensolaris release (0609, build 111b).<br /><br />Good luck!Max Bruninghttps://www.blogger.com/profile/17610253175691041309noreply@blogger.comtag:blogger.com,1999:blog-7245518.post-34556046981675108952010-01-03T00:18:30.302-08:002010-01-03T00:18:30.302-08:00Hi Max,
Thanks for the open solaris suggestion.....Hi Max, <br /><br />Thanks for the open solaris suggestion... I might give that a try before digging into the data structures any deeper... they seem inordinately and needlessly complex...<br /><br />Unfortunately zdb only works with live zfs disk sets and not exported ones, at least that is the case in Freebsd 7.2. Are you suggesting it is different in open solaris, that zdb can work with exported volumes that can't be imported due to vague claims of "data corruption" by zfs?<br /><br />This failure that my 8 disks are currently suffering due to zfs does mean that it is not just dangerous but a highly dangerous place to store precious data. Once I recover the data I'll be thinking twice about keeping my data on zfs. Any file system must provide recovery tools... part of that is that each on disk meta data structure (and even data chunks) must have a magic number to be able to identify the record object type out of the noise... and redundant pointers and info for reconstruction processes. Failure to provide that on a vendors part is in my view highly irresponsible and short sighted.<br /><br />I do look forward to the new version in open solaris to see if it can recover the data for me on an import.<br /><br />Thanks again... I'll keep you advised as I progress through this recovery of crucial data.<br /><br />Oh, I've noticed that zfs is very particular about which disk controller ports its various drives are on... why is that when each disk can be identified by it's on disk guid and other information? would this be what is causing the problem with the second set of four disks being corrupted? it's a real pain going though all the permutations of disks to controller ports... all that rebooting... I thought that export and import would then let them be moved to different controller ports?<br /><br />peterPeter William Lounthttps://www.blogger.com/profile/02305704440681618859noreply@blogger.comtag:blogger.com,1999:blog-7245518.post-83226386231752336472010-01-02T23:36:30.940-08:002010-01-02T23:36:30.940-08:00Peter,
There is a tool for examining metadata. It ...Peter,<br />There is a tool for examining metadata. It is zdb(1). As for sharing my tools, please send me email. I can be reached at max@bruningsystems.com. By the way, have you tried importing onto a recent build of opensolaris? There is supposed to be a recovery mechanism in the newest builds.Max Bruninghttps://www.blogger.com/profile/17610253175691041309noreply@blogger.comtag:blogger.com,1999:blog-7245518.post-87830523299035411402010-01-01T22:49:03.397-08:002010-01-01T22:49:03.397-08:00Using the following command I was able to verify t...Using the following command I was able to verify that uber blocks exist on all eight drives in the "damaged" pod1 (ad4, ad6, ad8, ad10, da0, da1, da2 and da3)!<br /><br />od -A x -x /dev/da0 | grep "b10c 00ba"<br /><br />(http://mail.opensolaris.org/pipermail/zfs-discuss/2008-December/024427.html)<br /><br />Note that I had to use four blanks and not just one as in the linked example.<br /><br />Note the "corrupted" set are da0-da3. They were on a four port rocket raid controller and now are on either a four port or sixteen port controller with the same import problem. Individual disk non-joined JBOD configuration.<br /><br />I was really worried that one of the disks might have been overwritten during bios configuration of the hard drive controllers. <br /><br />Well that's an important first step. There are zfs structures viable on all the drives! Could be recovered yet...Peter William Lounthttps://www.blogger.com/profile/02305704440681618859noreply@blogger.comtag:blogger.com,1999:blog-7245518.post-28659282494752845062010-01-01T22:30:44.934-08:002010-01-01T22:30:44.934-08:00Max I found a couple of your other articles and ha...Max I found a couple of your other articles and have started reading them. <br /><br />Is there a program to view the zfs data structures on an "exported" set of zfs disks?Peter William Lounthttps://www.blogger.com/profile/02305704440681618859noreply@blogger.comtag:blogger.com,1999:blog-7245518.post-70544954671233906982010-01-01T21:26:49.478-08:002010-01-01T21:26:49.478-08:00All the "protections" of zfs are useless...All the "protections" of zfs are useless since I can't "import" the eight zfs disks.<br /><br />I just "lost" 6TB of data when attempting to change controller cards for an upgrade. After exporting and shutting down, changing the controller, adding drives, and rebooting the import no longer works. It took a number of times to find the right controller card that would work with the motherboard so I had done this a number of times without any problems with the import. Clearly this controller works fine. It's a 16 port highpoint raid controller model 2340 on freebsd 7.2.<br /><br />The configuration is 8 drives in two groups of four raidz1, one of which shows online for all of it's drives and the raidz1 vdev level, however the second group of four drives shows online for each drive and "corrupted data" on the it's vdev. Thus the top level "pod1" vdev level shows "unavail" and "insufficient replicas". The state of the pool upon "zpool import" is "faulted" and the action says "The pool cannot be imported due to damaged devices or data."<br /><br />If you could share your methods of data recovery I'd surely appreciate it. Don't have any money to spend and the data is confidential so I can't send you the drives.<br /><br />Which programs did you use to view the data structures? Which data structures are the most important ones? What is the best source for information on the data structures? Any links? Docs? Or did you just use the source code?<br /><br />Fortunately none of the data is encrypted or compressed on this zfs installation and it's two groups of four disks each with a raidz1 so there should be a redundant copy of the data to recover files even if one disk in each set of four has been erased.<br /><br />By the way it's insane for any file system provider to not provide data recovery software (fsck or better as needed to recover data) as part of it's design and implementation. Claiming that users should rely on backups doesn't make sense especially for a file system that doesn't support live replication to a backup system as a simple option. The whole point is to protect the data... no matter how "good" a file system is claimed to be there will always be cases where meta data, disks, controllers, or data is damaged resulting in the need for recovery software to be used to attempt a rescue. <br /><br />File Systems must provide the ability for recovery tools to "scrape" as many "files" or fragments of files off of a disk with damaged meta data. A meta data inspector is a needed tool, one that can recover meta data even when some or all of it has been corrupted or destroyed. Clearly something better than zfs is needed. One thing is that meta data needs to be made hugely redundant even if it means a performance penalty.<br /><br />By the way my update was supposed to provide 8 more disks so that I could have an ongoing somewhat live backup copy in the same box with two "pods" of data. Sigh.<br /><br />Thanks.Peter William Lounthttps://www.blogger.com/profile/02305704440681618859noreply@blogger.comtag:blogger.com,1999:blog-7245518.post-4200430985704971002009-12-18T03:11:03.384-08:002009-12-18T03:11:03.384-08:00As to what happened... It started with a failure ...As to what happened... It started with a failure on a controller, then a failure during a scrub (I am not sure of the details). As to what was wrong, the labels were not consistent. I found a label with what appeared to be consistent data. Then I found an uberblock_t within the label that, by walking the data structures on the list, led me to what looked correct. Then I zeroed out all the uberblocks with transaction id greater than the txg id in the "good" uberblock. Then I imported.Max Bruninghttps://www.blogger.com/profile/17610253175691041309noreply@blogger.comtag:blogger.com,1999:blog-7245518.post-68160446769354555152009-12-17T20:32:51.633-08:002009-12-17T20:32:51.633-08:00Nice post:)Nice post:)Data Recoveryhttp://www.idatacare.com/noreply@blogger.comtag:blogger.com,1999:blog-7245518.post-15496598885962860162009-12-16T13:34:58.110-08:002009-12-16T13:34:58.110-08:00I agree with you about "Always have backups&q...I agree with you about "Always have backups". The person will continue to use ZFS. He says one of the best things about it is the performance. I did not mean to imply that "ZFS is dangerous".Max Bruninghttps://www.blogger.com/profile/17610253175691041309noreply@blogger.comtag:blogger.com,1999:blog-7245518.post-57327963556620351812009-12-16T11:42:02.376-08:002009-12-16T11:42:02.376-08:00The takeaway shouldn't be ZFS is dangerous — t...The takeaway shouldn't be ZFS is dangerous — the takeway should be: ALWAYS HAVE BACKUPS!Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-7245518.post-83564926471831222132009-12-16T10:33:59.728-08:002009-12-16T10:33:59.728-08:00Any details on what happened and what was done to ...Any details on what happened and what was done to recover? I figure that if it took you 1 week of tinkering to figure what's wrong, it's probably something more interesting than a case of corrupted data in the last TXG.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-7245518.post-86616558079452606922009-12-16T08:50:52.930-08:002009-12-16T08:50:52.930-08:00About 1 year ago we lost 4 zpools .. and a lot of ...About 1 year ago we lost 4 zpools .. and a lot of money (much more of $15k).<br />Now we moved back to Netapp.<br />Pace of mind :)Anonymousnoreply@blogger.com