Project Title Software to Extract Deleted Virtual Machine Files from a VMFS Volume
Project Start Date: 2016/06
Project Completion Date: 2016/08
Field of Science or Technology: Software Engineering and Technology
Purpose of work
To achieve technological advancement for the purpose of creating new or improving existing materials, devices, products or processes.
Experimental Development:
Overcoming Uncertainties, Work Performed, Achievements, and Outcome
Background:
We received a large multi-volume RAID storage array that utilized a Virtual Machine File System (VMFS) as the host file system. The file system contained files for multiple virtual machines (VMs) in the client’s environment. The array suffered a multiple drive failure, therefore the client and their IT consultants attempted to replace and rebuild the failed drives in the storage array. The overall storage system contained multiple independent RAID arrays and volumes, and drives from different arrays were placed in incorrect positions when rebuilt. Due to the extensive amount of damage resulting from the incorrect rebuild, the VMFS volumes would no longer mount.
Goal:
We attempted to recreate the original configuration in our labs by removing the drives that were known to be invalid and rebuilt those from parity. The result was a very damaged file system, but there was still a large amount of file system metadata and user data that was not damaged. There are very few companies that deal with highly damaged VMFS file systems, so any tools found on the internet were very basic and were only useful with undamaged or lightly damaged structures.
Our aim was to reverse engineer the VMFS metadata structures in order to see if it was possible to manually extract deleted/missing virtual machine (VM) files. The publicly available tools claimed to recover deleted files, but didn’t address file systems that were either reformatted or missing some of the original file metadata. If we were successful at extracting the deleted VM, we needed to see if it was possible to repair missing/corrupted structures since we highly suspected damage due to the incorrect RAID rebuild. If this wasn’t possible, we wanted to investigate if it was possible to create a utility to extract the virtual machine files and patch areas where the metadata traversed corrupted areas. With the lack of accurate technical documentation and an no publicly available utilities to recover highly damaged VMFS hosts, we were uncertain whether it would even be possible to reach our goal.
The overall (damaged) storage array submitted for recovery was suitable for our research purposes since it contained 3 different VMFS volumes, therefore we had a cross section of VMFS volumes of various sizes. This assisted us in the initial stages of reverse engineering the metadata and inode/pointer structures since the differing structure sizes provided additional valuable information.
Through detailed analysis of directory structures, inodes, secondary and tertiary indirect table files we were eventually able to gather enough details of the file system structure in order to navigate through the file structure and hence extract uncorrupted virtual machine (VMDK) files from the VMFS system.
During the examination of corrupted file structures, we noted a regular occurrence of invalid pointers within the indirect pointer files that we assumed were due to the incorrect RAID rebuild by the client. Some indirect pointers that should have been pointing to other pointer tables were either zeroed or contained illegal values. We devised strategies in our experimental utilities that would allow us to extract all valid data, while segregating the corrupt pointers and voiding these areas in the extracted virtual machine file.
In our recovery attempt, we repaired the damaged areas by filling them with legal values that pointed to a signature block that we created. Our hope was that this would essentially mark all invalid areas in the extracted VM files with an identifiable signature so that we would know that those areas corresponded to invalid data pointers. We attempted to mount the volume under Linux vmfs-fuse but it appeared that the modifications were not accepted and the volume would not mount. This was likely due to the severe damage in some other areas of the data and metadata and possibly the fact that many rebuilt pointers were pointing to the same location (this would normally not occur in an operational VMFS volume). Due to this failure, we then proceeded to modify our extraction utility and used it to extract the virtual machine. We had our utility disregard any duplicate pointers as well as some other issues in the volume structures.
The resulting extracted file had all of the retrievable VM data blocks that were possible with the metadata available. Due to data blocks that were corrupted during the rebuild though, corruption that couldn’t be detected earlier became apparent after extraction and examination of the VM file.
We were unable to successfully extract and rebuild a usable virtual machine file for our client since the combination of lost or bad data due to pointer corruption and data area corruption led to a larger amount of invalid areas than initially expected. However, we are able to extract deleted virtual machine files which was our main goal. We can reliably recover deleted files from a VMFS file system if the client submits their case without any further writing to the volume. With knowledge gained, we are also able to recover files from VMFS file systems with moderately damaged internal structures.
Great company!
I think it is the best company in the world regarding data recovery. They take all types of recovery cases with great responsibility and ensure they work with the best possible solution for your specific case. I highly recommend it.
Superb effort
Superb effort from werecoverdata. I had no idea how to fix it when I faced a horrible problem with my RAID NAS. But they solved it very quickly and superbly. I recommend them!
Fantastic results
Thanks to all staff of werecoverdata for excellent work and for getting my company data back! I got frustrated when a deadly drop damaged my hard drive. But they solved it!
Thanks
Few days ago my raid server is crashed dangerously. It was not starting any more. I tried in a various way whatever I know but I could not fix it. My uncle told me about WeRecoverdata. Then contact with them and they repaired it. Thank you guys.
Recover service
Those are suffering bad experience of getting a reliable and professional recovery service provider, definitely, they can choose werecoverdata. I highly recommend it.
Thanks a lot
The location is a bit hard to find out but eventually, I did. They recovered my hard drive, which was mostly dead. Thanks a lot.
Great response
Definitely weRecover team is great to solve that kind of hard drive problem. Great response and remarkable. They recovered my all data successfully within short time. I wanna give special thank to David. Thanks a lot man.
Liked Your Service
I have been using my HP laptop since 2016. Few months ago I could not open my laptop. After pressing the power button its showing nothing. My friend told me about that company. They recovered my laptop. Now its working perfectly. Thanks a lot WeRecoverData.
Pretty good experience
It was really pretty good experience. They are so professional, punctual and reliable. I strongly recommend it.
Remarkable Work
A lot of thanks to werecoverdata. You did really remarkable work. I was frustrated when lost my important data from a hard drive. I sent my hard drive to WRD then they got all the data back. At first, I couldn’t believe anyone can recover it since my computer tech said he can’t do it. All credit goes to werecoverdata.
Gorgeous service
Fantastic service from all the guys at werecoverdata. I was having pressure when corrupted data from my hard drive .Then I went there and they fixed it superbly.Thanks for removing the pressure!
Extremely well
I faced horrible problem but they solved it extremely well.My problem was that hard drive crashed dangerously.At this moment I thought the sky was broken on my head.Thank you for that.
Sensible
Thanks to WeRecoverData. You fixed the hard drive problem promptly. I am impressed… Anybody can rely on werecoverdata.
Well done
Thank you
Great work
Obviously great work from werecoverdata.Definitely no doubt about this company because they take seriously to solve any kinds of problem.I have lot of assurance at them.A lot of thanks to them.
Fantastic work
Oh Tom really fantastic work. You saved my important days. Raid 5 server damaged dangerously few days ago. But you fixed it well. Thank you for that!
Great efforts
All credits goes to werecoverdata for recovering raid server.I thought impossible to repair it but they was taken sensible and fixed it.Now I can fully trust on werecoverdata.All thanks to them.
Helpful Work
Werecoverdata, you did a really very helpful work.I was angry when my hard drive got dropped from my mistakes. I recommend the company and give a lot of thanks to werecoverdata.
Best service
After my hard drive crashed I was going to be totally mad. One of mu cousin suggested me to meet that company. I did and got the best service ever I had.
Remarkable Job
AWESOME service from WeRecoverData. I was almost mad when my pc crashed. A lot of valuable data in my hard drive. One of my friend recommend it. I thought it will not be possible.
But after all I got back my all data. Thank You WeRecoverData.