I am working on a backup solution for my Ubuntu web server and so far this is what I have. I am using rsync to create rotating snapshot style backups of my web files and sending them via SSH to a remote location in order to burn them for offsite storage. This is all working perfect. The remote machine is a Windows Server 2003 which has data that I combine with my web files before burning. I have cygwin installed on the remote server in order to archive and compress the entire backup using tar. (This is not a post about cygwin, I just thought I would mention it in case anyone was wondering how I was running Linux commands after transferring it to the Windows box). After compression, the backup is over 12gb. The next step in my process is to split this tar.gz file into smaller chunks in order to burn them to DVDs. I use dual layer DVDs which are 8.5gb storage size. I also use cygwin to split the tar.gz into multiple 2gb files using the split command. When I burn them, I only put 3 files on each disk totaling 6gb to leave some padding in case this was a problem. The burn completes and says successful, although it errors out when in verification. I have tried this multiple times and it seems to fail verification at the same point every time which leads me to believe that it has something to do with the data. I have also done tests such as creating smaller backups with completely different data and brning that to a CD-R which worked fine, so I'm convinced this process can work, I just cant get it to work in the right situation. I have also tried burning one of the 7 split files to a dual DVD which also worked fine. I'm wondering if their is a chunk of data that is causing this problem in one of the other split files? Any help would be very appreciated. I am using Ubuntu 8.04 LTS, SH-S182D Burner, and using Nero 6 for burning software. Thanks.
burning apps "round up" clusters.. so trying to verify a burn against a set of files will always fail. The way to actually verify a data backup made in this way is to md5 the input files before burning and md5 the output from the disk just the same way as you verify a download.
I am slightly familiar with MD5 encryption, but I don't know how to relate that to this. Can you give me an example or a link with information on how to do this. Thanks for the quick response.
Also, can you explain what you mean by the following: burning apps "round up" clusters.. so trying to verify a burn against a set of files will always fail. -thanks,
The first part is easy.. a burning app will round up a random length file to a complete disk sector by padding zeros on the end of the data till it reaches a full sector. They can only burn complete sectors, and as a result the file size hash result will not match the size of the input file... verify failure.. copy the file back off the disk and compare the actual data and it will match as the zeros will usually be dropped (or if not you can open in a hex editor and see them).. It's a geometry thing.. the disk sector size doesn't match the file sector size on the hdd. You can actually ignore the burning verify error.. it's an artifact of the difference between the way data is stored on a hdd compared to the way the burning app puts it on the disk. md5 summing may throw up the same problem.. the best way to verify is to copy the data back from the disk and then check it. Personally I never bother as long as the backed up files unpack and look right when run from my /backup directory with a ./compare command I'm happy. http://www.ghacks.net/2009/01/12/verify-file-integrity-of-backups/
Thanks for the reply. does this only happen because the files are split? Because I am able to burn multiple files to dvd in other formats such as .zip with no problem. I'm just trying pinpoint the problem. Is their an easier way of getting these backups on DVD than my method? Thanks,
So I have turned off the Nero verification and burned my data across 2 dual layer DVD+R disks. When going back to validate the data, I am unable to transfer the second disk's file to my hard drive. It starts transferring and eventually froze about midway through. This is exactly why I need this data verified. Please help, the coasters are building up quick and I have deadlines to meet. Thanks in advance!
Can anyone give me info on a different solution, maybe converting to an ISO image? I don't know a whole lot about ISO, but from what I understand, it is an alternative archive format such as tar only without the options to compress. In that case I don't think this would be a great solution. does anyone have something else?
Is there some way I can modify these split files to prevent them from getting altered? Maybe archiving the split files before burning?
sounds more like you have bad disks.. are you backing up just the updates from rsync or are you doing a full mirror each time? .. rsync updates are literally just that.. the bits that have changed since the last backup.. I don't see that much practical value to them apart from being incremental hourly backups.. any major change or structure change can cause odd problems.. really I never do disk backups (burned on domestic grade media.. I use a rotation on hdd) so you need to look for specific solutions for your particular need.. anyhow.. I don't get what the problem is or why there is a windoze server involved with what sounds like a system which should be pure *nix here is basically the same problem.. http://www.linux-archive.org/debian-isp/235268-maildir-backup.html worries me you say "deadlines are looming" and you don't know by instinct how the system needs to be backed up.. I would shove a usb or nfs system on the server and mirror the whole thing straight with tar.. then use rsync for incremental backups to modify the original data directory dumps. You have stop the system otherwise while you backup.
Thanks for the reply, First, I am using rsync to create a incremental cycling folders which take up minimal space by linking all existing files to the folder previous to it using the --link-dest option. Then any changes are also reflected in the backup as well. I am backing up on an external hard drive before burning to DVD as well, and I need these DVDs for offsite storage. Rsync can do much more than just updates, although this is a valuable feature of the tool. I fully understand the way that the data is being backed up, thats not the issue. And although I'm wouldn't rule out having bad disks, I have burnt successfully with these disks when testing other data sources. I agree that this should be a pure Linux process, but you clearly have never worked in the corporate world. I have done my best to integrate two Linux servers into the office, but everything else is Windows. And I am incorporating a backup from the windows server with the one from my Linux server all at the same time. Also this is the only server that has a DVD drive, so it needs to end up there either way. So as for the last comment, I already do that. and no I do not need to stop my system to do backups the way it is currently set up.
