Managing Files-A Loss-less Plean for the Future and Beyond!

OK, like most DV n00bs I am crawling my way forward, slowly forming a viable DV editing solution.

-I have a Sony DV Camera
-I have Firewire
-I have 200 Gigs of HDD space devoted to DV files
-I have a DVD burner
-I can make DVD movies

Today my problem is this:
I want LOSS-LESS storage of my DV files.

I see several possible solutions here (none I like)
1) Make 3 data DVD backups for each 12 Gig AVI file
2) Convert to Mpeg and make a data dvd backup
3) Keep on Tape (yuck)
4) Buy a ka-Jillion Gig Hard Drive and store raw avi files (double yuck)


Problems with each plan
1) I tried two ways to do this. I created a 12 gig avi file, but when I tried to burn it to DVD (as a data dvd) my burner program didn't span to muliple disks. (Do other programs work better?)

I then tried to MANUALLY CAPTURE the movie in three files. This was annoying and got old REALLY quick.

I can't find a program that will chop up an original into multiple files.

2) My second plan. I thought to myself "Jumbalia, why store avi files? It's not like you are going to make professional movies."
So...I was thinking that I'd convert it to "DVD Quality" and then I'd be OK because DVD's are my destination anyway....right?
But then I read comments like "..that Mpeg encoder sucks..."
So I am afraid that in 10 years I will be unhappy with the resulting conversion files.

3) I don't want to keep on tape for two reasons; Cost and deterioration.

4) Yeah....sure. I can buy a 300 gig HDD for 150 bucks, but that only like 20 tapes. And what if the HDD crashes?? Nah....DVD is best.

 

You’re wasting a lot of valuable space by using poorly encoded avi files. If you want to stay with avi files (although I can’t imagine why) so they take up much, much less room, then use a DivX encoder. It'll shrink a standard avi file considerably - to about 15-25% of it's original size without a loss in quality. But DivX may not be around in the 10 years you're talking about, although you could easily maintain a DivX codec on your DVD's for later use.

You're better off, though, using a good mpeg2 codec to compress your video files for safe keeping..but use a higher quality mpeg2 codec. Pegasys-inc.com offers their "TMPGEnc 3XPress" software that has a very good mpeg2 encoder, and is easy to use. Incidently, I've noticed that this software has become a "must have" item for serious video fanatics within the past year or so. 3XP will produce excellent mpeg2 files that will be better than the avi files you're now making. But to make a playable DVD from mpeg2 files you'll need to have a DVD Authoring program. It makes the .BUP, .IFO and .VOB files required by DVD players.

The 12 gig file you mention is unnecessary. A two hour movie encoded to mpeg2 properly will easily fit on a 4.3 gig DVD. Much less (about 2.7 gigs at 4000 kbits/sec encoding rate) if you want to drop the encode rate down to 4000-5000 kbits/sec. DVD players are mostly limited to 8000 kbits/sec, but encode rate above 4000 have the same visual quality usually.

There's no point in keeping your files in an uncompressed format using up far too much space on your hard drive!! And not too worry, mpeg2 will be around for quite a while. A lot of digital video on cable is now shifting to mpeg2 for quality. Currently, I capture exceptional quality mpeg2 video signals directly off a set top satellite receiver right now, for instance.

Tape: NEVER, NEVER store video on tape for any length of time!!!! Get your recording to a hard drive then onto media as soon as you can!!!! I've been burned in the past by tapes (as has every video fanatic that I know!!)

Which Sony camcorder do you have? If it has a ccd size of less than 680,000 bits than Firewire will work for you. If it's above that and you want to keep higher quality video then you're going to have to go to a video capture card to get the file into your computer with very little loss....

P.S. I chuckled over your comment: "Buy a ka-Jillion Gig Hard Drive and store raw avi files (double yuck)"

First off you would never want to store raw avi files. But the chuckle came when I read your comments, and then looked over my work area. I have 4 computers in front of me, two have six 300 gig hard drives (I use HiPoint pci extender cards to add extra drives). The other two systems have 2 and 4, respectively, 300 gig hard drives along with a couple 250 gig units. Most importantly, three of these systems have ATI video capture cards to capture video at very high resolution.

I started out nearly 6 years ago with essentially the same question that you have: How can I save valuable (to me and family) priceless family video's. As you can see, my hobby has gotten a bit out of hand!! So watch out!

 

Alright, there seem to be a few misconceptions floating around in these posts.

First: Firewire is an appropriate capture method for ANY video encoded in 4:1:1 25Mb/s video. This includes miniDV, Digital8, DVCAM and DVCPRO. CCD size has nothing to do with it (and CCDs are usually measured in physical size, because pixel resolution often has little bearing on overall quality).

Second: If you originally captured your video on DV tape, its already been compressed by a lossy codec, and there's nothing you can do about it. Very few cameras exist that capture video uncompressed, and they're all very, very expensive). Once captured to tape, your video has been compressed with the color sampling at 4:1:1, data compression of 5:1 and bit rate of 25 Mb/s. For more information on DV compression, check out: http://www.adamwilt.com/DV.html

If you do all your editing and rendering using a DV25 codec (Microsoft DV in Premiere and DV/DVCPRO NTSC in Final Cut), you should see no quality loss in your video. DV25 has a storage rate of about 13GB per hour vs. somewhere around 160GB/hour for RAW uncompressed video. Saving DV25 footage as uncompressed will not increase the quality one bit.

As far as storage is concerned, keep a few things in mind. Encoding in mpeg2 WILL further compress the video. Using a proper encoder you may be able to achieve pretty high quality, though. Its going to be a decision between video quality and how much you can fit on a disc. There is a visual difference between recording 1hr per disc and 2hrs per disc. Also, there's a lot of people experiencing issues with CD-R "rot" (myself included) after a few years. CD-Rs and DVD-Rs are not made the same way commercial CDs and DVDs are made (commercial discs have grooves written into them, CD-Rs use a transparent dye). So consumer DVD-R discs might not have the long term storage capability you seek.

Tapes are not necessarily a bad idea, if you buy high quality master DV tape, and store them in a proper environment, they may last quite a long time. Another option would be to transfer them to DVCAM or DVCPRO tapes, which use larger tape tracks, and higher quality materials to be less error prone.

Hard drives are also an excellent idea, but obviously expensive. One thing to keep in mind is that if a hard drive crashes or is corrupted, there's a good chance the data would be very difficult to recover. Often tape is easier to recover because only certain sections of it become corrupted.

This isn't an easy question to give a definite answer to, it all boils down to money and what quality you'll be happy with.

 

First: You don’t capture with Firewire!! Firewire is ONLY a data transmission method using the IEEE 1394 specifications.

Secondly: Pixel resolution has EVERYTHING to do with quality. While physical size is a poor judge of quality because the manufacturers continue to shrink the physical size and increase the pixel sensing ability to improve quality. Look at the quality between a 375,000 pixel ccd and a 1 meg pixel ccd.- there’s no comparison unless your quality acceptance level is pretty low.

Thirdly: The specific problem with transferring the compressed data on the video tape is not specifically with the Firewire data rate (it can keep up - but barely); it’s because most camcorder manufacturers use Microsoft’s DV25 codec!!!! Consequently, the software manufacturers use Microsoft spec’s to try to turn the file into a good quality video presentation. If you’re accepting the output of the currently available video massaging software as great, then you’ve got a lot to learn!! By playing back a video using the camcorder manufacturers processor and codec you are NEARLY nulling out the problems inherent with Microsoft’s codec. In other words, the camcorders own processor will undo the codecs “mistakes” as it converts the video to composite output. This is an easy item to prove. Capture the composite output of a high res camcorder using a high resolution video card, then transfer the same file via Firewire to ANY of the leading video programs. Blow up the frames from each and you will be startled by the difference. I, and a considerable number of others, have run through this sequence every time there’s been a “New, Improved” video editor on the market - with the same results.

Your concern’s about CD-R "rot" are well founded. However, the major CD producers claim they have gotten around this issue by better edge sealing techniques; but, only time will tell. The same companies have made statements that their DVD production will never have this problem because of the research they’ve done. Here again, we’ll just have to wait and see, although I’ve not read or heard about anyone running into this problem with DVD blanks - so far. But it is a concern that lurks in the background!

Tapes: Industrial archival tape will stand up to some severe handling - but even then, when I was still a corporate manager, tests indicated that large mag disc paks were more secure for long term data storage. But either required storage in controlled environments - something most users can’t do. All to frequently the camcorder tape is stuck in a storage container of some sort, then banged around over a few years time. Then, when you want to retrieve the data you have not-so-good quality. You could install a 9 track tape system for storage - but I doubt that many users will really want to go to this extreme.

One thing that we both agree on whole heartedly is your statement:
“This isn't an easy question to give a definite answer to, it all boils down to money and what quality you'll be happy with.”


.

 

ADDENDUM: Yesterday I recorded my wife's Choir group as they performed. I then did my normal procedure of capturing the video to a hard drive using the composite video output from the camcorder, then compressed it as an mpeg2 file using 3XPress, then authored it as a viewable DVD with DVDAuthor. The mpeg2 final file was 2.944 gigs in size.

Then, I recorded the same file from the camcorder tape to disk by using Firewire and Windows Movie Maker (with the resolution set to DVI-AVI at 25 mbps). The resultant file produced was 10.985 gigs in size. I then burned a DVD from this method.

Later, when the choir group came by our home to practice, I first played the Movie Maker version, asking most members what they thought of the DVD. Most comments were the polite "Very nice", "Looks good", etc., etc.

Then I put in the mpeg2 version for all to view, then asked for comments. "Wow, what did you do?", "The difference is amazing!!", "It looks like HDTV in comparision!", etc...

And there is even more of a difference if you blow the individual frames up. For instance: As I would zoom in on each memebers head and shoulders, the intricate design of the embroidery insignia vas clear and concise on the mpeg2 version - but on the DVI-AVI based version the same insignia was not clear at all - more of a fuzzy image that you could recognize only if you knew what it was.

AND - I could easily have reduced the mpeg2 encoding down to about 2 gig in size by dropping the encoding rate - and the output would still have been somewhat better than the DVI-AVI version.

Now, back to jumbalia's question... How can a video file be stored and retrieved later RELIABLY. My answer still stands - at this point in time, compress it to mpeg2 with a GOOD encodec. Mpeg2 will be around for quite some time as it is rapidly becoming THE standard that should last for the next few years at least. Regardless, there will always be archival codecs for these files for a long time to come anyway.


.

 

Mark7, you are awesome!
(ZombieHero, you need to have a few beers and chill out. I'm buying)

I will reiterate Mark7's advice to me and see if I have it right
BUT FIRST!
An anecdote from church. I asked around at my church and was directed to the guy who handles our video. Well his advice was VALID, but FLAWED.
His advice:
Step 1: Buy a $1800 video editing package
Step 2: Buy a $1000 optical disk drive.
Sure.....Maybe I should upgrade my $600 DV camera first.


Anyway, So this is what Mark7 is telling me:
-Transfer my DV (via FIREWIRE!!!!!) to my HDD
-Encode to Mpeg2 using a high quality Mpeg2 encoder such as the TMPGEnc program from Pegasys-inc
-Burn the MPeg to DVD for storage
-ERASE the DV tapes and ERASE the .avi files

Cool?


(BTW I searched for "TMPGEnc 3XPress" in Google and guess what I got....THIS VERY thread! Go figure.)

 

Well, for what it's worth. This is what I do.
Tranfer the video to my PC, make a nice edited movie (usually ending up with using not more than between 10 and 15% of my origianl material).
Encode it two MPEG-2 and put it on DVD (and a second one for backup).
Play my edited movie back to tape through DV-in to preserve an "original quality" backup.
Delete the avi's from my PC.
Maybe I should transfer the tape to a new one once every five years or so...
Unless, my house burns down I think I am covered well...

http://www.digitalvideoclub.com

 

Reading this thread again, one question comes up in my mind:
My assumption has always been that when I edit (with Premiere) and export my work with the option recompress off, non-edited frames are directly copied from the original files. Or do they actually get decoded and compressed again?
I would think it is the first, but I maybe wrong.

If my assumption is correct, only edited frames (which, in my productions will less than 1% of the total; mainly transitions and occasionaly PIP) are the only places where a codec can have a influence on quality on the path: transfer to PC - edit - transfer back to camera. This would definitely support my case to at least also send a copy of the edit back to tape and preserve it as "original quality".

BTW: I also assume that editing of audio does not require recompression of the video, because if that would be the case, suddenly more than 75% of my franes would become edited frames...

 

I specifically joined this forum just so I could say thanks....to all the posters, and especially to Mark7. I recently got a Panasonic PV-GS120 MiniDV camcorder (well, first I got a Sony DVD camcorder, then took it right the hell back, but that's another story).

I'm totally new to DV, and my first attempts at capturing and moviemaking, with Dazzle's MovieStar and Movie Maker, were a little unsettling.

However, going on information I gleaned from this thread, I tried using WinDV to capture, then TMPGEnc to encode to mpeg2, and the results were MUCH better.

Even in full screen with WinDVD, the picture was quite smooth and acceptibly sharp.

With my first attempts I seemed to be having a lot of problems with dropping frames, but WinDV fixed that. But then, pulling the avi files into Movie Maker (MovieStar) editing and outputting still seemed to result in quality problems in the final product.


But, at the very least, now I know I can get good results with
camera -> avi capture with WinDV -> mpeg2 encoding.

Now, I guess the question is, is there a way I can edit and combine and split and add effects while generally maintaining this quality? Seems when I try to use MovieMaker or MovieStar, quality goes downhill pretty quickly. Do I need better software or something? Or am I using the stuff I have incorrectly?

Thanks a bundle,

ding

 

I suggest you better start a new thread for this...
Anyway, if you use DV-AVI in your editing program and export it as DV-AVI again, you should have no loss of quality whatsoever. Just make sure that you export it in DV-AVI again.
For a list of some video editing programs check at the bottom of this article: http://www.digitalvideoclub.com/basics/editing.php

 

Yes, you're right about the off-topicness of my post, sorry about that....I just posted to say thanks, then got carried away and asked off-topic questions. I realized that just after I posted. Mea culpa

Anyway thanks for the additional info, and I'll research the site for more information regarding the questions I asked.

ding

 

Well Vurbal, technically you are right, but I disagree if you say "why this isn't really true for most people".
"Most people" (and specifically "most people" in this thread) don't edit all of their frames. They simply cut and paste and use some transitions and therefor probably end up with less than 1% of their frames recompressed. The other 99% will be "smart-rendered" as long as you don't go messing around with several DV-codecs installed on your system so they will remain unchanged.
Furthermore, I think that the poster of the question refers to another level of quality decrease as may be the result of recompressing DV-AVI to DV-AVI...

 

I again have to challenge that statement.
In the viseo export settings in Premiere there is a box called "recompress". As far as I know if you uncheck that, unchanged frames are copied. I did not perform any in-depth mathematical frame comparison, but that is what I see stated in several threads in the Adobe Premiere forum (I also would not know what else this box would be for).

I also found this official Adobe technical document:
http://www.adobe.com/support/techdocs/318277.html
Read the very last sentence of that article....

If you have hard evidence that we are being mislead by Adobe, I certainly would like to get that.

 

OK, thanks for letting me know. I recognized your comments here as knowledgeable, so you seriously made me doubt ;-)

I guess the only thing that can be wondered why the Adobe guys hide this setting somewhere in a corner of a window (where many people will not look) and why the default setting for "always recompress" is ON. I don't see why anybody would NOT want to use smart rendering...

 

Mark7 makes an extrodinary claim: That transferring video using the analog composite output of a camcorder is superior to a Firewire connection.

While not doubting his assertion, I'm not sure that I follow his explanation as to why this would be true.

Can anyone else confirm Mark7's experience with this?
Would the installation of a different codec on the PC make any difference?

Thanks