By popular request, this thread is specifically for tweaking the quality of your CCE encodes using DVD Rebuilder. If you have problems setting up or running any of the programs used please start a new thread as I'd rather not see it get cluttered with off topic posts. Believe it or not, this is actually very simple to do. Especially since the latest version of DVD-RB gives you the ability to insert lines to the AVS files. The key is is understanding what sorts of things require higher bitrates for MPEG encoding. Fast motion, sharp edges, and small details are the enemies of low bitrates. If you look at a movie that's been compressed with a program like DVD shrink, you'll probably notice that the worst parts of the movie have significant amounts of detail removed in a few areas. This is the right idea, just not the best implementation. Fortunately, since DVD-RB uses AviSynth to frameserve with, there area lot of options available to do a much better job than that. You'll still be removing detail, but instead of removing a lot of detail from small areas of a frame (which often makes it obvious due to the contrast with the nearly perfect parts) you want to spread the bitrate loss across as much of the frame as possible. To do this you need to use some additional AviSynth plugins, which you can download from http://www.avisynth.org/warpenterprises/ The plugins you want to look at are denoisers, because one side effect of denoising is to blur some of the details of the picture. Now instead of CCE trying to figure out which details can be kept and which should be removed, which is not it's primary purpose, you can use an AviSynth plugin designed expressly for that purpose. The first one to consider is Undot. Like the name implies, Undot removes small dots from the video. Its original purpose was removing noise from broadcast video captures, but dots are dots, and they're some of the best details to get rid of since by definition they're not as noticeable as say the edges of larger objects. This particular plugin does very little blurring, making it a great choice for most video. While Undot is great in terms of keeping picture quality high, it also has a relatively minimal effect on compressabililty. In other words your encode will still probably have some areas that CCE has problems with. To deal with this you can combine it with another denoiser called Deen. Deen is what's known as a 3D denoiser, meaning that it considers how each frame relates to the frames around it, unlike 2D denoisers which only look at a single frame at a time. Before you can use these filters you have to download them from the site referenced above. There are 2 different columns with downloads. Make sure you download from the column for AviSynth 2.5 otherwise they won't work properly. Once you have the zip files, extract the DLLs from each one and copy them into Crogram FilesAviSynthPlugins To add the filters to your DVD-RB project, before the prepare phase of your project go to the Options menu, go to AVS Options, then Advanced Options, and click on Filter Editor. A window will open for you to type additional lines for every segment to be encoded. Type in this line: Undot().Deen() Click the Save and Exit button and continue normally. More to come as time and brain power permit. Edit: I should probably mention that there are some cases where denoising in general, and the Undot/Deen combination in particular, isn't a good idea. Some movies have noise intentionally added as an effect. The 3 examples that come immediately to mind from my collection are Saving Private Ryan, Band of Brothers, and Pi. In the first 2, noise is used to add a period feel, and while it could probably be removed and still leave a good picture, you'd lose the feel that the noise gives you. Pi, on the other hand, is an independent black and white movie and a lot of noise is added to some scenes to reflect the emotional state of the main character. Once again, if you remove it you make a substantial change to the feel of the movie. _X_X_X_X_X_[small]Build a man a fire and he'll be warm for a night. Set him on fire and he'll be warm for the rest of his life Backup A DVD With DVD Rebuilder & CCE Basic: http://www.afterdawn.com/guides/archive/dvd_rebuilder_tutorial.cfm[/small]
I'm starting to use Rebuilder with CCE and the results are simply awesome. Great picture even at reductions of 50%. I got one question to you Vurbal, here goes: I have Agatha Christie's Poirot series. These are DVD5 with 2 episodes each, in 4:3. The image of the original is not very good compared with the actual standards. What i wanted to ask is the following, if I run Rebuilder with these two filters (Undot and Deen)and convert from 4:3 to 16:9, will the image improve by the use of these filters. I have made a project and run it, then it appeared a message saying the DVD will already fit in a DVD5, asking me if I want to continue. I said Yes and the normal progress of Rebuilder (one click mode) started analysing and then encoding, but the speed was so slow (Slow computer here) that i turned it off. Have you ever tried something like this? Btw, sorry for my english, i'm from Portugal.
Depending on your system it's certainly possible to get very slow speeds due to AviSynth filters. I've been cleaning up my Star Trek DVDs with Undot and DeScratch and my methods reduce the encode speed to 0.34 (from around 1.5) so I can understand your frustration. Unfortunately, in order to clean up a poor source it can take quite a bit of time if you want to do a good job. You could probably get it done faster with a filter like Convolution3D, but I don't think the results will be as good. If you'd like to try other filters you might want to read through my AVSEdit/AviSynth guide: http://www.afterdawn.com/guides/archive/avisynth_tutorial.cfm And your English is a lot better than my Portugese so I wouldn't worry about that ;-)
So you use undot and descratch, not undot and deen? I've tried what i've told you in the last post, this night with rebuilder. When i got up I went to see and rebuilder has made a Video_TS folder with ~230 MB from a source of ~4 GB. My source was DVD files with 4 GB. I started Rebuilder with one click mode and in the filter editor i've put undot().deen(). In Avysinth options i've put to transform from 4:3 to 16:9. All the other options were normal. I pressed Transcode and a message appeared saying the source was already in size to fit in a DVD5, and asked me if i wanted to continue. I pressed yes and went to sleep. When I got up the result was a 230 MB video_TS folder wich ran OK to the menus, but then when i press to watch the episode nothing happens. You're improving your Star Trek DVD's with filters, how do you do it? Am I only to use Avisynth with AVSEdit. Because if that is the way I don't know if I'm able to do it. Fro what i read from your avisynth guide it doesn't look that easy. Enlight me, please. Is it possible to do it with rebuilder or do I have to use only Avisynth?
Just a warning. You hear a lot of varying stories about filters -- and they are good tools. But remember that a filter always distorts the original picture, it's just that sometimes the distortion looks better than what you started with. Filters work good in controlled circumstances -- undot() is a good example. Sometimes when recording from a TV you get those momentary single pixel flashes that last for one frame and undot() can clear them up. But, if you have an ugly source, filters usually can't help -- except in the same way that the bottom of a beer bottle can sometimes make the lady across the bar look better... a distorted view can sometimes be better than clarity.
@bigqual: I've been using DeScratch for my Star Trek encodes because normal denoising is far too destructive on some scenes because the transporter effect is essentially noise. I don't remember what kind of speed I normally get with Undot and Deen, but it will certainly mean a performance hit because it's examining each frame multiple times. @jdobbs: That's a good point. I generally tell people that for the most part using filters doesn't actually improve the picture, but it may trade imperfections that are unacceptable for ones that aren't as bad.
Some earlier rock concerts were filmed with video cameras (3/4 inch Beta and VHS) in dark concert halls. The earliest ones used tubes to capture video, and then came magnetic oxide semiconductors, closely followed by the Charge Coupled Device. All of them had problems in low light settings although the later CCD’s did, and are still doing a pretty good job. In low light frames the dark background becomes less detailed, snowy. Can Undot affect a positive outcome and remove the snow without affecting the clearer scenes on these types of DVD’s?
Actually I find that undot().deen() does a pretty good job of this on DV source that I've done myself in very low light.
jdobbs I'd hoped that was the case because I have a lot of concerts on DVD that needs a little ajustment, thank you. Vurbal This thread was ahead of its time, we were just in the 1 click trancoder phase when you started it. We've all come a long ways since then thanks to you and jdobbs. We can now ask the right questions. _ _X_X_X_X_X_[small] Once you eliminate the impossible, whatever remains, no matter how improbable, must be the truth." Sherlock Holmes (by Sir Arthur Conan Doyle, 1859-1930)[/small]
Well then now is a good time to buy some of the Star Trek original series DVDs. At one time the big challenge for encoding was Star Trek TNG because of it's hybrid of film and video (for effects). As <he whose name I won't mention> correctly said, the challenge was that you ended up either changing telecined film to actual interlaced video or turning interlaced video into telecined progressive frames with a combination of deinterlacing artifacts and choppiness from removing fields. Now that jdobbs has come up with such a simple (and obvious in hindsight) solution that challenge has been effectively eliminated. The original series, on the other hand, presents a set of problems that make those issues seem a little trivial IMO. If you want to learn both the benefits and drawbacks to AviSynth filters you'd be hard pressed to find a better source for it. For starters, the film has a lot of defects from age that have never been cleaned up. It has a lot of scratches and what appears to be deterioration of the celluloid, which is compounded by some portions that are just incredibly dirty. The stock footage clips, like the Enterprise flying through space or orbiting planets, is even worse than the rest. To make things even worse, despite being 100% film, every episode I've seen is encoded interlaced. This is a challenge to clean up no matter how you look at it. My initial thought, before looking too closely, was that it should be easy to make at least slight improvements. I started by using UnDot. Unfortunately it didn't make as much difference as I hoped it would, and the more I looked at the source, the more I realized how big the problems were. Due to the dirty film, there really aren't very many small dots. The vast majority of the flaws are far too big for UnDot to have any effect on. As I've said before, the benefit and drawback to UnDot is that it doesn't make major changes. It helped, but I don't know if I'd notice the difference if I didn't have so much practice looking for flaws. After several hours of research on Doom9 I thought I had found the solution. DeSpot is a plugin made specifically for removing spots from film. It's a lot like UnDot (in fact I think with the default settings it does exactly the same thing), but you can tweak some settings and make it filter a little more heavily to remove bigger spots (UnDot only fixes single pixel anomalies from what I understand). I was very excited with the results I got - that is until I watched more of the filtered video. The big problem is with the transporter. The tranporter effects are made out of noise, and as soon as I got the settings correctly set for a slight, but definitely noticeable, reduction of the ugly spots, the transporter effects were being seen as noise and blurred together. I tried lightening the settings, but until I got them back down to UnDot levels the transporter issues remained. So back to the drawing board (and searching Doom9) I went. This time I came up with DeScratch. Since there are a lot of scratches on the film it seemed promising, and in fact it has given me the biggest improvements in quality without damaging the picture. Ironically, I can't use the settings I'd like to on it because part of using it to optimal effect is to use DeSpot to remove general noise first! When I used the settings I would like to, which would allow me to remove most of the scratches, it started detecting the edges of objects as scratches because there were so many defects around them. I eventually found some settings that are very non-destructive, but unfortunately they also leave the majority of the scratches untouched. Any scratch bigger than 3 pixels wide (and there are a lot of them) gets ignored completely. On the other hand there's a significant improvement to some areas of the video so I'm not exactly unhappy with this solution, it's just not as much improvement as I had anticipated. Now for the fun part - dealing with the itnerlaced video. There are basically 2 types of denoising - spatial and temporal. Spatial denoisers compare pixels to other pixels in the same frame. that means that if you have interlaced video you have to separate it into individual fields so the denoiser knows what pixels are actually next to each other. See the second post on this thread for more details on that issue: http://forums.afterdawn.com/thread_view.cfm/99448 The other type of denoising is temporal. Instead of making comparisons to pixels in the same frame, each pixel is compared to the same pixel in adjacent frames. For this type of denoising separating the frame into fields isn't necessary because the pixels above and below can't cause erronious noise detection. Some denoisers actually use a combination of the two, and are called spatial-temporal denoisers. Spatial Temporal Denoisers tend to be the most accurate because sometimes a pixel that looks like noise when it's only compared to adjacent pixels can be determined not to be because it matches the same pixel in adjacent frames. Since there is a spatial component, you need to separate frames. However, the temporal component adds an additional level of complexity because top and bottom fields are made up of pixels from different spatial locations. If you use the standard SeparateFields filter you'll get a stream with top fields followed by bottom fields, making it unsuitable for accurate temporal filtering. The solution for this is to separate the fields, but keep them in the same frame, stacking one on top of another. Once again you run into a problem. If you stack the top field on top of the bottom field you now have pixels at the bottom of the top field being spatially filtered by comparing them against the pixels at the top of the bottom field (re-read that if it's confusing). Fortunately there's a solution to this as well. One of the many bright AviSynth coders has created a pair of functions called FoldFieldVertical and UnfoldFieldsVertical. By using these functions you can separate fields, flip the bottom field vertically, and stack them in the same frame. By doing this you keep the bottom pixels of each field in the same position relative to each other. You can find more information on this, as well as the code and instructions for using it, in my AVSEdit/AviSynth guide on our Guides page: http://www.afterdawn.com/guides/archive/avisynth_tutorial.cfm Note: Alternately you can separate fields and place them in top-bottom-top-bottom order and filter just the odd fields first and just the odd fields in a separate step, but it's more complicated to do things this way. It was also problematic for a series of filters called Dust that could only be used once in a script. UnDot is a purely spatial filter since it only compares pixels to adjacent pixels in the same frame. DeScratch, on the other hand, is a spatial-temporal denoiser because it not only has to determine possible scratches by comparing adjacent pixels, but make sure they're not part of the signal by finding out if they occur in 2 adjacent frames. Therefore you can't filter a sequence of fields that alternates top-bottom-top-bottom. In order to add these filters to my AVS files, I run prepare normally and then open the project in RB-Opt and use the AVS Editor to make changes. I have all my filters saved as favorites (you can do this ahead of time by adding each one as a separate line in filters.txt in the RB-Opt directory) and I can quickly add them. I start by removing the last line, which is ConvertToYUY2(interlaced=true) because once I separate fields I can do a progressive conversion which I've found to be slightly faster. Then I add my own filters, test, and save. Here are the lines I add (with comments added): #Separate and stack fields with the bottom field flipped vertically UnfoldFieldsVertical(true) #Remove single pixel anomalies UnDot() #Required Because C Plugins Don't Autoload From Plugin Directory LoadCPlugin("C:\Program Files\AviSynth 2.5\plugins\C\descratch.dll") #Remove Scratches 3 Pixels Wide Or Less DeScratch(mindif=54,maxgap=3) #Progressive Colorspace Conversion ConvertToYUY2() #Put Fields Back Together FoldFieldsVertical(true) Edit: As I've mentioned before, my scripts with this filter chain run at a fraction of the speed they would without them. The original scripts encode at a speed of around 1.5 while the modified ones encode at a speed of around .35 If you think that's bad you should see what some of the script gurus at Doom9 do. Some folks, like mf, have been known to write scripts that encode at 1-2fps, and I wouldn't be surprised to find scripts there that you could actually calculate seconds per frame on instead. _X_X_X_X_X_[small]Looks like I picked the wrong week to stop sniffing glue DVD Rebuilder Guides: http://www.afterdawn.com/guides/archive/dvd_rebuilder_tutorial.cfm http://www.afterdawn.com/guides/archive/dvd_rebuilder_tutorial_advanced.cfm[/small]
Vurbal You weren't exaggerating when you said this was a long post even for you. Although I’m a big Star Trek fan this may be what I need to clean up some of those 1970’s rock concerts. The old ¾ inch camcorders just could do it in a dark concert hall. I let you know how it works out.
Yeah, I'd be interested too. (Good money says there's a few Grateful Dead's in that group of tapes Sophocles has.)
Vurbal Never got to try it yet, right after I posted I realized that doom9 was down so I couldn't download. Then along came a hurricane Charlie and I live in its forecasted path so I had to prepare my property, then when it passed I had to undo everything. I just downloaded the plugins yesterday. But I will get there and I will post my results. Doc Yes some of the backups will be old concerts of the Dead, so no ones betting.
