Back with XP I had used Handbrake with no issue to encode a bunch of movies with no issue. They had been ripped to my hard drive, put through all the proper shrinking processes, and had the Video_TS directory loaded into Handbrake and encoded into Divx/Xvid with no issue. I went to Windows 7 and now those same steps produce a file that when watched looks like it has gone through a fun house mirror with parts of the right side over lapping the left edge and everything all shifted around. The sound carries over fine but the video is wacked out. I know the ripped files are fine because I can convert them using AutoGK and a couple over free tools so that part isn't the problem. I could just go on using them and forget about it but I have been using Handbrake for so long that I am curious to why all of a sudden it stopped working.
if you havent already, try a different playback program like media player classic homecinema. i think its probably codec related.