I just purchased Anydvd HD and ripped a BD but have no idea what files were produced and what to do with them. I want to use Handbrake or Ripbot264 to shrink and convert to MKV but dont know how to start.

Is there a guide someone can point me to?

I'm not all that familiar with either Handbrake or Ripbot, but with AnyDVD running in the background and decrypting the disc, you may be able to encode using the largest m2ts file on the disc as the source (it's usually the movie). If either will work directly with m2ts file it might bypass the need for all that ripping.

Personally I use MeGUI for converting. It also has a HDStreams extractor which (with AnyDVD running in the background) will extract the video to an MKV file and extract/convert the appropriate audio stream(s). Once again you'd just work with the largest m2ts file on the disc, and when they're extracted you use the MKV file containing the original video as the source, adding the converted/extracted audio to the output file.

It's probably impossible to explain the whole process in detail in a brief forum post, but if you Google any of the above programs along with the words "blu-ray guide" you're sure to find one. The only problem is, if they're anything like the guides I found when searching for a MeGUI guide, they're mostly outdated enough to be misleading.

Maybe try opening the largest m2ts file for converting with your program of choice, then if you have any questions, ask them here.