This sounds like a pretty well thought out system.
It will definitely do you nicely.
The external box idea is IMHO preferable to an all in one traditional soundcard as well. It seems that breakout boxes usually give you more options, and TerraTec are a very good make for the money. Not sure how they manage it.
If, however, you are considering Surround you may want to rethink at one point but for 24/96 you are going to be just fine.
Media players.
There are a couple of them that are quite good - it all depends on budget really.
WinDVD 6 Platinum will allow playback of just sbout everything you can throw at it, including DTS-CD and DVD-Audio, as well as DVD-Video discs.
As for WaveLab, get v4 if it is cheap enough, and upgrade it to v% which will give you DVD-A authoring should you require it.
If all you want to do is store everything to HDD and play back from there, you don't need DVD-A though.
There is a freeware Madia player called VideoLAN, at
http://www.videolan.org/vlc/which will also happily play back just about everything you can throw at it.
You will definitely hear an improvement re recording your vinyl at 24/96, as the dynamic range of 24 bit audio is 144dB. The wider word really makes a difference.
I'm not 100% convinced about 96KHz samplerates, as on most vinyl there is a limited frequency response and to be blunt - it just don't go up that high! To use 96KHz to my mind implies needing to reproduce frequencies at 48KHz, and I know of nobody who can reliably hear that high without perhaps spending many thousands of pounds on seriously specialist equipment. Then, of course, your amp & speakers also need to be 48KHz capable, or else the extra disc space is wasted. But this is just my personal opinion. I'd stick to 48KHz tops, 44.1 if you intend to write CD's.