The most painful/hardest thing I have ever done (that I can say on the internet) is SASR selection. I trained for it for over 7 years and it was still extremely hard. It's not just physically straining either, some of the exercises screw with your mind. And most of the time you are sleep deprived. Then you have 18 months of training.
SASR stands for Special Air Service Regiment, it's basically Australia's version of the British SAS. I've heard the selection process for SF is a lot easier now though with the commando direct entry scheme bullsh*t.