Torrent Speeds upload and download

i have noticed something while downloading torrents of any tracker when looking at my peers in your torrent. What i notice is the only time that i tend to get fast speeds of any other peer leecher or seeder is if they are on the same ISP as me (rogers.com). The only time i get a fast donwload speed like above 20 kB/s form on peer is if they use the same ISP. and th eonly time i can upload alot is if that peer is on my ISP aswell, so the only times i can upload above 20 kB/s is if i am uplaoding to a peer on my ISP. why is this? is my isp cheap or is this just a normal thing. i am on a torrent with 200 leechers and the only people that i can upload fast to our peers using the same ISP/host as me and same applies for downloading. Is my isp just cheap ebcasue bandiwdth on their own network costs nothing? I am properly connected i use uTorrent port 54000 and also forwarded port 54000 on my router. i have like 15 upload slots availible and can only send to 6 usually. I have a linksys WRT54G router and installed a custom firmware and a custom start up script so this router doesnt crash. (it is a huge porblem at linksys with all those routers becasue they track old connections and crash)