On that point sendshare, rapidshare and a few others have api's for checking links which you should use. For Rapidshare you can group the links and then check them all with one api call. This really speeds it all up. Say it's a large movie or game with 20 links. Instead of you having to scrape 20 pages you just make one lighting fast api call. This really helps your server.


For FileSonic who keep banning your IP or whatever all you have to do is use their link checker http://www.filesonic.com/link-checker Group all the links, $_POST them and get the result. Don't go curling a page for every single link where possible and that goes for all file hosts as it's every innefficent.

I used to have a site like this called linkzbot.com of which their are a few topics here on KWWH about two years back. http://www.besthostingforums.com/64-...ll-forums.html
It got incredibly popular after 2 months and was checking almost a million links a day before the server packed up so ya concentrate now making it as efficient as possible before it get's too big.