Link checker idea

[Whoo]

Hey,

I was just thinking about a cool feature for checking links on my DDL website but wanted to get your input on it.


The idea is to whenever a user clicks the "Report dead links" on a submissions, a new window opens.

In order for the download database to stay consistent I could crawl the "check download links" website using curl and display that page to the visitor. This way I could also check if a certain percentage of links is down and remove the submission from the database. The only negative point about this is that my server will have to crawl the "check download links" page everytime a visitor reports a dead submissions.

An alternative I had in mind is to load the "check download links" page into an iframe and then check the amount of dead links via jquery/javascript.

I would have to do a on document ready/on document load event on that page and check the amount of dead links this way.

This way it's all done via the visitor and thus no extra load on the server.

Would this be do-able? I mean grab contents of a page loaded in an iframe and iterate over the document elements? By the color of each link you can make a % of how many dead links there are.

If it passes a certain percentage I would just do an AJAX request to an private API to remove that certain submission.

Inputs are appreciated

-Whoo

Whoo Reviewed by Whoo on 27th Dec 2010. Link checker idea Hey, I was just thinking about a cool feature for checking links on my DDL website but wanted to get your input on it. The idea is to whenever a user clicks the "Report dead links" on a submissions, a new window opens. In order for the download database to stay consistent I could crawl the "check download links" website using curl and display that page to the visitor. This way I could also check if a certain percentage of links is down and remove the submission from the database. The Rating: 5

[SplitIce]

An alternative I had in mind is to load the "check download links" page into an iframe and then check the amount of dead links via jquery/javascript.

Cant be done. Google Same origin policy.

[litewarez]

+1 Splitice.

[desiboy]

Good idea
YQL does allow us to make cross-domain GET requests..

http://www.wait-till-i.com/2010/01/1...query-and-yql/

&
https://github.com/codepo8/crossdoma...jquery-and-yql


best way is to use JSONP , but I highly doubt that file hosting sites offer that service .

[SplitIce]

except there are service limits on that, I think you get something like 1k requests per day (5k, but external fetches cost X)

[desiboy]

I don't support the use of external services..but Yahoo is a reliable company and the limits are pretty much practical ..and can be used on live sites with decent traffic ..

[SplitIce]

hehe you found them, so 1,000 links per houw. Might as well do it on the server. I just know there are limits from when I utilised their term extracter interface.

Although if he does it clientside he will use the clients IP I suppose. Still serverside would be better.

[Whoo]

The most efficient method would be the best for me, ill look into YQL.

Thanks
-Whoo

[MR HARRY]

Good idea

[Momoe]

Why not use some link-checking script within your site? Server load shouldn't cause much trouble. The hosters offer an Api or at least a special page were you can check like 25-100 links in one step - so no big network load. Simply run a cronjob every 5 minute which checks 500 links. Can be combined with your site's db so that offline downloads get invisible to your user.