2

Some websites completely block Tor (for example, Yelp gives a 503), while some allow me to browse, but not use more advanced functionality (Wikipedia allows me to read content but not edit it, for obvious reasons).

I'd like to obtain a list of websites that block Tor. To this end, I've written two scripts in Python using the requests module which can query a url and interact with it. I'd like some help- what parameters can I look for while trying to categorize these websites? HTTP Status Codes might not be well suited (as already demonstrated by the example), so I would like to consider a few more things here.

Any help, advice or suggestion is appreciated. Thanks.

Tarun Verma
  • 21
  • 2

2 Answers2

1

This isn't a question about Tor. It really boils down to: How do I programmatically check whether two websites are the same?

Where those two websites are, 1) the site when browsing without Tor, 2) the site when browsing with Tor. (Assuming you won't be comprimised in any way by browsing the site without using Tor.)

At the most simplistic level, you could just compare the contents of the Response objects returned by the requests API. To do a complete job would be a lot more work. You'd effectively have to crawl each of the two sites, following links to child pages that may or may not be in different sub-domains (and so may have different firewall/filtering profiles), and deal with situations where user interaction is required (which would need to be handled by your script).

I'm not one for pessimism, and I certainly value the problem-solving/fun aspects of such a project, but I'd be inclined to find a pre-existing tool that does the job for you. There are a few threads on the Stack Overflow site that might also be of help.

Finally, on the Tor wiki there's a list of services known to block Tor, together with a list of software that can be used to do the blocking. Perhaps working out a way to check for signatures of the blocking software would be another approach. (This might be more complicated that just checking for Captchas.)

Richard Horrocks
  • 3,155
  • 1
  • 15
  • 24
0

I don't think there is a generic way to test whether a web site will 'let you do everything'.

Other than the HTTP response code, you can check that the Content-Location is the same, and that the Content-Length header is roughly the same. You might do some heuristics on the content itself to recognise captcha's.

Also, to begin with, you would need a list of sites to test.

Jobiwan
  • 3,685
  • 2
  • 19
  • 31