Verifying the Astiga podcast crawler

You can verify that a HTTP request is coming from the Astiga podcast crawler by running though the following checks. All of the checks should pass to be confident that it is the Astiga podcast crawler making the request.

  1. Check the User-Agent of the request contains “Astiga”.
  2. Perform a reverse DNS lookup on the request’s origin IP. The domain name should be asti.ga.
  3. Perform a DNS lookup for the IP address, and verify that the IP is the same as the request’s origin IP tested in the previous step.
  4. Check that the IP address matches one of the prefixes listed on the Astiga IP list.