Website Crawler

This Is Source Code You Have Full Resale Rights

  • Dev and Pro version both come with every purchase
  • Source Code video included

This website crawler has been in development for months. I wanted my next product to be big and this is one of the best products I’ve ever made if not the best.

There are so many uses for this product. I built the website crawler “engine” so that it would be a beast that can crawl tens of thousands of webpages looking for information. What is that information? That is the question. This can easily be modified to scrape whatever you wish off of the web pages. It is so easy that I will even write the code for you (as long as it is a reasonable request).

Some ideas off the top of my head that you can scrape:

  • External domains from authority websites
  • Emails
  • Phone numbers
  • Any kind of videos to make a video site
  • Any kind of images to make a large image site
  • links that contain certain words
  • PDF files to make a document website
  • And more, the list is endless!

Here are some of the features you can see in the program:

  • Dev version comes with a simply coded HTML design – easy to modify
  • Multi-threaded
  • Extension filter to filter out pages you do not want to crawl
  • Custom user agent to tell the websites what your crawler is about
  • Ability to retry failed requests x number of times
  • Ability to crawl multiple websites in one run (may be taken away in the future)

Here are some of the “behind the scenes” features:

  • Intelligent filtering system that only acquires internal links to keep the program running
  • Unique ways of acquiring the links, both internal and external (took me lots of head scratching to figure this out)
  • Elegant solutions for the filtering and link acquisition
  • Comments where they make sense

Watch the video for more information.

Price may rise at any time.

Plugins Required: HTTP Post (Paid)Advanced File (Free) / File Management (Free) / Threads Counter (Free) / Large Data (Free) / Local Dictionary (Free)

Made and tested in Ubot Studio 4