Tampilkan postingan dengan label crawler. Tampilkan semua postingan
Tampilkan postingan dengan label crawler. Tampilkan semua postingan

Rabu, 04 Desember 2013

Spider - PHP Website Crawler (Search)

This PHP script allows you to search for files in ANY website. All you have to do is specify what website you want to search, the kind of files you are looking for then click Crawl to watch spider handle the rest.

It comes packed with more than 150 file types pre-installed and multiple ways to view results, Spider is the tool at heart for people building crawlers and web scrappers.


Features



  • Over 150 file tpyes included

  • Accepts custom file types

  • Crawler Depth Managemment

  • Preview and export HTML

  • Saves results on server

  • Crawling Time Limits

  • Exports to SQL File

  • Bootstrap 3.0 Ready

  • Responsive Design

  • Excel CSV Ready

  • Eports to JSON

  • Much more…



Updates


29/08/2013 v1.0 – Updated Documentaion. Improved code comments. Fixed php errors and warnings.
30/08/2013 v1.0 – Updated to support PHP 5.3.

Mini Bots PHP Class (Miscellaneous)

ADDED NEW BOTS AND METHODS THAT:
- get images and user informations from Instagram
- get twitter user informations
- functions for parsing html pages as a spider




This PHP class is a growing collection of small spiders and web bot that go out on the web and make some small useful jobs.




Actually the mini bots can do those things:





  1. Copy a remote url to your local server: copyFile

  2. Get IP address of the user: getIP

  3. Get geographic informations from IP address: ipToGeo

  4. Get spelling for a word using Google: doSpelling

  5. Make a Tiny URL using tiny url service: doShortURL

  6. Decode a short URL of any service: doShortURLDecode

  7. Check if an URL to an mp3 is really an mp3 file: checkMp3

  8. Check if an URL exists: url_exists

  9. Do SMTP email validation: doSMTPValidation

  10. Get infos of an URL (keywords, title, images, description, favicon…): getUrlInfo

  11. Get infos of a VIDEO URL (title, description, thumb for youtube and vimeo services): getVideoUrlInfo

  12. Read facebook counters for likes, shares, comments, clicks…: readFacebookCounters

  13. Read the numbers of time a URL has been twitted: readTwitterCounter

  14. Read Google +1s counter for an URL: readGooglePlusCounter

  15. Get Google suggested keywords for a keyword: googleSuggestKeywords

  16. Get Latitude and Longitude (georeference) from an address with Google: getLatLong

  17. Get Wikipedia definition: wikiDefinition

  18. Get Vimeo Info on a video: getVimeoInfo

  19. Get exchange rates, currency converter: getExchangeRateFromTo

  20. Get push notification on android devices: notifyNma

  21. Ping pingomatic services: pingomatic

  22. Get images from a keyword: getImage

  23. Get images and user data from Instagram: getInstagramPics NEW!

  24. Get twitter user data (followers/followings): twitterInfo NEW!

  25. Get a page, find links, find emails… and more methods for spidering. NEW!



Future updates will bring new mini bots for you!
To fully work need cURL module installed on your server.




You can play with some methods in this interactive demo or check the complete test demo (same as live preview button).