webcrawler

KO | EN

gitlite — search

#hacktoberfest #react #ai #llm #golang #rust #deep-learning #linux #docker #macos #cli #kubernetes #vue #awesome-list

(★ 23)

A web crawler oriented to infosec.

webcrawler 최신버젼 다운로드

최종 버전 다운로드 (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

Web Crawler Security Tool
=========================
![GitHub last commit (branch)](https://img.shields.io/github/last-commit/verovaleros/webcrawler/main)

The Web Crawler Security Tool is a Python-based tool that automatically crawls a website. It is a web crawler oriented to help in penetration testing tasks. The main task of this tool is to search and list all the links (pages and files) in a website. The crawler has been completely rewritten in v2.0. The current version is still a work in progress, and the features shown below are still not fully ported.

Ported Features
========
* Implements a `--resume` option where you can resume a crawling activity.
* Crawl HTTP and HTTPS websites (even those not using common ports).
* It allows to determine the depth of the crawling (-C <depth> option).
* Generates a summary at the end of the crawling with statistics about the crawl results, including the number of crawled URLs, external URLs, files, errors, failed requests, and total transferred data.
* Uses CTRL-C to stop current crawler stages and save the status.
* Export the files identified in separate files and the errors and failed requests.
* Uses beautifulsoup4 for finding absolute and relative links.
* Implemented HEAD method for analyzing file types before crawling. This feature improves the speed of the crawler significantly.
* Does not crawl non-html files.
  
Unported features
========
* Identifies non-html files and shows them.
* Identifies directory indexing.
* Crawl directories with indexing (not yet implemented in v1.0)
* Identifies all kinds of files by reading the content-type header field of the response.
* Select the type of files to download (-d option). Ex.: png, pdf, jpeg, gif or png, jpeg.
* Select in an interactive way which type of files to download (-i option).
* Save the downloaded files into a directory. It only creates the output directory if at least one file is downloaded.
* Generates an output log in CLF (Common Log Format) of all the requests done during crawling.
* (beta) Login with basic authentication. Feedback is welcome!
* Tries to detect if the website uses a CMS (like WordPress, Joomla, etc) (not yet implemented in v1.0)
* It looks for '.bk' or '.bak' files of php, asp, aspx, jps pages. (not yet implemented in v1.0)
* It works in Windows but doesn't save results.

The original project was on SourceForge: http://sourceforge.net/projects/webcrawler-py.

webcrawler

webcrawler 최신버젼 다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

webcrawler

webcrawler 최신버젼 다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

Download README