C

crawler

BigPhish - An all-in-one solution to identify and monitor phishing kit deployment

Spremioloon.Com data scraper

Dark Web Search Engine : a dark net crawler and database viewer.

Web crawler for checking the validity of your documents.

Findpapers is an application that helps researchers who are looking for references for their work

Novel Feast is the website where you can read Chinese, Japanese, and Korean Web Novels, Web Comics Light Novels Online Free https://novelfeast.com/novels

Finds webpages containing recipes

HTTP request module customized for crawlers.

Discover apps by different mehtods. Mass download app packages and metadata.

Bilingual TED-TALK, Crawler, Crawling

A data scraping and manipulation library with command line interface with multi-database support for MSSQL, MySQL, PSQL, MariaDB and SQLite. View jsdoc and coverage reports at, https://rdfedor.gitlab.io/datahorde

A Python crawler to gather information about augmented reality applications. (For a research project.)

A Python app for scanning large data sets of URLs for a given signature and storing the results to an ElasticSearch index. Useful applications for CERTs and security researchers, maybe others.

Python tools for ethical hacking

Ethereum Internal Transaction Crawler lets you extract internal transactions from etherscan.io and transform them into CSV format.

A simple email extractor for obfuscated emails.

A crawler that looks like a browser for distant servers.

A Python module for collecting and providing popular user agent strings, with a requests session which rotates user agents.