C

crawler

Worldplay's core library for extracting data from streaming sites

PublicStack AI is an open-source system that analyzes software projects and recommends full open-source repositories to enhance or replace parts of a stack. Instead of suggesting code snippets, it focuses on system-level composition—helping developers build better software by assembling proven repositories from across the open-source ecosystem, with built-in license awareness and automatic attribution tracking.

OpenCrawler is an open-source search engine built for transparency, fairness, and user control. It delivers explainable search results through a transparent ranking system, verifies business entities using public records, and protects user privacy with encryption and consent-based location access. Designed as a community-driven alternative to traditional search engines, OpenCrawler focuses on honest indexing, accountable ranking, and privacy-first infrastructure for the open web.

This is a python web miner. This actually crawles or mine the hidden dir and files using async it is faster then the normal def sync.

Website Downloader MCP Server in Python

BigPhish - An all-in-one solution to identify and monitor phishing kit deployment

Spremioloon.Com data scraper

Dark Web Search Engine : a dark net crawler and database viewer.

Web crawler for checking the validity of your documents.

Findpapers is an application that helps researchers who are looking for references for their work

Novel Feast is the website where you can read Chinese, Japanese, and Korean Web Novels, Web Comics Light Novels Online Free https://novelfeast.com/novels

Finds webpages containing recipes

HTTP request module customized for crawlers.

Discover apps by different mehtods. Mass download app packages and metadata.

Bilingual TED-TALK, Crawler, Crawling

A data scraping and manipulation library with command line interface with multi-database support for MSSQL, MySQL, PSQL, MariaDB and SQLite. View jsdoc and coverage reports at, https://rdfedor.gitlab.io/datahorde

A Python crawler to gather information about augmented reality applications. (For a research project.)

A Python app for scanning large data sets of URLs for a given signature and storing the results to an ElasticSearch index. Useful applications for CERTs and security researchers, maybe others.

Python tools for ethical hacking