Skip to content

Repository files navigation

SoSearch API

A lightning-fast pseudo Web Search Engine API written in Rust โ€” also works as an MCP server for AI agents. This project emulates popular APIs like SerpAPI or Tavily without needing official and expensive API keys, by multiplexing requests to popular engines directly and scraping the results concurrently.

Philosophy

  • Performance: Powered by tokio for async concurrent I/O.
  • Bot Bypass: Leverages rquest with TLS impersonation (e.g., simulating a Chrome 124 browser footprint at the TLS/HTTP2 layer) to minimize blocking vs standard HTTP clients (the Rust equivalent of curl_cffi).
  • Standardized: Normalizes DuckDuckGo, Yahoo, and Brave HTML results into a standardized SearchResult JSON array.

Core Stack

๐Ÿ” Supported Search Engines

  • DuckDuckGo (Primary standard search)
  • Yahoo (Powered by Bing)
  • Brave Search (Independent index)

๐Ÿ“ Project Structure

src/
โ”œโ”€โ”€ main.rs              # Entry point: HTTP server or MCP mode (--mcp)
โ”œโ”€โ”€ search.rs            # Shared concurrent search logic
โ”œโ”€โ”€ mcp.rs               # MCP stdio server (JSON-RPC 2.0)
โ”œโ”€โ”€ models.rs            # SearchResultItem, SearchResponse structs
โ””โ”€โ”€ engines/
    โ”œโ”€โ”€ mod.rs            # SearchEngine enum + trait dispatch
    โ”œโ”€โ”€ duckduckgo.rs     # DuckDuckGo scraper
    โ”œโ”€โ”€ yahoo.rs          # Yahoo scraper (Bing-powered)
    โ””โ”€โ”€ brave.rs          # Brave Search scraper
examples/
โ”œโ”€โ”€ fetch_html.rs        # Download raw HTML for offline debugging
โ””โ”€โ”€ test_parser.rs       # Offline CSS selector iteration
.gemini/                 # Gemini CLI agent config
โ”œโ”€โ”€ GEMINI.md            # Project-level system prompt
โ”œโ”€โ”€ settings.json        # MCP server configuration
โ””โ”€โ”€ skills/              # Project-level agent skills
    โ”œโ”€โ”€ sosearch-engine-dev/  # Scraper development workflow
    โ””โ”€โ”€ sosearch-api-ops/     # API operations & deployment
.agents/                 # Generic agent config (compatible with multiple AI tools)
โ””โ”€โ”€ skills/              # Same skills, alternative discovery path
    โ”œโ”€โ”€ sosearch-engine-dev/
    โ””โ”€โ”€ sosearch-api-ops/

๐Ÿค– Agent Skills & MCP Support

This project includes built-in AI agent support for both Gemini CLI and other tools that follow the .agents/ convention.

Available Skills

Skill Description
sosearch-engine-dev Full workflow for adding/debugging search engine scrapers: fetch HTML โ†’ test selectors offline โ†’ decode URLs โ†’ integrate
sosearch-api-ops Operations guide: build, run, test, deploy (local + Docker), troubleshoot

MCP Servers

Configured in .gemini/settings.json:

Server Package Purpose
filesystem @modelcontextprotocol/server-filesystem Scoped file access to project directory

Usage with Gemini CLI

cd /path/to/SoSearch
gemini
# Skills are auto-discovered. Ask: "How do I add a new search engine?"

๐Ÿ”Œ MCP Server Mode

Run SoSearch as an MCP server for AI agents (Claude, Gemini, Cursor, etc.):

./SoSearch --mcp

Configuration

Claude Desktop (claude_desktop_config.json):

{
  "mcpServers": {
    "sosearch": {
      "command": "/path/to/SoSearch",
      "args": ["--mcp"]
    }
  }
}

Gemini CLI (.gemini/settings.json):

{
  "mcpServers": {
    "sosearch": {
      "command": "/path/to/SoSearch",
      "args": ["--mcp"]
    }
  }
}

This exposes a web_search tool that AI agents can call to search the web.

Windows Configuration

Claude Desktop (%APPDATA%\Claude\claude_desktop_config.json):

{
  "mcpServers": {
    "sosearch": {
      "command": "C:\\path\\to\\SoSearch.exe",
      "args": ["--mcp"]
    }
  }
}

๐Ÿš€ Quick Start

Linux / macOS

# Download pre-built binary or build from source
cargo run --release
curl "http://localhost:10080/search?q=hello+world"

Windows

ไฝฟ็”จ้ข„็ผ–่ฏ‘ไบŒ่ฟ›ๅˆถ๏ผš

ไปŽ GitHub Releases ไธ‹่ฝฝ SoSearch-windows-amd64.zip๏ผŒ่งฃๅŽ‹ๅŽ๏ผš

# ๅฏๅŠจ HTTP ๆœๅŠก
.\SoSearch.exe

# ๅฆๅผ€ไธ€ไธช็ปˆ็ซฏๆต‹่ฏ•
Invoke-RestMethod "http://localhost:10080/search?q=hello+world" | ConvertTo-Json

# ๆˆ–ไฝฟ็”จ curl
curl.exe "http://localhost:10080/search?q=hello+world"

ไปŽๆบ็ ็ผ–่ฏ‘๏ผˆ้œ€่ฆๅฎ‰่ฃ… Rustใ€CMakeใ€NASMใ€LLVM/Clang๏ผ‰๏ผš

# ๅฎ‰่ฃ…ไพ่ต– (ไฝฟ็”จ Chocolatey)
choco install cmake nasm llvm -y

# ็ผ–่ฏ‘่ฟ่กŒ
cargo run --release

MCP ๆจกๅผ๏ผš

.\SoSearch.exe --mcp

Refer to QUICK_START.md for full instructions.

๐Ÿ“„ License

CC BY-NC 4.0 โ€” ้žๅ•†ไธš็”จ้€”ๅฏ่‡ช็”ฑไฝฟ็”จใ€ไฟฎๆ”นๅ’Œๅˆ†ๅ‘ใ€‚

๐Ÿ  Community

้ฆ–ๅ‘ไบŽ LINUX DO ็คพๅŒบ๏ผŒๆฌข่ฟŽ Star โญ ๅ’Œ PR๏ผ

About

No description, website, or topics provided.

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages