Skip to content

feat(search): add Keenable as a search engine, keyless by default - #1180

Open
KGKallasmaa wants to merge 1 commit into
ScrapeGraphAI:pre/betafrom
KGKallasmaa:feat/keenable-search-engine
Open

KGKallasmaa wants to merge 1 commit into
ScrapeGraphAI:pre/betafrom
KGKallasmaa:feat/keenable-search-engine

Conversation

@KGKallasmaa

Copy link
Copy Markdown

What this changes

Adds keenable as a search_engine for SearchGraph / SearchInternetNode (search_on_web), next to duckduckgo, bing, searxng and serper. Keenable is an independent web index built for agents. It works without an API key, which makes it the second zero-credential engine after DuckDuckGo. It also returns structured JSON from an API, so there's no HTML scraping and no ddgs dependency.

graph_config = {
    "llm": {...},
    "search_engine": "keenable",       # keyless by default
    # "keenable_api_key": "keen_...",  # optional: lifts the hourly limit (or set KEENABLE_API_KEY)
    "max_results": 5,
}
SearchGraph(prompt="...", config=graph_config).run()
  • research_web.py: new _search_keenable().
    • With no key (argument or KEENABLE_API_KEY env var), it calls the public endpoint POST https://api.keenable.ai/v1/search/public. That endpoint is rate-limited per IP and requires an X-Keenable-Title app header, which is set to scrapegraphai.
    • With a key, it calls POST /v1/search with X-API-Key.
    • max_results is capped at Keenable's per-call limit of 50.
    • The proxy setting is honoured, PDF links are filtered as for every other engine, and errors raise SearchRequestError.
    • SearchConfig gains keenable_api_key and accepts "keenable".
  • SearchInternetNode reads keenable_api_key from the node config, and SearchGraph passes it through, the same way serper_api_key is handled.
  • The defaults don't change. DuckDuckGo stays the default engine.

API reference: https://docs.keenable.ai/api-reference/search (see "Without a key").

Tests

  • New tests/utils/research_web_keenable_test.py: 6 mock-based tests with no network. They cover the keyless endpoint and title header, the keyed endpoint, the env-var key, the 50 cap, trimming to max_results, and HTTP error → SearchRequestError.
    • I couldn't add the file to the unit-tests list in .github/workflows/test-suite.yml, because my token can't push workflow changes. If you'd like it in CI, it's one line under tests/utils/parse_state_keys_test.py.
  • The CI unit-test list plus the new file: 101 passed locally (uv run pytest …).
  • ruff check and black --check are clean on the changed files.
  • Live, no key: search_on_web("Lightpanda headless browser for AI agents", search_engine="keenable", max_results=5) returned 5 URLs, starting with https://github.com/lightpanda-io/browser.

The same keyless pattern is the default search engine in Lightpanda's headless browser (lightpanda-io/browser#3265). Disclosure: I work with the Keenable team. Targeting pre/beta per the repo's flow.

@KGKallasmaa

Copy link
Copy Markdown
Author

@VinciGit00 @lurenss this adds an optional keenable search engine. It's keyless by default and mirrors how the Serper path is wired. Default behaviour is unchanged, mock tests are included, and the CI unit list passes locally. Happy to adjust.

@KGKallasmaa

Copy link
Copy Markdown
Author

@VinciGit00 @lurenss gentle ping: checks are green and it's mergeable. Let me know if you'd like any changes.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant