Skip to content

Add Mastodon data source integration#103

Open
xingzihai wants to merge 1 commit intomvanhorn:mainfrom
xingzihai:add-mastodon-data-source
Open

Add Mastodon data source integration#103
xingzihai wants to merge 1 commit intomvanhorn:mainfrom
xingzihai:add-mastodon-data-source

Conversation

@xingzihai
Copy link
Copy Markdown

Summary

This PR adds Mastodon as a new data source to the last30days-skill project.

Mastodon is a federated social network that uses public APIs without requiring authentication. This integration searches multiple popular instances including mastodon.social, mas.to, fosstodon.org, mstdn.social, and techhub.social.

Changes

New Files

  • scripts/lib/mastodon.py - Mastodon search module with:
    • search_mastodon() - searches multiple Mastodon instances via public APIs
    • parse_mastodon_response() - parses API responses into normalized items
    • No authentication required (uses public APIs)

Modified Files

  • scripts/lib/schema.py - Added MastodonItem dataclass
  • scripts/lib/normalize.py - Added normalize_mastodon_items()
  • scripts/lib/score.py - Added compute_mastodon_engagement_raw() and score_mastodon_items()
  • scripts/lib/dedupe.py - Added dedupe_mastodon()
  • scripts/lib/env.py - Added is_mastodon_available() (always returns True)
  • scripts/lib/query_type.py - Added mastodon to SOURCE_TIERS and TIEBREAKER_BY_TYPE
  • scripts/last30days.py - Integrated Mastodon search into the main pipeline
  • SKILL.md - Updated documentation with Mastodon source

Features

  • No API Key Required: Uses public Mastodon APIs that don't require authentication
  • Multi-Instance Search: Searches multiple popular Mastodon instances in parallel
  • Full Integration: Complete integration with scoring, deduplication, and cross-source linking
  • Federated Network: Covers the fediverse, complementing centralized platforms like Bluesky and Truth Social

Testing

  • All Python files pass syntax validation
  • Follows the existing data source patterns (similar to Bluesky and Truth Social implementations)

Version

Bumps version from 2.9.5 to 2.10.0

- Add scripts/lib/mastodon.py with search_mastodon() and parse_mastodon_response()
- Add MastodonItem dataclass to schema.py
- Add normalize_mastodon_items() to normalize.py
- Add compute_mastodon_engagement_raw() and score_mastodon_items() to score.py
- Add dedupe_mastodon() to dedupe.py
- Add is_mastodon_available() to env.py (always returns True - uses public APIs)
- Add mastodon to SOURCE_TIERS and TIEBREAKER_BY_TYPE in query_type.py
- Integrate Mastodon search in last30days.py main script
- Update SKILL.md documentation with Mastodon source

Mastodon is a federated social network that uses public APIs without
authentication. Searches multiple popular instances: mastodon.social,
mas.to, fosstodon.org, mstdn.social, techhub.social.
@mvanhorn
Copy link
Copy Markdown
Owner

See my reply on #109 covering all your PRs. Can't commit to anything right now with the v3.0 refactor underway, but I promise I'll consider each one once it lands. Appreciate the work.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

2 participants