Give your agent X threads, not CAPTCHAs.
xthread-agent turns any public X/Twitter status URL into a reconstructed self-reply thread, an enveloped machine-readable JSON manifest, and every video & photo on disk. It never logs in, never touches cookies, never opens a browser — and it tells you honestly what it could not get.
pip install xthread-agent
X locked its front door. The content is still public.
X's auth wall guards discovery APIs, while its media CDN serves bytes to anyone holding a resolved URL. xthread-agent routes through the open services that grew up around X — and never touches x.com itself. No credentials, ever.
No login. No cookies. No browser.
The tool never presents credentials or a browser fingerprint. Public content only — deleted and protected posts fail closed, by design.
Honest accessBuilt for agents, not humans
Logs on stderr, data on stdout, stable exit codes, a JSON Schema contract, and an agent.md manual — a headless agent runs it end-to-end without asking a human.
Deterministic. No LLM inside.
A pure state machine: same input, same steps, same honest envelope. Zero pip dependencies — one stdlib-only file that runs anywhere Python 3.9+ exists.
ReproducibleEverything a thread-harvesting agent needs
Every claim below is backed by the offline test suite and re-verified against live endpoints.
True thread reconstruction
Chain membership comes only from replying_to_status — walk up to the thread start, down through self-replies. Same-author recommendations are filtered, not harvested.
Dual-homed discovery
Two walker slots (UnrollNow, ThreadReaderApp). If one dies or drifts, the other takes the walk transparently — and the manifest names who served it (walker_slot).
Fallback decoder slot
FixTweet decodes each post; vxtwitter is the automatic network-failure fallback with an honest, provenance-tagged field subset.
Best-quality media
4K-ready mp4 selection (highest-bitrate variant when the source is HLS), poster frames, photos with alt text, resumable + size-verified + atomic downloads.
t.co resolution
Paste a t.co shortlink: one hop is followed (HTTP redirect or the embedded-interstitial fallback), then validated — non-status destinations are rejected, never guessed.
Envelope contract
thread_manifest.json — a versioned, schema-validated envelope with posts, authors, timestamps, metrics, media, per-stage errors, and explicit nulls. Never fabricated data.
Quoted posts included
Quoted content is captured one level deep with author, text, timestamp and media URLs — recorded, not silently dropped.
Defense in depth
Media downloads restricted to *.twimg.com over HTTPS, remote IDs validated before touching the filesystem, response-body caps, no shell, no eval.
Degrades, never lies
Lose the walker → root-only harvest, flagged. Lose a video variant → hls_only reason. Lose everything → status: empty + exit 1. The envelope always says what it did not get.
Three replaceable tiers, one honest envelope
Each tier is a slot with a narrow contract. X itself is never touched; every auth burden in X's stack sits in front of discovery, not delivery.
Thread Walk
Ordered candidate IDs from the walker slots. Root always kept, capped at 50. Candidates only — never trusted as truth.
Metadata
Per-post JSON: text, author, timestamps, metrics, media URLs, quoted posts. 404 = filter signal, never retried.
Media Fetch
No auth needed once a URL is known. Streamed, size-verified, atomically renamed — a crash never corrupts a file.
| Supported input | Example | Behavior |
|---|---|---|
| x.com / twitter.com | https://x.com/user/status/123 | Full pipeline |
| mobile / www hosts | https://mobile.twitter.com/user/status/123 | Normalized |
| i/web + suffixes | …/status/123/photo/1, ?s=20 | Stripped to the ID |
| Bare status ID | 123456789012345678 | Full pipeline |
| t.co shortlinks | https://t.co/abc123 | One hop, then validated |
| Anything else | profiles, hashtags, other hosts | Rejected with a stable error code |
Plug it straight into your agent host
mcp_server.py exposes the harvester as MCP tools over the standard stdio transport — still stdlib-only, still no login. Each tool call shells out to the CLI with a hard timeout, so one contract stays the single source of truth.
extract_thread
Full harvest — thread reconstruction + media downloads; returns the complete envelope.
lookup_status
Metadata only — text, authors, timestamps, media URLs, no files on disk.
read_manifest
Return an existing manifest verbatim. Refuses any other filename.
get_schema
The JSON Schema (draft-07) behind the envelope contract.
{
"mcpServers": {
"xthread-agent": {
"command": "python3",
"args": ["/path/to/mcp_server.py"]
}
}
}
Any agent, any language, one pipeline
Logs always on stderr, data always on stdout. Exit 0 = harvested, 1 = nothing, 2 = usage error.
# 1 · run — machine-readable summary on stdout python3 xthread-agent.py "<status_url>" --json --quiet # 2 · inspect — the full envelope, schema-validated cat <out>/thread_manifest.json # 3 · branch on the contract status = "ok" | "partial" | "empty" # errors[] tells you why
The honest answers
Does it really need no login?expand_more
Is this allowed by X's Terms of Service?expand_more
What happens when an upstream service dies?expand_more
errors[] with a stable code. Worst case is a root-only harvest or an honest status: empty. Slots are designed to be replaced behind their contracts.What can't it do?expand_more
Why stdlib-only Python?expand_more
How is it tested?expand_more
Stop scraping. Start harvesting.
One file, zero dependencies, an MCP server included — and an envelope that never lies to your agent.
star Star on GitHub menu_book Read agent.mdpip install xthread-agent