Reddit Fetch

Reddit Fetch

Reddit's public JSON API works by appending .json to any Reddit URL — but Reddit now hard-blocks automated access. curl 403s essentially every time (host AND container, regardless of User-Agent), and even a cold Playwright navigation to reddit.com hits a "You've been blocked by network security" challenge page. The reliable method is to arrive at Reddit through a DuckDuckGo result redirect: that sets a Reddit session cookie which unlocks direct .json access for the rest of the browser session.

Primary method: the DuckDuckGo-hop unlock

Step 1 - the DDG hop. Do this once per session before any .json fetch.

1. mcp__playwright__browser_navigate to https://html.duckduckgo.com/html/?q=site:reddit.com/r/SUBREDDIT+YOUR+QUERY
2. Grab the first result's full href - it's a DDG redirect that includes a rut token (https://duckduckgo.com/l/?uddg=...&rut=...). The token is required; navigating to the bare /l/?uddg= without it 400s.

   jsCopy code
   () => document.querySelector('.result__a')?.href
   

3. browser_navigate to that full redirect href. It lands on a real www.reddit.com page (title = the post/subreddit title, not "Blocked") and sets the session cookie. The result doesn't have to be the exact thread you want - landing on any real Reddit page sets the cookie.

If Playwright errors with Browser is already in use, a stale instance is holding the profile - pkill -f ms-playwright-mcp (or the profile dir named in the error) and retry.

Step 2 - direct .json now works. For the rest of the session, navigate Playwright straight to any .json URL and JSON.parse(document.body.innerText). Use www.reddit.com (not old.reddit.com) for browser navigation. Full recency sorting (sort=new&t=week) is available.

1. browser_navigate to e.g. https://www.reddit.com/r/SUBREDDIT/search.json?q=QUERY&restrict_sr=on&sort=new&t=week&limit=25
2. browser_evaluate, always wrapped in try/catch (return document.body.innerText.slice(0,200) on failure so you can see a challenge page if the session lapsed - just re-do the hop):

   jsCopy code
   () => {
     try {
       const data = JSON.parse(document.body.innerText);
       return data.data.children.map(c => ({
         t: c.data.title, s: c.data.score, n: c.data.num_comments, id: c.data.id
       }));
     } catch (e) { return document.body.innerText.slice(0, 200); }
   }
   

3. For a thread, navigate to .../comments/POST_ID.json?limit=30&sort=top and parse data[0] (post) and data[1].data.children (comments).

JSON shapes (same for browser and curl)

textCopy code
# Listing - swap hot for new/top/rising; for top add &t=day|week|month|year|all
/r/SUBREDDIT/hot.json?limit=15
# Post + comments - JSON array where [0]=post, [1]=comment tree
/r/SUBREDDIT/comments/POST_ID.json?limit=20
# Search within a subreddit
/r/SUBREDDIT/search.json?q=QUERY&restrict_sr=on&sort=new&limit=15

• Listings: .data.children[].data has title, score, num_comments, author, id.
• Threads: [0].data.children[0].data is the post; [1].data.children[] (filter kind == "t1") are comments with author, score, body, and nested replies of the same shape.
• Truncate long comment bodies (e.g. body.slice(0, 300) in JS, .body[:300] in jq 1.7+) to keep output readable.

Fallbacks

• More comments per thread: load the rendered thread page (after the hop) and scrape the shreddit DOM - it returns more comments than .json?limit=:

  jsCopy code
  () => ({
    title: document.querySelector('shreddit-post')?.getAttribute('post-title'),
    comments: [...document.querySelectorAll('shreddit-comment')].map(c => ({
      author: c.getAttribute('author'),
      score:  c.getAttribute('score'),
      text:   c.querySelector('.md')?.innerText
    }))
  })
  

• No Playwright at all: use Claude for Chrome to open the thread / .json URL and read it off the page.
• curl (last resort, expect 403): direct curl with a browser User-Agent used to work and is faster when it does, but it now gets 403'd essentially always - changing the UA doesn't help. Only worth a single quick try if you're already shelling out and a browser isn't available; on 403, go straight to the DDG hop.

  bashCopy code
  UA="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"
  curl -s -L -o /tmp/reddit_result.txt -w "%{http_code}" -H "User-Agent: $UA" \
    'https://old.reddit.com/r/SUBREDDIT/hot.json?limit=15'
  jq -r '.data.children[] | .data | "\(.title)\n   \(.score) pts | \(.num_comments) comments | u/\(.author) | id: \(.id)\n"' /tmp/reddit_result.txt
  

  Fetch to a temp file (-o) then parse; -w "%{http_code}" prints the status; -L follows redirects; single-quote the URL so the shell doesn't eat &.

Rate limiting

Reddit rate-limits aggressively even once you're unblocked:

• Don't fire parallel requests - run them sequentially with sleep 2/sleep 3 (or brief pauses between navigations). Fetch one listing, parse it, then fetch threads one at a time.
• Empty response (0 bytes): wait 3-5s and retry. HTTP 429: back off 10-15s. A challenge page mid-session means the cookie lapsed - re-do the DDG hop.