Yandex SERP scraper: how to collect search results

Yandex is a major search engine in Russia. SEO teams use its search results to study other websites. They also check where their own pages appear. A Yandex SERP scraper collects this data for them. It saves links, titles, and short descriptions.

In this guide, you will use 2Captcha Scraper API with Python. You will collect results for several keywords. You will then save the data to a CSV file.

What is a Yandex SERP scraper?

SERP means search engine results page. A Yandex SERP scraper reads these pages and collects data from them.

It can collect:

• URLs — links to pages in the search results.
• Titles — titles that Yandex shows for each page.
• Snippets — short text about each page.
• Domains — websites found for a keyword.
• Result order — the order in which the script finds the results.

SEO teams use this data to find competing websites and watch changes in search results. However, the order found by a script is not always the true search rank. Check that the script finds all organic results before you report exact rankings.

Why scrape Yandex search results?

Checking search results by hand takes time. You must enter each keyword, open the results, and copy the data. A scraper can do this for many keywords.

You can use it for these tasks:

1. Competitor research. Find websites that appear for your keywords.
2. Search visibility. Check if your pages appear in the results.
3. SERP monitoring. Compare results from different dates.
4. Regional research. Collect results for different cities.
5. Content research. Read the titles and snippets of competing pages.

For example, you can collect results for "seo tools", "keyword research", and "web scraping tools".

Ways to scrape Yandex search results

There are three common ways to collect the data.

Manual collection: Open Yandex and copy the results. This is useful for a few searches.

Python requests and BeautifulSoup: Download the HTML and read it with Python. Yandex can restrict direct requests or show a captcha.

Scraper API: Ask an API to fetch the page. Your Python script then reads the HTML that the API returns.

This guide uses the third method.

How 2Captcha Scraper API works with Yandex

2Captcha Scraper API can fetch pages and return HTML, Markdown, or screenshots. It can process pages that use JavaScript. It can also handle supported captcha challenges automatically.

For Yandex, use the general scrape method. The API does not provide a separate Yandex SERP result with search ranks. You must read the HTML and find the search results yourself.

The process has four steps:

1. Create a Yandex search URL.
2. Send the URL to Scraper API.
3. Get the HTML from the response.
4. Read the search results and save them to CSV.

How to scrape Yandex with Python

Step 1. Get your API key

Sign in to 2Captcha and copy your API key. Save the key in an environment variable named TWO_CAPTCHA_API_KEY. Do not add your key to code that you share.

Step 2. Install the libraries

Install requests to send API requests. Install beautifulsoup4 to read HTML.

pip install requests beautifulsoup4

You do not need Selenium or Playwright for this example.

Step 3. Build a Yandex search URL

Yandex uses these URL parameters:

This example URL searches for "seo tools" in Moscow. It requests the first page:

https://yandex.ru/search/?text=seo+tools&lr=213&p=0

The script uses Python's urlencode() function to build the URL.

Step 4. Send a Scraper API request

Send a POST request to this endpoint:

POST https://scraper.2captcha.com/tasks/sync

Here is a request example:

curl -X POST "https://scraper.2captcha.com/tasks/sync" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "task_type": "scrape",
    "url": "https://yandex.ru/search/?text=seo+tools&lr=213&p=0",
    "data_format": "raw",
    "format": "json",
    "waitFor": {"element": "#search-result li.serp-item"},
    "timeout": 120
  }'

Set data_format to raw to get HTML. The waitFor setting tells the API to wait for a search result card. The wait limit is 30 seconds. If the card does not appear, the API may return the page with a warning.

The timeout value controls how long the synchronous API request can wait for the task. It does not change the 30-second waitFor limit.

The JSON response includes three useful fields. status shows the scrape status. http_code shows the HTTP status returned by Yandex. body contains the HTML.

Sometimes the API returns HTTP 408 and a response_id. This means the task is still running. Keep the ID and get the result from GET /tasks/result/{response_id}. Do not create another paid task for the same request.

Step 5. Create the Python scraper

Create a file named yandex_serp_parser.py. Copy the code below into the file.

The script searches for three keywords. It requests two pages for each keyword. It reads the HTML and collects likely organic results. It skips ads and AI answer blocks. It also saves HTML when a page needs review.

We tested the parsing code with a real Yandex HTML page from 2Captcha Scraper API on November 3, 2026. The page had 10 organic result candidates, 11 ads, and one AI answer block. The script kept the 10 candidates and skipped the other blocks.

Yandex can change its page layout. Check the extracted results before you use them for ranking reports.

"""Collect Yandex SERP cards via 2Captcha Scraper API.

Requires: pip install requests beautifulsoup4
Set TWO_CAPTCHA_API_KEY to your API key.
Yandex selectors tested on a saved real SERP HTML (2026-11-03).
Validate again if Yandex changes its layout.
"""

import csv
import json
import os
import re
import time
from datetime import datetime, timezone
from pathlib import Path
from urllib.parse import urlencode, urljoin, urlparse

import requests
from bs4 import BeautifulSoup

API_BASE = "https://scraper.2captcha.com"
API_KEY = os.environ.get("TWO_CAPTCHA_API_KEY", "")
QUERIES = ["seo tools", "keyword research", "web scraping tools"]
REGION = 213
PAGES_PER_QUERY = 2
OUTPUT_FILE = Path("yandex_serp_results.csv")
DEBUG_DIR = Path("yandex_debug")

class ScrapeError(RuntimeError):
    pass

def search_url(query: str, page: int) -> str:
    return "https://yandex.ru/search/?" + urlencode(
        {"text": query, "lr": REGION, "p": page}
    )

def response_json(response: requests.Response) -> dict:
    try:
        result = response.json()
    except ValueError as exc:
        raise ScrapeError(
            f"Expected JSON, got HTTP {response.status_code}: "
            f"{response.text[:250]}"
        ) from exc
    if not isinstance(result, dict):
        raise ScrapeError("The API returned a non-object JSON response")
    return result

def poll_task(session: requests.Session, response_id: str) -> dict:
    """Fetch the existing task instead of creating duplicate paid tasks."""
    for _ in range(90):
        time.sleep(3)
        try:
            response = session.get(
                f"{API_BASE}/tasks/result/{response_id}", timeout=(10, 30)
            )
        except requests.RequestException as exc:
            raise ScrapeError(
                f"Cannot check task {response_id}: {exc}. "
                "Retain the response ID and check the task again."
            ) from exc

        if response.status_code == 202:
            continue
        if response.status_code == 200:
            return response_json(response)
        error = response_json(response)
        raise ScrapeError(
            f"Task {response_id} failed (HTTP {response.status_code}): "
            f"{error.get('error', error)}"
        )
    raise ScrapeError(f"Task {response_id} is still running; retrieve it later")

def scrape_html(session: requests.Session, target_url: str) -> tuple[str, str]:
    payload = {
        "task_type": "scrape",
        "url": target_url,
        "data_format": "raw",
        "format": "json",
        "waitFor": {"element": "#search-result li.serp-item"},
        "timeout": 120,
    }
    headers = {"Authorization": f"Bearer {API_KEY}"}

    for attempt in range(3):
        try:
            response = session.post(
                f"{API_BASE}/tasks/sync",
                json=payload,
                headers=headers,
                timeout=(15, 135),
            )
        except requests.RequestException as exc:
            raise ScrapeError(
                "Network error during task creation. The task may already "
                "exist; do not blindly retry the paid POST request: " + str(exc)
            ) from exc

        if response.status_code != 503 or attempt == 2:
            break
        retry_after = response.headers.get("Retry-After", "5")
        try:
            delay = max(1, min(120, int(retry_after)))
        except ValueError:
            delay = 5
        print(f"API busy. Retrying in {delay} seconds.")
        time.sleep(delay)

    if response.status_code == 408:
        pending = response_json(response)
        response_id = pending.get("response_id")
        if not response_id:
            raise ScrapeError("HTTP 408 without response_id")
        print(f"Task pending: {response_id}")
        result = poll_task(session, response_id)
    elif response.status_code == 200:
        result = response_json(response)
    else:
        error = response_json(response)
        raise ScrapeError(
            f"API HTTP {response.status_code}: {error.get('error', error)}"
        )

    status = result.get("status")
    http_code = result.get("http_code")
    if status not in ("success", "warn") or http_code != 200:
        raise ScrapeError(
            f"Target failed: status={status}, http_code={http_code}, "
            f"error={result.get('error', '')}"
        )
    html = result.get("body")
    if not isinstance(html, str) or not html.strip():
        raise ScrapeError("Empty or invalid HTML in body")
    return html, str(result.get("warning", ""))

def clean_text(node) -> str:
    return " ".join(node.stripped_strings) if node else ""

# Yandex can place ads inside ordinary-looking OrganicTitle cards.
# Check the primary click URL, not only the card CSS classes.
AD_TRACKING_HOSTS = {"yabs.yandex.ru"}
AD_TRACKING_PATHS = {"/an/count", "/count"}
KNOWN_SPECIAL_BLOCKS = {"neuro_answer"}

def is_ad(item, primary_link=None) -> bool:
    fast_name = str(item.get("data-fast-name", "")).lower()
    if fast_name.startswith("adv_") or fast_name in {"direct", "serp-adv"}:
        return True

    if item.select_one(
        ".serp-adv-item, .serp-adv__found, "
        "[data-fast-name='direct'], [data-fast-name='serp-adv'], "
        "[aria-label='Реклама'], [title='Реклама']"
    ):
        return True

    if primary_link is None:
        primary_link = item.select_one(
            ".OrganicTitle a.OrganicTitle-Link[href], "
            ".OrganicTitle a.Link[href]"
        )
    if primary_link is None:
        return False

    href = primary_link.get("href", "")
    url = urlparse(urljoin("https://yandex.ru", href))
    host = (url.hostname or "").lower()
    path = url.path.lower()

    # These click-tracking URLs were found in the user's live SERP HTML.
    if host in AD_TRACKING_HOSTS:
        return True
    if host in {"yandex.ru", "www.yandex.ru"} and any(
        path == prefix or path.startswith(prefix + "/")
        for prefix in AD_TRACKING_PATHS
    ):
        return True
    return False

def parse_yandex(html: str) -> tuple[list[dict], dict]:
    soup = BeautifulSoup(html, "html.parser")
    captcha = soup.select_one(
        ".CheckboxCaptcha, .AdvancedCaptcha, [class*='SmartCaptcha'], "
        "form[action*='captcha'], iframe[src*='captcha'], "
        "#checkbox-captcha-form"
    )
    if captcha or "captcha" in clean_text(soup.title).lower():
        raise ScrapeError("Yandex returned a verification page")

    container = soup.select_one("#search-result")
    if container is None:
        raise ScrapeError("Could not find #search-result")
    items = container.select("li.serp-item")
    if not items:
        raise ScrapeError("No SERP cards found")

    results = []
    ad_count = 0
    special_count = 0
    unrecognized_count = 0

    for item in items:
        fast_name = str(item.get("data-fast-name", "")).lower()
        if fast_name in KNOWN_SPECIAL_BLOCKS:
            special_count += 1
            continue

        link = item.select_one(
            ".OrganicTitle a.OrganicTitle-Link[href], "
            ".OrganicTitle a.Link[href]"
        )
        if is_ad(item, link):
            ad_count += 1
            continue

        title = item.select_one(".OrganicTitle-LinkText")
        if link is None or title is None:
            unrecognized_count += 1
            continue

        target = urljoin("https://yandex.ru", link.get("href", ""))
        parsed = urlparse(target)
        title_text = clean_text(title)
        if (parsed.scheme not in {"http", "https"}
                or not parsed.hostname or not title_text):
            unrecognized_count += 1
            continue

        snippet = item.select_one(
            ".Organic-ContentWrapper .OrganicTextContentSpan, "
            ".OrganicTextContentSpan"
        )
        results.append({
            "title": title_text,
            "url": target,
            "domain": parsed.hostname.lower(),
            "snippet": clean_text(snippet),
        })

    if not results:
        raise ScrapeError("No recognizable organic result cards")

    quality = {
        "total_cards": len(items),
        "ads": ad_count,
        "special_blocks": special_count,
        "unrecognized": unrecognized_count,
    }
    return results, quality

def save_debug(query_index: int, page: int, html: str) -> None:
    DEBUG_DIR.mkdir(parents=True, exist_ok=True)
    path = DEBUG_DIR / f"query_{query_index}_page_{page + 1}.html"
    path.write_text(html, encoding="utf-8")
    print(f"HTML saved for inspection: {path}")

def main() -> None:
    if not API_KEY:
        raise SystemExit("Set TWO_CAPTCHA_API_KEY in your environment")

    all_rows = []
    failures = []
    with requests.Session() as session:
        for query_index, query in enumerate(QUERIES, 1):
            seen = set()
            ordinal = 0
            for page in range(PAGES_PER_QUERY):
                html = ""
                try:
                    html, warning = scrape_html(session, search_url(query, page))
                    found, quality = parse_yandex(html)
                    if warning:
                        print(f"Yandex request warning: {warning}")
                    print(f"{query} page {page + 1}: {len(found)} parsed, {quality}")
                    if quality["unrecognized"] or warning:
                        save_debug(query_index, page, html)
                    for result in found:
                        if result["url"] in seen:
                            continue
                        seen.add(result["url"])
                        ordinal += 1
                        all_rows.append({
                            "keyword": query,
                            "extracted_order": ordinal,
                            "page": page + 1,
                            "region": REGION,
                            **result,
                            "checked_at": datetime.now(timezone.utc).isoformat(timespec="seconds"),
                            "needs_review": bool(warning or quality["unrecognized"]),
                        })
                except (ScrapeError, requests.RequestException) as exc:
                    failures.append((query, page + 1, str(exc)))
                    print(f"Failed: {query}, page {page + 1}: {exc}")
                    if html:
                        save_debug(query_index, page, html)
                    break

    if all_rows:
        columns = ["keyword", "extracted_order", "page", "region", "title", "url", "domain", "snippet", "checked_at", "needs_review"]
        with OUTPUT_FILE.open("w", encoding="utf-8-sig", newline="") as out:
            writer = csv.DictWriter(out, fieldnames=columns)
            writer.writeheader()
            writer.writerows(all_rows)
        print(f"Saved {len(all_rows)} rows to {OUTPUT_FILE}")

    if failures:
        print("Incomplete collection:")
        for query, page, reason in failures:
            print(f"  {query}, page {page}: {reason}")
        raise SystemExit(1)
    if not all_rows:
        raise SystemExit("No results collected")
    print("Finished. Check result order against Yandex before using as rankings.")

if __name__ == "__main__":
    main()

Step 6. Run the scraper

First, set your API key.

On macOS or Linux:

export TWO_CAPTCHA_API_KEY="YOUR_API_KEY"
python yandex_serp_parser.py

On Windows PowerShell:

$env:TWO_CAPTCHA_API_KEY="YOUR_API_KEY"
python yandex_serp_parser.py

After the script runs, it creates yandex_serp_results.csv.

The CSV file has these columns:

If a page needs review, the script can save its HTML in yandex_debug. It can do this only if it received HTML.

If a request fails, the script prints an error. It also reports that the collection is incomplete. It may still save results from earlier requests.

How to find your website in the results

Use the CSV file to check if your website appears in the collected results.

Create a file named check_results.py:

import csv
from urllib.parse import urlparse

TARGET_DOMAIN = "example.com"

with open("yandex_serp_results.csv", encoding="utf-8-sig", newline="") as file:
    for row in csv.DictReader(file):
        hostname = (urlparse(row["url"]).hostname or "").lower()
        if hostname == TARGET_DOMAIN or hostname.endswith("." + TARGET_DOMAIN):
            print(
                f"Keyword: {row['keyword']} | "
                f"Extracted order: {row['extracted_order']} | "
                f"URL: {row['url']}"
            )

Run python check_results.py. Change example.com to your domain.

The program shows the order in which the scraper found your pages. This is not always the exact organic rank. The parser may miss some organic results or special result types. Check the HTML before you use this order as a ranking.

How to scrape Yandex results by region

Yandex results can change by location. Set the lr parameter to request a region.

To request results for Saint Petersburg, change this line:

REGION = 2

Use the same keywords and search settings when you compare regions. Yandex may still return different results for different sessions. Personalization can also affect the page.

How to monitor Yandex results over time

Run the script each day, week, or month. Save each CSV file with the date in its name. Compare the files to find new competitors or changes in titles and snippets.

For regular checks, you can save the data in a database. Keep the full search URL and collection date for each request.

Common Yandex scraping errors

The API returns HTTP 401 or 402. Check your API key and account balance. Do not repeat the request until you fix the problem.

The API returns HTTP 408. Keep the response_id. Use GET /tasks/result/{response_id} to get the task result. Do not send a second paid request for the same task.

Yandex shows a verification page. The page can return HTTP 200 and still contain a captcha. Check the HTML, not only the status code. Automatic handling of supported challenges does not always give access.

The script finds no search cards. Yandex may have changed the HTML. Open the saved page in yandex_debug. Check the HTML selectors and update them if needed.

Some results are missing. The script skips ads and cards it cannot read. Check the needs_review value and the terminal output before you use the data as rankings.

The region differs from your manual search. Check the lr value and other search settings. The results can also differ between sessions.

Frequently asked questions

Can I scrape Yandex search results without Selenium?

Yes. Scraper API fetches the page. Python and BeautifulSoup read the returned HTML.

Does 2Captcha Scraper API have a Yandex SERP method?

The documentation does not list a separate Yandex SERP method. Use the general scrape method. Then read the search results from the HTML.

Can I collect the top 100 Yandex results?

You can request more pages with the p parameter. Yandex may show a different number of results for each search. If you need a verified top 100, check that your script found every organic result in the correct order.

What is the difference between Scraper API and Browser API?

Scraper API fetches web pages and returns their content. Browser API gives you control of a cloud browser. You can use tools such as Playwright to click buttons, enter text, and run longer tasks.

Use Browser API when your task needs these browser actions.

Conclusion

A Yandex SERP scraper helps you collect search results and study competing websites. 2Captcha Scraper API fetches the pages. Python reads the results and saves them to CSV.

Check the HTML selectors and result order before you use the data for SEO ranking reports.