clawoxy Website Logo
English
English简体中文
Start free

glossary

Author: Victoria

  • Git Bash

    Git Bash is a command-line environment for Windows that provides Git plus a collection of Unix-style shell tools. It lets developers use familiar commands such as ls, grep, and shell scripts alongside Git without switching to a Linux machine.

    Where it fits

    Git Bash is useful when a project documents commands for Bash or when a small script needs common Unix utilities. It is not the same thing as Git itself: Git is the version control program, while Git Bash provides a shell environment in which Git and other commands can run.

    Windows paths and shell syntax can differ from PowerShell or Command Prompt. A command that uses environment variable syntax, quoting, or path separators in Bash may need adjustment in another shell. If a team shares setup instructions, state which shell they assume.

    For crawler work, Git Bash can run repository commands and simple scripts. For longer or platform-specific automation, choose the shell and runtime that match the project and document the expected environment.

    Sources

  • Googlebot

    Googlebot is the name used for Google’s crawlers that discover and fetch pages for Google Search and related services. Site owners can review crawl activity in server logs and Search Console, but a request that merely claims to be Googlebot should not automatically be trusted.

    Verifying crawler requests

    Google documents a verification process for requests claiming to come from Google crawlers. Depending on the method, this can involve checking the source IP against published ranges or performing a reverse DNS lookup followed by a forward lookup. Follow Google’s current instructions rather than relying only on the user-agent string, which any client can set.

    Googlebot’s behavior is separate from the schedule of a crawler you operate. A site can influence crawling through availability, internal links, and supported controls, but it does not directly set Google’s internal crawl schedule. If Google’s requests cause a critical load problem, use Google’s documented crawl-rate guidance.

    Do not use the Googlebot name for unrelated automated requests. Accurate identification helps site operators distinguish legitimate crawler traffic from other clients.

    Sources

  • Web Scraping

    Web scraping is the process of collecting selected information from web pages or related interfaces and turning it into structured data. It can involve direct HTTP requests, HTML parsing, browser automation, or an API, depending on how the source exposes its content.

    A reliable collection workflow

    Define the fields and scope first. Inspect a representative response, choose selectors or a documented endpoint, validate the extracted records, and preserve the source URL and collection time. Separate extraction from normalization so the original value remains available when a transformation needs to be checked.

    Before running a collection, review the site’s terms and published access rules. Use conservative request pacing, handle 429 responses by backing off, and avoid retry loops. Public accessibility does not automatically settle whether a particular use is permitted.

    Scraping quality depends on more than parsing. Coverage, freshness, duplicates, and source changes all affect whether the resulting dataset is useful. Measure them explicitly and mark partial results honestly.

    Sources

  • Zero-Result Page

    A zero-result page is a search, filter, or category page that contains no matching records. In a crawl, it can be a valid outcome, a sign that a filter is too narrow, or evidence that the request landed on an error or challenge page.

    Distinguish an empty result from a failed fetch

    Check the HTTP status, page title, expected result container, and any explicit “no results” message. A 200 status alone does not prove that the page represents a normal empty search; some sites return error or verification pages with the same status.

    Store zero-result outcomes separately from fetch failures and parser failures. This keeps a legitimate empty query from being retried endlessly and makes coverage reports more accurate. If filters are user-controlled, validate them and record the exact query state.

    For site owners, make empty states clear and provide a useful way to adjust filters. For crawlers, set a stopping rule so a sequence of empty pages does not continue indefinitely.

    Sources

  • Hydration

    Hydration is the process by which client-side JavaScript attaches behavior to HTML that was already rendered, often on the server. A page may therefore contain readable HTML before the browser finishes initializing its interactive components.

    Why it matters when collecting pages

    If the first response includes the target text, a parser may not need to wait for hydration. If the page renders only a shell or fills fields after JavaScript runs, a browser may be necessary. Inspect the raw HTML and network requests first; this reveals whether the data is already present or arrives later from an API.

    Hydration can cause timing issues in automation. Waiting for the document’s load event may be too early for a particular component, while waiting for all network activity to stop may never finish on pages with analytics or long-polling. Wait for a specific selector or state that corresponds to the content you need.

    Framework behavior varies, so avoid relying on internal hydration scripts as a stable data interface. Prefer documented endpoints or semantic page content where available.

    Sources

  • XPath Selector

    An XPath selector is an XPath expression used to choose nodes from an XML or HTML document. A crawler can use it to extract text, attributes, or groups of elements such as product cards or article links.

    Make the selector match the page’s structure

    Use stable attributes and relationships instead of counting nested div elements. For example, select a card by a known class and then find its title within that card, rather than querying every title on the page and hoping order stays fixed.

    Check whether the selector returns zero, one, or many results, and handle each case deliberately. A zero result may mean the page changed, the wrong response was parsed, or the content is loaded later with JavaScript. Inspect the raw response before rewriting the selector.

    Different libraries expose XPath results in different forms. Read the framework’s documentation and test a saved response sample before running at scale.

    Sources

  • Beautiful Soup

    Beautiful Soup is a Python library for navigating and searching HTML or XML that you already have. It turns markup into a tree of tags and text, so code can find an element by tag, attribute, or text without walking through the document by hand.

    What It Does (and What It Doesn’t)

    Beautiful Soup parses a response body; it does not fetch the page or run its JavaScript. A common setup is to retrieve HTML with an HTTP client, pass the response text to BeautifulSoup, then select the fields you need. If a value appears only after client-side JavaScript runs, the original HTML may not contain it.

    For example, given a page with <h1 class="title">Books</h1>, you could find the heading with soup.find("h1", class_="title") and read its text with .get_text(strip=True). The selectors are part of the library; the network request is a separate step.

    Choosing a Parser

    Beautiful Soup provides one interface for several parser backends, including Python’s built-in html.parser, lxml, and html5lib. They may build different trees from malformed markup. If your extraction depends on how broken HTML is repaired, name the parser explicitly and use the same one in each environment.

    The library is a good fit for modest parsing tasks and scripts where readable code matters. For a large crawl, the request scheduler, concurrency, retries, and storage usually need their own tools. Parsing with Beautiful Soup alone does not provide those pieces.

    Sources

  • API Scraping

    API scraping is collecting data from an HTTP API instead of extracting it from a page’s rendered HTML. A crawler sends a request to an endpoint and parses the response, often JSON, into records. The term is informal: some people use it for documented APIs, while others mean requests a website’s own frontend makes to retrieve data.

    The Request and Response Workflow

    The basic workflow is to identify the endpoint, reproduce the request your use is authorized to make, and turn the response into the fields your task needs. A request may include a method, query parameters, headers, or an access token. Responses may contain pagination links or cursors, so retrieving one page is not always enough.

    For a page that loads data dynamically, browser developer tools can show which requests the page makes. That can help explain the data flow, but it does not grant permission to use a private endpoint. Check the API documentation, access terms, and required authentication before collecting data.

    API Endpoints and HTML Parsing

    Use an official API when it provides the fields you need and your use is allowed. API responses are often easier to parse than page markup, and changes to layout may not affect them. HTML scraping may be necessary when no suitable API is available or the required information appears only in the page.

    An API is not automatically stable or unrestricted. It may require credentials, impose quotas, omit fields shown on the page, or change its schema. An endpoint used internally by a site may also lack public documentation or support.

    Checks Before Collecting API Data

    • Confirm that the endpoint and intended use are permitted.
    • Read the API documentation and authentication requirements.
    • Handle pagination, rate limits, errors, and retries deliberately.
    • Check the response status and content type before parsing JSON.
    • Validate field names and types; a successful HTTP response can still contain an error object or unexpected data.
    • Store only the fields needed for the task, with source and collection time when useful.

    API scraping is most useful when a permitted endpoint returns the data in a structured form. The practical work is often less about parsing and more about understanding access rules, pagination, and response behavior.

    Sources

  • Headless Browser

    A headless browser runs a browser engine without displaying a normal graphical window. It can load pages, execute JavaScript, and expose the resulting DOM to automation code, which is useful for testing and for collection tasks that genuinely depend on browser rendering.

    When to use one

    First check whether the needed data is already present in the HTML or a documented endpoint. A direct HTTP request is often simpler and faster. Use a headless browser when content appears only after client-side execution or when the task requires a browser interaction that you are authorized to perform.

    Headless does not mean stateless or identical to every visible browser. Rendering can depend on viewport, locale, cookies, fonts, and timing. Wait for a page-specific condition rather than a fixed sleep, and keep browser versions consistent when reproducing a result.

    Browsers consume more resources than basic HTTP clients. Limit concurrency, close pages when done, and handle timeouts explicitly. A browser also does not bypass a site’s access rules.

    Sources

  • Git Bash

    Git Bash is a command-line environment for Windows that provides Git plus a collection of Unix-style shell tools. It lets developers use familiar commands such as ls, grep, and shell scripts alongside Git without switching to a Linux machine.

    Where it fits

    Git Bash is useful when a project documents commands for Bash or when a small script needs common Unix utilities. It is not the same thing as Git itself: Git is the version control program, while Git Bash provides a shell environment in which Git and other commands can run.

    Windows paths and shell syntax can differ from PowerShell or Command Prompt. A command that uses environment variable syntax, quoting, or path separators in Bash may need adjustment in another shell. If a team shares setup instructions, state which shell they assume.

    For crawler work, Git Bash can run repository commands and simple scripts. For longer or platform-specific automation, choose the shell and runtime that match the project and document the expected environment.

    Sources