# Technical Documentation: omkarcloud/botasaurus

> ℹ️ **Provenance:** Multi-Tier Ingestion · [CodeWiki Reference](https://codewiki.google/github.com/omkarcloud/botasaurus)

### Overview (README.md)

<p align="center">
  <img src="https://raw.githubusercontent.com/omkarcloud/botasaurus/master/images/mascot.png" alt="botasaurus" />
</p>
  <div align="center" style="margin-top: 0;">
  <h1>🤖 Botasaurus 🤖</h1>
  </div>

<h3 align="center">
  The All in One Framework to Build Undefeatable Scrapers
</h3>

<p align="center">
  <b>The web has evolved. Finally, web scraping has too.</b>
</p>


<p align="center">
  <img src="https://views.whatilearened.today/views/github/omkarcloud/botasaurus.svg" width="80px" height="28px" alt="View" />
</p>


<p align="center">
  <a href="https://gitpod.io/#https://github.com/omkarcloud/botasaurus-starter">
    <img alt="Run in Gitpod" src="https://gitpod.io/button/open-in-gitpod.svg" />
  </a>
</p>

# IMPORTANT: We recently released an update with new features and bug fixes. Please run the following command to upgrade all Botasaurus packages to their latest versions:
```shell
python -m pip install bota botasaurus botasaurus-api botasaurus-requests botasaurus-driver botasaurus-proxy-authentication botasaurus-server botasaurus-humancursor --upgrade
```

## 🐿️ Botasaurus In a Nutshell

How wonderful that of all the web scraping tools out there, you chose to learn about Botasaurus. Congratulations! 

And now that you are here, you are in for an exciting, unusual, and rewarding journey that will make your web scraping life a lot easier.

Now, let me tell you about Botasaurus in bullet points. (Because as per marketing gurus, YOU as a member of the Developer Tribe have a VERY short attention span.)

*So, what is Botasaurus?*

Botasaurus is an all-in-one web scraping framework that enables you to build awesome scrapers in less time, with less code, and with more fun.

We have put all our web scraping experience and best practices into Botasaurus to save you hundreds of hours of development time! 

Now, for the magical powers awaiting you after learning Botasaurus:

- In terms of humaneness, what Superman is to Man, Botasaurus is to Selenium and Playwright. Easily pass every (Yes, E-V-E-R-Y) bot test, and build undetected scrapers.  
    
In the video below, watch as we **bypass some of the best bot detection systems**:

- ✅ [Cloudflare Web Application Firewall (WAF)](https://nopecha.com/demo/cloudflare)  
- ✅ [BrowserScan Bot Detection](https://www.browserscan.net/bot-detection)  
- ✅ [Fingerprint Bot Detection](https://fingerprint.com/products/bot-detection/)  
- ✅ [Datadome Bot Detection](https://antoinevastel.com/bots/datadome)  
- ✅ [Cloudflare Turnstile CAPTCHA](https://turnstile.zeroclover.io/)

<p align="center">
<video src='https://github.com/user-attachments/assets/b4f6171f-f2a2-4255-9feb-2973ee9a25ae'/>
</p>

🔗 Want to try it yourself?
See the code behind these tests [here](https://github.com/omkarcloud/botasaurus/blob/master/bot_detection_tests.py)

- Perform realistic, human-like mouse movements and say sayonara to detection


- Convert your scraper into a desktop app for Mac, Windows, and Linux in 1 day, so not only developers but everyone can use your web scraper.



- Turn your scraper into a beautiful website, making it easy for your customers to use it from anywhere, anytime.



- Save up to 97%, yes 97%, on browser proxy costs by using [browser-based fetch requests.](https://github.com/omkarcloud/botasaurus#how-to-significantly-reduce-proxy-costs-when-scraping-at-scale)

- Easily save hours of development time with easy parallelization, profiles, extensions, and proxy configuration. Botasaurus makes asynchronous, parallel scraping child's play.

- Use caching, sitemap, data cleaning, and other utilities to save hours of time spent writing and debugging code.

- Easily scale your scraper to multiple machines with Kubernetes, and get your data faster than ever.

And those are just the highlights. I mean! 

There is so much more to Botasaurus that you will be amazed at how much time you will save with it.

## 🚀 Getting Started with Botasaurus

Let's dive right in with a straightforward example to understand Botasaurus.

In this example, we will go through the steps to scrape the heading text from [https://www.omkar.cloud/](https://www.omkar.cloud/).



### Step 1: Install Botasaurus

First things first, you need to install Botasaurus. Run the following command in your terminal:

```shell
python -m pip install --upgrade botasaurus
```

### Step 2: Set Up Your Botasaurus Project

Next, let's set up the project:

1. Create a directory for your Botasaurus project and navigate into it:

```shell
mkdir my-botasaurus-project
cd my-botasaurus-project
code .  # This will open the project in VSCode if you have it installed
```

### Step 3: Write the Scraping Code

Now, create a Python script named `main.py` in your project directory and paste the following code:

```python
from botasaurus.browser import browser, Driver

@browser
def scrape_heading_task(driver: Driver, data):
    # Visit the Omkar Cloud website
    driver.get("https://www.omkar.cloud/")
    
    # Retrieve the heading element's text
    heading = driver.get_text("h1")

    # Save the data as a JSON file in output/scrape_heading_task.json
    return {
        "heading": heading
    }
     
# Initiate the web scraping task
scrape_heading_task()
```

Let's understand this code:

- We define a custom scraping task, `scrape_heading_task`, decorated with `@browser`:
```python
@browser
def scrape_heading_task(driver: Driver, data):
```  

- Botasaurus automatically provides a Humane Driver to our function:
```python
def scrape_heading_task(driver: Driver, data):
```  

- Inside the function, we:
    - Visit Omkar Cloud
    - Extract the heading text
    - Return the data to be automatically saved as `scrape_heading_task.json` by Botasaurus:
```python
    driver.get("https://www.omkar.cloud/")
    heading = driver.get_text("h1")
    return {"heading": heading}
```  

- Finally, we initiate the scraping task:
```python
# Initiate the web scraping task
scrape_heading_task()
```  

### Step 4: Run the Scraping Task

Time to run it:

```shell
python main.py
```

After executing the script, it will:
- Launch Google Chrome
- Visit [omkar.cloud](https://www.omkar.cloud/)
- Extract the heading text
- Save it automatically as `output/scrape_heading_task.json`.



Now, let's explore another way to scrape the heading using the `request` module. Replace the previous code in `main.py` with the following:

```python
from botasaurus.request import request, Request
from botasaurus.soupify import soupify

@request
def scrape_heading_task(request: Request, data):
    # Visit the Omkar Cloud website
    response = request.get("https://www.omkar.cloud/")

    # Create a BeautifulSoup object    
    soup = soupify(response)
    
    # Retrieve the heading element's text
    heading = soup.find('h1').get_text()

    # Save the data as a JSON file in output/scrape_heading_task.json
    return {
        "heading": heading
    }     
# Initiate the web scraping task
scrape_heading_task()
```

In this code:

- We scrape the HTML using `request`, which is specifically designed for making browser-like humane requests.
- Next, we parse the HTML into a `BeautifulSoup` object using `soupify()` and extract the heading.

### Step 5: Run the Scraping Task (which makes Humane HTTP Requests)

Finally, run it again:

```shell
python main.py
```

This time, you will observe the exact same result as before, but instead of opening a whole browser, we are making browser-like humane HTTP requests.

## 💡 Understanding Botasaurus

### What is Botasaurus Driver, and why should I use it over Selenium and Playwright?

Botasaurus Driver is a web automation driver like Selenium, and the single most important reason to use it is because it is truly humane. You will not, and I repeat NOT, have any issues accessing any website.

Plus, it is super fast to launch and use, and the API is designed by and for web scrapers, and you will love it.

### How do I access Cloudflare-protected pages using Botasaurus?

Cloudflare is the most popular protection system on the web. So, let's see how Botasaurus can help you solve various Cloudflare challenges.

**Connection Challenge**

This is the single most popular challenge and requires making a browser-like connection with appropriate headers. It's commonly used for:
- Product Pages
- Blog Pages 
- Search Result Pages



#### What Works?

- Visiting the website via Google Referrer (which makes it seem as if the user has arrived from a Google search).

```python
from botasaurus.browser import browser, Driver

@browser
def scrape_heading_task(driver: Driver, data):
    # Visit the website via Google Referrer
    driver.google_get("https://www.cloudflare.com/en-in/")
    driver.prompt()
    heading = driver.get_text('h1')
    return heading

scrape_heading_task()
```

- Use the request module. The Request Object is smart and, by default, visits any link with a Google Referrer. Although it works, you will need to use retries.

```python
from botasaurus.request import request, Request

@request(max_retry=10)
def scrape_heading_task(request: Request, data):
    response = request.get("https://www.cloudflare.com/en-in/")
    print(response.status_code)
    response.raise_for_status()
    return response.text

scrape_heading_task()
```

**JS with Captcha Challenge**

This challenge requires performing JS computations that differentiate a Chrome controlled by Selenium/Puppeteer/Playwright from a real Chrome. It also involves solving a Captcha. It's used to for pages which are rarely but sometimes visited by people, like:
- 5th Review page
- Auth pages

Example Page: https://nopecha.com/demo/cloudflare

#### What Does Not Work?
Using `@request` does not work because although it can make browser-like HTTP requests, it cannot run JavaScript to solve the challenge.

#### What Works?
Pass the `bypass_cloudflare=True` argument to the `google_get` method.

```python
from botasaurus.browser import browser, Driver

@browser
def scrape_heading_task(driver: Driver, data):
    driver.google_get("https://nopecha.com/demo/cloudflare", bypass_cloudflare=True)
    driver.prompt()

scrape_heading_task()
```


### What are the benefits of a UI scraper?

Here are some benefits of creating a scraper with a user interface:

- Simplify your scraper usage for customers, eliminating the need to teach them how to modify and run your code.
- Protect your code by hosting the scraper on the web and offering a monthly subscription, rather than providing full access to your code. This approach:
  - Safeguards your Python code from being copied and reused, increasing your customer's lifetime value.
  - Generate monthly recurring revenue via subscription from your customers, surpassing a one-time payment.
- Enable sorting, filtering, and downloading of data in various formats (JSON, Excel, CSV, etc.).
- Provide access via a REST API for seamless integration.
- Create a polished frontend, backend, and API integration with minimal code.

### How to run a UI-based scraper?

Let's run the Botasaurus Starter Template (the recommended template for greenfield Botasaurus projects), which scrapes the heading of the provided link by following these steps:

1. Clone the Starter Template:
   ```
   git clone https://github.com/omkarcloud/botasaurus-starter my-botasaurus-project
   cd my-botasaurus-project
   ```

2. Install dependencies (will take a few minutes):
   ```
   python -m pip install -r requirements.txt
   python run.py install
   ```

3. Run the scraper:
   ```
   python run.py
   ```

Your browser will automatically open up at http://localhost:3000/. Then, enter the link you want to scrape (e.g., https://www.omkar.cloud/) and click on the Run Button.



After some seconds, the data will be scraped.


Visit http://localhost:3000/output to see all the tasks you have started.



Go to http://localhost:3000/about to see the rendered README.md file of the project.



Finally, visit http://localhost:3000/api-integration to see how to access the scraper via API.



The API documentation is generated dynamically based on your scraper's inputs, sorts, filters, etc., and is unique to your scraper. 

So, whenever you need to run the scraper via API, visit this tab and copy the code specific to your scraper.

### How to create a UI scraper using Botasaurus?

Creating a UI scraper with Botasaurus is a simple 3-step process:
1. Create your scraper function
2. Add the scraper to the server using 1 line of code 
3. Define the input controls for the scraper

To understand these steps, let's go through the code of the Botasaurus Starter Template that you just ran.

#### Step 1: Create the Scraper Function

In `src/scrape_heading_task.py`, we define a scraping function that basically does the following:

1. Receives a `data` object and extracts the "link".
2. Retrieves the HTML content of the webpage using the "link".
3. Converts the HTML into a BeautifulSoup object.
4. Locates the heading element, extracts its text content, and returns it.

```python
from botasaurus.request import request, Request
from botasaurus.soupify import soupify

@request
def scrape_heading_task(request: Request, data):
    # Visit the Link
    response = request.get(data["link"])

    # Create a BeautifulSoup object    
    soup = soupify(response)
    
    # Retrieve the heading element's text
    heading = soup.find('h1').get_text()

    # Save the data as a JSON file in output/scrape_heading_task.json
    return {
        "heading": heading
    }
```

#### Step 2: Add the Scraper to the Server

In `backend/scrapers.py`, we:
- Import our scraping function
- Use `Server.add_scraper()` to register the scraper

```python
from botasaurus_server.server import Server
from src.scrape_heading_task import scrape_heading_task

# Add the scraper to the server
Server.add_scraper(scrape_heading_task)
```

#### Step 3: Define the Input Controls

In `backend/inputs/scrape_heading_task.js`, we:
- Define a `getInput` function that takes the controls parameter
- Add a link input control to it
- Use JSDoc comments to enable IntelliSense Code Completion in VSCode as you won't be able to remember all the controls in botasaurus.


```js
/**
 * @typedef {import('../../frontend/node_modules/botasaurus-controls/dist/index').Controls} Controls
 */

/**
 * @param {Controls} controls
 */
function getInput(controls) {
    controls
        // Render a Link Input, which is required, defaults to "https://stackoverflow.blog/open-source". 
        .link('link', { isRequired: true, defaultValue: "https://stackoverflow.blog/open-source" })
}
```

Above was a simple example; below is a real-world example with multi-text, number, switch, select, section, and other controls.

```
/* Detailed source-code truncated for AI context efficiency. */
```

I encourage you to paste the above code into `backend/inputs/scrape_heading_task.js` and reload the page, and you will see a complex set of input controls like the image shown.



Now, to use the Botasaurus UI for adding new scrapers, remember these points:

1. Create a `backend/inputs/{your_scraping_function_name}.js` file for each scraping function.
2. Define the `getInput` function in the file with the necessary controls.
3. Use JSDoc comments to enable IntelliSense code completion in VSCode, as you won't be able to remember all the controls in Botasaurus.

Use this template as a starting point for new scraping function's input controls js file:

```js
/**
 * @typedef {import('../../frontend/node_modules/botasaurus-controls/dist/index').Controls} Controls
 */

/**
 * @param {Controls} controls
 */
function getInput(controls) {
    // Define your controls here.
}
```

That's it! With these simple steps, you can create a fully functional UI scraper using Botasaurus.

Later, you will learn how to add sorts and filters to make your UI scraper even more powerful and user-friendly.



### What is a Desktop Extractor?
A **Desktop Extractor** is a standalone application that runs on your computer and extracts specific data from websites, PDFs, Excel files, and other documents. Unlike web-based tools, desktop extractors run locally, giving **faster performance** and **zero cloud costs**.



### What advantages do Desktop Scrapers have over web-based scrapers?
**Desktop Scrapers** offer key advantages over web-based scraper solutions like Outscraper:  

- **Zero Infrastructure Costs**: 
   - Runs on the user's machine, eliminating expensive cloud computing fees.  
   - Lower cloud costs allow you to offer lower pricing, attracting more customers and increasing revenue.

- **Faster Execution**:  
  - Instant execution, no delays for cloud resource allocation.  
  - Uses the user's system, which is much faster than shared cloud servers.  

- **Increased Customer Engagement**:  
  The app sits right on the user's desktop, encouraging frequent use compared to web tools they must actively visit via browser.  

- **Cross-Platform Deployment in 1 Day**:  
  With **Botasaurus**, you can launch a desktop scraper for **Windows, macOS, and Linux** within a day. No need to build a website, manage servers, or handle scaling issues. Bota Desktop includes built-in features such as:
  - Task management
  - Data Table
  - Data export (Excel, CSV, etc.)
  - Sorting & Filtering
  - Caching and many more

With zero usage costs, faster performance, and easier development, Desktop Scrapers outperform web-based alternatives.

### How to Build a Desktop Extractor
Creating Desktop Extractors is easier than you think! All you need is a basic understanding of JavaScript. Once you're ready, read the [Desktop Extraction Tutorial](https://github.com/omkarcloud/botasaurus/blob/master/botasaurus-desktop-tutorial.md#what-will-be-built-and-why), where we'll guide you through building two practical extractors:
- **Yahoo Finance Stock Scraper** – Extracts real-time stock prices from Yahoo Finance.

 

- **Amazon Invoice PDF Extractor** – Automates the extraction of key invoice data like Document Number, Document Date, and Place of Supply from Amazon PDFs.  

  

As a web scraper, you might naturally want to focus on web scraping. Still, I want you to create the **Amazon Invoice PDF Extractor** project. Why? Because many developers overlook the immense potential of extracting data from PDFs, Excel files, and other documents.  

**Document Data Extraction is a large untapped market.** For example, even in most developed countries, accountants often spend hundreds of hours manually entering invoice data for tax filings. A desktop extractor can transform this tedious, error-prone process into a task that takes just minutes, delivering 100% accurate results.

Please read the step-by-step tutorial [here](https://github.com/omkarcloud/botasaurus/blob/master/botasaurus-desktop-tutorial.md#what-will-be-built-and-why). By the end of this short guide, you'll be able to create powerful desktop extractors in very little time.

### What is Botasaurus, and what are its main features?

Botasaurus is an all-in-one web scraping framework designed to achieve three main goals:  

1. Provide essential web scraping utilities to streamline the scraping process.

To accomplish these goals, Botasaurus gives you 3 decorators:
- `@browser`: For scraping web pages using a humane browser.
- `@request`: For scraping web pages using lightweight and humane HTTP requests.
- `@task`: 
  - For scraping web pages using third-party libraries like `playwright` or `selenium`.
  - or, For running non-web scraping tasks, such as data processing (e.g., converting video to audio). Botasaurus is not limited to web scraping tasks; any Python function can be made accessible with a stunning UI and user-friendly API.

In practice, while developing with Botasaurus, you will spend most of your time in the following areas:
- Configuring your scrapers via decorators with settings like:
  - Which proxy to use
  - How many scrapers to run in parallel, etc.
- Writing your core web scraping logic using BeautifulSoup (bs4) or the Botasaurus Driver.

Additionally, you will utilize the following Botasaurus utilities for debugging and development:
- `bt`: Mainly for writing JSON, EXCEL, and HTML temporary files, and for data cleaning.
- `Sitemap`: For accessing the website's links and sitemap.
- Minor utilities like:
  - `LocalStorage`: For storing scraper state.
  - `soupify`: For creating BeautifulSoup objects from Driver, Requests response, Driver Element, or HTML string.
  - `IPUtils`: For obtaining information (IP, country, etc.) about the current IP address.
  - `Cache`: For managing the cache.

By simply configuring these three decorators (`@browser`, `@request`, and `@task`) with arguments, you can easily create `real-time scrapers` and `large-scale datasets`, thus saving you countless hours that would otherwise be spent writing and debugging code from scratch.

2. Offering a Python-based UI scraper that allows non-technical users to run scrapers online by simply visiting a website link. (As described in the previous FAQ)  

3. Make it easy to create desktop applications for Mac, Windows, and Linux, using JavaScript. More details can be found in the [Botasaurus Desktop Documentation here](https://github.com/omkarcloud/botasaurus/blob/master/botasaurus-desktop-tutorial.md).

### How to use decorators in Botasaurus?

Decorators are the heart of Botasaurus. To use a decorator function, you can call it with:
- A single item
- A list of items

If a scraping function is given a list of items, it will sequentially call the scraping function for each data item.

For example, if you pass a list of three links to the `scrape_heading_task` function:
```python
from botasaurus.browser import browser, Driver

@browser
def scrape_heading_task(driver: Driver, link):
    driver.get(link)
    heading = driver.get_text("h1")
    return heading

scrape_heading_task(["https://www.omkar.cloud/", "https://www.omkar.cloud/blog/", "https://stackoverflow.com/"]) # <-- list of items
```

Then, Botasaurus will launch a new browser instance for each item, and the final results will be stored in `output/scrape_heading_task.json`.



### How does Botasaurus help me in debugging?

Botasaurus helps you in debugging by:

- Easily viewing the result of the scraping function, as it is saved in `output/{your_scraping_function_name}.json`. Say goodbye to print statements.



- Bringing your attention to errors in browser mode with a beep sound and pausing the browser, allowing you to debug the error on the spot.



- Even if an exception is raised in headless mode, it will still open the website in your default browser, making it easier to debug code in a headless browser. (Isn't it cool?)



### How to configure the Browser Decorator?

The Browser Decorator allows you to easily configure various aspects of the browser, such as:

- Blocking images and CSS
- Setting up proxies
- Specifying profiles
- Enabling headless mode
- Using Chrome extensions
- Captcha Solving
- Selecting language
- Passing Arguments to Chrome

#### Blocking Images and CSS

Blocking images is one of the most important configurations when scraping at scale. Blocking images can significantly: 
- Speed up your web scraping tasks
- Reduce bandwidth usage
- And save money on proxies. (Best of All!)

For example, a page that originally takes 4 seconds and 12 MB to load might only take one second and 100 KB after blocking images and CSS.

To block images, use the `block_images` parameter:

```python
@browser(
    block_images=True,
)
```

To block both images and CSS, use `block_images_and_css`:

```python
@browser(
    block_images_and_css=True,
)    
```

#### Proxies

To use proxies, simply specify the `proxy` parameter:

```python
@browser(
    proxy="http://username:password@proxy-provider-domain:port"
)    
def visit_what_is_my_ip(driver: Driver, data):
    driver.get("https://whatismyipaddress.com/")
    driver.prompt()

visit_what_is_my_ip()
```

You can also pass a list of proxies, and the proxy will be automatically rotated:

```python
@browser(
    proxy=[
        "http://username:password@proxy-provider-domain:port", 
        "http://username2:password2@proxy-provider-domain:port"
    ]
)
def visit_what_is_my_ip(driver: Driver, data):
    driver.get("https://whatismyipaddress.com/")
    driver.prompt()

visit_what_is_my_ip() 
```

#### Profile

Easily specify the Chrome profile using the `profile` option:

```python
@browser(
    profile="pikachu"
)    
```

However, each Chrome profile can become very large (e.g., 100 MB) and can eat up all your computer storage. 

To solve this problem, use the `tiny_profile` option, which is a lightweight alternative to Chrome profiles. 

When creating hundreds of Chrome profiles, it is highly recommended to use the `tiny_profile` option because:

- Creating 1000 Chrome profiles will take at least 100 GB, whereas 1000 tiny profiles will take up only 1 MB of storage, making tiny profiles easy to store and back up.
- Tiny profiles are cross-platform, meaning you can create profiles on a Linux server, copy the `./profiles` folder to a Windows PC, and easily run them.

Under the hood, tiny profiles persist cookies from visited websites, making them extremely lightweight (around 1 KB) while providing the same session persistence.

Here's how to use the tiny profile:

```python
@browser(
    tiny_profile=True, 
    profile="pikachu",
)    
```

#### Headless Mode

Enable headless mode with `headless=True`:
```python
@browser(
    headless=True
)    
```
Note that if you use headless mode, you will surely be identified by services like Cloudflare and Datadome. Therefore, use headless mode only when scraping websites that don't use such services.

#### Chrome Extensions

Botasaurus allows the use of ANY Chrome Extension with just 1 line of code. The example below shows how to use the Mouse Coordinates Chrome Extension to show current mouse X and Y coordinates on web pages:

```python
from botasaurus.browser import browser, Driver
from chrome_extension_python import Extension

@browser(
    extensions=[
        Extension(
            "https://chromewebstore.google.com/detail/mouse-coordinates/mfohnjojhopfcahiddmeljeholnciakl"
        )
    ],
)
def scrape_while_blocking_ads(driver: Driver, data):
    driver.get("https://example.com/")
    driver.prompt()

scrape_while_blocking_ads()
```

In some cases, an extension may require additional configuration, such as API keys or credentials. For such scenarios, you can create a custom extension. Learn more about creating and configuring custom extensions [here](https://github.com/omkarcloud/chrome-extension-python).

#### Captcha Solving

Encountering captchas is common in web scraping. You can use the [capsolver_extension_python](https://github.com/omkarcloud/capsolver-extension-python?tab=readme-ov-file#installation) package to automatically solve CAPTCHAs with Capsolver.  

To use it, first install the package:  

```bash
python -m pip install capsolver_extension_python
```

Then, integrate it into your code as follows:  

```python
from botasaurus.browser import browser, Driver
from capsolver_extension_python import Capsolver

# Replace "CAP-MY_KEY" with your actual CapSolver API key
@browser(extensions=[Capsolver(api_key="CAP-MY_KEY")])  
def solve_captcha(driver: Driver, data):
    driver.get("https://recaptcha-demo.appspot.com/recaptcha-v2-checkbox.php")
    driver.prompt()

solve_captcha()
```  

#### Language

Specify the language using the `lang` option:

```python
from botasaurus.lang import Lang

@browser(
    lang=Lang.Hindi,
)
```

#### User Agent and Window Size

To make the browser really humane, Botasaurus does not change browser fingerprints by default, because using fingerprints makes the browser easily identifiable by running CSS tests to find mismatches between the provided user agent and the actual user agent.

However, if you need fingerprinting, use the `user_agent` and `window_size` options:

```python
from botasaurus.browser import browser, Driver
from botasaurus.user_agent import UserAgent
from botasaurus.window_size import WindowSize

@browser(
    user_agent=UserAgent.RANDOM,
    window_size=WindowSize.RANDOM,
)
def visit_whatsmyua(driver: Driver, data):
    driver.get("https://www.whatsmyua.info/")
    driver.prompt()

visit_whatsmyua()
```

When working with profiles, you want the fingerprints to remain consistent. You don't want the user's user agent to be Chrome 106 on the first visit and then become Chrome 102 on the second visit. 

So, when using profiles, use the `HASHED` option to generate a consistent user agent and window size based on the profile's hash:

```python
from botasaurus.browser import browser, Driver
from botasaurus.user_agent import UserAgent
from botasaurus.window_size import WindowSize

@browser(
    profile="pikachu",
    user_agent=UserAgent.HASHED,
    window_size=WindowSize.HASHED,
)
def visit_whatsmyua(driver: Driver, data):
    driver.get("https://www.whatsmyua.info/")
    driver.prompt()
    
visit_whatsmyua()

# Everytime Same UserAgent and WindowSize
visit_whatsmyua()
```
#### Passing Arguments to Chrome

To pass arguments to Chrome, use the `add_arguments` option:

```python
@browser(
    add_arguments=['--headless=new'],
)
```

To dynamically generate arguments based on the `data` parameter, pass a function:

```python
def get_arguments(data):
    return ['--headless=new']

@browser(
    add_arguments=get_arguments,
)
```

#### Wait for Complete Page Load

By default, Botasaurus waits for all page resources (DOM, JavaScript, CSS, images, etc.) to load before calling your scraping function with the driver.

However, sometimes the DOM is ready, but JavaScript, images, etc., take forever to load. 

In such cases, you can set `wait_for_complete_page_load` to `False` to interact with the DOM as soon as the HTML is parsed and the DOM is ready:

```python
@browser(
    wait_for_complete_page_load=False,
)
```

#### Reuse Driver

Consider the following example:

```python
from botasaurus.browser import browser, Driver

@browser
def scrape_data(driver: Driver, link):
    driver.get(link)

scrape_data(["https://www.omkar.cloud/", "https://www.omkar.cloud/blog/", "https://stackoverflow.com/"])
```

If you run this code, the browser will be recreated on each page visit, which is inefficient. 



To solve this problem, use the `reuse_driver` option which is great for cases like:
- Scraping a large number of links and reusing the same browser instance for all page visits.
- Running your scraper in a cloud server to scrape data on demand, without recreating Chrome on each request.

Here's how to use `reuse_driver` which will reuse the same Chrome instance for visiting each link.

```python
from botasaurus.browser import browser, Driver

@browser(
    reuse_driver=True
)
def scrape_data(driver: Driver, link):
    driver.get(link)

scrape_data(["https://www.omkar.cloud/", "https://www.omkar.cloud/blog/", "https://stackoverflow.com/"])
```
**Result**

 
---

Also, by default, whenever the program ends or is canceled, Botasaurus smartly closes any open Chrome instances, leaving no instances running in the background.

In rare cases, you may want to explicitly close the Chrome instance. For such scenarios, you can use the `.close()` method on the scraping function:

```python
scrape_data.close()
```

This will close any Chrome instances that remain open after the scraping function ends.


### How to Significantly Reduce Proxy Costs When Scraping at Scale?

Recently, we had a project requiring access to around 100,000 pages from a well-protected website, necessitating the use of Residential Proxies.

Even after blocking images, we still required 250GB of proxy bandwidth, costing approximately $1050 (at $4.2 per GB with IP Royal). 

This was beyond our budget :(

To solve this, we implemented a smart strategy:
- We first visited the website normally.
- We then made requests for subsequent pages using the browser's `fetch` API.

Since we were only requesting the HTML, which was well compressed by the browser, we reduced our proxy bandwidth needs to just 5GB, costing only $30.

This resulted in savings of around $1000!

Here's an example of how you can do something similar in Botasaurus:

```python
from botasaurus.browser import browser, Driver
from botasaurus.soupify import soupify

@browser(
    reuse_driver=True,  # Reuse the browser
    max_retry=5,        # Retry up to 5 times on failure
)
def scrape_data(driver: Driver, link):
    # If the browser is newly opened, first visit the link
    if driver.config.is_new:
        driver.google_get(link)
    
    # Make requests using the browser fetch API
    response = driver.requests.get(link)
    response.raise_for_status()  # Ensure the request was successful
    html = response.text

    # Parse the HTML to extract the desired data
    soup = soupify(html)
    stock_name = soup.select_one('[data-testid="quote-hdr"] h1').get_text()
    stock_price = soup.select_one('[data-testid="qsp-price"]').get_text()
    
    return {
        "stock_name": stock_name,
        "stock_price": stock_price,
    }

# List of URLs to scrape
links = [
    "https://finance.yahoo.com/quote/AAPL/",
    "https://finance.yahoo.com/quote/GOOG/",
    "https://finance.yahoo.com/quote/MSFT/",
]

# Execute the scraping function for the list of links
scrape_data(links)
```

Note:
1. **Dealing with 429 (Too Many Requests) Errors**

   If you encounter a 429 error, add a delay before making another request. Most websites using Nginx, setting a rate limit of 1 request per second. To respect this limit, a delay of 1.13 seconds is recommended.

   ```python
   driver.sleep(1.13)  # Delay to respect the rate limit
   response = driver.requests.get(link)
   ```

2. **Handling 400 Errors Due to Large Cookies**

   If you encounter a 400 error with a "cookie too large" message, delete the cookies and retry the request.

   ```python
   response = driver.requests.get(link)

   if response.status_code == 400:
       driver.delete_cookies()  # Delete cookies to resolve the error
       driver.short_random_sleep()  # Short delay before retrying
       response = driver.requests.get(link)
   ```
3. You can also use `driver.requests.get_mank(links)` to make multiple requests in parallel, which is faster than making them sequentially.

### How to Configure the Browser's Chrome Profile, Language, and Proxy Dynamically Based on Data Parameters?

The decorators in Botasaurus are really flexible, allowing you to pass a function that can derive the browser configuration based on the data item parameter. This is particularly useful when working with multiple Chrome profiles.

You can dynamically configure the browser's Chrome profile and proxy using decorators in two ways:

1. Using functions to extract configuration values from data:
   - Define functions to extract the desired configuration values from the `data` parameter.
   - Pass these functions as arguments to the `@browser` decorator.

   Example:
   ```python
   from botasaurus.browser import browser, Driver

   def get_profile(data):
       return data["profile"]

   def get_proxy(data):
       return data["proxy"]

   @browser(profile=get_profile, proxy=get_proxy)
   def scrape_heading_task(driver: Driver, data):
       profile, proxy = driver.config.profile, driver.config.proxy
       print(profile, proxy)
       return profile, proxy

   data = [
       {"profile": "pikachu", "proxy": "http://142.250.77.228:8000"},
       {"profile": "greyninja", "proxy": "http://142.250.77.229:8000"},
   ]

   scrape_heading_task(data)
   ```

2. Directly passing configuration values when calling the decorated function:
   - Pass the profile and proxy values directly as arguments to the decorated function when calling it.

   Example:
   ```python
   from botasaurus.browser import browser, Driver

   @browser
   def scrape_heading_task(driver: Driver, data):
       profile, proxy = driver.config.profile, driver.config.proxy
       print(profile, proxy)
       return profile, proxy

   scrape_heading_task(
       profile='pikachu',  # Directly pass the profile
       proxy="http://142.250.77.228:8000",  # Directly pass the proxy
   )
   ```

PS: Most Botasaurus decorators allow passing functions to derive configurations from data parameters. Check the decorator's argument type hint to see if it supports this functionality.


### What is the best way to manage profile-specific data like name, age across multiple profiles?

To store data related to the active profile, use `driver.profile`. Here's an example:

```python
from botasaurus.browser import browser, Driver

def get_profile(data):
    return data["profile"]

@browser(profile=get_profile)
def run_profile_task(driver: Driver, data):
    # Set profile data
    driver.profile = {
        'name': 'Amit Sharma',
        'age': 30
    }

    # Update the name in the profile
    driver.profile['name'] = 'Amit Verma'

    # Delete the age from the profile
    del driver.profile['age']

    # Print the updated profile
    print(driver.profile)  # Output: {'name': 'Amit Verma'}

    # Delete the entire profile
    driver.profile = None

run_profile_task([{"profile": "amit"}])
```

For managing all profiles, use the `Profiles` utility. Here's an example:

```python
from botasaurus.profiles import Profiles

# Set profiles
Profiles.set_profile('amit', {'name': 'Amit Sharma', 'age': 30})
Profiles.set_profile('rahul', {'name': 'Rahul Verma', 'age': 30})

# Get a profile
profile = Profiles.get_profile('amit')
print(profile)  # Output: {'name': 'Amit Sharma', 'age': 30}

# Get all profiles
all_profiles = Profiles.get_profiles()
print(all_profiles)  # Output: [{'name': 'Amit Sharma', 'age': 30}, {'name': 'Rahul Verma', 'age': 30}]

# Get all profiles in random order
random_profiles = Profiles.get_profiles(random=True)
print(random_profiles)  # Output: [{'name': 'Rahul Verma', 'age': 30}, {'name': 'Amit Sharma', 'age': 30}] in random order

# Delete a profile
Profiles.delete_profile('amit')
```

Note: All profile data is stored in the `profiles.json` file in the current working directory.


### What are some common methods in Botasaurus Driver?

Botasaurus Driver provides several handy methods for web automation tasks, such as:

- Visiting URLs:
  ```python
  driver.get("https://www.example.com")
  driver.google_get("https://www.example.com")  # Use Google as the referer [Recommended]
  driver.get_via("https://www.example.com", referer="https://duckduckgo.com/")  # Use custom referer
  driver.get_via_this_page("https://www.example.com")  # Use current page as referer
  ```

- Finding elements:
  ```python
  from botasaurus.browser import Wait
  search_results = driver.select(".search-results", wait=Wait.SHORT)  # Wait for up to 4 seconds for the element to be present, return None if not found
  all_links = driver.select_all("a")  # Get all elements matching the selector
  search_results = driver.wait_for_element(".search-results", wait=Wait.LONG)  # Wait for up to 8 seconds for the element to be present, raise exception if not found
  hello_mom = driver.get_element_with_exact_text("Hello Mom", wait=Wait.VERY_LONG)  # Wait for up to 16 seconds for an element having the exact text "Hello Mom"
  ```

- Interacting with elements:
  ```python
  driver.type("input[name='username']", "john_doe")  # Type into an input field
  driver.click("button.submit")  # Click an element
  element = driver.select("button.submit")
  element.click()  # Click on an element
  element.select_option("select#fruits", index=2)  # Select an option
  ```

- Retrieving element properties:
  ```python
  header_text = driver.get_text("h1")  # Get text content
  error_message = driver.get_element_containing_text("Error: Invalid input")
  image_url = driver.select("img.logo").get_attribute("src")  # Get attribute value
  ```

- Working with parent-child elements:
  ```python
  parent_element = driver.select(".parent")
  child_element = parent_element.select(".child")
  child_element.click()  # Click child element
  ```

- Executing JavaScript:
  ```python
  result = driver.run_js("script.js") # Run a JavaScript file located in the current working directory.
  result = driver.run_js("return document.title")
  pikachu = driver.run_js("return args.pokemon", {"pokemon": 'pikachu'}) # args can be a dictionary, list, string, etc.
  text_content = driver.select("body").run_js("(el) => el.textContent")
  ```

- Enable human mode to perform, human-like mouse movements and say sayonara to detection:
  ```python
  # Navigate to Cloudflare's Turnstile Captcha demo
  driver.get(
    "https://nopecha.com/demo/cloudflare",
  )

  # Wait for page to fully load
  driver.long_random_sleep()
  
  # Locate iframe containing the Cloudflare challenge
  iframe = driver.get_element_at_point(160, 290)
  
  # Find checkbox element within the iframe
  checkbox = iframe.get_element_at_point(30, 30)

  # Enable human mode for realistic, human-like mouse movements
  driver.enable_human_mode()

  # Click the checkbox to solve the challenge
  checkbox.click()

  # (Optional) Disable human mode if no longer needed  
  driver.disable_human_mode()

  # Pause execution, for inspection
  driver.prompt()
  ```
  
  

- Drag and Drop:
  ```python
  # Open React DnD tutorial  
  driver.get("https://react-dnd.github.io/react-dnd/examples/tutorial")  

  # Select draggable and droppable elements  
  draggable = driver.select('[draggable="true"]')  
  droppable = driver.select('[data-testid="(3,6)"]')  

  # Perform drag-and-drop  
  draggable.drag_and_drop_to(droppable)  

  # Pause execution, for inspection
  driver.prompt()  
  ```

  

- Selecting Shadow Root Elements:
  ```python
  # Visit the website
  driver.get("https://nopecha.com/demo/cloudflare")
  
  # Wait for page to fully load
  driver.long_random_sleep()
  
  # Locate the element containing shadow root
  shadow_root_element = driver.select('[name="cf-turnstile-response"]').parent
  
  # Access the iframe
  iframe = shadow_root_element.get_shadow_root()

  # Access the nested shadow DOM inside the iframe 
  content = iframe.get_shadow_root()
  
  # print the text content of the "label" element.
  print(content.select("label", wait = 8).text)

  # Pause execution, for inspection
  driver.prompt()
  ```


- Monitoring requests:
  ```
/* Detailed source-code truncated for AI context efficiency. */
```  

- Working with iframes:
  ```python
  driver.get("https://www.freecodecamp.org/news/using-entity-framework-core-with-mongodb/")
  iframe = driver.get_iframe_by_link("www.youtube.com/embed") 
  # OR the following works as well
  # iframe = driver.select_iframe(".embed-wrapper iframe") 
  freecodecamp_youtube_subscribers_count = iframe.select(".ytp-title-expanded-subtitle").text
  print(freecodecamp_youtube_subscribers_count)
  ```
- Executing CDP Command:
  ```python
  from botasaurus.browser import browser, Driver, cdp
  driver.run_cdp_command(cdp.page.navigate(url='https://stackoverflow.blog/open-source'))
  ```

- Miscellaneous:
  ```python
  form.type("input[name='password']", "secret_password")  # Type into a form field
  container.is_element_present(".button")  # Check element presence
  page_html = driver.page_html  # Current page HTML
  driver.select(".footer").scroll_into_view()  # Scroll element into view
  driver.close()  # Close the browser
  ```

### How Can I Pause the Browser to Inspect Website when Developing the Scraper?

To pause the scraper and wait for user input before proceeding, use `driver.prompt()`:

```python
driver.prompt()
```

### How do I configure authenticated proxies with SSL in Botasaurus?

Proxy providers like BrightData, IPRoyal, and others typically provide authenticated proxies in the format "http://username:password@proxy-provider-domain:port". For example, "http://greyninja:awesomepassword@geo.iproyal.com:12321".

However, if you use an authenticated proxy with a library like seleniumwire to visit a website using Cloudflare, or Datadome, you are GUARANTEED to be identified because you are using a non-SSL connection.

To verify this, run the following code:

First, install the necessary packages:
```bash 
python -m pip install selenium_wire
```

Then, execute this Python script:
```python
from seleniumwire import webdriver  # Import from seleniumwire

# Define the proxy
proxy_options = {
    'proxy': {
        'http': 'http://username:password@proxy-provider-domain:port', # TODO: Replace with your own proxy
        'https': 'http://username:password@proxy-provider-domain:port', # TODO: Replace with your own proxy
    }
}

# Install and set up the driver
driver = webdriver.Chrome(seleniumwire_options=proxy_options)

# Visit the desired URL
link = 'https://fingerprint.com/products/bot-detection/'
driver.get("https://www.google.com/")
driver.execute_script(f'window.location.href = "{link}"')

# Prompt for user input
input("Press Enter to exit...")

# Clean up
driver.quit()
```

You will SURELY be identified:



However, using proxies with Botasaurus solves this issue. See the difference by running the following code:

```python
from botasaurus.browser import browser, Driver

@browser(proxy="http://username:password@proxy-provider-domain:port") # TODO: Replace with your own proxy 
def scrape_heading_task(driver: Driver, data):
    driver.google_get("https://fingerprint.com/products/bot-detection/")
    driver.prompt()

scrape_heading_task()    
```  

Result: 


Important Note: To run the code above, you will need [Node.js](https://nodejs.org/en) installed.

### Why am I getting a socket connection error when using a proxy to access a website?

Certain proxy providers like BrightData will block access to specific websites. To determine if this is the case, run the following code:

```python
from botasaurus.browser import browser, Driver

@browser(proxy="http://username:password@proxy-provider-domain:port")  # TODO: Replace with your own proxy
def visit_what_is_my_ip(driver: Driver, data):
    driver.get("https://whatismyipaddress.com/")
    driver.prompt()

visit_what_is_my_ip()
```

If you can successfully access [whatismyipaddress.com](https://whatismyipaddress.com/) but not the website you're attempting to scrape, it means the proxy provider is blocking access to that particular website. 

In such situations, the only solution is to switch to a different proxy provider.

Some good proxy providers we personally use are:

- For Rotating Datacenter Proxies: **BrightData Datacenter Proxies**, which cost around $0.6 per GB on a pay-as-you-go basis. No KYC is required.
- For Rotating Residential Proxies: **IPRoyal Royal Residential Proxies**, which cost around $7 per GB on a pay-as-you-go basis. No KYC is required.

As always, nothing good in life comes free. Proxies are expensive, and will take up almost all of your scraping costs. 

So, use proxies only when you need them, and prefer request-based scrapers over browser-based scrapers to save bandwidth.

Note: BrightData and IPRoyal have not paid us. We are recommending them based on our personal experience.

### Which country should I choose when using proxies for web scraping?

The United States is often the best choice because:
- The United States has a highly developed internet infrastructure and is home to numerous data centers, ensuring faster internet speeds.
- Most global companies host their websites in the US, so using a US proxy will result in faster scraping speeds.

### Should I use a proxy for web scraping?

ONLY IF you encounter IP blocks.

Sadly, most scrapers unnecessarily use proxies, even when they are not needed. Everything seems like a nail when you have a hammer.

We have seen scrapers which can easily access hundreds of thousands of protected pages using the @browser module on home Wi-Fi without any issues.

So, as a best practice scrape using the @browser module on your home Wi-Fi first. Only resort to proxies when you encounter IP blocks. 

This practice will save you a considerable amount of time (as proxies are really slow) and money (as proxies are expensive as well).

### How to configure the Request Decorator?

The Request Decorator is used to make humane requests. Under the hood, it uses botasaurus-requests, a library based on hrequests, which incorporates important features like:
- Using browser-like headers in the correct order.
- Makes a browser-like connection with correct ciphers.
- Uses `google.com` referer by default to make it appear as if the user has arrived from google search.

Also, The Request Decorator allows you to configure proxy as follows:

```python
@request(
    proxy="http://username:password@proxy-provider-domain:port"
)    
```


### What Options Can I Configure in all 3 Decorators?

All 3 decorators allow you to configure the following options:

- Parallel Execution:
- Caching Results
- Passing Common Metadata
- Asynchronous Queues
- Asynchronous Execution
- Handling Crashes
- Configuring Output
- Exception Handling

Let's dive into each of these options and in later sections we will see their real-world applications.

#### `parallel`

The `parallel` option allows you to scrape data in parallel by launching multiple browser/request/task instances simultaneously. This can significantly speed up the scraping process.

Run the example below to see parallelization in action:
```python
from botasaurus.browser import browser, Driver

@browser(parallel=3, data=["https://stackoverflow.blog/open-source", "https://stackoverflow.blog/ai", "https://stackoverflow.blog/productivity",])
def scrape_heading_task(driver: Driver, link):
    driver.get(link)
    heading = driver.get_text('h1')
    return heading

scrape_heading_task()    
```

#### `cache`

The `cache` option enables caching of web scraping results to avoid re-scraping the same data. This can significantly improve performance and reduce redundant requests.


Run the example below to see how caching works:
```python
from botasaurus.browser import browser, Driver

@browser(cache=True, data=["https://stackoverflow.blog/open-source", "https://stackoverflow.blog/ai", "https://stackoverflow.blog/productivity",])
def scrape_heading_task(driver: Driver, link):
    driver.get(link)
    heading = driver.get_text('h1')
    return heading

print(scrape_heading_task())
print(scrape_heading_task())  # Data will be fetched from cache immediately 
```

Note: Caching is one of the most important features of Botasaurus.

#### `metadata`

The metadata option allows you to pass common information shared across all data items. This can include things like API keys, browser cookies, or any other data that remains constant throughout the scraping process.

It is commonly used with caching to exclude details like API keys and browser cookies from the cache key.

Here's an example of how to use the `metadata` option:
```python
from botasaurus.task import task

@task()
def scrape_heading_task(driver: Driver, data, metadata):
    print("metadata:", metadata)
    print("data:", data)

data = [
    {"profile": "pikachu", "proxy": "http://142.250.77.228:8000"},
    {"profile": "greyninja", "proxy": "http://142.250.77.229:8000"},
]
scrape_heading_task(
  data, 
  metadata={"api_key": "BDEC26..."}
)
```

#### `async_queue`

In the world of web scraping, there are only two types of scrapers:

1. Dataset Scrapers: These extract data from websites and store it as datasets. Companies like Bright Data use them to build datasets for Crunchbase, Indeed, etc.

2. Real-time Scrapers: These fetch data from sources in real-time, like SERP APIs that provide Google and DuckDuckGo search results.

When building real-time scrapers, speed is paramount because customers are waiting for requests to complete. The `async_queue` feature is incredibly useful in such cases.

`async_queue` allows you to run scraping tasks asynchronously in a queue and gather the results using the `.get()` method. 

A great use case for `async_queue` is scraping Google Maps. Instead of scrolling through the list of places and then scraping the details of each place sequentially, you can use `async_queue` to:
1. Scroll through the list of places.
2. Simultaneously make HTTP requests to scrape the details of each place in the background.
    
By executing the scrolling and requesting tasks concurrently, you can significantly speed up the scraper.

Run the code below to see browser scrolling and request scraping happening concurrently (really cool, must try!):
```
/* Detailed source-code truncated for AI context efficiency. */
```

#### `run_async`

Similarly, the `run_async` option allows you to execute scraping tasks asynchronously, enabling concurrent execution.

Similar to `async_queue`, you can use the `.get()` method to retrieve the results of an asynchronous task.

Code Example:
```python
from botasaurus.browser import browser, Driver
from time import sleep

@browser(run_async=True)
def scrape_heading(driver: Driver, data):
    sleep(5)
    return {}

if __name__ == "__main__":
    result1 = scrape_heading()  # Launches asynchronously
    result2 = scrape_heading()  # Launches asynchronously

    result1.get()  # Wait for the first result
    result2.get()  # Wait for the second result
```

#### `close_on_crash`

The `close_on_crash` option determines the behavior of the scraper when an exception occurs:
- If set to `False` (default):
  - The scraper will make a beep sound and pause the browser.
  - This makes debugging easier by keeping the browser open at the point of the crash.
  - Use this setting during development and testing.
- If set to `True`:
  - The scraper will close the browser and continue with the rest of the data items.
  - This is suitable for production environments when you are confident that your scraper is robust.
  - Use this setting to avoid interruptions and ensure the scraper processes all data items.

```python
from botasaurus.browser import browser, Driver

@browser(
    close_on_crash=False  # Determines whether the browser is paused (default: False) or closed when an error occurs
)
def scrape_heading_task(driver: Driver, data):
    raise Exception("An error occurred during scraping.")

scrape_heading_task()  
```

#### `output` and `output_formats`

By default, Botasaurus saves the result of scraping in the `output/{your_scraping_function_name}.json` file. Let's learn about various ways to configure the output.

1. **Change Output Filename**: Use the `output` parameter in the decorator to specify a custom filename for the output.
```python
from botasaurus.task import task

@task(output="my-output")
def scrape_heading_task(data): 
    return {"heading": "Hello, Mom!"}

scrape_heading_task()
```

2. **Disable Output**: If you don't want any output to be saved, set `output` to `None`.
```python
from botasaurus.task import task

@task(output=None)
def scrape_heading_task(data): 
    return {"heading": "Hello, Mom!"}

scrape_heading_task()
```

3. **Dynamically Write Output**: To dynamically write output based on data and result, pass a function to the `output` parameter:
```python
from botasaurus.task import task
from botasaurus import bt

def write_output(data, result):
    json_filename = bt.write_json(result, 'data')
    excel_filename = bt.write_excel(result, 'data')
    bt.zip_files([json_filename, excel_filename]) # Zip the JSON and Excel files for easy delivery to the customer

@task(output=write_output)  
def scrape_heading_task(data): 
    return {"heading": "Hello, Mom!"}

scrape_heading_task()
```

4. **Upload File to S3**: Use `bt.upload_to_s3` to upload file to S3 bucket.
```python
from botasaurus.task import task
from botasaurus import bt

def write_output(data, result):
    json_filename = bt.write_json(result, 'data')
    bt.upload_to_s3(json_filename, 'my-magical-bucket', "AWS_ACCESS_KEY", "AWS_SECRET_KEY")

@task(output=write_output)  
def scrape_heading_task(data): 
    return {"heading": "Hello, Mom!"}

scrape_heading_task()
```

5.**Save Outputs in Multiple Formats**: Use the `output_formats` parameter to save outputs in different formats like JSON and EXCEL.
```python
from botasaurus.task import task

@task(output_formats=[bt.Formats.JSON, bt.Formats.EXCEL])  
def scrape_heading_task(data): 
    return {"heading": "Hello, Mom!"}

scrape_heading_task()
```

PRO TIP: When delivering data to customers, provide the dataset in JSON and Excel formats. Avoid CSV unless the customer asks, because Microsoft Excel has a hard time rendering CSV files with nested JSON.

**CSV vs Excel**


#### Exception Handling Options

Botasaurus provides various exception handling options to make your scrapers more robust:
- `max_retry`: By default, any failed task is not retried. You can specify the maximum number of times to retry scraping when an error occurs using the `max_retry` option.
- `retry_wait`: Specifies the waiting time between retries.
- `raise_exception`: By default, Botasaurus does not raise an exception when an error occurs during scraping, because let's say you are keeping your PC running overnight to scrape 10,000 links. If one link fails, you really don't want to stop the entire scraping process, and ruin your morning by seeing an unfinished dataset.
- `must_raise_exceptions`: Specifies exceptions that must be raised, even if `raise_exception` is set to `False`.
- `create_error_logs`: Determines whether error logs should be created when exceptions occur. In production, when scraping hundreds of thousands of links, it's recommended to set `create_error_logs` to `False` to avoid using computational resources for creating error logs.

```python
@browser(
    raise_exception=True,  # Raise an exception and halt the scraping process when an error occurs
    max_retry=5,  # Retry scraping a failed task a maximum of 5 times
    retry_wait=10,  # Wait for 10 seconds before retrying a failed task
    must_raise_exceptions=[CustomException],  # Definitely raise CustomException, even if raise_exception is set to False
    create_error_logs=False  # Disable the creation of error logs to optimize scraper performance
)
def scrape_heading_task(driver: Driver, data):
  # ...
```

### What are some examples of common web scraping utilities provided by Botasaurus that make scraping easier?

#### bt Utility

The `bt` utility provides helper functions for:

- Writing and reading JSON, EXCEL, and CSV files
- Data cleaning

Some key functions are:

- `bt.write_json` and `bt.read_json`: Easily write and read JSON files.
```python
from botasaurus import bt

data = {"name": "pikachu", "power": 101}
bt.write_json(data, "output")
loaded_data = bt.read_json("output")
```

- `bt.write_excel` and `bt.read_excel`: Easily write and read EXCEL files.
```python
from botasaurus import bt

data = {"name": "pikachu", "power": 101}
bt.write_excel(data, "output")
loaded_data = bt.read_excel("output")
```

- `bt.write_csv` and `bt.read_csv`: Easily write and read CSV files.
```python
from botasaurus import bt

data = {"name": "pikachu", "power": 101}
bt.write_csv(data, "output")
loaded_data = bt.read_csv("output")
```

- `bt.write_html` and `bt.read_html`: Write HTML content to a file.
```python
from botasaurus import bt

html_content = "<html><body><h1>Hello, Mom!</h1></body></html>"
bt.write_html(html_content, "output")
```

- `bt.write_temp_json`, `bt.write_temp_csv`, `bt.write_temp_html`: Write temporary JSON, CSV, or HTML files for debugging purposes.
```python
from botasaurus import bt

data = {"name": "pikachu", "power": 101}
bt.write_temp_json(data)
bt.write_temp_csv(data)
bt.write_temp_html("<html><body><h1>Hello, Mom!</h1></body></html>")
```

- Data cleaning functions like `bt.extract_numbers`, `bt.extract_links`, `bt.remove_html_tags`, and more.
```python
text = "The price is $19.99 and the website is https://www.example.com"
numbers = bt.extract_numbers(text)  # [19.99]
links = bt.extract_links(text)  # ["https://www.example.com"]
```

#### Local Storage Utility

The Local Storage utility allows you to store and retrieve key-value pairs, which can be useful for maintaining state between scraper runs. 

Here's how to use it:

```python
from botasaurus.local_storage import LocalStorage

LocalStorage.set_item("credits_used", 100)
print(LocalStorage.get_item("credits_used", 0))
```

#### soupify Utility

The `soupify` utility creates a BeautifulSoup object from a Driver, Requests response, Driver Element, or HTML string.

```python
from botasaurus.soupify import soupify
from botasaurus.request import request, Request
from botasaurus.browser import browser, Driver

@request
def get_heading_from_request(req: Request, data):
   """
   Get the heading of a web page using the request object.
   """
   response = req.get("https://www.example.com")
   soup = soupify(response)
   heading = soup.find("h1").text
   print(f"Page Heading: {heading}")

@browser
def get_heading_from_driver(driver: Driver, data):
   """
   Get the heading of a web page using the driver object.
   """
   driver.get("https://www.example.com")

   # Get the heading from the entire page
   page_soup = soupify(driver)
   page_heading = page_soup.find("h1").text
   print(f"Heading from Driver's Soup: {page_heading}")

   # Get the heading from the body element
   body_soup = soupify(driver.select("body"))
   body_heading = body_soup.find("h1").text
   print(f"Heading from Element's Soup: {body_heading}")

# Call the functions
get_heading_from_request()
get_heading_from_driver()
```

#### IP Utils

IP Utils provide functions to get information about the current IP address, such as the IP itself, country, ISP, and more:

```python
from botasaurus.ip_utils import IPUtils

# Get the current IP address
current_ip = IPUtils.get_ip()
print(current_ip)
# Output: 47.31.226.180

# Get detailed information about the current IP address
ip_info = IPUtils.get_ip_info()
print(ip_info)
# Output: {
#     "ip": "47.31.226.180",
#     "country": "IN",
#     "region": "Delhi",
#     "city": "Delhi",
#     "postal": "110001",
#     "coordinates": "28.6519,77.2315",
#     "latitude": "28.6519",
#     "longitude": "77.2315",
#     "timezone": "Asia/Kolkata",
#     "org": "AS55836 Reliance Jio Infocomm Limited"
# }
```

#### Cache Utility
The Cache utility in Botasaurus allows you to manage cached data for your scraper. You can put, get, has, remove, and clear cache data.

*Basic Usage*

```python
from botasaurus.task import task
from botasaurus.cache import Cache

# Example scraping function
@task
def scrape_data(data):
    # Your scraping logic here
    return {"processed": data}

# Sample data for scraping
input_data = {"key": "value"}

# Adding data to the cache
Cache.put('scrape_data', input_data, scrape_data(input_data))

# Checking if data is in the cache
if Cache.has('scrape_data', input_data):
    # Retrieving data from the cache
    cached_data = Cache.get('scrape_data', input_data)
    print(f"Cached data: {cached_data}")

# Removing specific data from the cache
Cache.remove('scrape_data', input_data)

# Clearing the complete cache for the scrape_data function
Cache.clear('scrape_data')
```

**Advanced Usage for large-scale scraping projects**

*Count Cached Items*

You can count the number of items cached for a particular function, which can serve as a scraping progress bar.

```python
from botasaurus.cache import Cache

Cache.print_cached_items_count('scraping_function')
```

*Filter Cached/Uncached Items*

You can filter items that have been cached or not cached for a particular function.

```python
from botasaurus.cache import Cache

all_items = ['1', '2', '3', '4', '5']

# Get items that are cached
cached_items = Cache.filter_items_in_cache('scraping_function', all_items)
print(cached_items)

# Get items that are not cached
uncached_items = Cache.filter_items_not_in_cache('scraping_function', all_items)
print(uncached_items)
```

*Delete Cache*
The cache for a function is stored in the `cache/{your_scraping_function_name}/` folder. To delete the cache, simply delete that folder.




*Delete Specific Items*

You can delete specific items from the cache for a particular function.

```python
from botasaurus.cache import Cache

all_items = ['1', '2', '3', '4', '5']
deleted_count = Cache.delete_items('scraping_function', all_items)
print(f"Deleted {deleted_count} items from the cache.")
```

*Delete Items by Filter*  

In some cases, you may want to delete specific items from the cache based on a condition. For example, if you encounter honeypots (mock HTML served to dupe web scrapers) while scraping a website, you may want to delete those items from the cache.

```python
def should_delete_item(item, result):
    if 'Honeypot Item' in result:
        return True  # Delete the item
    return False  # Don't delete the item

all_items = ['1', '2', '3', '4', '5']
# List of items to iterate over, it is fine if the list contains items which have not been cached, as they will be simply ignored.
Cache.delete_items_by_filter('scraping_function', should_delete_item, all_items)
```

Importantly, be cautious and first use `delete_items_by_filter` on a small set of items which you want to be deleted. Here's an example:

```python
from botasaurus import bt
from botasaurus.cache import Cache

def should_delete_item(item, result):
    # TODO: Update the logic
    if 'Honeypot Item' in result:
        return True # Delete the item
    return False # Don't delete the item

test_items = ['1', '2'] # TODO: update with target items
scraping_function_name = 'scraping_function' # TODO:  update with target scraping function name
Cache.delete_items_by_filter(scraping_function_name, test_items, should_delete_item)

for item in test_items:
    if Cache.has(scraping_function_name, item):
        bt.prompt(f"Item {item} was not deleted. Please review the logic of the should_delete_item function.")
```

### How to Extract Links from a Sitemap?

In web scraping, it is a common use case to scrape product pages, blogs, etc. But before scraping these pages, you need to get the links to these pages.

Sadly, many developers unnecessarily increase their work by writing code to visit each page one by one and scrape links, which they could have easily obtained by just looking at the Sitemap.

The Botasaurus Sitemap Module makes this process easy as cake by allowing you to get all links or sitemaps using:
- The homepage URL (e.g., `https://www.omkar.cloud/`)
- A direct sitemap link (e.g., `https://www.omkar.cloud/sitemap.xml`)
- A `.gz` compressed sitemap

For example, if you're an Angel Investor seeking innovative tech startups to invest in, G2 is an ideal platform to find such startups. You can run the following code to fetch over 190K+ product links from G2:

```python
from botasaurus import bt
from botasaurus.sitemap import Sitemap, Filters, Extractors

links = (
    Sitemap("https://www.g2.com/sitemaps/sitemap_index.xml.gz")
    .filter(Filters.first_segment_equals("products"))
    .extract(Extractors.extract_link_upto_second_segment())
    .write_links('g2-products')
)
```

**Output:** 



Or, let's say you're in the mood for some reading and looking for good stories. The following code will get you over 1000+ stories from [moralstories26.com](https://moralstories26.com/):

```python
from botasaurus import bt
from botasaurus.sitemap import Sitemap, Filters

links = (
    Sitemap("https://moralstories26.com/")
    .filter(
        Filters.has_exactly_1_segment(),
        Filters.first_segment_not_equals(
            ["about", "privacy-policy", "akbar-birbal", "animal", "education", "fables", "facts", "family", "famous-personalities", "folktales", "friendship", "funny", "heartbreaking", "inspirational", "life", "love", "management", "motivational", "mythology", "nature", "quotes", "spiritual", "uncategorized", "zen"]
        ),
    )
    .write_links('moral-stories')
)
```

**Output:** 



Also, before scraping a site, it's useful to identify the available sitemaps. This can be easily done with the following code:

```python
from botasaurus import bt
from botasaurus.sitemap import Sitemap

sitemaps = Sitemap("https://www.omkar.cloud/").write_sitemaps('omkar-sitemaps')
```

**Output:** 



To ensure your scrapers run super fast, we cache the Sitemap, but you may want to periodically refresh the cache. To do so, pass the Cache.REFRESH parameter. 

```python
from botasaurus import bt
from botasaurus.sitemap import Sitemap, Filters, Extractors
from botasaurus.cache import Cache

links = (
    Sitemap("https://moralstories26.com/", cache=Cache.REFRESH)
    .write_links('moral-stories')
)
```
### How can I filter a list of links, similar to working with Sitemaps?

Filtering links from a webpage is a common requirement in web scraping. For example, you might want to filter out all non-product pages.


Botasaurus's `Links` module simplifies link filtering and extraction:

```python
from botasaurus.links import Links, Filters, Extractors

# Sample list of links
links = [
    "https://finance.yahoo.com/topic/stock-market-news/",
    "https://finance.yahoo.com/topic/morning-brief/", 
    "https://finance.yahoo.com/quote/AAPL/", 
    "https://finance.yahoo.com/quote/GOOG/"
]

# Filter and extract links
filtered_links = (
    Links(links)
    .filter(Filters.first_segment_equals("quote"))
    .extract(Extractors.extract_link_upto_second_segment())
    .write('stocks')
)
```

### What is the best way to use caching in Botasaurus?

Sadly, when using caching, most developers write a scraping function that scrapes the HTML and extracts the data from the HTML in the same function, like this:

```python
from botasaurus.request import request, Request
from botasaurus.soupify import soupify

@request
def scrape_data(request: Request, data):
    # Visit the Link
    response = request.get(data)
    
    # Create a BeautifulSoup object
    soup = soupify(response)
    
    # Retrieve the heading element's text
    heading = soup.find('h1').get_text()
    
    # Save the data as a JSON file in output/scrape_data.json
    return {"heading": heading}

data_items = [
    "https://stackoverflow.blog/open-source",
    "https://stackoverflow.blog/ai",
    "https://stackoverflow.blog/productivity",
]

scrape_data(data_items)
```

Now, let's say, after 50% of the dataset has been scraped, what if:
- Your customer wants to add another data point (which is very likely), or
- One of your BeautifulSoup selectors happens to be flaky and needs to be updated (which is super likely)?

In such cases, you will have to scrape all the pages again, which is painful as it will take a lot of time and incur high proxy costs.

To resolve this issue, you can:
1. Write a function that only scrapes and caches the HTML.
2. Write a separate function that calls the HTML scraping function, extracts data using BeautifulSoup, and caches the result.

Here's a practical example:

```python
from bs4 import BeautifulSoup
from botasaurus.task import task
from botasaurus.request import request, Request
from botasaurus.soupify import soupify

@request(cache=True)
def scrape_html(request: Request, link):
    # Scrape the HTML and cache it
    html = request.get(link).text
    return html

def extract_data(soup: BeautifulSoup):
    # Extract the heading from the HTML
    heading = soup.find("h1").get_text()
    return {"heading": heading}

# Cache the scrape_data task as well
@task(cache=True)
def scrape_data(link):
    # Call the scrape_html function to get the cached HTML
    html = scrape_html(link)
    # Extract data from the HTML using the extract_data function
    return extract_data(soupify(html))

data_items = [
    "https://stackoverflow.blog/open-source",
    "https://stackoverflow.blog/ai",
    "https://stackoverflow.blog/productivity",
]

scrape_data(data_items)
```

With this approach:
- If you need to add data points or fix BeautifulSoup bugs, delete the `cache/scrape_data` folder and re-run the scraper.

- You only need to re-run the BeautifulSoup extraction, not the entire HTML scraping, saving time and proxy costs. Yahoo!

**PRO TIP**: This approach also makes your `extract_data` code easier and faster to test, like this:

```python
from bs4 import BeautifulSoup
from botasaurus import bt

def extract_data(soup: BeautifulSoup):
    heading = soup.find('h1').get_text()
    return {"heading": heading}

if __name__ == '__main__':
    # Will use the cached HTML and run the extract_data function again.
    bt.write_temp_json(scrape_data("https://stackoverflow.blog/open-source", cache=False))
```
### What are the recommended settings for each decorator to build a production-ready scraper in Botasaurus?

For websites with minimal protection, use the `Request` module.

Here's a template for creating production-ready datasets using the `Request` module:

```
/* Detailed source-code truncated for AI context efficiency. */
```

For visiting well protected websites, use the `Browser` module. 

Here's a template for creating production-ready datasets using the `Browser` module:

```
/* Detailed source-code truncated for AI context efficiency. */
```

### Why Am I Getting Detected on Some Websites?

If you're getting detected, it's likely due to:

- Using a non-residential proxy — Services like Datadome and Cloudflare often flag datacenter IPs/VPNs.
- Clicking without Human Mode enabled — Unnatural mouse movements can trigger detection.
- Visiting websites too quickly — Rapid, bot-like navigation is easy to detect.

To reduce detection, follow these best practices:

1. Upgrade all Botasaurus packages to the latest version:
   ```bash
   python -m pip install --upgrade bota botasaurus botasaurus-api botasaurus-requests botasaurus-driver botasaurus-proxy-authentication botasaurus-server botasaurus-humancursor
   ```

2. Enable Human Mode for human-like mouse movements:
   ```python
   driver.enable_human_mode()
   ```

3. Avoid rapid `driver.get` calls. Instead, try:
   - Clicking within pages with Human Mode enabled, if possible.
   - Using `driver.google_get` or `driver.get_via_this_page`.
   - Using [`driver.requests.get`](https://github.com/omkarcloud/botasaurus?tab=readme-ov-file#how-to-significantly-reduce-proxy-costs-when-scraping-at-scale) to fetch the page HTML content.

4. Slow down your scraper with random delays:
   ```python
   driver.short_random_sleep()
   driver.long_random_sleep()
   ```

5. Avoid using `headless` mode, as it can be easily detected by Cloudflare, Datadome, and Imperva.

6. Use a residential proxy, as datacenter IPs are often flagged.

7. Remove the `--no-default-browser-check` argument as it is detectable by systems like Datadome, as follows:
   ```python
   @browser(remove_default_browser_check_argument=True)
   ```

8. If your IP has been flagged, you can perform this technique to change it:
1. Visit [whatismyipaddress.com](https://whatismyipaddress.com/) to see your current IP Address.
2. Connect your PC to a smartphone's hotspot.
3. On your smartphone, turn airplane mode on and off.
4. Turn the hotspot on again.
5. Now visit [whatismyipaddress.com](https://whatismyipaddress.com/). You'll see a new IP address.

### How Do I Close All Running Chrome Instances?

While developing a scraper, multiple browser instances may remain open in the background (because of interrupting it with CTRL + C). This situation can cause your computer to hang.



To prevent your PC from hanging, you can run the following command to close all Chrome instances:

```shell
python -m close_chrome
```

### How to Run Scraper in Docker?

To run a Scraper in Docker, use the Botasaurus Starter Template, which includes the necessary Dockerfile and Docker Compose configurations.

Use the following commands to clone the Botasaurus Starter Template, build a Docker image from it, and execute the scraper within a Docker environment.

```
git clone https://github.com/omkarcloud/botasaurus-starter my-botasaurus-project
cd my-botasaurus-project
docker-compose build
docker-compose up
```

### How to Run Scraper in Gitpod?

Running a scraper in Gitpod offers several benefits:

- Allows your scraper to use a powerful 8-core machine with 1000 Mbps internet speed
- Makes it easy to showcase your scraper to customers without them having to install anything, by simply sharing the Gitpod machine link

In this example, we will run the Botasaurus Starter template in Gitpod:

1. First, visit [this link](https://gitpod.io/#https://github.com/omkarcloud/botasaurus-starter) and sign up using your GitHub account.
   
   
  
2. Once signed up, open the starter project in Gitpod.   

   

3. To use UI Scraper, run the following command in Terminal:
   ```bash
   python run.py
   ```
4. You will see a popup notification with the heading "A service is available on port 3000". In the popup notification, click the **"Open Browser"** button to open the UI Dashboard in your browser

   

5. Now, you can press the `Run` button to get the results.
   
   

Note: Gitpod is not suitable for long-running tasks, as the environment will automatically shut down after a short period of inactivity. Use your local machine or a cloud VM for long-running scrapers.

## Should I Scrape Datasets Locally or in the Cloud?  
For most scraping tasks, we recommend running the scraper **locally** on your system because:  
- It requires far fewer setup steps  
- It saves time and costs  
- Most importantly, it allows you to quickly fix bugs and continue scraping.

However, consider cloud scraping when:  
- Running tasks longer than 5 days.  
- Scraping large-scale data (terabytes).  
- Performing recurring monthly scrapes.  
- Having slow Internet or data caps

Cloud scraping is also significantly faster due to superior internet speeds (often 10x+ faster than home Wi-Fi).  

---

## How to Run a Data Scraper in a Virtual Machine? 

To run a scraper in a virtual machine (VM), follow these steps:

### 1. Prepare Your Scraper
1. Use the [Botasaurus Starter Template](https://github.com/omkarcloud/botasaurus-starter) to create your dataset scraper.
2. For large datasets, ensure memory efficiency (e.g., by using file formats like `ndjson`).  
3. Add project dependencies to `requirements.txt`.
4. Push your project to GitHub.

### 2. Set Up a Virtual Machine

1. If you don't already have one, create a Google Cloud Account. You'll receive a $300 credit to use over 3 months.
   


2. Go to [Google Click to Deploy](https://console.cloud.google.com/marketplace/product/click-to-deploy-images/nodejs), create a deployment and configure it as follows or as appropriate based on your workload:
   ```
   Zone: us-central1-a # Use us-central1 (Iowa) for the lowest-cost VMs
   Series: N1
   Machine Type: n1-standard-2 (2 vCPU, 1 core, 7.5 GB memory)
   Boot Disk Type: Standard persistent disk	# This is the most cost-effective disk option.
   Boot disk size in GB: 20 GB # Adjust based on storage needs  
   ```
   

3. Navigate to [VM Instances](https://console.cloud.google.com/compute/instances) and click the SSH button to SSH into the VM.
   

4.  Now, run the following command in the terminal and wait for it to complete:

   ```bash
   curl -sL https://raw.githubusercontent.com/omkarcloud/botasaurus/master/vm-scripts/install-bota.sh | bash
   ```
   
5.  Install your scraper by running the following command. It may take 5 minutes to install the scraper, so grab a coffee or watch [this awesome video](https://www.youtube.com/watch?v=nwAYpLVyeFU):

   ```bash
   python3 -m bota install-scraper --repo-url https://github.com/omkarcloud/botasaurus-starter
   ```
   

   Note: If you are using a different repo, replace `https://github.com/omkarcloud/botasaurus-starter` with your repo URL.
   
That's it! You have successfully installed the scraper in a virtual machine. The scraper will now start running and succeed.



*Notes*
- The `main.py` file serves as the scraper's entry point.
- Update your project's `requirements.txt` to ensure it has all the dependencies required to run the scraper.
- Ensure your VM has enough memory for your scraper's needs.
- If running a headful browser, enable a virtual display by setting `enable_xvfb_virtual_display=True`. This creates a virtual display required for running a headful browser in a VM.
  ```python
  @browser(enable_xvfb_virtual_display=True)
  ```
- The scraper will run until it completes successfully or fails three times. You can also configure retries as follows:
  For example, to allow a maximum of 5 retries:
  ```bash
  python3 -m bota install-scraper --repo-url <your-repo> --max-retry=5
  ```
  or, to allow unlimited retries:
  ```bash
  python3 -m bota install-scraper --repo-url <your-repo> --max-retry=unlimited
  ```
- If your scraper fails, you can check the logs of the current boot by running:
  ```bash
  journalctl -u botasaurus-starter.service -b # replace botasaurus-starter with your repo name
  ```

*Downloading Data*
To download the scraped data, you can either:
1. Download it directly from the VM.

2. Upload it to Amazon S3:
   ```python
   from botasaurus import bt
   bt.upload_to_s3('data.json', 'my-bucket', "AWS_ACCESS_KEY", "AWS_SECRET_KEY")
   ```

## How to Stop the Scraper
If you are performing recurring monthly scrapes, it's best to stop the scraper instead of deleting it. Note that you will only incur storage costs (~$0.4 per month for a 10GB Standard Persistent Disk) but not compute costs.

To stop the scraper:
1. Visit [VM Instances](https://console.cloud.google.com/compute/instances).
2. Select your VM and stop it.



To restart later:
1. Start the VM from [VM Instances](https://console.cloud.google.com/compute/instances).


2. SSH into it.
3. Delete caches and update sitemaps if needed.
4. Restart with:
   ```bash
   shutdown -r now
   ```

## How to Delete the Scraper and Avoid Incurring Further Charges

If you no longer need the scraper, please ensure you have downloaded your data before deleting it.

Next, follow these steps to delete the scraper:
1. Go to [Deployments](https://console.cloud.google.com/products/solutions/deployments).
2. Delete your deployment.



That's it! You have successfully deleted the scraper, and you will not incur any disk or compute charges.

## How to Run a UI Scraper in a Virtual Machine

To run your scraper in a virtual machine, we will:
- Create a static IP
- Create a VM with that IP
- SSH into the VM
- Install the scraper

Now, follow these steps to run your scraper in a virtual machine:

1. Create a Google Cloud Account if you don't already have one.
   

2. Visit the [Google Cloud Console](https://console.cloud.google.com/welcome?cloudshell=true) and click the Cloud Shell button. A terminal will open up.
   

3. Run the following commands in the terminal:

   ```bash
   python -m pip install bota
   python -m bota create-ip
   ```

   You will be asked for a VM name. Enter any name you like, such as "pikachu".

   > Name: pikachu

   Then, you will be asked for the region for the scraper. Press Enter to go with the default, which is "us-central1", as it has the lowest-cost VMs.

   > Region: Default

   

4. Go to [Google Click to Deploy](https://console.cloud.google.com/marketplace/product/click-to-deploy-images/nodejs), create a deployment and configure it as follows or as appropriate based on your workload:
   ```
   Zone: us-central1-a # Use the zone from the region you selected in the previous step.
   Series: N1
   Machine Type: n1-standard-2 (2 vCPU, 1 core, 7.5 GB memory)
   Boot Disk Type: Standard persistent disk	# This is the most cost-effective disk option.
   Boot disk size in GB: 20 GB # Adjust based on storage needs  
   Network Interface [External IP]: pikachu-ip # Use the IP name you created in the previous step.
   ```
   

5. Navigate to [VM Instances](https://console.cloud.google.com/compute/instances) and click the SSH button to SSH into the VM.
   

6.  Now, run the following command in the terminal:

   ```bash
   curl -sL https://raw.githubusercontent.com/omkarcloud/botasaurus/master/vm-scripts/install-bota.sh | bash
   ```

7. Finally, install the UI scraper by running the following command, then wait for 5 minutes for it to complete:
   ```bash
   python3 -m bota install-ui-scraper --repo-url https://github.com/omkarcloud/botasaurus-starter
   ```
   
   
   Note: If you are using a different repo, replace `https://github.com/omkarcloud/botasaurus-starter` with your repo URL.

That's it! You have successfully launched the scraper in a virtual machine. When the previous commands are done, you will see a **link** to your scraper. Visit it to run your scraper.



*Notes*
    - Update your project's `requirements.txt` to ensure it has all the dependencies required to run the scraper.
    - Ensure your VM has enough memory for your scraper's needs.
    - If running a headful browser, enable a virtual display by setting `enable_xvfb_virtual_display=True`. This creates a virtual display required for running a headful browser in a VM.
  ```python
  @browser(enable_xvfb_virtual_display=True)
  ```
    - The UI scraper will run indefinitely and will be available at the printed link.
    - If your UI scraper fails, you can check the logs of the current boot by running:
  ```bash
  journalctl -u backend.service -b 
  ```

##  How to Delete the UI Scraper and Avoid Incurring Further Charges

If you no longer need the scraper, please ensure you have downloaded your data before deleting it.

Next, follow these steps to delete the scraper:

1. Delete the static IP by running the following command:

   ```bash
   python -m bota delete-ip
   ```

   You will be asked for the name of the VM you created in the first step. Enter the name and press Enter.

   

   Note: If you forgot the name of the IP, you can also delete all the IPs by running `python -m bota delete-all-ips`.

2. Go to [Deployments](https://console.cloud.google.com/products/solutions/deployments) and delete your deployment.

   

That's it! You have successfully deleted the scraper, and you will not incur any further charges.

### How to Run Scraper in Kubernetes?
Visit [this link](https://github.com/omkarcloud/botasaurus/blob/master/run-scraper-in-kubernetes.md) to learn how to run scraper at scale using Kubernetes.


### I have a feature request!

We'd love to hear it! Share them on [GitHub Discussions](https://github.com/omkarcloud/botasaurus/discussions). 

[](https://www.omkar.cloud/l/botasaurus-make-discussions/)


### ❓ Advanced Questions

Congratulations on completing the Botasaurus Documentation! Now, you have all the knowledge needed to effectively use Botasaurus. 

You may choose to read the following questions based on your interests:

1. [How to Run Botasaurus in Google Colab?](https://github.com/omkarcloud/botasaurus/blob/master/advanced.md#how-to-run-botasaurus-in-google-colab)

2. [How can I allow users to filter the scraped data?](https://github.com/omkarcloud/botasaurus/blob/master/advanced.md#how-can-i-allow-users-to-filter-the-scraped-data)

3. [How can I allow the user to sort the scraped data?](https://github.com/omkarcloud/botasaurus/blob/master/advanced.md#how-can-i-allow-the-user-to-sort-the-scraped-data)

4. [How can I present the scraped data in different views?](https://github.com/omkarcloud/botasaurus/blob/master/advanced.md#how-can-i-present-the-scraped-data-in-different-views)

5. [When building a large dataset, customers often request data in different formats like overview and review. How can I do that?](https://github.com/omkarcloud/botasaurus/blob/master/advanced.md#when-building-a-large-dataset-customers-often-request-data-in-different-formats-like-overview-and-review-how-can-i-do-that)

6. [What more can I configure when adding a scraper?](https://github.com/omkarcloud/botasaurus/blob/master/advanced.md#what-more-can-i-configure-when-adding-a-scraper)

7. [How to control the maximum number of browsers and requests running at any point of time?](https://github.com/omkarcloud/botasaurus/blob/master/advanced.md#how-to-control-the-maximum-number-of-browsers-and-requests-running-at-any-point-of-time)

8. [How do I change the title, header title, and description of the scraper?](https://github.com/omkarcloud/botasaurus/blob/master/advanced.md#how-do-i-change-the-title-header-title-and-description-of-the-scraper)

9. [How can I use a database like PostgreSQL with UI Scraper?](https://github.com/omkarcloud/botasaurus/blob/master/advanced.md#how-can-i-use-a-database-like-postgresql-with-ui-scraper)

10. [Which PostgreSQL provider should I choose among Supabase, Google Cloud SQL, Heroku, and Amazon RDS?](https://github.com/omkarcloud/botasaurus/blob/master/advanced.md#which-postgresql-provider-should-i-choose-among-supabase-google-cloud-sql-heroku-and-amazon-rds)

11. [How to create a PostgreSQL database on Supabase?](https://github.com/omkarcloud/botasaurus/blob/master/advanced.md#how-to-create-a-postgresql-database-on-supabase)

12. [How to create a PostgreSQL database on Google Cloud?](https://github.com/omkarcloud/botasaurus/blob/master/advanced.md#how-to-create-a-postgresql-database-on-google-cloud)

13. [I am a Youtuber, Should I create YouTube videos about Botasaurus? If so, how can you help me?](https://github.com/omkarcloud/botasaurus/blob/master/advanced.md#i-am-a-youtuber-should-i-create-youtube-videos-about-botasaurus-if-so-how-can-you-help-me)

## Thank You
- To That, who has given me a sufficiently intelligent mind to create Botasaurus and do a lot of good.
- I made Botasaurus because I would be really happy if you could use it to successfully complete your project. So, a Gigantic Thank you for using Botasaurus!
- A heartfelt thank you to [Cheng Zhao](https://zcbenz.com/) from GitHub for creating Electron, which powers Botasaurus Desktop.
- Kudos to the Apify Team for creating the `proxy-chain` library. The implementation of SSL-based Proxy Authentication wouldn't have been possible without their groundbreaking work on `proxy-chain`.
- Shout out to [ultrafunkamsterdam](https://github.com/ultrafunkamsterdam) for creating `nodriver`, which inspired the creation of Botasaurus Driver.
- A big thank you to [daijro](https://github.com/daijro) for creating [hrequest](https://github.com/daijro/hrequests), which inspired the creation of botasaurus-requests.
- Deepest gratitude to [Flori Batusha](https://github.com/riflosnake) and [Ambri](https://github.com/iLeaf30/) for their contributions in creating **Botasaurus Humancursor**, which brings human-like mouse movements to Botasaurus.
- A humongous thank you to Cloudflare, DataDome, Imperva, and all bot recognition systems. Had you not been there, we wouldn't be either 😅.

*Now, what are you waiting for? 🤔 Go and make something mastastic! 🚀*

## Love It? [Star It! ⭐](https://github.com/omkarcloud/botasaurus)

Become one of our amazing stargazers by giving us a star ⭐ on GitHub!

It's just one click, but it means the world to me.

<a href="https://github.com/omkarcloud/botasaurus/stargazers">
    <img src="https://bytecrank.com/nastyox/reporoster/php/stargazersSVG.php?user=omkarcloud&repo=botasaurus" alt="Stargazers for @omkarcloud/botasaurus">
</a>


## Disclaimer for Botasaurus Project

> By using Botasaurus, you agree to comply with all applicable local and international laws related to data scraping, copyright, and privacy. The developers of Botasaurus are not responsible for any misuse of this software. It is the sole responsibility of the user to ensure adherence to all relevant laws regarding data scraping, copyright, and privacy, and to use Botasaurus in an ethical and legal manner.

We take the concerns of the Botasaurus Project very seriously. For any inquiries or issues, please contact Chetan Jain at [chetan@omkar.cloud](mailto:chetan@omkar.cloud). We will take prompt and necessary action in response to your emails.

---

### README (`docs/README.md`)

# Website

This website is built using [Docusaurus 2](https://docusaurus.io/), a modern static website generator.

### Installation

```
$ yarn
```

### Local Development

```
$ yarn start
```

This command starts a local development server and opens up a browser window. Most changes are reflected live without having to restart the server.

### Build

```
$ yarn build
```

This command generates static content into the `build` directory and can be served using any static contents hosting service.

---

### Docs Getting Started (`docs/docs/getting-started.md`)

---
sidebar_position: 2
---

# Getting Started

:::info Prerequisites

To use Botasaurus effectively, it is helpful to have a familiarity with Python and Selenium concepts.

:::

This section will guide you through the steps to create a new Botasaurus project using starter template, which is recommended for most Greenfield automation projects.

---

Clone Starter Template

```bash
git clone https://github.com/omkarcloud/botasaurus-starter my-botasaurus-project
```

Then change into `my-botasaurus-project` directory, install dependencies, open vscode, and start the project:

```bash
cd my-botasaurus-project
python -m pip install -r requirements.txt
code . # Optionally, open the project in VSCode
python main.py
```


Once started it will scrape data and store the results in `output/finished.json`. 


You can edit the `task.py` file based on your Project Needs. 

## Next Steps

If you are new to Botasaurus, we encourage you to learn about Botasaurus [here](google-maps-scraping-tutorial.md).

---

### Docs What Is Botasaurus (`docs/docs/what-is-botasaurus.md`)

---
sidebar_position: 1
title: What is Botasaurus?
description: Botasaurus is a Swiss Army knife 🔪 for web scraping and browser automation 🤖 that helps you create bots fast. ⚡️
---
## In a nutshell

Botasaurus is an all in 1 web scraping framework built for the modern web. We
address the key pain points web scrapers face when scraping the web.

Our aim it to make web scraping extremely easy and save you hours of Development Time.

## Features

Botasaurus comes fully baked, with batteries included. Here is a list of things it can do that no other web scraping framework can:

- **Anti Detect:** Make Anti Detect Requests and Selenium Visits.
- **Debuggability:** When a crash occurs due to an incorrect selector, etc., Botasaurus pauses the browser instead of closing it, facilitating painless on-the-spot debugging.
- **Caching:** Botasaurus allows you to cache web scraping results, ensuring lightning-fast performance on subsequent scrapes.
- **Easy Configuration:** Easily save time with parallelization, profile, and proxy configuration.
- **Time-Saving Selenium Shortcuts:** Botasaurus comes with numerous Selenium shortcuts to make web scraping incredibly easy.

## 🚀 Getting Started with Botasaurus

Welcome to Botasaurus! Let’s dive right in with a straightforward example to understand how it works.

In this tutorial, we will go through the steps to scrape the heading text from [https://www.omkar.cloud/](https://www.omkar.cloud/).



### Step 1: Install Botasaurus

First things first, you need to install Botasaurus. Run the following command in your terminal:

```shell
python -m pip install --upgrade botasaurus 
```

### Step 2: Set Up Your Botasaurus Project

Next, let’s set up the project:

1. Create a directory for your Botasaurus project and navigate into it:

```shell
mkdir my-botasaurus-project
cd my-botasaurus-project
code .  # This will open the project in VSCode if you have it installed
```

### Step 3: Write the Scraping Code

Now, create a Python script named `main.py` in your project directory and insert the following code:

```python
from botasaurus import bt

@browser
def scrape_heading_task(driver: AntiDetectDriver, data):
    # Navigate to the Omkar Cloud website
    driver.get("https://www.omkar.cloud/")
    
    # Retrieve the heading element's text
    heading = driver.text("h1")

    # Save the data as a JSON file in output/all.json
    return {
        "heading": heading
    }
     
if __name__ == "__main__":
    # Initiate the web scraping task
    scrape_heading_task()
```

Let’s dissect this code:

- We define a custom scraping task, `scrape_heading_task`, decorated with `@browser`:
```python
@browser
def scrape_heading_task(driver: AntiDetectDriver, data):
```  

- Botasaurus automatically provides an Anti Detection Selenium driver to our function:
```python
def scrape_heading_task(driver: AntiDetectDriver, data):
```  

- Inside the function, we:
    - Navigate to Omkar Cloud
    - Extract the heading text
    - Prepare the data to be automatically saved as JSON and CSV files by Botasaurus:
```python
    driver.get("https://www.omkar.cloud/")
    heading = driver.text("h1")
    return {"heading": heading}
```  

- Finally, we initiate the scraping task:
```python
if __name__ == "__main__":
    scrape_heading_task()
```  

### Step 4: Run the Scraping Task

Time to run your bot:

```shell
python main.py
```

After executing the script, it will:
- Launch Google Chrome
- Navigate to [omkar.cloud](https://www.omkar.cloud/)
- Extract the heading text
- Save it automatically as `output/finished.json`.



Now, let’s explore another way to scrape the heading using the `request` module. Replace the previous code in `main.py` with the following:

```python
from botasaurus import *

@request
def scrape_heading_task(request: AntiDetectRequests, data):
    # Navigate to the Omkar Cloud website
    soup = request.bs4("https://www.omkar.cloud/")
    
    # Retrieve the heading element's text
    heading = soup.find('h1').get_text()

    # Save the data as a JSON file in output/all.json
    return {
        "heading": heading
    }
     
if __name__ == "__main__":
    # Initiate the web scraping task
    scrape_heading_task()
```

In this code:

- We are using the BeautifulSoup (bs4) module to parse and scrape the heading.
- The `request` object provided is not a standard Python request object but an Anti Detect request object, which also preserves cookies.

### Step 5: Run the Scraping Task (Using Anti Detect Requests)

Finally, run the bot again:

```shell
python main.py
```

This time, you will observe the same result as before, but instead of using Anti Detect Selenium, we are utilizing the Anti Detect request module.

## 💡 Understanding Botasaurus

Let's learn about the features of Botasaurus that assist you in web scraping and automation.

### How to Scrape Multiple Data Points/Links?

To scrape multiple data points or links, define the `data` variable and provide a list of items to be scraped:

```python
@browser(data=["https://www.omkar.cloud/", "https://www.omkar.cloud/blog/", "https://stackoverflow.com/"])
def scrape_heading_task(driver: AntiDetectDriver, data):
  # ...
```

Botasaurus will launch a new browser instance for each item in the list and merge and store the results in `scrape_heading_task.json` at the end of the scraping.



Please note that the `data` parameter can also handle items such as dictionaries.


```python
@browser(data=[{"name": "Mahendra Singh Dhoni", ...}, {"name": "Virender Sehwag", ...}])
def scrape_heading_task(driver: AntiDetectDriver, data: dict):
    # ...
```

### How to Scrape in Parallel?

To scrape data in parallel, set the `parallel` option in the browser decorator:

```python
@browser(parallel=3, data=["https://www.omkar.cloud/", ...])
def scrape_heading_task(driver: AntiDetectDriver, data):
  # ...
```

### How to know how many scrapers to run parallely?

To determine the optimal number of parallel scrapers, pass the `bt.calc_max_parallel_browsers` function, which calculates the maximum number of browsers that can be run in parallel based on the available RAM:

```python
@browser(parallel=bt.calc_max_parallel_browsers, data=["https://www.omkar.cloud/", ...])
def scrape_heading_task(driver: AntiDetectDriver, data):
  # ...
```

*Example: If you have 5.8 GB of free RAM, `bt.calc_max_parallel_browsers` would return 10, indicating you can run up to 10 browsers in parallel.*

### How to Cache the Web Scraping Results?

To cache web scraping results and avoid re-scraping the same data, set `cache=True` in the decorator:

```python
@browser(cache=True, data=["https://www.omkar.cloud/", ...])
def scrape_heading_task(driver: AntiDetectDriver, data):
  # ...  
```


### How Botasaurus helps me in debugging?

Botasaurus enhances the debugging experience by pausing the browser instead of closing it when an error occurs. This allows you to inspect the page and understand what went wrong, which can be especially helpful in debugging and removing the hassle of reproducing edge cases.

Botasaurus also plays a beep sound to alert you when an error occurs.



### How to Block Images?

Blocking images can significantly speed up your web scraping tasks. For example, a page that originally takes 4 seconds to load might only take one second to load after images have been blocked.

To block images, simply set `block_images=True` in the decorator:

```python
@browser(block_images=True)
def scrape_heading_task(driver: AntiDetectDriver, data):
  # ...
```

### How to Configure UserAgent, Proxy, Chrome Profile, Headless, etc.?

To configure various settings such as UserAgent, Proxy, Chrome Profile, and headless mode, you can specify them in the decorator as shown below:

```python
@browser(
  headless=True, 
  profile='my-profile', 
  proxy="http://your_proxy_address:your_proxy_port",
  user_agent=bt.UserAgents.user_agent_106
)
def scrape_heading_task(driver: AntiDetectDriver, data):
  # ...
```

### How Can I Save Data With a Different Name Instead of 'all.json'?

To save the data with a different filename, pass the desired filename along with the data in a tuple as shown below:

```python
@browser(block_images=True, cache=True)
def scrape_article_links(driver: AntiDetectDriver, data):
    # Visit the Omkar Cloud website
    driver.get("https://www.omkar.cloud/blog/")
    
    links = driver.links("h3 a")

    filename = "links"
    return filename, links
```

### I want to Scrape a large number of Links, a new selenium driver is getting created for each new link, this increases the time to scrape data. How can I reuse Drivers?

Utilize the `reuse_driver` option to reuse drivers, reducing the time required for data scraping:

```python
@browser(reuse_driver=True)
def scrape_heading_task(driver: AntiDetectDriver, data):
  # ...
```

### Could you show me a practical example where all these Botasaurus Features Come Together to accomplish a typical web scraping project?

Below is a practical example of how Botasaurus features come together in a typical web scraping project to scrape a list of links from a blog, and then visit each link to retrieve the article's heading and date:

```python
@browser(block_images=True,
         cache=True, 
         parallel=bt.calc_max_parallel_browsers, 
         reuse_driver=True)
def scrape_articles(driver: AntiDetectDriver, link):
    driver.get(link)

    heading = driver.text("h1")
    date = driver.text("time")

    return {
        "heading": heading, 
        "date": date, 
        "link": link, 
    }

@browser(block_images=True, cache=True)
def scrape_article_links(driver: AntiDetectDriver, data):
    # Visit the Omkar Cloud website
    driver.get("https://www.omkar.cloud/blog/")
    
    links = driver.links("h3 a")

    filename = "links"
    return filename, links

if __name__ == "__main__":
    # Launch the web scraping task
    links = scrape_article_links()
    scrape_articles(links)
```


### How to Read/Write JSON and CSV Files?

Botasaurus provides convenient methods for reading and writing data:

```python
# Data to write to the file
data = [
    {"name": "John Doe", "age": 42},
    {"name": "Jane Smith", "age": 27},
    {"name": "Bob Johnson", "age": 35}
]

# Write the data to the file "data.json"
bt.write_json(data, "data.json")

# Read the contents of the file "data.json"
print(bt.read_json("data.json"))

# Write the data to the file "data.csv"
bt.write_csv(data, "data.csv")

# Read the contents of the file "data.csv"
print(bt.read_csv("data.csv"))
```

### How Can I Pause the Scraper to Inspect Things While Developing?

To pause the scraper and wait for user input before proceeding, use `bt.prompt()`:

```python
bt.prompt()
```

### How Does AntiDetectDriver Facilitate Easier Web Scraping?

AntiDetectDriver is a patched Version of Selenium that has been modified to avoid detection by bot protection services such as Cloudflare. 

It also includes a variety of helper functions that make web scraping tasks easier.

You can learn about these methods [here](https://github.com/omkarcloud/botasaurus/blob/master/anti-detect-driver.md).

### What Features Does @request Support, Similar to @browser?

Similar to @browser, @request supports features like 
 - asynchronous execution [Will Learn Later]
 - parallel processing
 - caching
 - user-agent customization
 - proxies, etc.

Below is an example that showcases these features:


```python
@request(parallel=40, cache=True, proxy="http://your_proxy_address:your_proxy_port", data=["https://www.omkar.cloud/", ...])
def scrape_heading_task(request: AntiDetectDriver, link):
  soup = request.bs4(link)
  heading = soup.find('h1').get_text()
  return {"heading": heading}
```

### I have an existing project into which I want to integrate the AntiDetectDriver/AntiDetectRequests.

You can create an instance of `AntiDetectDriver` as follows:
```python
driver = bt.create_driver()
# ... Code for scraping
driver.quit()
```

You can create an instance of `AntiDetectRequests` as follows:
```python
anti_detect_request = bt.create_request()
soup = anti_detect_request.bs4("https://www.omkar.cloud/")
# ... Additional code
```

### How to Run Botasaurus in Docker?

To run Botasaurus in Docker, use the Botasaurus Starter Template, which includes the necessary Dockerfile and Docker Compose configurations:

```
git clone https://github.com/omkarcloud/botasaurus-starter my-botasaurus-project
cd my-botasaurus-project
docker-compose build && docker-compose up
```

### How to Run Botasaurus in Gitpod?

Botasaurus Starter Template comes with the necessary `.gitpod.yml` to easily run it in Gitpod, a browser-based development environment. Set it up in just 5 minutes by following these steps:

1. Open Botasaurus Starter Template, by visiting [this link](https://gitpod.io/#https://github.com/omkarcloud/botasaurus-starter) and sign up using your GitHub account.
   
   
  
2. To speed up scraping, select the Large 8 Core, 16 GB Ram Machine and click the `Continue` button.   

   

3. In the terminal, run the following command to start scraping:
   ```bash
   python main.py
   ```

Please understand:
   - The internet speed in Gitpod is extremely fast at around 180 Mbps.

      

   - When Scraping, You need to interact with Gitpod, such as clicking within the environment, every 30 minutes to prevent the machine from automatically closing.


*Advanced Features*


### How Do I Configure the Output of My Scraping Function in Botasaurus?
To configure the output of your scraping function in Botasaurus, you can customize the behavior in several ways:

1. **Change Output Filename**: Use the `output` parameter in the decorator to specify a custom filename for the output. 
   ```python
   @browser(output="my-output")
   def scrape_heading_task(driver: AntiDetectDriver, data): 
       # Your scraping logic here
   ```

2. **Disable Output**: If you don't want any output to be saved, set `output` to `None`.
   ```python
   @browser(output=None)
   def scrape_heading_task(driver: AntiDetectDriver, data): 
       # Your scraping logic here
   ```

3. **Dynamically Write Output**: To dynamically write output based on data and result, pass a function to the `output` parameter:
   ```python
   def write_output(data, result):
       bt.write_json(result, 'data')
       bt.write_csv(result, 'data')

   @browser(output=write_output)  
   def scrape_heading_task(driver: AntiDetectDriver, data): 
       # Your scraping logic here
   ```

4. **Save Outputs in Multiple Formats**: Use the `output_formats` parameter to save outputs in different formats like CSV and JSON.
   ```python
   @browser(output_formats=[bt.Formats.CSV, bt.Formats.JSON])  
   def scrape_heading_task(driver: AntiDetectDriver, data): 
       # Your scraping logic here
   ```

These options provide flexibility in how you handle the output of your scraping tasks with Botasaurus.

### How to Run Drivers Asynchronously from the Main Process?

To execute drivers asynchronously, enable the `async` option and use `.get()` when you're ready to collect the results:

```python
from time import sleep

@browser(
    run_async=True,  # Specify the Async option here
)
def scrape_heading(driver: AntiDetectDriver, data):
    print("Sleeping for 5 seconds.")
    sleep(5)
    print("Slept for 5 seconds.")
    return {}

if __name__ == "__main__":
    # Launch web scraping tasks asynchronously
    result1 = scrape_heading()  # Launches asynchronously
    result2 = scrape_heading()  # Launches asynchronously

    result1.get()  # Wait for the first result
    result2.get()  # Wait for the second result
```

With this method, function calls run concurrently. The output will indicate that both function calls are executing in parallel.

### How to Asynchronously Add Multiple Items and Get Results?

The `async_queue` feature allows you to perform web scraping tasks in asyncronously in a queue, without waiting for each task to complete before starting the next one. To gather your results, simply use the `.get()` method when all tasks are in the queue.

#### Basic Example:

```python
from time import sleep
from your_scraping_library import browser, AntiDetectDriver  # Replace with your actual scraping library

@browser(async_queue=True)
def scrape_data(driver: AntiDetectDriver, data):
    print("Starting a task.")
    sleep(1)  # Simulate a delay, e.g., waiting for a page to load
    print("Task completed.")
    return data

if __name__ == "__main__":
    # Start scraping tasks without waiting for each to finish
    async_queue = scrape_data()  # Initializes the queue

    # Add tasks to the queue
    async_queue.put([1, 2, 3])
    async_queue.put(4)
    async_queue.put([5, 6])

    # Retrieve results when ready
    results = async_queue.get()  # Expects to receive: [1, 2, 3, 4, 5, 6]
```

#### Practical Application for Web Scraping:

Here's how you could use `async_queue` to scrape webpage titles while scrolling through a list of links:

```python
from your_scraping_library import browser, AntiDetectDriver  # Replace with your actual scraping library

@browser(async_queue=True)
def scrape_title(driver: AntiDetectDriver, link):
    driver.get(link)  # Navigate to the link
    return driver.title  # Scrape the title of the webpage

@browser()
def scrape_all_titles(driver: AntiDetectDriver):
    # ... Your code to visit the initial page ...

    title_queue = scrape_title()  # Initialize the asynchronous queue
    
    while not end_of_page_detected(driver):  # Replace with your end-of-list condition
        title_queue.put(driver.links('a'))  # Add each link to the queue
        driver.scroll(".scrollable-element")
        

    return title_queue.get()  # Get all the scraped titles at once

if __name__ == "__main__":
    all_titles = scrape_all_titles()  # Call the function to start the scraping process
```

**Note:** The `async_queue` will only invoke the scraping function for unique links, avoiding redundant operations and keeping the main function (`scrape_all_titles`) cleaner.

### I want to repeatedly call the scraping function without creating new Selenium drivers each time. How can I achieve this?

Utilize the `keep_drivers_alive` option to maintain active driver sessions. Remember to call `.close()` when you're finished to release resources:

```python
@browser(
    keep_drivers_alive=True, 
    parallel=bt.calc_max_parallel_browsers,  # Typically used with `keep_drivers_alive`
    reuse_driver=True,  # Also commonly paired with `keep_drivers_alive`
)
def scrape_data(driver: AntiDetectDriver, data):
    # ... (Your scraping logic here)

if __name__ == "__main__":
    for i in range(3):
        scrape_data()
    # After completing all scraping tasks, call .close() to close the drivers.
    scrape_data.close()
```


### How do I manage the Cache in Botasaurus?

You can use The Cache Module in Botasaurus to easily manage cached data. Here's a simple example explaining its usage:

```python
from botasaurus import *
from botasaurus.cache import Cache

# Example scraping function
@request
def scrape_data(data):
    # Your scraping logic here
    return {"processed": data}

# Sample data for scraping
input_data = {"key": "value"}

# Adding data to the cache
Cache.put(scrape_data, input_data, scrape_data(input_data))
# Checking if data is in the cache
if Cache.has(scrape_data, input_data):
    # Retrieving data from the cache
    cached_data = Cache.get(scrape_data, input_data)
# Removing specific data from the cache
Cache.remove(scrape_data, input_data)
# Clearing the complete cache for the scrape_data function
Cache.clear(scrape_data)
```

---



### Conclusion

Botasaurus is a powerful, flexible tool for web scraping. 

Its various settings allow you to tailor the scraping process to your specific needs, improving both efficiency and convenience. Whether you're dealing with multiple data points, requiring parallel processing, or need to cache results, Botasaurus provides the features to streamline your scraping tasks.

---

### Docs Botasaurus Desktop Future Roadmap (`docs/docs/botasaurus-desktop/future-roadmap.md`)

---
sidebar_position: 100
---

# Future Roadmap

## Upcoming Features (2025-2026)
- Improve the tasks dashboard with search functionality and multi-select capabilities
- Replace Google Chrome dependency with a portable Chromium bundle similar to Playwright's implementation
- Add browser fingerprinting
- Add XPath support and other Playwright features currently missing from Botasaurus Driver
- Port Botasaurus Driver from Python to JavaScript
- Develop a Node.js version of Botasaurus Web
- Create video and docs showing code signing using DigiCert certificates.
- Create video and docs showing code signing for Mac OS, specific to Botasaurus Desktop.

## How to Contribute?  
1. Open a [GitHub Discussion](https://github.com/omkarcloud/botasaurus) to propose your feature.  
2. We'll review and guide you on implementation.  
3. Once approved, send a pull request!  

## Why Contribute?  
Your work simplifies the lives of thousands of developers.

---

### Docs Botasaurus Desktop Introduction (`docs/docs/botasaurus-desktop/introduction.md`)

---
sidebar_position: 1
---

# Introduction

:::info[header]

**Build Production Ready Desktop Scrapers in 1 Day**

:::


### What is a Desktop Extractor?
A **Desktop Extractor** is a standalone application that runs locally on your computer to extract data from websites, PDFs, Excel files, and other documents. Unlike web-based tools, it operates offline, offering:  
- **Faster performance** (no cloud latency)  
- **Zero cloud costs** (runs on your machine)  

  

### Why Do Desktop Scrapers Outperform Web-Based Scrapers?
A web-based scraper sends your work to someone else's server farm; a Desktop Extractor keeps the entire workflow on your laptop. That one shift leads to the following big wins:

**1. Zero Cloud Bills**: 
   - Runs on the user's machine, eliminating expensive cloud computing fees  
   - Charge less than competitors while keeping 100% margins  

**2. Faster Execution**:  
   - No waiting for cloud VMs to spin up  
   - User's 4-core CPU beats shared cloud servers  

**3. Higher User Engagement**:  
   The app sits right on the user's desktop, encouraging frequent use compared to web tools they must actively visit via browser.  

**4. Cross-Platform Deployment in 1 Day**:  
    With **Botasaurus**, you can create a production-ready desktop scraper for **Windows (.exe), macOS (.dmg), and Linux (.deb/.rpm)** within 1 day. No need to build a website, manage servers, or handle scaling issues. Bota Desktop includes built-in features such as:

    - Task management
    - Data table
    - Data export (Excel, CSV, etc.)
    - Sorting & filtering
    - Caching
    - Auto updates and many more

With zero cloud costs, faster performance, and easier development, Desktop Scrapers simply offer a better cost/performance envelope than the cloud.

### But I Don't Know Electron/React/Databases!
Here's the magic:  

✅ **You write scraping logic** (simple JavaScript/TypeScript)  
✅ **Botasaurus handles the rest**: Task scheduling, data export, caching, sorting and filtering, installers, auto updates, etc.  

**Example**: A stock price scraper needs just 3 steps:  
1. Write a 10-line function to scrape Yahoo Finance  
2. Add input controls (1 file)  
3. Click "npm run package" – your app is ready  

**→ Next:** Let's build your first extractor in minutes.

---

### Docs Botasaurus Desktop Packaging Publishing (`docs/docs/botasaurus-desktop/packaging-publishing.md`)

---
sidebar_position: 21
---

# Packaging & Publishing

Once you've built your desktop extractor, you'll want to create installers that users can easily download and install.

This guide shows you how to package your app for different operating systems and distribute it to users.

## Creating an Installer for Your Current OS

To create an installer for your operating system, run the following command in your project's root directory:

```bash
npm run package
```

After the command completes, you can find the installer for your OS in the `release/build` folder.



### Resolving Windows Build Errors

On Windows, if you encounter the "resource busy or locked" error:



Follow these steps to resolve it:

1. Run Command Prompt as **Administrator**
2. Navigate to your project directory using `cd`
3. Run the package command again:
   ```bash
   npm run package
   ```

This will resolve the error and allow the installer to be created successfully.

## Creating Installers for All Platforms

To reach users on all major operating systems, you'll need to create 4 installers:

- **`.dmg`** for macOS
- **`.exe`** for Windows
- **`.deb`** for Ubuntu/Debian
- **`.rpm`** for Fedora/CentOS/Red Hat

Building each installer requires its native OS, which is challenging because you'd need access to multiple machines running different operating systems.

To solve this problem, we'll use **GitHub Actions** to automate the process. The workflow will:

- Build installers for all platforms in parallel
- Upload them to **AWS S3** for distribution
- Trigger automatically whenever you push code to GitHub

Here's how to set it up:

### Step 1: GitHub Repository Setup

First, create a new repository on GitHub to host your application's source code.



### Step 2: Create an S3 Bucket

Next, set up an AWS S3 bucket to store your installers.

1. Open the **AWS Console** and navigate to **S3**.
   
   

2. Click **Create bucket**.
   
   

3. **Configure the bucket** with these settings:
   1. **Bucket name**: Enter a unique bucket name like `yahoo-finance-extractor`. Conventionally, this name matches your product's name in kebab case. For example, if your product's name is "Yahoo Finance Extractor," your bucket name will be `yahoo-finance-extractor`.
   2. **Object Ownership**: Select "ACLs enabled"
   3. **Block Public Access settings for this bucket**: Uncheck "Block all public access"

   :::warning[Important]
   Ensure **Object Ownership** is set to **"ACLs enabled"**. Otherwise, Electron Builder will fail with "The Bucket does not allow ACLs" error.
   
   
   :::

4. Click **Create bucket**.
   
   

5. If you don't have an **AWS access key and secret key**, create them through the IAM service as described in the [AWS documentation](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_root-user_manage_add-key.html).

### Step 3: Configure GitHub Secrets

In your GitHub repository, navigate to **Settings** → **Secrets and variables** → **Actions** → **Repository secrets** and create the following secrets:

```
AWS_ACCESS_KEY_ID            # Your AWS access key
AWS_SECRET_ACCESS_KEY        # Your AWS secret key
```



### Step 4: Configure Electron Builder

In your project's `package.json` file, add the following to the `build` section:

```json
"build": {
// highlight-start  
  "publish": {
    "provider": "s3",
    "bucket": "your-bucket-name"
  }
// highlight-end  
}
```

Replace `your-bucket-name` with the name of your S3 bucket.

### Step 5: Deploy and Build

1. **Push your code** to GitHub.
2. Go to the repository's **Actions** tab to see the build process in action.
   
   

3. After a **successful build**, the installer files will be available in your S3 bucket. These files are **publicly accessible** in the following format:
   ```
   https://<your-bucket-name>.s3.amazonaws.com/<your-product-name>.<extension>
   ```

**Example URLs:**
- https://yahoo-finance-extractor.s3.amazonaws.com/Yahoo+Finance+Extractor.dmg
- https://yahoo-finance-extractor.s3.amazonaws.com/Yahoo+Finance+Extractor.exe
- https://yahoo-finance-extractor.s3.amazonaws.com/Yahoo+Finance+Extractor.deb
- https://yahoo-finance-extractor.s3.amazonaws.com/Yahoo+Finance+Extractor.rpm

### ✅ You're Ready to Distribute

With this automated pipeline, you can package and publish your app across all major platforms with just a push of a commit.

---

### Docs Botasaurus Desktop Pre Launch Checklist (`docs/docs/botasaurus-desktop/pre-launch-checklist.md`)

---
sidebar_position: 31
---
import ShortCut from '@site/src/components/ShortCut';

# Pre-Launch Checklist

Before making your app public, follow these steps to ensure a perfect launch:  

### 1. Replace Placeholder Text

Search your codebase for the following placeholders and replace them with your app's actual details:

| Placeholder                 | Example Replacement                                                       |
| --------------------------- | ------------------------------------------------------------------------- |
| `todo-my-app-name`          | `yahoo-finance-extractor`                                                 |
| `Todo My App Name`          | `Yahoo Finance Extractor`                                                 |
| `Todo my app description`   | `Extract real-time stock prices from Yahoo Finance quickly and accurately.` |
| `todo-my-organization-name` | `head-first`                                                              |
| `Todo My Organization Name` | `Head First`                                                              |
| `todo-my-email@gmail.com`   | `head-first-python@gmail.com`                                             |

:::tip[Use Find & Replace All]
Use your IDE's global search feature (<ShortCut/>+Shift+F) to replace all placeholder text across your codebase.


:::

### 2. Upload to S3

If you haven't already, [upload installers to S3](./packaging-publishing.md)

### 3. Replace Icon Assets

Replace default icons with your own brand icons. 

You will need to create the following icons:

**Standard PNG Icons:**
1. `assets/icons/16x16.png`
2. `assets/icons/24x24.png`
3. `assets/icons/32x32.png`
4. `assets/icons/48x48.png`
5. `assets/icons/64x64.png`
6. `assets/icons/96x96.png`
7. `assets/icons/128x128.png`
8. `assets/icons/256x256.png`
9. `assets/icons/512x512.png`
10. `assets/icons/1024x1024.png`

**Platform-Specific Icons:**
1. `assets/icon.icns` (macOS)
2. `assets/icon.ico` (Windows)
3. `assets/icon.png` (Linux)

**Web Assets:**
1. `public/icon-256x256.png`



#### How to Create Icons

**Step 1: Design Your Icon**
1. Use [Figma](https://www.figma.com/) or another design tool to create your app icon

**Step 2: Export PNG Icons**

Export your icon into the following PNG sizes and place them in the appropriate folders:

  In `assets/icons/`:
    1. `assets/icons/16x16.png`
    2. `assets/icons/24x24.png`
    3. `assets/icons/32x32.png`
    4. `assets/icons/48x48.png`
    5. `assets/icons/64x64.png`
    6. `assets/icons/96x96.png`
    7. `assets/icons/128x128.png`
    8. `assets/icons/256x256.png`
    9. `assets/icons/512x512.png`
    10. `assets/icons/1024x1024.png`

  In `public/`
    1. `public/icon-256x256.png`

**Step 3: Generate Platform-Specific Icons**

**For macOS (.icns):**
- Go to [CloudConvert PNG to ICNS](https://cloudconvert.com/png-to-icns)
- Upload `assets/icons/1024x1024.png` file ([recommended size for macOS](https://www.electronforge.io/guides/create-and-add-icons#supported-formats))
- Save as `assets/icon.icns`

**For Windows (.ico):**
- Go to [FreeConvert PNG to ICO](https://www.freeconvert.com/png-to-ico)
- Upload `assets/icons/256x256.png` file ([recommended size for Windows](https://www.electronforge.io/guides/create-and-add-icons#supported-formats))
- Save as `assets/icon.ico`

**For Linux:**
- Copy `assets/icons/256x256.png` to `assets/icon.png`.

**Step 4: Backup Icons**

Save the following icons in a secure location.

- `assets/icons/256x256.png` 
- `assets/icons/512x512.png`
- `assets/icons/1024x1024.png`

You will need them for your organization's LinkedIn, Twitter, your website, and other marketing materials.


:::info[Quick Launch Alternative]
If you don't have a brand icon yet and don't want to invest time in creating one, you can use the existing default icons, they're professional-looking. This allows you to launch faster with a polished appearance. 

Once you have users, you can invest time in creating a custom icon.
:::

### 4. Add Customer Support

Giving great customer support is your duty. Give it your best shot.

I recommend offering not only email but also WhatsApp, as most users prefer WhatsApp over email.

We provide WhatsApp support for our apps, and users appreciate it.


Your support options will appear in your app's **Support** menu:




#### Adding WhatsApp Support

To add WhatsApp support, update `src/scraper/backend/server.ts` with the following code:

```ts title="src/scraper/backend/server.ts"
import { Server } from 'botasaurus-server/server';
import { config } from '../../main/config';

// highlight-start
// Add WhatsApp support details
Server.addWhatsAppSupport({
  number: '1234567890', // Your 10-digit phone number (without the country code)
  countryCallingCode: '81', // Your country calling code (e.g., 81 for Japan, 1 for US)
  message: `Hi, I need help with using the ${config.productName} Tool`, // Default message for WhatsApp
});
// highlight-end
```

#### Adding Email Support

Similarly to add email support, update `src/scraper/backend/server.ts` with the following code:  

```ts title="src/scraper/backend/server.ts"
import { Server } from 'botasaurus-server/server';
import { config } from '../../main/config';

// highlight-start
// Add Email support details
Server.addEmailSupport({
  email: 'happy.to.help@my-app.com', // Replace with your support email
  subject: `Help with ${config.productName} Tool`, // Default email subject
  body: `Hi, I need help with using the ${config.productName} Tool`, // Default email body
});
// highlight-end
```

#### Add Support Section to Product Page

It's a best practice to add a dedicated support section to your product page. You can adapt the following template, which has proven effective for us.

To customize the template:
- Replace `811234567890` with your full WhatsApp number (country code + phone number).
- Replace `happy.to.help@my-app.com` with your support email.
- Replace `Yahoo%20Finance%20Extractor` with your product name, with spaces replaced by `%20`.


```markdown
### ❓ Need More Help or Have Additional Questions?

For further help, feel free to contact us via:

- **WhatsApp:** If you prefer WhatsApp, simply message us [here](https://api.whatsapp.com/send?phone=811234567890&text=I%20need%20help%20with%20using%20the%20Yahoo%20Finance%20Extractor%20Tool.). Please include as much detail as possible so we can help you effectively.

  [](https://api.whatsapp.com/send?phone=811234567890&text=I%20need%20help%20with%20using%20the%20Yahoo%20Finance%20Extractor%20Tool.)

- **Email:** Prefer email? Send your questions to [happy.to.help@my-app.com](mailto:happy.to.help@my-app.com?subject=Help%20with%20Yahoo%20Finance%20Extractor%20Tool&body=I%20need%20help%20with%20using%20the%20Yahoo%20Finance%20Extractor%20Tool.). Also, please include as much detail as possible so we can help you effectively.

  [](mailto:happy.to.help@my-app.com?subject=Help%20with%20Yahoo%20Finance%20Extractor%20Tool&body=I%20need%20help%20with%20using%20the%20Yahoo%20Finance%20Extractor%20Tool.)

We look forward to helping you and will reply within 1 working day.
```

**Preview:**




> 💡 Once you start getting lots of support queries, consider hiring a support agent and writing an SOP (Standard Operating Procedure) to help them respond effectively.

**🎉 That's it! You're all set! Now go!** 🚀  
- Create something amazing that helps people.  
- Market it well, so it reaches people.  
- Achieve financial freedom and live life on your own terms!  

Wishing you all the success in life!

---

### Docs Botasaurus Desktop Quick Start (`docs/docs/botasaurus-desktop/quick-start.md`)

---
sidebar_position: 2
---
import LinuxOnly from '@site/src/components/LinuxOnly';

# Quick Start

In the next 10 minutes you'll have two fully-working desktop tools:

**1. Stock-Price Scraper** – Enter a ticker (e.g. AAPL) and get the live price from Yahoo Finance.  


**2. Amazon PDF-Invoice Extractor** – Drag-and-drop an invoice PDF and extract GST No, Document No, Date and Place of Supply, etc.  


Both extractors share the same desktop shell, so once you build one, you already know how to build the other—and the next one you have in mind.

## 👨‍💻 Project Setup — Clone, Install, Run

**1️⃣ Clone the Magic 🧙‍♀️:**
```bash
git clone https://github.com/omkarcloud/botasaurus-desktop-starter my-botasaurus-app
cd my-botasaurus-app
```

**2️⃣ Install Packages 📦:**
```bash
npm install
```

**3️⃣ Launch the Desktop App 🚀:**
```bash
npm run dev
```

A window opens with a heading scraper ready to test. Let's explore what you just launched.
<LinuxOnly>
--- 
**Note:** On **Ubuntu 24.04** and later, you will face the following error:



To resolve it, run:

```bash
sudo chown root:root ./node_modules/electron/dist/chrome-sandbox
sudo chmod 4755 ./node_modules/electron/dist/chrome-sandbox
```

Then, relaunch the app using:

```bash
npm run dev
```
</LinuxOnly>

## 👀 Touring the Starter App

The starter app demonstrates all the core features through a simple heading scraper:


**1. Home (Input)**  
   - Enter any URL and click Run – the demo scraper grabs the page’s first `<h1>`.  
   

**2. Results**  
   - See the data returned by the task.
   - Export to CSV / XLSX / JSON.  
   

**3. Tasks**  
   - This Page shows all tasks you've started.
   - Abort or delete any task.  
   

**4. About**  
   - Renders your project’s README.md, servers as app docs.  
   


## 📂 Project Structure at a Glance

Before we build our extractors, here's a simplified view of the key files you'll work with:

```
my-botasaurus-app/
├─ assets/                        ← Icons for the installers
├─ inputs/                        
│   └─ scrapeHeadingTask.js       ← Renders UI controls (one file per extractor)
├─ src/
│  └─ scraper/
│      ├─ backend/
│      │   └─ server.ts           ← Central registry – add your scrapers here
│      └─ src/
│          └─ scraper.ts          ← Extraction logic (e.g., grab <h1> with Playwright)
├─ release/                       ← Installers (.exe, .dmg, .deb) after "npm run package"
└─ package.json                   ← Scripts + build settings
```

**Mental model:**  
- `inputs/` = UI controls  
- `src/scraper/src/` = extraction logic  
- `src/scraper/backend/server.ts` = add scraper here

Now that you understand the structure, let's build your first Stock-Price Scraper!

---

### Docs Botasaurus Desktop Understanding App Features (`docs/docs/botasaurus-desktop/understanding-app-features.md`)

---
sidebar_position: 26
---

# Understanding App Features
This guide explains the Support Menu and the Auto Update system.

## Support Menu

The Support Menu, located in the top-right corner of the app, provides options for debugging, resetting the app, and getting support.

It contains the following options:


Here’s a breakdown of each option:

### Show User Data

Opens the folder containing all application data, which is useful for debugging any issues your users encounter in production.

It includes:

-   **`task_results/`**: Stores cached data and scraped results.
-   **`botasaurus_storage.json`**: Persistent JSON storage for user settings. You can also use this storage in your code as follows:
    ```javascript
    import { getBotasaurusStorage } from 'botasaurus/botasaurus-storage';

    const storage = getBotasaurusStorage();
    
    storage.setItem('userId', 10);
    const userId = storage.getItem('userId');
    ```
-   **`db.nedb`**: The application's database.

> **Note:** These files appear in your root folder when running the app in development (`npm run dev`).

### Factory Reset App and Clear All Data

Resets the app to its original state by deleting:  
- `task_results/`  
- `botasaurus_storage.json`  
- `db.nedb`

**When to use:**
- To free up storage space
- To start with a clean installation

When clicked, the app will request confirmation before deleting any data.


### Update to Latest Version

Manually checks for updates and installs the latest version if the app is not already up-to-date.

### Contact Support via WhatsApp & Contact Support via Email

These options provide your users with direct ways to contact you for help.

-   **Contact Support via WhatsApp**: Opens WhatsApp Web in the user's browser with a pre-filled message.
-   **Contact Support via Email**: Launches the user's default email app with a pre-filled message.



You can learn how to configure these with your contact details in the [Pre-Launch Checklist](./pre-launch-checklist.md) section.

## Auto Updates

Botasaurus Desktop automatically updates itself when new versions are released. This ensures your users always have the latest features and bug fixes without manual installation.

To enable this feature, you need to host your application installers on an **AWS S3 bucket**, as detailed in the [Packaging & Publishing guide](./packaging-publishing.md).

Once configured, pushing new code to your GitHub repository will trigger a new build. The application will then automatically update itself, on next app launch.

Auto updates are supported on the following platforms:
-   **macOS**
-   **Windows**: Only supported when the app is signed with a code signing certificate.
-   **Linux**

---

### Docs Botasaurus Desktop Botasaurus Desktop Api Adding Api (`docs/docs/botasaurus-desktop/botasaurus-desktop-api/adding-api.md`)

---
sidebar_position: 1
---

# Adding an API

Botasaurus allows you to expose your scrapers as APIs, enabling you to:

- Consume scrapers from web apps, cron jobs, or other workflows
- Run your scrapers on cloud VMs for scalable, always-on automation
- Monetize your scrapers by listing them on marketplaces like RapidAPI

## How to Add an API to Your App

To enable API call `ApiConfig.enableApi()` in `src/scraper/backend/api-config.ts` file:

```ts title="src/scraper/backend/server.ts"
import ApiConfig from "botasaurus-server/api-config";

// Enable the API
ApiConfig.enableApi();
```

**What happens after enabling the API:**

1. Your app displays an `API Integration` tab containing auto-generated documentation and code examples specific to your scrapers.

   

2. When your app launches, an API server starts at `http://localhost:8000` (default). This URL also also displays your API documentation.
   

:::info Key Features

- The API is built on [Fastify](https://www.fastify.io/) for low latency and production-ready performance
- It automatically enforces any [rate limits](../enhancing-scrapers/rate-limiting.md) you've defined for your scrapers
- It validates incoming requests against the controls defined in your scraper's `inputs` folder, rejecting invalid requests with a `400` error to protect your scrapers from bad data


:::

We recommend reading the generated API documentation to understand and effectively configure your API.

## Configuration Options

The `ApiConfig` class provides 5 methods to configure your API's behavior:

### `setApiPort(port: number)`

Changes the port where the API server listens. The default is port 8000.

**Example:**

```ts
// Set API to run on port 8080
ApiConfig.setApiPort(8080);
```

**When to use:** When port 8000 is already in use or you want to run on a specific port.

### `disableApiAutostart()`

Prevents the API server from starting automatically when the app launches. Users must manually start the API from the "API" tab.

**Example:**

```ts
// API will not run until manually started from the desktop GUI
ApiConfig.disableApiAutostart();
```

**Note:** Botasaurus remembers toggle settings. If a user turns the API **on**, it will start automatically on the next app launch.


**When to use:** When you want users to explicitly enable the API before using it. Note that disabling API autostart doesn't reduce startup time, as the API server is lightweight and launches after the desktop app finishes loading.

### `setApiBasePath(path: string)`

Adds a prefix to all API routes, useful for mounting the API under a specific subpath like `/v1` or `/api`.

**Example:**

```ts
// Prefixes all routes with /v1
ApiConfig.setApiBasePath("/v1");
```

**When to use:**
- Hosting multiple APIs on one server
- Implementing API versioning (`/v1`, `/v2`)

### `addScraperAlias(scraper: Scraper, endpoint: string)`

Creates a direct GET endpoint for running scrapers without the overhead of creating, scheduling, and running tasks.

**Example:**

```ts
import { hotelsSearchScraper } from "../src/scrapers";

// Creates direct GET endpoint at /hotels/search
ApiConfig.addScraperAlias(hotelsSearchScraper, "/hotels/search");
```

This allows you to:
- Make direct **GET** requests to the `/hotels/search` endpoint
- Execute the scraper immediately, bypassing task creation, scheduling, and running overhead
- Validate input data before execution, returning a `400` error for invalid requests
- Cache results based on the provided parameters
- Respect the [rate limits](../enhancing-scrapers/rate-limiting.md) defined for your scraper

**When to use:**
- Simple scraper calls without task management overhead
- Custom endpoint naming
- Reselling APIs via platforms like **RapidAPI**, where task abstraction is unnecessary

### `addCustomRoutes((server: FastifyInstance) => void)`

Extends the API with custom endpoints and middleware using Fastify's routing system. This method receives a Fastify instance, allowing you to define any route you need.

**Example: Adding a custom health check endpoint**

```ts
ApiConfig.addCustomRoutes((server) => {
  server.get('/health', (request, reply) => {
    return reply.send({ status: 'OK'});
  });
});
```

**Example: Adding validation middleware**

```ts
ApiConfig.addCustomRoutes((server) => {
  server.addHook('onRequest', (request, reply, done) => {
    // Check for secret
    const secret = request.headers['x-secret'] as string;

    if (secret === '49cb1de3-419b-4647-bf06-22c9e1110313') {
      // Valid secret, proceed
      return done(); 
    } else {
      return reply.status(401).send({
        message: 'Unauthorized: Invalid secret.',
      });
    }
  });
});
```

**When to use:**
- Adding authentication middleware
- Creating custom endpoints
- Implementing webhook receivers

## Complex Example

Here's a comprehensive configuration that demonstrates multiple options working together:

```ts title="src/scraper/backend/server.ts"
import ApiConfig from 'botasaurus-server/api-config';
import { hotelsSearchScraper } from "../src/scrapers";

// Enable API functionality
ApiConfig.enableApi();

// Production configuration
ApiConfig.setApiPort(3000);
ApiConfig.setApiBasePath("/v1");

// Add scraper aliases for direct access
ApiConfig.addScraperAlias(hotelsSearchScraper, '/hotels/search');

// Add custom routes
ApiConfig.addCustomRoutes((server) => {
  // Health check for monitoring
  server.get('/health', (request, reply) => {
    return reply.send({ status: 'OK'});
  });

  // Authentication middleware  
  server.addHook('onRequest', (request, reply, done) => {
    // Check for secret
    const secret = request.headers['x-secret'] as string;

    if (secret === '49cb1de3-419b-4647-bf06-22c9e1110313') {
      // Valid secret, proceed
      return done(); 
    } else {
      return reply.status(401).send({
        message: 'Unauthorized: Invalid secret.',
      });
    }
  });
});
```

With this configuration:
- The API runs on port 3000
- All routes are prefixed with `/v1`
- Hotel search is available at GET `/v1/hotels/search`
- Health check is available at GET `/v1/health`
- All requests require authentication via the `x-secret` header

---

### Docs Botasaurus Desktop Botasaurus Desktop Api Adding Domain And Ssl (`docs/docs/botasaurus-desktop/botasaurus-desktop-api/adding-domain-and-ssl.md`)

---
sidebar_position: 50
---

# Adding a Domain Name and SSL

This guide shows you how to:

1. Point a domain name to the VM that hosts your Botasaurus Desktop API.
2. Secure that domain with a free SSL certificate from [Let's Encrypt](https://letsencrypt.org/).

## Why Use a Custom Domain?

A custom domain with SSL provides two key benefits:

- **Professional appearance**: `api.yourcompany.com` looks more professional than an IP address
- **Trust**: HTTPS encryption builds trust and prevents browser security warnings

### When to Use a Custom Domain?

**You DON'T need a domain if:**
- You're using the API for personal scripts or automation
- You're listing your API on marketplaces like RapidAPI, which provides its own domain and SSL

**You DO need a domain if:**
- You're providing API access to customers
- You want a professional, branded endpoint


## How to Add a Domain Name and SSL?

The process involves two main steps:

- Pointing your domain to the VM's IP address
- Installing a free SSL certificate



### Step 1: Get a Domain and Point it to Your VM


1. If you don't already have a domain, purchase one from a registrar.

   I recommend these registrars:

   - **Cloudflare Domains** - Most Cheapest option (offers at-cost pricing, charging only the ICANN registry fees with no markup)
   - **Porkbun** - Second cheapest option


   Avoid GoDaddy and Namecheap:
   - Like HP/Canon with printers and ink, they follow the [Razor and Blades](https://www.investopedia.com/terms/r/razor-razorblademodel.asp) pricing model. Their domain registration costs are low, but their renewal prices are significantly higher than Porkbun and Cloudflare Domains. It is unwise to buy from them.
   - Also, do not even search for a domain on GoDaddy. They may purchase the domain themselves and then offer it to you at a much higher price.

   I use Porkbun because:
   - Unlike Cloudflare Domains, which is primarily a CDN provider, Porkbun specializes in domain registration. I want a specialist to manage my domains.
   
   - It sends multiple renewal reminders (45 days and 30 days before expiry), so I don't forget to renew my domains.

   

   

2. Next, log in to your domain registrar and create DNS **A records** pointing to your VM's IP address.

    - To host on a root domain and `www` subdomain (e.g., `yahoo-finance.com` and `www.yahoo-finance.com`):

    | Type | Host | Value |
    |------|------|-------|
    | A    | w&#8203;ww.yahoo-finance.com | YOUR_VM_EXTERNAL_IP |
    | A    | yahoo-finance.com | YOUR_VM_EXTERNAL_IP |

    - To host on a subdomain like `yahoo-finance.john-doe-it-solutions.com` (allows hosting multiple scrapers on a single domain):

    | Type | Host | Value |
    |------|------|-------|
    | A    | yahoo-finance.john-doe-it-solutions.com | YOUR_VM_EXTERNAL_IP |

    Here's an example DNS configuration in Porkbun:
    

After a few minutes, test your domain by accessing it via HTTP:


### Step 2: Install SSL Certificate

Next, we'll use **Certbot** from [Let's Encrypt](https://letsencrypt.org/) to automatically issue and install a free SSL certificate.

1. SSH into your VM instance.


2. Install Certbot:
    ```bash
    sudo apt install certbot python3-certbot-apache -y
    ```

3. Run Certbot to obtain and install the certificate:
    ```bash
    sudo certbot --apache --agree-tos --register-unsafely-without-email
    ```
   - `--apache` configures Apache to use the certificate
   - `--agree-tos` automatically accepts the Let's Encrypt terms of service
   - `--register-unsafely-without-email` skips email registration

4. When prompted for domain names, enter them separated by commas:
   - For root + www: `example.com,www.example.com`
   - For subdomain only: `subdomain.example.com`

   

After successful installation, you'll see a confirmation message:


That's it! You can now access your API securely over HTTPS.



:::info[Automatic Renewal]
Certbot automatically renews your SSL certificate 30 days before expiration. No manual renewal is required.
:::

### I am facing "Some challenges have failed." error while running certbot command. How do I fix it?

If you encounter the "Some challenges have failed" error:



This error is almost always caused by one of two issues:
1. The DNS **A record** doesn't point to the correct VM IP address (common with ephemeral IP addresses)
2. DNS changes haven't propagated across the internet yet

To resolve this error:

1. Find your VM's IP address by running:
   ```bash
   curl icanhazip.com
   ```
2. Log in to your domain registrar and update the DNS A record to point to this IP address

3. Wait 5 minutes for DNS propagation

4. Retry the Certbot command:
    ```bash
    sudo certbot --apache --agree-tos --register-unsafely-without-email
    ```

This should resolve the error and successfully install your SSL certificate.

---

### Docs Botasaurus Desktop Botasaurus Desktop Api Deploying On Aws (`docs/docs/botasaurus-desktop/botasaurus-desktop-api/deploying-on-aws.md`)

---
sidebar_position: 31
---

# Deploying on AWS EC2

This guide will help you deploy your Botasaurus Desktop API on an AWS EC2 instance. The process is straightforward and involves creating an Elastic IP, setting up an EC2 instance, and installing your desktop application.

### Step-by-Step Deployment Guide

### 1. Reserve a Static IP Address

First, we'll reserve an Elastic IP address. An Elastic IP ensures your EC2 instance is always reachable at the same IP address.

1. Create an [AWS Account](https://signin.aws.amazon.com/signup?request_type=register) if you don't already have one.
   

2. Go to the [Elastic IP addresses](https://console.aws.amazon.com/ec2/home?#Addresses:) page.
   

3. Click the "Allocate Elastic IP address" button.
   

4. Keep the default settings and click "Allocate"
   

This creates an Elastic IP address that you'll use to access your app. You will assign this IP to your EC2 instance in the next step.

   
### 2. Create an AWS EC2 Instance

1. Go to the [EC2 Dashboard](https://console.aws.amazon.com/ec2/v2/home#Instances:) and click "Launch Instance"


2. Configure your instance with these recommended settings. You may change the Machine Type and Boot Disk Size based on your scraping needs:

| Section | Setting | Value | Notes |
|---------|---------|-------|-------|
| Name and tags | Name | yahoo-finance | Use your app name as the instance name |
| Application and OS Images (AMI) | Amazon Machine Image (AMI) | Ubuntu Server 24.04 LTS | A `.deb` installer requires a Debian-based OS like Ubuntu. |
| Instance type | Instance type | t3.medium | 2 vCPU, 4 GB memory. Change based on needs |
| Key pair (login) | Key pair name | Create new key pair > RSA (.pem) | Required for SSH access |
| Network settings | Allow SSH traffic from | ✓ | Required for terminal access |
| Network settings | Allow HTTP traffic from the internet | ✓ | Required for accessing API via HTTP |
| Network settings | Allow HTTPS traffic from the internet | ✓ | Required for accessing API via HTTPS |
| Configure storage | Storage | 20 GiB - Magnetic | **Magnetic** is the cheapest disk type. |

    

3. Optionally, if you're scraping data for your own needs, enable Spot Instances as they are 70-90% cheaper than On-Demand instances. Enable them with these settings:

| Section | Setting | Value | Notes |
|---------|---------|-------|-------|
| Advanced details | Purchasing option | Spot Instances | Enables Spot Instances |
| Customize Spot instance options | Request type | Persistent | - |
| Customize Spot instance options | Interruption behavior | Stop | - |

:::warning
Don't use Spot VMs for customer-facing APIs or mission-critical applications, as they can be stopped by AWS at any time if the resources are needed elsewhere.
:::

4. Click **Launch instance**.

5. Now, associate the Elastic IP:
   - Go to [Elastic IPs](https://console.aws.amazon.com/ec2/v2/home#Addresses:)
   - Select your previously created Elastic IP
   - Click **Actions** > **Associate Elastic IP address**
   - Select your newly created instance and click **Associate**

   

6. Connect to your instance via SSH:
   - Go to [EC2 Instances](https://console.aws.amazon.com/ec2/v2/home#Instances:)
   - Select your instance
   - Click **Connect** > **EC2 Instance Connect**
   - Click the **Connect** button

   


### 3. Installing Your Desktop App

Now that your EC2 instance is ready, let's install your Botasaurus Desktop API.

1. First, install the necessary packages on your instance by running the command below. This script installs Botasaurus CLI and the Apache web server to manage requests to your app.

   ```bash
   curl -sL https://raw.githubusercontent.com/omkarcloud/botasaurus/master/vm-scripts/install-bota-desktop.sh | bash
   ```
   

2. Upload your app to S3 following the [packaging and publishing guide](../packaging-publishing.md). A Debian installer will be saved in the S3 bucket, which we will use in the next step. It looks like this: 

   ```bash 
   https://your-bucket.s3.amazonaws.com/Your-App-amd64.deb
   ```

3. Next, install the desktop application on the EC2 instance.

   - If you have already uploaded your app to S3, replace the link in the command below with your own Debian installer URL.
   - If you only want to test the installation, keep the sample URL as is.

   ```bash
   python3 -m bota install-desktop-app --debian-installer-url https://yahoo-finance-extractor.s3.us-east-1.amazonaws.com/Yahoo+Finance+Extractor-amd64.deb
   ```

   

   :::info[Configuration Options]
    The `install-desktop-app` command supports these options:
    - `--port`: Sets the port for the app (e.g., `8001`). An alternative to `ApiConfig.setApiPort`.
    - `--api-base-path`: Adds a prefix to all API routes (e.g., `/yahoo-finance`). An alternative to `ApiConfig.setApiBasePath`.
    - `--skip-apache-request-routing`: Disables automatic Apache configuration for the API. Use it if you want to manually configure other load balancers like Nginx.
   :::

When the installation completes, you'll see a **link** to your API documentation. Visit it see the api.



:::tip[Hosting Multiple Desktop APIs on One EC2 Instance]
To host additional applications on the same instance, you must use the `--port` and `--api-base-path` flags to prevent conflicts:
- `--port` with different ports (e.g., `--port 8001`)
- `--api-base-path` with unique paths (e.g., `--api-base-path /amazon-invoices`)

For example, run this command to add an Amazon Invoice extractor alongside the Yahoo Finance extractor:
```sh
python3 -m bota install-desktop-app \
  --debian-installer-url https://amazon-invoice-extractor.s3.us-east-1.amazonaws.com/Amazon+Invoice+Extractor-amd64.deb \
  // highlight-next-line
  --port 8001 \
  // highlight-next-line
  --api-base-path /amazon-invoices
```
The Amazon Invoice API will now be available at: `http://<your-elastic-ip>/amazon-invoices`.
:::

### How to Uninstall the Desktop App?
:::warning[Backup Your Data]
Before uninstalling, download any important data. This action is irreversible and will result in permanent data loss.
:::

To uninstall the application from the EC2 instance, use one of the following methods:

**Method 1: Using Debian Installer URL**

Replace `https://yahoo-finance-extractor.s3.us-east-1.amazonaws.com/Yahoo+Finance+Extractor-amd64.deb` with your app's Debian installer URL:

```bash 
python3 -m bota uninstall-desktop-app --debian-installer-url https://yahoo-finance-extractor.s3.us-east-1.amazonaws.com/Yahoo+Finance+Extractor-amd64.deb
```

**Method 2: Using Package Name**

1. Find your package **name** in `package.json`:
   

2. Uninstall using the package **name**:
   ```bash
   python3 -m bota uninstall-desktop-app --package-name yahoo-finance-extractor
   ```
### How to Delete the EC2 Instance and Avoid Incurring Further Charges?

:::warning[Backup Your Data]
Before deleting the instance, download any important data to avoid permanent loss.
:::

To prevent ongoing costs, you must delete both the EC2 instance and release the Elastic IP address as follows:

1. **Cancel the Spot Request** *(Spot VMs only)*
   If you have enabled Spot Instances, cancel the Spot request before deleting the instance. Otherwise, they will keep respawning, again and again. To cancel the Spot request:

   - Go to [Spot Requests](https://console.aws.amazon.com/ec2/v2/home#SpotInstances:)
   - Select your Spot request
   - Click **Actions** > **Cancel request**
   - Click **Confirm** to cancel

   

2. **Terminate the EC2 Instance**

   - Go to [EC2 Instances](https://console.aws.amazon.com/ec2/v2/home#Instances:)
   - Select your instance
   - Click **Instance state** > **Terminate (delete) instance**
   - Click **Terminate (delete)** to confirm

   

3. **Release the Elastic IP Address**

   - Go to [Elastic IPs](https://console.aws.amazon.com/ec2/v2/home#Addresses:)
   - Select your Elastic IP
   - Click **Actions** > **Release Elastic IP address**
   - Click **Release** to confirm

   

That's it! You have successfully deleted the EC2 instance and released the Elastic IP. You will not incur any further charges.

---

### Docs Botasaurus Desktop Botasaurus Desktop Api Deploying On Google Cloud (`docs/docs/botasaurus-desktop/botasaurus-desktop-api/deploying-on-google-cloud.md`)

---
sidebar_position: 30
---
# Deploying on Google Cloud VM

Deploying your API on Cloud VM allows you to:
- Run your scrapers 24/7
- Access your scrapers from anywhere via HTTP Endpoints
- Scale to multiple instances based on your scraping needs

This guide walks you through deploying your **Botasaurus Desktop API** on a Google Cloud VM instance.

### Prerequisites

This guide uses **Google Cloud VM**, but the steps apply to any cloud provider (AWS, Azure, DigitalOcean, etc.) as long as the VM runs a Debian or Ubuntu-based OS.

### Step-by-Step Deployment Guide

### 1. Reserve a Static IP Address

First, we'll reserve a static IP address. A static IP ensures your VM is always reachable at the same IP address.

1. Create a Google Cloud Account if you don't already have one.
   

2. Visit the [Google Cloud Console](https://console.cloud.google.com/welcome?cloudshell=true) and click the Cloud Shell button. A terminal will open up.
   

3. Install the Botasaurus CLI and create a static IP address by running the following commands:

   ```bash
   python -m pip install bota --upgrade 
   python -m bota create-ip # Create a static IP address for your VM
   ```

   The CLI will prompt you for a name for your VM. Use a relevant name like your app name, such as `yahoo-finance`.

   > Name: yahoo-finance

   Next, it will ask for a region. Press **Enter** to accept the default (`us-central1`), which offers the lowest-cost VMs. 

   > Region: Default

   

   This creates a static IP address that you'll use to access your app. You will assign this IP to your VM in the next step.
   
### 2. Create a Google Cloud VM

1. Go to [VM Instances](https://console.cloud.google.com/compute/instances) and create a VM with these specifications. You may change the Machine Type and Boot Disk Size based on your scraping needs:

   | Section | Setting | Value | Notes |
   |---------|---------|-------|-------|
   | Machine configuration | Name | yahoo-finance | Use your app name as the instance name |
   | Machine configuration | Region | us-central1 | **Must match** the region you selected for the static IP. |
   | Machine configuration | Zone | Any | Leave it any zone within selected region |
   | Machine configuration | Series | E2 | Cost-effective choice |
   | Machine configuration | Machine Type | e2-medium | 2 vCPU, 4 GB memory. Change based on your needs |
   | OS and storage | Boot Disk Type | Standard persistent disk | Cheapest disk, suitable for scraping |
   | OS and storage | Boot Disk Size | 20 GB | Change based on your storage needs |
   | Data protection | Back up your data | No backups | No backups needed for scraping, as the data can be easily regenerated. |
   | Networking | Firewall/Allow HTTP traffic | ✓ | Required for accessing API via HTTP |
   | Networking | Firewall/Allow HTTPS traffic | ✓ | Required for accessing API via HTTPS |
   | Networking | Firewall/Allow Load Balancer Health Checks | ✓ | - |
   | Networking | Network Interfaces/External IPv4 address | yahoo-finance-ip | Select the static IP you created in the previous step. |
   | Observability | Ops Agent | Uncheck Install Ops Agent for Monitoring and Logging | Helps reduce logging costs |

    

2. Optionally, if you're scraping data for your own needs, enable Spot VMs as they are 60-91% cheaper than standard VMs. Enable them by going to **Advanced** > **Provisioning model** and selecting **Spot**.

   

   :::warning
   Don't use Spot VMs for customer-facing APIs or mission-critical applications, as they can be stopped by Google at any time if the resources are needed elsewhere.
   :::

3. Click the **Create** button. The VM will be provisioned in a few minutes.
   


4. Once the VM is running, connect to it by clicking the **SSH** button on the [VM Instances page](https://console.cloud.google.com/compute/instances).
   

### 3. Installing Your Desktop App

Now that your VM is ready, let's install your Botasaurus Desktop API.

1. First, install the necessary packages on your VM by running the command below. This script installs Botasaurus CLI and the Apache web server to manage requests to your app.

   ```bash
   curl -sL https://raw.githubusercontent.com/omkarcloud/botasaurus/master/vm-scripts/install-bota-desktop.sh | bash
   ```
   

2. Upload your app to S3 following the [packaging and publishing guide](../packaging-publishing.md). A Debian installer will be saved in the S3 bucket, which we will use in the next step. It looks like this: 

   ```bash 
   https://your-bucket.s3.amazonaws.com/Your-App-amd64.deb
   ```


3. Next, install the desktop application on the VM.

   - If you have already uploaded your app to S3, replace the link in the command below with your own Debian installer URL.
   - If you only want to test the installation, keep the sample URL as is.

   ```bash
   python3 -m bota install-desktop-app --debian-installer-url https://yahoo-finance-extractor.s3.us-east-1.amazonaws.com/Yahoo+Finance+Extractor-amd64.deb
   ```

   

   :::info[Configuration Options]
    The `install-desktop-app` command supports these options:
    - `--port`: Sets the port for the app (e.g., `8001`). An alternative to `ApiConfig.setApiPort`.
    - `--api-base-path`: Adds a prefix to all API routes (e.g., `/yahoo-finance`). An alternative to `ApiConfig.setApiBasePath`.
    - `--skip-apache-request-routing`: Disables automatic Apache configuration for the API. Use it if you want to manually configure other load balancers like Nginx.
   :::

When the installation completes, you'll see a **link** to your API documentation. Visit it see the api.

 highlight


:::tip[Hosting Multiple Desktop APIs on One VM]
To host additional applications on the same VM, you must use the `--port` and `--api-base-path` flags to prevent conflicts:
- `--port` with different ports (e.g., `--port 8001`)
- `--api-base-path` with unique paths (e.g., `--api-base-path /amazon-invoices`)

For example, run this command to add an Amazon Invoice extractor alongside the Yahoo Finance extractor:
```sh
python3 -m bota install-desktop-app \
  --debian-installer-url https://amazon-invoice-extractor.s3.us-east-1.amazonaws.com/Amazon+Invoice+Extractor-amd64.deb \
  // highlight-next-line
  --port 8001 \
  // highlight-next-line
  --api-base-path /amazon-invoices
```
The Amazon Invoice API will now be available at: `http://<your-vm-ip>/amazon-invoices`.
:::

### How to Uninstall the Desktop App?

:::warning[Backup Your Data]
Before uninstalling, download any important data. This action is irreversible and will result in permanent data loss.
:::

To uninstall the application from the VM, use one of the following methods:


**Method 1: Using Debian Installer URL**

Replace `https://yahoo-finance-extractor.s3.us-east-1.amazonaws.com/Yahoo+Finance+Extractor-amd64.deb` with your app's Debian installer URL:

```bash 
python3 -m bota uninstall-desktop-app --debian-installer-url https://yahoo-finance-extractor.s3.us-east-1.amazonaws.com/Yahoo+Finance+Extractor-amd64.deb
```


**Method 2: Using Package Name**

1. Find your package **name** in `package.json`:
   

2. Uninstall using the package **name**:
   ```bash
   python3 -m bota uninstall-desktop-app --package-name yahoo-finance-extractor
   ```

### How to Delete the VM and Avoid Incurring Further Charges?

:::warning[Backup Your Data]
Before deleting VM, download any important data to avoid permanent loss.
:::

To prevent ongoing costs, you must delete both the VM instance and its static IP address as follows:

1. Delete the static IP by running the following command:

   ```bash
   python -m bota delete-ip
   ```

   The CLI will prompt you for the name of the VM you created in the first step. Enter the name and press Enter.

   

2. Go to [VM Instances](https://console.cloud.google.com/compute/instances) and delete your VM.

   

That's it! You have successfully deleted the VM and the IP. You will not incur any further charges.


### How to Reduce Compute Costs?


Running VMs is expensive, but there are a few decent ways to reduce costs:

- Choose the right machine type. Avoid overprovisioning CPU and memory. Start with a smaller instance like `e2-medium` for browser-based scrapers and `e2-small` for requests/task-based scrapers.

   Then, check the CPU and memory usage on the VM Instance Observability Page.

   

   If you find your VM's CPU and memory are underutilized, you can downgrade to a smaller machine type. If they're overutilized, you can upgrade to a larger machine type.

- Use Spot VMs for personal/non-critical workloads. Spot VMs are up to 60-91% cheaper than standard VMs, but they can be stopped by Google at any time if the resources are needed elsewhere.

   The Botasaurus task management system is fault-tolerant, making it well-suited for Spot VMs.

   **Spot VMs are perfect for:**
   - Personal scraping
   - Batch jobs where you can check the results later

   **Avoid Spot VMs when:**
   - Running customer-facing APIs that require high availability
   - Running mission-critical applications that cannot tolerate downtime

   To use a Spot VM, when creating VM, go to **Advanced** > **Provisioning model** and select **Spot**.

   

- If you have a stable, long-term workload, you can achieve significant savings by committing to 1-year (37% discount) or 3-year terms (55-70% discount).


      :::tip CUDs: Pros and Cons
      **Pro:** Significant discounts for predictable workloads.
      **Con:** You are locked into paying for the committed resources for the entire term, even if you stop using the VM. We recommend running a VM with on-demand pricing for 2 months to confirm your needs before committing.
      :::


### Should I Choose On-Demand, 1-Year CUD, or 3-Year CUD?

Excellent question. Choosing the right GCP pricing model can save you thousands of dollars in compute costs, so read this carefully.

Let's simplify the decision by calculating the "break-even" point—the moment when a longer commitment becomes the cheaper option.

#### Step 1: Find Your Monthly Costs

First, you need the numbers. Go to the [GCP Pricing Calculator](https://cloud.google.com/products/calculator?dl=CjhDaVEyWVRGa1lqVmlOeTFtTXpsakxUUXpOVGd0T1dNM1ppMDBaRGcyWm1NelkyTXlaVE1RQVE9PRAIGiQyOEQ1QjQwNi05RjNGLTRBNkQtODczMS0wNTYzQzQ4QjgxRUU) and find the monthly costs for your machine type across all three pricing models.



For this example, we'll use an **e2-medium** instance with a **20GB Standard Persistent Disk**:
-   **On-Demand Price:** $24.46/month
-   **1-Year CUD Price:** $15.41/month
-   **3-Year CUD Price:** $11.01/month

#### Step 2: Calculate the Break-Even Points

With these prices, we can answer two key questions by setting up simple equations.

#### When does a 1-Year CUD beat On-Demand pricing?

When you choose a CUD, you're committed to paying for the full term, regardless of usage. The break-even point is the number of months it would take for your on-demand spending to equal the total fixed cost of the 1-year commitment.

-   **Total cost of a 1-Year Commitment:** 12 months × $15.41/month = **$184.92**

Let's find the break-even point in months (X):
-   `X months × On-Demand Price = Total 1-Year Cost`
-   `X × $24.46 = $184.92`
-   `X = $184.92 / $24.46`
-   `X = 7.5 months`

**Conclusion:** If you expect to need the virtual machine for **8 months or more**, the 1-Year CUD is the wise choice.

#### When does a 3-Year CUD beat consecutive 1-Year CUDs?

This is the most important decision for long-term projects. We want to find when locking in for 3 years becomes more profitable than renewing 1-year CUD commitments.

-   **Total cost of a 3-Year Commitment:** 36 months × $11.01/month = **$396.36**

Let's find how many months (X) of the 1-year plan this total 3-year cost would cover:
-   `X months × 1-Year CUD Price = Total 3-Year Cost`
-   `X × $15.41 = $396.36`
-   `X = $396.36 / $15.41`
-   `X = 25.7 months`

**Conclusion:** The financial break-even point is approximately **2 years and 2 months**. If you're confident your resource will be needed for 26 months or longer, the 3-Year CUD is the more financially sound choice from day one.

#### Step 3: Ask the Critical Question

Now, look at your project and your business, then ask one simple question:

**"How long will my app need to be online?"**

#### Step 4: Choose Your Scenario

Based on your answer, choose the most appropriate pricing model:

-   **Choose On-Demand Pricing if:** You're just starting out, have no customers, or expect the server to run for **less than 8 months**. This offers maximum flexibility with no commitment.

-   **Choose a 1-Year CUD if:** You expect your app to run for **at least 8 months but are unsure about a 2+year timeframe**. This provides significant savings over on-demand pricing without a multi-year lock-in. Recommended scenarios include:
      - A new B2B service with 10-50 paying customers

-   **Choose a 3-Year CUD if:** You're confident your application will run for **at least 2 years and 2 months**. This is the best choice for stable, long-term workloads. Recommended scenarios include:
      - A business website that you're sure will be online for the foreseeable future (e.g., for omkar.cloud, I chose a 3-year CUD)
      - A popular scraping API with 100+ paying customers

#### Summary Table

| Plan | Best For | Risk | Savings |
|------|----------|------|---------|
| **On-Demand** | < 8 months usage, uncertain projects | None | 0% |
| **1-Year CUD** | 8-26 months usage, moderate certainty | Low | ~37% |
| **3-Year CUD** | 26+ months usage, high certainty | Medium | ~55% |

#### Break-Even Table

Here's a table showing the approximate break-even months for different E2 machine types (with a 20GB persistent disk). Note that the break even months are the same across all E2 types:

| Machine Type | Break-even: On-Demand vs 1-Year CUD | Break-even: 1-Year CUD vs 3-Year CUD |
|:---|:---:|:---:|
| **e2-small** | ~8 months (7.5) | ~2 years, 2 months (25.7) |
| **e2-medium** | ~8 months (7.5) | ~2 years, 2 months (25.7) |
| **e2-standard-2** | ~8 months (7.5) | ~2 years, 2 months (25.7) |

#### How to Calculate for Any Machine Type

For any other machine type, simply plug the prices from the GCP Pricing Calculator into these formulas:

**Break-even months for On-Demand vs 1-Year CUD:**
```
= (1-Year CUD Monthly Cost × 12) / On-Demand Monthly Cost
```

**Break-even months for 1-Year CUD vs 3-Year CUD:**
```
= (3-Year CUD Monthly Cost × 36) / 1-Year CUD Monthly Cost
```

Maths is awesome, isn't it? Use this knowledge to choose the right CUDs, and save significant money on compute costs.

### How to Apply Committed Use Discounts?

#### Step 1: Find your CUD Requirements

Before purchasing CUDs, carefully analyze your resource needs:

1. Monitor your VM's Observability Page for at least 1 month to identify underutilized resources. If your VM consistently uses less than 50% of its RAM/CPU, consider downgrading to a smaller machine type first.
   
   

2. Visit the [CUD Analysis Page](https://console.cloud.google.com/billing/commitments/analysis) to see your current memory and vCPU usage. 

   

3. Next, find your instance's vCPU count using the [Pricing Calculator](https://cloud.google.com/products/calculator?hl=en&dl=CjhDaVF5WldVMFlUZzNPUzFqTm1Ka0xUUTBOelF0WVRBM1pDMW1NREE1TURNNVltTTBaRE1RQVE9PRAIGiRFNUZGNEQ3OC1EMTdELTRGMDktOUMzOC00NUIyNDhDOTU1MTU).

   

   For quick reference, here are the vCPU counts for common E2 machine types:

   | Machine Type | vCPUs |
   |--------------|-------|
   | e2-micro     | 0.25  |
   | e2-small     | 0.5   |
   | e2-medium    | 1     |
   | e2-standard-2| 2     |
   | e2-standard-4| 4     |

   :::warning[Use Pricing Calculator vCPU Values]
   The vCPU count displayed when creating an instance is incorrect:

   

   **Always** use the vCPU count from the [Pricing Calculator](https://cloud.google.com/products/calculator?hl=en&dl=CjhDaVF5WldVMFlUZzNPUzFqTm1Ka0xUUTBOelF0WVRBM1pDMW1NREE1TURNNVltTTBaRE1RQVE9PRAIGiRFNUZGNEQ3OC1EMTdELTRGMDktOUMzOC00NUIyNDhDOTU1MTU) when purchasing CUDs:

   
   :::
4. Next, find your instance's memory size in GB using the same [Pricing Calculator](https://cloud.google.com/products/calculator?hl=en&dl=CjhDaVF5WldVMFlUZzNPUzFqTm1Ka0xUUTBOelF0WVRBM1pDMW1NREE1TURNNVltTTBaRE1RQVE9PRAIGiRFNUZGNEQ3OC1EMTdELTRGMDktOUMzOC00NUIyNDhDOTU1MTU).

5. Calculate your total resource needs by multiplying the per-instance resources by the number of instances you plan to run.

   For example, if you need 2 `e2-medium` instances (1 vCPU, 4GB memory each):
   - **Total Memory Needed:** 2 × 4 GB = 8 GB
   - **Total vCPUs Needed:** 2 × 1 vCPU = 2 vCPUs

6. Finally, choose your commitment duration:
   - **1 Year** – Ideal for workloads running at least 8 months
   - **3 Years** – Ideal for long-term projects running over 2 years and 2 months

7. CUDs are non-refundable commitments. So, take **at least 2 days** to consider:
   - Have you right-sized your instances?
   - Is the commitment duration right for your project?
   - Will your resource needs change during the commitment period?

   There's a good chance you might reconsider the **commitment duration** or **instance types** after **2 days**.

   **Don't skip** this step — it's a crucial pause that could save you from a costly mistake.

#### Step 2: Create Your CUDs

1. Go to the [Compute Commitments page](https://console.cloud.google.com/compute/commitments).

   

2. Click **Purchase Commitments**.

   

3. Create a memory CUD with these settings:

   | Field | Value |
   |-------|-------|
   | Name | your-app-memory-cud (use a descriptive name) |
   | Region | us-central1 (must match your VM's region) |
   | Commitment Type | General-purpose E2 (must match your VM series) |
   | Duration | 1 year or 3 years |
   | Memory | Total GB calculated in Step 1 |
   | Reservations | Don't attach reservations |

   

4. Create a vCPU CUD with these settings:

   | Field | Value |
   |-------|-------|
   | Name | your-app-vcpu-cud (use a descriptive name) |
   | Region | us-central1 (must match your VM's region) |
   | Commitment Type | General-purpose E2 (must match your VM series) |
   | Duration | 1 year or 3 years |
   | vCPUs | Total vCPUs calculated in Step 1 |
   | Reservations | Don't attach reservations |

   

After creation, both CUDs will appear on the [Commitments page](https://console.cloud.google.com/compute/commitments):




You can see your CUD utilization and savings on the [CUD Analysis Page](https://console.cloud.google.com/billing/commitments/analysis).

### When creating a CUD, I get "The COMMITMENTS-per-project-region quota maximum in region us-central1 has been exceeded." error. How do I fix it?

If you encounter an error like **Creating commitment "commitment-20250727-141128" failed. Error: The COMMITMENTS-per-project-region quota maximum in region us-central1 has been exceeded.**:


This occurs because your maximum number of CUDs allowed in a region is 0. To resolve this:

1. Visit the [Quotas page](https://console.cloud.google.com/iam-admin/quotas) and in search enter **commitments**.


2. Scroll down to your region (e.g., 6us-central1`) and click the **Edit Quotas** button.


3. Enter a higher quota, such as **6**. 

4. For **Request Description**, enter:

```
We have reached the commitment quota limit in us-central1. An increase is required to purchase a new commitment for a production workload.
```



5. Click **Next**, then **Submit Request**.



6. Within a minute, you'll receive a confirmation email. Wait ~15 minutes more, and then you will be able to create the CUD.


### Can I apply Committed-Use Discounts (CUDs) to Spot VMs?
No—**Committed Use Discounts (CUDs)** apply only to standard VMs. Spot VMs are charged separately.

If you have an active CUD and also run Spot VMs, you will essentially be billed twice:
1. For the Spot VM usage at its current (low) price
2. For the resources you committed to via the CUD, whether you use them or not

Therefore, do not mix Spot VMs with CUDs for the same resource types.

---

### Docs Botasaurus Desktop Code Signing Budget Friendly Strategy (`docs/docs/botasaurus-desktop/code-signing/budget-friendly-strategy.md`)

---
sidebar_position: 25
---

# Budget Friendly Strategy for Early Stage Developers

## Code signing is quite expensive. Is there a cost-effective alternative for an early-stage developer?

Yes, it is expensive ($99–$1,068 per year), and for early-stage developers, it is financially unwise to invest in a certificate when your app has no users yet.

If you are just starting out, here's how you can launch with **0 code-signing costs**:

1. **Temporarily adjust your app download page:**  
   - **For macOS:** Do not provide a download link. Installation is impossible without changing settings that most macOS users do not know how to change.  
   - **For Windows:** Provide the download link and include a message such as:  
     > "Temporarily, you will see a blue popup from Windows Defender stating that the app is from an **unknown publisher**. This is because we are not yet a verified publisher. Rest assured, the app is 100% safe. To proceed, click 'More Info' and then 'Run Anyway.' We aim to become a verified publisher soon to resolve this."
2. Focus on promoting your app and building a user base.  
3. Monetize once you have enough users.  
4. After a few months, when your app generates revenue, invest in code signing certificates for macOS and Windows.

## Why is this financially wise?

This approach lets you start with 0 code-signing costs and invest only after your app generates revenue.

Once again, **do not invest in code signing** if you are just starting out and are **short on cash**. Windows users can still install an unsigned app, giving you a large enough user base to monetize your product.

---

### Docs Botasaurus Desktop Code Signing Macos Code Signing Guide (`docs/docs/botasaurus-desktop/code-signing/macos-code-signing-guide.md`)

---
sidebar_position: 2
---

# macOS Code Signing Guide

To sign a macOS installer (.dmg), you need:

- A Mac  
- A subscription to the [Apple Developer Program](https://developer.apple.com/support/compare-memberships/), which costs $99 per year  

Once you have these, watch this detailed video to learn how to sign apps for macOS:

[](https://www.youtube.com/watch?v=hYBLfjT57hU)

The video covers:  
- Enrolling in the Apple Developer Program  
- Creating a certificate  
- Configuring your app for code signing  
- Building and signing your app  
- Using GitHub Actions and S3 for upload  

:::info[Note]  
While the video demonstrates general Electron macOS signing, you will need to adjust the steps specifically for Botasaurus Desktop, which is easy. A dedicated tutorial is coming soon.  
:::

:::tip[Tip]  
If you are a nonprofit organization or educational institution, you may qualify for Apple's Developer Program **fee waiver**. The waiver removes the $99 annual charge, saving you about $1,000 over a decade.

Check your eligibility and the [application process here](https://developer.apple.com/help/account/membership/fee-waivers/).  
:::

---

### Docs Botasaurus Desktop Code Signing What Is Code Signing (`docs/docs/botasaurus-desktop/code-signing/what-is-code-signing.md`)

---  
sidebar_position: 1  
title: What Is Code Signing  
---

# What Is Code Signing & Why Does It Matter?

Apple and Microsoft both want to ensure that the apps running on their operating systems are safe and free of viruses. To achieve this, they use a process called code signing, which:

- Verifies your identity  
- Scans your code for viruses  
- After confirming that the code is safe, signs it with a certificate that tells the OS the app is safe to run  

## What happens if I skip code signing?

### macOS  
Users **will not** be able to install your app. macOS displays a message stating that the app **cannot be opened because the developer cannot be verified**.



### Windows  

Users can install your app, but they will see a Windows Defender SmartScreen pop-up saying **Windows protected your PC**. To proceed, they must:  
1. Click **More info** on the warning dialog.  
2. Click **Run anyway** to continue the installation.  



This deters less-technical users and reduces installations.

### Linux  
Code signing is not required. Users can install and run your app without any security warnings.

:::info[Note on Local Development]

When you **build** with `npm run package` and **install it on the same machine**, security warnings do not appear.

However, these warnings **will definitely appear** when users download and run your unsigned installer on their own computers.  
:::

## Can one certificate sign many apps?

Yes. A single certificate can sign an unlimited number of installers, which is great if you have multiple projects.

---

### Docs Botasaurus Desktop Code Signing Windows Code Signing Guide (`docs/docs/botasaurus-desktop/code-signing/windows-code-signing-guide.md`)

---
sidebar_position: 3
---

# Windows Code Signing Guide

This guide explains the two main options for obtaining a Windows code signing certificate.

## ✅ If You're Based in the US or Canada

**If your business is in the US or Canada and has 3+ years of tax history**, the best option is **Azure Trusted Signing** — it's the cheapest at just **\$10/month**.

For a detailed walkthrough, follow this excellent guide: **[Code Signing on Windows with Azure Trusted Signing by melatonin.dev](https://melatonin.dev/blog/code-signing-on-windows-with-azure-trusted-signing/)**

## 🌍 For Everyone Else
You will need an **Extended Validation (EV) Code Signing Certificate** (not an Organization Validation (OV)) from a Certification Authority (CA) like DigiCert or Sectigo.

### Important Requirements

Code signing requirements are stringent and designed for companies — **not solo developers**. While requirements vary by country, here are the standard ones:

- **Registered Business:** You must be a legally registered company. (Sole proprietorships usually don't qualify, but check with your chosen CA to confirm.)
- **Physical Address:** A physical business address (not a P.O. box or virtual office). This address should be listed in public records like Google Maps or DUNS.
- **Professional Web Presence:** A business website and a company email address (e.g., `your.name@yourcompany.com`) are required. Personal email addresses (such as Gmail or Outlook) will not be accepted for registration.

:::warning[Contact Sales Before You Purchase]
Contact the CA's validation or sales team to ask exactly which documents are required for your country and entity type. 

Don't purchase first and then ask about the requirements, as I did. In my case, it led to a failed validation — a waste of time I hope you can avoid.
:::

### Use Cloud Signing

When purchasing your certificate, select the "Cloud Signing" or "Cloud HSM" option. While more expensive than hardware tokens, it enables automated signing in GitHub Actions and avoids hardware token hassles. For example, DigiCert's cloud-based EV certificate costs $1,068 per year.

### Recommended Certification Provider

If you're wondering which CA to choose, we recommend DigiCert despite their higher cost ($1,068 for EV Cloud Signing) for these reasons:

- DigiCert is the most popular CA. Following the wisdom of the crowd makes it a safe choice.
    - Electron uses DigiCert [to sign their applications](https://www.electronjs.org/docs/latest/tutorial/code-signing#signing-windows-builds).
    
    - The popular npm package **[@electron/windows-sign](https://github.com/electron/windows-sign)**, used for signing Windows executables, features DigiCert extensively in its documentation.
    

- They offer a 30-day refund policy, which is a key benefit if you're unsure whether you'll pass the validation process. I attempted to obtain an EV Certificate through DigiCert but failed validation because I wasn't registered as a company, which is required for Indian businesses. They refunded my payment within a few days.

:::info[Note for Solo Developers] 
We recommend DigiCert because it's the wisdom-of-the-crowd option. We were not paid by DigiCert. 

CA Code Signing is definitely too expensive, and I wouldn't recommend the CA Code Signing process if you're just starting out. 
:::

## ✅ Final Recommendation

- If you're an American or Canadian business with **3+ years of tax history** (or have a family member or friend with such a business), choose **Azure Trusted Signing** (\$120/year).

- Otherwise, if you qualify and can afford it, go with **DigiCert EV Cloud Code Signing** ($1,068/year). To proceed:

    1. First, contact DigiCert's Validation Team to confirm the specific requirements for your country and entity type [here](https://www.digicert.com/support). 

    
    
    Send them this message via chat, replacing YOUR_COUNTRY and YOUR_ENTITY_TYPE with your details:
    ```
    Kindly share the documents required specifically for YOUR_COUNTRY, registered as YOUR_ENTITY_TYPE, for obtaining an EV Code Signing Certificate. 
    ```
    
    2. Once you've confirmed you can meet the requirements, purchase the certificate from this link, which has the Cloud Signing EV option pre-selected:
    
    https://www.digicert.com/signing/code-signing-certificates#code_signing_ev_key_locker

---

### Docs Botasaurus Desktop Enhancing Scrapers Custom Views (`docs/docs/botasaurus-desktop/enhancing-scrapers/custom-views.md`)

---
sidebar_position: 9
---

# Custom Views

**Custom Views** display your data in multiple, easy-to-read tables in the `Results` page.

For example, if your scraper returns product data with nested reviews, you can create  
- An **"Overview"** table with key details like name and price
- A **"Reviews"** table that expands each review into its own row

This not only enhances readability but also allows users to export the data in structured format (CSV/JSON) by selecting the view.



## Creating a Simple View
The code below adds a view called "Overview" that shows `name` and `price` fields.

```ts title="src/scraper/backend/server.ts"
import { Server } from 'botasaurus-server/server';
import { View, Field } from 'botasaurus-server/ui';
import { yourScraper } from '../src/yourScraper';   // your scraper function

/* 1. Define the view */
// highlight-start
const overviewView = new View('Overview', [
  new Field('name'),
  new Field('price'),
]);
// highlight-end
/* 2. Register scraper + view */
Server.addScraper(yourScraper, { views: [overviewView] });
```

How it works  
- `new View('Overview', …)` creates a tab called **Overview**  
- `new Field('name')` / `new Field('price')` creates columns  
- `Server.addScraper(…, { views: [overviewView]})` adds the `Overview` view to the scraper



## Types of Fields

You can use **4** types of fields to build views.

### `Field`

Displays a single value from your data record. It's the most common field.

**Options**
- `outputKey` renames the column header, such as changing `price` to `product_price`
- `map` transforms the value using a function. Perfect for calculating an average, formatting a date, or adding a currency symbol
- `showIf` conditionally omits the column based on input form data

**Example**
```ts
new Field("reviews_per_rating", { 
    outputKey: "average_rating", 
    map: (value, record) => {
        // ... Logic to calculate average rating from the 'value' object
    }, 
    // Only show this column if the user checked "scrape_prices"
    showIf: (inputData) => inputData.scrape_prices === true 
})
```

### `CustomField`
Creates a new column from multiple values in the record.

*Supports `showIf` option only.*

**Example**
```ts
// Combines two fields into one
new CustomField("full_name", (record) => `${record.first_name} ${record.last_name}`
)
```

### `ExpandDictField`
Expands a single dictionary object into multiple columns.

*Supports `showIf` option only.*

**Example**
```ts
// Convert a rating like {1: 0, 2: 0, 3: 0, 4: 100, 5: 900} into 5 new columns
new ExpandDictField("reviews_per_rating",[
        new Field("1", { outputKey: "rating_1" }),
        new Field("2", { outputKey: "rating_2" }),
        new Field("3", { outputKey: "rating_3" }),
        new Field("4", { outputKey: "rating_4" }),
        new Field("5", { outputKey: "rating_5" }),
    ],
)
```

### `ExpandListField`
Expands an array of objects into multiple rows. Each item in the array becomes a new row. 

For example, if a product has 3 reviews, it creates 3 rows in the table.

*Supports `showIf` option only.*

**Example**
```ts
new ExpandListField('reviews', [
  new Field('rating'),
  new Field('content'),
])
```

## Real World Example 

Let's say a scraper returns the following product data.

```ts
const taskScraper = task({
  name: "taskScraper",
  run: () => {
    // highlight-start
    return [
      {
        id: 1,
        name: "T-Shirt",
        price: 16, // in US Dollar
        reviews: 1000,
        reviews_per_rating: {
          1: 0,
          2: 0,
          3: 0,
          4: 100,
          5: 900,
        },
        featured_reviews: [
          {
            id: 1,
            rating: 5,
            content: "Awesome t-shirt!",
          },
          {
            id: 2,
            rating: 5,
            content: "Amazing t-shirt!",
          },
        ],
      },
      {
        id: 2,
        name: "Laptop",
        price: 700,
        reviews: 500,
        reviews_per_rating: {
          1: 0,
          2: 0,
          3: 0,
          4: 100,
          5: 400,
        },
        featured_reviews: [
          {
            id: 1,
            rating: 5,
            content: "Best laptop ever!",
          },
          {
            id: 2,
            rating: 5,
            content: "Great laptop!",
          },
        ],
      },
    ];
    // highlight-end
  },
})
```

The following code creates **3** views from this data  
-  **"Overview"** shows a summary of each product with expanded rating data
    
-  **"Featured Reviews"** displays a detailed list where each review is its own row
    

```
/* Detailed source-code truncated for AI context efficiency. */
```
**Result**


:::info
An **`All Fields`** view is always included by default, showing all raw data returned by the scraper without any transformations.
:::

---

### Docs Botasaurus Desktop Enhancing Scrapers Enable Caching (`docs/docs/botasaurus-desktop/enhancing-scrapers/enable-caching.md`)

---
sidebar_position: 5
---
# Enabling Caching

Caching saves the results of a scraper run. When you run the scraper again with the same inputs, Botasaurus serves the cached result instead of running the scraper again, saving both time and compute resources.

## How to Enable Caching

To enable caching for all scrapers, add `Server.enableCache()` to `src/scraper/backend/server.ts` file:

```ts title="src/scraper/backend/server.ts"
import { Server } from "botasaurus-server/server"

// highlight-next-line
Server.enableCache();
```

## How Caching Works

Botasaurus uses input data to determine whether to use a cached result:

1. When you run a scraper, Botasaurus creates a unique "cache key" from the `data` object
2. If the same `data` values are used in a future run, the cached result is returned instantly
3. If the `data` values change, the scraper runs again

**Example:** If you scrape a product page with `data: { productId: "123" }`, the result is cached. Running the scraper again with the same `productId` returns the cached result without re-running the scraper.

## Excluding Fields from the Cache Key

Some inputs should not affect caching decisions. For example:
- API keys
- Session cookies  
- Proxy URLs

These values may change between runs without affecting the scraper's results. Including them in the cache key would cause unnecessary re-runs.

To exclude these fields from caching, mark an input control with `isMetadata: true`:

```js
// In your inputs/myScraper.js
.text("api_key", {
   label: "API Key",
   // highlight-next-line
   isMetadata: true,
})
```

When `isMetadata` is set to `true`, the value is not included in the cache key.

This approach ensures that only essential, data-defining inputs determine caching behavior, making your cache more effective.

## Bypassing Cache Control

When caching is enabled, users see a **"Use cache"** checkbox in the input form:



This checkbox:
   - Defaults to `true` (using cached results if available)
   - Allows users to bypass the cache for a specific run by unchecking it

---

### Docs Botasaurus Desktop Enhancing Scrapers Filtering And Sorting (`docs/docs/botasaurus-desktop/enhancing-scrapers/filtering-and-sorting.md`)

---
sidebar_position: 10
---

# Filtering and Sorting

**Filtering and Sorting** help users quickly find and organize scraped data in the `Results` page.

For example, if your scraper returns product data, users can:  
- **Filter** to show only **available** items under $50  
- **Sort** by price (low to high) or reviews (high to low)

These controls appear above the data table in the **Results** page.



## Creating a Simple Filter

To add filters, pass a `filters` array when adding your scraper.

```ts title="src/scraper/backend/server.ts"
import { Server } from "botasaurus-server/server"
import { filters } from "botasaurus-server/ui"
import { scrapeProductData } from "../src/scrapeProductData"

// highlight-start
Server.addScraper(scrapeProductData, { 
    filters: [new filters.SearchTextInput("product_name")] 
})
// highlight-end
```

**How it works**  
- `filters.SearchTextInput('product_name')` adds a search box for `product_name` field
- `Server.addScraper(..., { filters: [...] })` adds the filter to your scraper

With this, users can now filter products by typing in the search box.



## Types of Filters

There are 12 types of filters. All filters accept an optional `label` parameter to customize the display text. If `label` is not provided, the field name appears in Title Case.


### `MinNumberInput`
A number input that shows items where the target field is ≥ the entered number

**Example**
```ts
new filters.MinNumberInput("price")
```

### `MaxNumberInput`
A number input that shows items where the target field is ≤ the entered number

**Example**
```ts
new filters.MaxNumberInput("price")
```

### `IsTrueCheckbox`
A checkbox that shows items where the target field is strictly `true`.

**Example**
```ts
new filters.IsTrueCheckbox("is_available")
```

### `IsFalseCheckbox`
A checkbox that shows items where the target field is strictly `false`.

**Example**
```ts
new filters.IsFalseCheckbox("is_available")
```

### `IsTruthyCheckbox`
A checkbox that shows items where the target field is truthy (e.g., `true`, `1`, `"hello"`).

**Example**
```ts
new filters.IsTruthyCheckbox("description")
```

### `IsFalsyCheckbox` 
A checkbox that shows items where the target field is falsy (e.g., `false`, `0`, `""`, `null`).

**Example**
```ts
new filters.IsFalsyCheckbox("description")
```

### `IsNullCheckbox`
A checkbox that shows items where the target field is strictly `null`.

**Example**
```ts
new filters.IsNullCheckbox("description")
```

### `IsNotNullCheckbox`
A checkbox that shows items where the target field is not strictly `null`.

**Example**
```ts
new filters.IsNotNullCheckbox("description")
```

### `SingleSelectDropdown`
A dropdown for picking a **single** option. It requires an `options` array.

The filter matches if:
- The target field is a `String` and exactly matches the selected option.
- The target field is an `Array` and *includes* the selected option.

**Additional Options**
- `caseInsensitive`: Perform case-insensitive matching. When `true`, "Electronics" will match "electronics", "ELECTRONICS", etc. Defaults to `false`.

**Example**
```ts
new filters.SingleSelectDropdown("category", {
    options: [
        { value: "apparel",     label: "Apparel" },
        { value: "electronics", label: "Electronics" }
    ]
})
```

### `MultiSelectDropdown`
A dropdown for picking **multiple** options. It requires an `options` array.

The filter matches if:
- The target field is a `String` and matches **any** of the selected options.
- The target field is an `Array` and includes **any** of the selected options.

**Additional Options**
- `caseInsensitive`: Perform case-insensitive matching. When `true`, "Cotton" will match "cotton", "COTTON", etc. Defaults to `false`.

**Example**
```ts
new filters.MultiSelectDropdown("tags", {
    options: [
        { value: "cotton",   label: "Cotton" },
        { value: "casual",   label: "Casual" },
        { value: "computer", label: "Computer" },
        { value: "portable", label: "Portable" }
    ]
})
```

### `BoolSelectDropdown`
A dropdown with "Yes" and "No" options for filtering truthy/falsy values.

- "Yes" keeps rows where the target field is *truthy* (e.g., `true`, `1`, `"hello"`).  
- "No" keeps rows where the target field is *falsy* (e.g., `false`, `0`, `""`, `null`).

It is an alternative to `checkbox`, often clearer to users than a `checkbox`.

**Additional Options**
- `prioritizeNo`: If `true`, "No" will be the first option in the dropdown instead of "Yes". Defaults to `false`.
- `invertFilter`: If `true`, "Yes" filters for falsy values and "No" filters for truthy ones. Defaults to `false`.

**Example**
```ts
new filters.BoolSelectDropdown("is_available")
```

### `SearchTextInput`
A text input that shows items where the target field's value contains the search term (case-insensitive).

**Example**
```ts
new filters.SearchTextInput("name")
```

## Creating a Simple Sort

Sorts are added exactly the same way as filters.  

```ts title="src/scraper/backend/server.ts"
import { Server } from "botasaurus-server/server"
import { sorts } from "botasaurus-server/ui"
import { scrapeProductData } from "../src/scrapeProductData"

Server.addScraper(scrapeProductData, {
    // highlight-start
    sorts: [
        // A simple sort by price
        new sorts.NumericAscendingSort("price"),
        
        // A custom, multi-level sort preset
        new sorts.Sort({    
            label: "Top Rated",
            isDefault: true,
            sorts: [
                new sorts.NumericDescendingSort("reviews"),
                new sorts.TrueFirstSort("is_available"),
            ]
        })
    ]
    // highlight-end
})
```

**How it works:**
- `sorts.NumericAscendingSort('price')` creates a sort option for the `price` field.
- `sorts.Sort()` combines multiple sorting criteria into a single sort option, it first sorts by reviews, then by products where `is_available` is True.
- In `sorts.Sort`, `isDefault` is set to `true`. With this, the "Top Rated" sort will be automatically applied to data when the Results page first loads.
- `Server.addScraper(..., { sorts: [...] })` adds the sorts to your scraper

With this, users can now sort products



## Types of Sorts

There are 13 types of sorts. 

Each sort accepts two optional properties:  
- `label`: text shown in the dropdown. If not provided, it is automatically generated based on field name.
- `isDefault`: apply this sort automatically when the Results page loads. Defaults to `false`. 

We recommend setting one of the sorts as default (`isDefault: true`) to ensure data is meaningfully organized when users first view the Results page.


### `AlphabeticAscendingSort`
Sorts items by the target field in ascending alphabetical order (A-Z).

**Example**
```ts
new sorts.AlphabeticAscendingSort("name")
```

### `AlphabeticDescendingSort`
Sorts items by the target field in descending alphabetical order (Z-A).

**Example**
```ts
new sorts.AlphabeticDescendingSort("name")
```

### `NumericAscendingSort`
Sorts items by the target field in ascending numerical order.

**Example**
```ts
new sorts.NumericAscendingSort("price")
```

### `NumericDescendingSort`
Sorts items by the target field in descending numerical order.

**Example**
```ts
new sorts.NumericDescendingSort("price")
```

### `NewestDateFirstSort`
Sorts items by a date field in newest date first order.

**Example**
```ts
new sorts.NewestDateFirstSort("created_at")
```

### `OldestDateFirstSort`
Sorts items by a date field in oldest date first order.

**Example**
```ts
new sorts.OldestDateFirstSort("created_at")
```

### `TrueFirstSort`
Sorts items where the target field is `true` first.

**Example**
```ts
new sorts.TrueFirstSort("is_available")
```

### `FalseFirstSort`
Sorts items where the target field is `false` first.

**Example**
```ts
new sorts.FalseFirstSort("discounted")
```

### `TruthyFirstSort`
Sorts items where the target field has a `truthy` value first.

**Example**
```ts
new sorts.TruthyFirstSort("description")
```

### `FalsyFirstSort`
Sorts items where the target field has a `falsy` value first.

**Example**
```ts
new sorts.FalsyFirstSort("description")
```

### `NullsFirstSort`
Sorts items where the target field is `null` first.

**Example**
```ts
new sorts.NullsFirstSort("end_date")
```

### `NullsLastSort`
Sorts items where the target field is `null` last.

**Example**
```ts
new sorts.NullsLastSort("end_date")
```

### `Sort`

Combines multiple sorting criteria into a single sort option. Sorts are applied in the order they are listed.

**Example**
```ts
new sorts.Sort({
  label: "Top Products",
  sorts: [
      new sorts.AlphabeticAscendingSort("name"),
      new sorts.NumericDescendingSort("reviews"),
      new sorts.TrueFirstSort("is_available")
  ]
})
```


## Real World Example

Let's say your scraper returns the following product data:

```ts
const taskScraper = task({
  name: "taskScraper",
  run: () => {
    // highlight-start
    return [
        {
            "id": 1,
            "name": "T-Shirt",
            "price": 16,  // in US Dollar
            "reviews": 1000,
            "is_available": true,
            "category": "apparel",
            "tags": ["cotton", "casual"]
        },
        {
            "id": 2,
            "name": "Laptop",
            "price": 700,
            "reviews": 500,
            "is_available": false,
            "category": "electronics",
            "tags": ["computer", "portable"]
        }
    ];
    // highlight-end
  },
})
```

Now you want to add filters to help users find specific products. The following code adds filters that allow users to:
- Search for products by name
- Filter by minimum number of reviews
- Filter by product category
- Filter by product tags
- Show only available products

```ts title="src/scraper/backend/server.ts"
import { Server } from "botasaurus-server/server"
import { filters } from "botasaurus-server/ui"
import { scrapeProductData } from "../src/scrapeProductData"

// highlight-start
const allFilters = [
    new filters.SearchTextInput("name"),
    new filters.MinNumberInput("reviews", { label: "Minimum Reviews" }),
    new filters.SingleSelectDropdown("category", {
        options: [
            { value: "apparel", label: "Apparel" },
            { value: "electronics", label: "Electronics" }
        ]
    }),
    new filters.MultiSelectDropdown("tags", {
        options: [
            { value: "cotton", label: "Cotton" },
            { value: "casual", label: "Casual" },
            { value: "computer", label: "Computer" },
            { value: "portable", label: "Portable" }
        ]
    }),
    new filters.IsTrueCheckbox("is_available", { label: "Is Available" })
]
// highlight-end

Server.addScraper(
    scrapeProductData,
    { filters: allFilters }
)
```

With these filters in place, users can easily get the data they want:



Next, you want to add sorts to order products. The following code allows users to:
- Sort products by name in ascending order
- Use a combined sort that orders products by:
  1. Name alphabetically
  2. Number of reviews (highest first)
  3. Availability (available products first)

```ts title="src/scraper/backend/server.ts"
import { Server } from "botasaurus-server/server"
import { sorts } from "botasaurus-server/ui"
import { scrapeProductData } from "../src/scrapeProductData"
// highlight-start
const allSorts = [
    new sorts.AlphabeticAscendingSort("name"),
    new sorts.Sort({
        label: "Top Products",
        isDefault: true,
        sorts: [
            new sorts.AlphabeticAscendingSort("name"),
            new sorts.NumericDescendingSort("reviews"),
            new sorts.TrueFirstSort("is_available")
        ]
    })
]
// highlight-end

Server.addScraper(
    scrapeProductData,
    { sorts: allSorts }
)
```

Now users can sort the product data:

---

### Docs Botasaurus Desktop Enhancing Scrapers Input Controls (`docs/docs/botasaurus-desktop/enhancing-scrapers/input-controls.md`)

---
sidebar_position: 8
---
# Input Controls

**Input controls** are interactive UI elements (such as text fields, file pickers, and dropdowns) that allow users to provide data to scrapers. With minimal JavaScript, you can create complex forms, making your desktop extractors more user-friendly.

## Key Concepts

- **Scraper-Input Binding**: Each scraper's controls are defined in a matching `.js` file within the `inputs/` folder. For example, `stockPriceScraper.js` for a scraper named `stockPriceScraper`.
- **Dynamic UI**: Change controls dynamically using `isShown` or `isDisabled` to hide or disable fields based on user input.
- **Validation**: Enforce mandatory fields with `isRequired` or implement custom validation logic (e.g., regex checks) using `validate`.

## Types of Controls

### `text`
A single-line text field.

**Example**
```ts
.text("stock_symbol", {
  placeholder: "Enter a stock symbol (e.g., AAPL)",
  defaultValue: "AAPL",
  isRequired: true,
})
```

### `listOfTexts`
Multiple single-line text fields. Use the optional `limit` parameter to cap the maximum number of items that can be entered.

**Example**
```ts
.listOfTexts("search_queries", {
  placeholder: "python",
  defaultValue: ["python", "javascript"],
  limit: 5,         // maximum 5 queries
  isRequired: true, // at least 1 query must be provided
})
```

### `link`
A single-line URL field.

**Example**
```ts
.link("website", {
  placeholder: "https://example.com",
})
```

### `listOfLinks`
Multiple single-line URL fields. Use the optional `limit` parameter to cap the maximum number of links that can be entered.

**Example**
```ts
.listOfLinks("product_urls", {
  placeholder: "https://example.com/product/item",
  limit: 5,          // maximum 5 links
  isRequired: true,  // at least 1 link must be provided
})
```

### `number`
A numeric input (integer or float). The `placeholder`, `min`, `max`, and `defaultValue` parameters are optional.

**Example**
```ts
.number("max_results", {
  placeholder: 100,
  min: 1,
  max: 1000,
  defaultValue: 50,
})
```

### `numberGreaterThanOrEqualToZero`
A number field that enforces a minimum value of 0.

**Example**
```ts
.numberGreaterThanOrEqualToZero("retry_delay_in_seconds", {
  placeholder: 5,
})
```

### `numberGreaterThanOrEqualToOne`
A number field that enforces a minimum value of 1.

**Example**
```ts
.numberGreaterThanOrEqualToOne("retries", {
  placeholder: 3,
})
```

### `select`
A single-select dropdown menu. It requires either an `options` array or a `searchMethod` for dynamic option fetching.


**Example - Static Options**
```ts
.select("category", {
  options: [
    { value: "tech",     label: "Technology" },
    { value: "robotics", label: "Robotics"   },
    { value: "ai",       label: "AI"         },
  ],
  defaultValue: "tech",
})
```

**Example - Dynamic Options**
```ts
.select("city", {
  searchMethod: "searchCityOptions",  // Backend method that fetches options
  canCreateOptions: true,          // Allow users to create custom options
  defaultValue: {value: 'US__CA__SF', label: "San Francisco"},
})
```

**Default Value Format:**
- For static `options`: Use a string value (e.g., `"tech"`)
- For `searchMethod`: Use an object with `value` and `label` properties:
  ```ts
  defaultValue: {value: 'US__CA__SF', label: "San Francisco"}
  ```

**Additional Parameters:**
- `canCreateOptions` (optional, default: `false`) - Allows users to type and create custom values
- `searchMethod` (optional) - Name of the backend handler function for dynamic option fetching

#### How to Dynamically fetch Options with `searchMethod`

To use `searchMethod`, you need to define async handler functions and add them using `Server.addSearchOptionsEndpoints()`.

**Input JS Implementation:**
```ts
.select("city", {
  searchMethod: "searchCityOptions",  // Backend method that fetches options
  canCreateOptions: true,          // Allow users to create custom options
  defaultValue: {value: 'US__CA__SF', label: "San Francisco"},
})
```

**`server.ts` Implementation:**
```ts
import { Server } from 'botasaurus-server/server';
import axios from 'axios';
import { config } from '../../main/config';

const baseUrl = config.isDev ? "http://0.0.0.0:3000" : "https://api.my-app.com";

async function searchCityOptions(query: string, data: any) {
  const params = {
    query,
  };

  try {
    const response = await axios.get(`${baseUrl}/search/cities`, { params });
    return response.data; // Must return array of { value, label }
  } catch (error) {
    throw error; // Propagate errors for proper handling
  }
}

// Register the search endpoint
Server.addSearchOptionsEndpoints({
  searchCityOptions,
});
```

**Handler Requirements:**
- Receives two parameters: `query` (string) and `data` (object with all form field values)
- Must return an array of objects with `value` and `label` properties
- Errors (including Axios errors) are automatically caught and displayed to users

### `multiSelect`
A multi-select dropdown menu. It requires either an `options` array or a `searchMethod` for dynamic option fetching. The optional `limit` parameter sets the maximum number of selections.


**Example - Static Options**
```ts
.multiSelect("tags", {
  options: [
    { value: "tech",     label: "Technology" },
    { value: "robotics", label: "Robotics"   },
    { value: "ai",       label: "AI"         },
  ],
  defaultValue: ["tech", "ai"],
})
```

**Example - Dynamic Search with Bulk Add**
```ts
.multiSelect("cities", {
  searchMethod: "searchCityOptions",   // Backend method that fetches options
  canCreateOptions: true,           // Allow users to create custom options
  canBulkAdd: true,                 // Enable bulk add/edit functionality
  defaultValue: [{value: 'US__CA__SF', label: "San Francisco"}],
})
```

**Default Value Format:**
- For static `options`: Use an array of strings (e.g., `["tech", "ai"]`)
- For `searchMethod`: Use an array of objects with `value` and `label` properties:
  ```ts
  defaultValue: [
    {value: 'US__CA__SF', label: "San Francisco"},
    {value: 'US__NY__NYC', label: "New York"}
  ]
  ```

**Additional Parameters:**
- `limit` (optional) - Maximum number of items that can be selected
- `canCreateOptions` (optional, default: `false`) - Allows users to type and create custom values
- `canBulkAdd` (optional, default: `false`) - Enables bulk add/edit functionality with a "Bulk Add" or "Bulk Edit" button below the control
- `searchMethod` (optional) - Name of the backend handler function for dynamic option fetching

**Bulk Add Functionality:**

When `canBulkAdd` is enabled, a button appears below the multi-select control allowing users to paste multiple values at once. Bulk add supports multiple input formats:
- **JSON array**: `["New York", "San Francisco", "Chicago"]`
- **Comma separated**: `New York, San Francisco, Chicago`
- **Newline separated**:
  ```
  New York
  San Francisco
  Chicago
  ```

### `checkbox`
A Boolean checkbox. Defaults to `false` if no `defaultValue` is provided.

:::warning
Do not use `checkbox` to reveal hidden controls via [isShown](#isshown). This is an anti-pattern. Use [`switch`](#switch) instead for toggling visibility of other controls.
:::

**Example**
```ts
.checkbox("include_reviews", {
  defaultValue: true,
})
```

### `textarea`
A multi-line text field.

**Example**
```ts
.textarea("description", {
  placeholder: "Enter a detailed description",
})
```

### `switch`
A Boolean toggle switch (an alternative to `checkbox`). Defaults to `false` if no `defaultValue` is provided.

:::tip[Best Practice]
Use `switch` instead of `checkbox` when the toggle is meant to reveal hidden controls via [isShown](#isshown). For example, showing **Advanced Settings**. Using `checkbox` for this purpose is an anti-pattern.
:::

**Example**
```ts
.switch("show_browser")
```

### `search`
A text input with search-specific styling.

**Example**
```ts
.search("search_query", {
  isRequired: true,
})
```

### `choose`
Displays options as clickable buttons (an alternative to `select`). It requires an `options` array.

Use `choose` instead of `select` when you have fewer than 3 options for better user experience.

**Example**
```ts
.choose("theme", {
  options: [
    { value: "light", label: "Light" },
    { value: "dark",  label: "Dark"  },
  ],
  defaultValue: "light",
})
```

### `filePicker`
Allows users to upload one or more files. Supports the `accept`, `multiple`, and `limit` parameters.

**Example**
```js
/**
 * @typedef {import('botasaurus-controls').Controls} Controls
 * @typedef {import('botasaurus-controls').FileTypes} FileTypes
 */

const { FileTypes } = require('botasaurus-controls');

/**
 * @param {Controls} controls
 */
function getInput(controls) {
  controls.filePicker("product_images", {
    accept: FileTypes.IMAGE, // restrict to image files ('jpeg','jpg','png','gif','bmp','svg','webp')
    limit: 20,               // maximum 20 files
    isRequired: true,        // at least 1 file is required
  })
}
```

To allow only one file to be selected, set `multiple` to `false`:

```js
/**
 * @typedef {import('botasaurus-controls').Controls} Controls
 * @typedef {import('botasaurus-controls').FileTypes} FileTypes
 */

const { FileTypes } = require('botasaurus-controls');

/**
 * @param {Controls} controls
 */
function getInput(controls) {
  controls.filePicker("product_image", {
    accept: FileTypes.IMAGE,
    multiple: false,
    isRequired: true, // a file is required
  })
}
```

### `section`
Groups related controls into a collapsible section. Keep in mind that sections cannot be nested inside other sections.

**Example**
```ts
.section("User Details", (section) => {
  section
    .text("first_name", {
      placeholder: "John",
      isRequired: true,
    })
    .text("last_name", {
      placeholder: "Doe",
      isRequired: true,
    })
    .number("age", {
      min: 18,
      isRequired: true,
    })
})
```

### `addLangSelect`
A prebuilt language dropdown with **187** languages.

**Example**
```ts
.addLangSelect({
  label: "Preferred Language",
  defaultValue: "en",
})
```
The selected language is accessible via `data["lang"]` in the scraping function.

### `addCountrySelect`
A prebuilt country dropdown with **252** countries.

**Example**
```ts
.addCountrySelect({
  label: "Target Country",
  defaultValue: "US",
})
```
The selected country is accessible via `data["country"]` in the scraping function.

### `addProxySection`
Adds a section with proxy controls.

**Example (single proxy)**
```ts
.addProxySection()
```

Set `isList` to `true` to allow users to enter multiple proxies.

**Example (proxy list)**
```ts
.addProxySection({
  isList: true,
})
```

Also, you can get the proxy (or proxy list) from the `proxy` field in the metadata:
```ts
const taskScraper = task({
  name: "taskScraper",
  run: ({ data, metadata }) => {
    // highlight-next-line
    const proxy = metadata["proxy"]
    return data
  },
})
```

## Control Options

All controls support a common set of options that allow you to customize their label, default value, validation logic, visibility, and more.

### `label`

The label is displayed above the control. If omitted, the control ID (e.g., `stock_symbol`) is shown in Title Case.

**Example**
```ts
.text("stock_symbol", {
   // highlight-next-line
  label: "Ticker Symbol",
})
```



### `defaultValue`

Sets the initial value for the control when the form loads.

**Example**
```ts
.select("category", {
  options: [
    { value: "tech",     label: "Technology" },
    { value: "robotics", label: "Robotics"   },
    { value: "ai",       label: "AI"         },
  ],
  // highlight-next-line
  defaultValue: "tech",
})
```



### `isRequired`

Enforces mandatory field validation. This can be a boolean value or a function that decides based on other inputs.

**Example - Boolean isRequired**
```ts
.link("website", { 
   // highlight-next-line
   isRequired: true,
 })
```



**Example - Dynamic Requirement**
```ts
.text("phone_number", {
   // highlight-next-line
  isRequired: (data) => data["contact_method"] === "phone", // Required only if contact_method is phone
})
```



### `validate`

A custom validation function that returns an error message if validation fails. It can return a string or an array of strings for multiple errors.

**Note:**
- A control value can be `null`, but is never `undefined`, so you never need to check for `undefined`.
- If a control is hidden or disabled, validation is skipped.
- If a control fails the `isRequired` check, validation is skipped. This is why we don't need to check for `null` in the second **multiple errors** example.

**Example - Validate an Email Address**
```ts
.text("email", {
  // highlight-start
  validate: (value) => {
    
    const emailRegex =  /^(([^<>()[\]\\.,;:\s@\"]+(\.[^<>()[\]\\.,;:\s@\"]+)*)|(\".+\"))@((\[[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\])|(([a-zA-Z\-0-9]+\.)+[a-zA-Z]{2,}))$/
    if (!emailRegex.test(value)) {
      return "Please enter a valid email address"
    }
  },
  // highlight-end
})
```


**Example - Multiple Errors**
```ts
.number("age", {
  isRequired: true,
  // highlight-start
  validate: (value) => {
    const errors = []
    if (value < 18) {
      errors.push("Must be 18 or older")
    }
   if (!Number.isInteger(value)) {
      errors.push("Age must be a number")
   }
    return errors
  },
  // highlight-end
})
```


### `helpText`

Shows a question mark icon that displays help text when hovered over.

**Example**
```ts
.text("api_key", {
   label: "API Key",
   // highlight-next-line
   helpText: "Find API key in Dashboard → Settings → API"
})
```



### `isDisabled`

Dynamically disables the control based on other inputs.

**Example**
```ts
.checkbox("use_custom_settings", {
  label: "Use Custom Settings"
})
.number("timeout", {
  label: "Timeout (seconds)",
  // highlight-next-line
  isDisabled: (data) => !data["use_custom_settings"],
})
```



### `disabledMessage`

Tooltip explaining why a control is disabled. It's recommended to use this option when using `isDisabled`.

**Example**
```ts
.checkbox("use_custom_settings", {
  label: "Use Custom Settings"
})
.number("timeout", {
  label: "Timeout (seconds)",
  isDisabled: (data) => !data["use_custom_settings"],
  // highlight-next-line
  disabledMessage: "Enable custom settings to set this value"
})
```



### `isShown`

Conditionally shows or hides a control based on other inputs. This is often better than `isDisabled` for optional sections like "Advanced settings" because it creates a cleaner UI by hiding complexity until needed.

A `switch` or `choose` control is recommended to toggle visibility.

**Example**
```ts
.switch('enable_reviews_extraction')
.numberGreaterThanOrEqualToZero('max_reviews', {
  label: 'Max Reviews per Place (Leave empty to extract all reviews)',
  placeholder: 20,
  isShown: (data) => data['enable_reviews_extraction'],
  defaultValue: 20,
})
.choose('reviews_sort', {
  label: "Sort Reviews By",
  isRequired: true,
  isShown: (data) => data['enable_reviews_extraction'],
  defaultValue: 'newest',
  options: [
    { value: 'newest', label: 'Newest' },
    { value: 'most_relevant', label: 'Most Relevant' },
    { value: 'highest_rating', label: 'Highest Rating' },
    { value: 'lowest_rating', label: 'Lowest Rating' },
  ],
})
```



### `isMetadata`

When set to `true`, the control's value is passed in the `metadata` instead of the `data` parameter in the scraping function.

**Example**
```ts
.text("api_key", {
   label: "API Key",
   // highlight-next-line
   isMetadata: true,
})
```

In the scraper function, use it as follows:

```ts
const task = task({
  name: "taskScraper",
  run: ({ data, metadata }) => {
    // highlight-next-line
    const scraperId = metadata["api_key"]
  }
})
```

The `metadata` parameter is useful when you have enabled caching and want to exclude a field from being included in the cache key, such as `api_key`, `cookies`, `proxy`, etc.

Keep this in mind, as you will learn about [caching](./enable-caching.md) in a later section.

## Real World Example

The following example demonstrates how to use `listOfTexts`, `number`, `switch`, `choose`, `section`, and other controls to build an advanced form:

```
/* Detailed source-code truncated for AI context efficiency. */
```

---

### Docs Botasaurus Desktop Enhancing Scrapers Rate Limiting (`docs/docs/botasaurus-desktop/enhancing-scrapers/rate-limiting.md`)

---
sidebar_position: 17
---

# Rate Limiting

Rate limiting controls how many scrapers can run simultaneously. This helps prevent system overload and reduces the risk of getting blocked.

You can set rate limits in two ways:
- By the scraper's name (recommended)  
- By the scraper's type (browser or task)

## By Scraper Name (Recommended)

This is the preferred method for controlling scraper execution. It allows you to set specific limits for individual scrapers by name.

For example, to allow:
- 1 concurrent instance of `stockPriceScraper`  
- 40 concurrent instances of `amazonPdfExtractor`

Use this configuration:

```ts title="src/scraper/backend/server.ts"
import { Server } from 'botasaurus-server/server';

Server.setRateLimit({ 
  stockPriceScraper: 1, 
  amazonPdfExtractor: 40,
});
```

## By Scraper Type

Alternatively, you can set limits based on the scraper type:

- `browser`: Limits concurrent Playwright scrapers
- `task`: Limits concurrent Task scrapers

For example, to allow:
- 1 concurrent browser scraper  
- 40 concurrent task scrapers  

Use this configuration:

```ts title="src/scraper/backend/server.ts"
import { Server } from 'botasaurus-server/server';

Server.setRateLimit({ 
  browser: 1, 
  task: 40,
});
```

## Default Behavior

If you don't set any rate limits, Botasaurus applies these defaults:
- **Browser scrapers:** 1 at a time
- **Task scrapers:** Unlimited

This default configuration prevents accidental resource overload from running too many browser instances simultaneously.

---

### Docs Botasaurus Desktop Enhancing Scrapers Scraper Types (`docs/docs/botasaurus-desktop/enhancing-scrapers/scraper-types.md`)

---
sidebar_position: 5
---

# Scraper Types  

Botasaurus Desktop provides two types of scrapers to handle different extraction scenarios:

### Playwright Scrapers for Web Scraping

Playwright Scrapers launch a Chrome browser to scrape websites. It is a stealthier version of Playwright that's optimized for web scraping.

Use a Playwright scraper when you need to:

- Scrape JavaScript-heavy sites (React, Vue, Angular)
- Scroll through infinite lists or click buttons/forms
- Bypass bot-protection

**Example** 
```ts
import { playwright } from 'botasaurus/playwright';

export const webScraper = playwright({
  name: 'webScraper',
  headless: true,  // Set to false in production to hide the window
  run: async ({ page }) => {
    await page.goto('http://example.com');
    const h1 = await page.textContent('h1');
    return { h1 };
  }
})
```

### Task Scrapers for File Parsing & API Calls
Task scrapers run in pure Node.js with no browser overhead. They are ideal for operations that can be handled through network requests or file parsing.

Use a Task scraper when you need to:
- Extract data from PDFs or Excel files
- Call REST endpoints

**Example** 
```ts
import { task } from 'botasaurus/task';

export const taskScraper = task({
  name: 'taskScraper',
  run: ({ data }) => {
    // Return data as is
    return data;
  }
})
```

---

Use a **Playwright Scraper** if you need to open a website, and a **Task-Based Scraper** for everything else, like processing files or calling APIs.

---

### Docs Botasaurus Desktop Enhancing Scrapers Task Configuration (`docs/docs/botasaurus-desktop/enhancing-scrapers/task-configuration.md`)

---
sidebar_position: 11
---
# Task Configuration

**Task Configuration** lets you customize how tasks are displayed and processed.

You can:  
- Control task naming  
- Split large tasks into smaller ones  
- Combine results from multiple tasks into one task
- Remove duplicates from task results  

## Configuration Options

There are 4 options available for task configuration:

- `getTaskName` - Customize how tasks are named
- `splitTask` - Split a single input into multiple, smaller tasks
- `createAllTask` - Combine results from all split tasks into a single "All" task
- `removeDuplicates` - Remove duplicates from task results

Let's explore each option in detail.

## Custom Task Naming

By default, tasks on the `Tasks` page are named using their task ID, which makes it difficult to identify what data each task contains.



You can use `getTaskName` to generate meaningful task names based on input data. For example, you could name a task after the stock symbol it processes.

**How to Use It?**

Provide a function to `getTaskName` that receives the input `data` and returns a string.

```ts title="src/scraper/backend/server.ts"
import { Server } from "botasaurus-server/server"
import { stockPriceScraper } from "../src/scraper"

// Function to generate custom task name
function getTaskName(data) {
    return `${data.stock_symbol}`;
}

Server.addScraper(stockPriceScraper, {
    getTaskName: getTaskName
});
```

**Result:**


## Task Splitting

When scraping multiple items (like 200 URLs or 50 stock symbols), you may want to track each item separately. The `splitTask` option creates individual tasks for each item.

For example, if your input accepts multiple stock symbols, you can split them into separate tasks.


**How to Use It?**

Provide a function to `splitTask` that receives the input `data` and returns an array of items. Botasaurus will create a new task for each item in the returned array.

```ts title="src/scraper/backend/server.ts"
import { Server } from "botasaurus-server/server"
import { stockPriceScraper } from "../src/scraper"

// Function to split input data into multiple tasks
function splitTask({stock_symbols, ...otherControlSettings}) {
    return stock_symbols.map(symbol => ({ stock_symbol: symbol, ...otherControlSettings }));
}

Server.addScraper(stockPriceScraper, {
    splitTask: splitTask
});
```

**Result:**


## Creating an "All Task"

When tasks are split, users often need to view all results in one place. Setting `createAllTask: true` automatically generates a combined view of all split task results, improving usability.

```ts title="src/scraper/backend/server.ts"
import { Server } from "botasaurus-server/server"
import { stockPriceScraper } from "../src/scraper"

function splitTask(data) {
    ...
}

Server.addScraper(
    stockPriceScraper,
    {
        splitTask: splitTask,
        createAllTask: true // Add this line
    }
);
```

**Result:**


## Removing Duplicate Tasks

If your scraper produces duplicate items in its results, use the `removeDuplicatesBy` option to automatically filter out duplicates based on a specific field.

**How to Use It?**

Set `removeDuplicatesBy` to the field that uniquely identifies an item. 

For example, if each item has a unique `link`, you can remove duplicates like this:
```ts title="src/scraper/backend/server.ts"
Server.addScraper(searchResultsScraper, {
  splitTask: splitTask,
  createAllTask: true,
  removeDuplicatesBy: 'link' // Deduplicates based on this field  
});
```

This ensures that only one result item per unique `link` is kept.

**Notes on Deduplication:**
When using `removeDuplicatesBy` with `splitTask` and `createAllTask`, deduplication occurs in two stages:
1. **Within each task:** Duplicates are removed from each split task's results as it completes.
2. **In "All Task":** After all split tasks finish, their results are aggregated. Deduplication is then applied to this combined dataset, removing any duplicates that exist across different split tasks.

Due to this second stage, you may notice the total count in the "All Task" drop after the final split task completes.

---

### Docs Botasaurus Desktop Your First Extractors Amazon Pdf Invoice Extractor (`docs/docs/botasaurus-desktop/your-first-extractors/amazon-pdf-invoice-extractor.md`)

---
sidebar_position: 2
---


# Amazon PDF Invoice Extractor

{/* Build-time constant */}
export const currentYear = new Date().getFullYear();

While web scraping is common, many high-value automation opportunities involve parsing documents like PDFs and Excel files. This is a large untapped market even in {currentYear}.

This extractor will turn hours of manual work into seconds of automation.

We'll build a desktop tool that extracts key details from Amazon invoice PDFs such as:

- Place of Supply
- GST Identification Number (Think of it like Business ID)
- Document Number
- Document Date

Unlike our stock scraper, this extractor never opens a browser—it simply parses the file on the user's machine.

Creating it takes 4 simple steps:

1. Install a PDF parsing package
2. Write the extractor function
3. Add the extractor to the server
4. Create a file picker for PDF selection

### Step 1: Install PDF Parsing Package

First, we need a tool to read PDF file content. We'll use `electron-pdf-parse`, a fork of the `pdf-parse` package that's designed to work seamlessly with Electron.

Open your terminal and install it:

```bash
npm install electron-pdf-parse
```

### Step 2: Write the extractor function

Create a new file `src/scraper/src/amazonPdfExtractor.ts` and paste the code below.

```
/* Detailed source-code truncated for AI context efficiency. */
```

This code:
1. Reads each PDF the user selects
2. Extracts the full text using `electron-pdf-parse`
3. Uses regular expressions to extract 4 fields:
    - Place of Supply
    - GST Identification Number
    - Document Number
    - Document Date
4. Returns the results

**Key points**
- We use `task()` instead of `playwright()` since no browser is needed
- The function name `amazonPdfExtractor` connects to the input controls that will be defined in `inputs/amazonPdfExtractor.js`

### Step 3: Add the extractor to the server

Now let's connect our extractor to the desktop app. Open `src/scraper/backend/server.ts` and add the extractor:

```ts title="src/scraper/backend/server.ts"
import { Server } from 'botasaurus-server/server';
// highlight-next-line
import { amazonPdfExtractor } from '../src/amazonPdfExtractor';

// highlight-next-line
Server.addScraper(amazonPdfExtractor);     // 👈 makes it appear in the UI
```

### Step 4: Create the file-picker input

Create a new file `inputs/amazonPdfExtractor.js` and paste following code:

```js title="inputs/amazonPdfExtractor.js"
/**
 * @typedef {import('botasaurus-controls').Controls} Controls
 * @typedef {import('botasaurus-controls').FileTypes} FileTypes
 */
const { FileTypes } = require('botasaurus-controls');

/**
 * Renders the form users see on the Home page.
 * @param {Controls} controls
 */
function getInput(controls) {
  // Render a File Input for uploading PDFs
  controls.filePicker('files', {
    label: 'Invoice PDFs',
    accept: FileTypes.PDF,
    isRequired: true,
    helpText: 'Drag one or more Amazon invoice PDFs here',
  });
}
```

This creates a drag-and-drop file picker that only accepts PDF files:



### Test with a Sample Invoice

Time to see your extractor in action:

1. Reload the desktop app with `Ctrl/⌘ + R`
2. Click on **Amazon PDF Extractor**



3. Download this [sample Amazon invoice PDF](https://raw.githubusercontent.com/omkarcloud/botasaurus-desktop-tutorial/master/test/sample-invoice.pdf)
4. Drop the PDF into the file picker and click **Run**



The app will parse the file and display a table like this:

| Place of Supply | GSTIN              | Document Number | Document Date |
|-----------------|--------------------|-----------------|---------------|
| MAHARASHTRA     | 27AICPR2880F1Z5    | FBA15HKNSOZZ   | 06-12-2023    |

### 🎉 Congratulations!

You've built a document processor that can extract data from hundreds of PDFs in seconds. The same approach works for any structured document—bank statements, receipts, and more.

Find the complete code for this tutorial, including the **Stock Pricer Scraper** and **Amazon Invoice PDF Extractor**, in the [Botasaurus Desktop Tutorial GitHub Repository](https://github.com/omkarcloud/botasaurus-desktop-tutorial)

---

You've now mastered both extraction styles: `playwright` for websites and `task` for local files and API calls.

Next, we'll explore advanced features like filtering, sorting, and custom data views to make your extractors even more easy to use.

---

### Docs Botasaurus Desktop Your First Extractors Stock Price Scraper (`docs/docs/botasaurus-desktop/your-first-extractors/stock-price-scraper.md`)

---
sidebar_position: 1
---

# Stock Price Scraper

This first extractor shows the "Playwright way" of scraping: open a browser, navigate to a page, grab a value, and return the data.

Creating this extractor is a simple 3-step process:

**1. Write a Playwright scraper function**  
**2. Add the function to the server**  
**3. Create a simple text input for the user interface**

Follow along and you'll have a desktop app fetching real-time stock prices in under 10 minutes.

### Step 1: Create the Scraper Function

Open `src/scraper/src/scraper.ts` and replace all its contents with the code below. This creates our `stockPriceScraper` function that will:

1. Receive a stock symbol from user input
2. Build the Yahoo Finance URL for that specific stock
3. Navigate to the page using Playwright
4. Extract the current stock price from this specific element:
  
5. Return the data

```ts title="src/scraper/src/scraper.ts"
import { playwright } from 'botasaurus/playwright';

/**
 * Grabs the live quote for a given stock symbol from Yahoo Finance.
 */
const stockPriceScraper = playwright({
  // 👇 MUST match the input file name we'll create in Step 3
  // highlight-next-line
  name: 'stockPriceScraper',    
  headless: false,              // Show the browser window while scraping
  reuseDriver: true,            // Reuse the same browser instance for multiple tasks
  run: async ({ data, page }) => {
    // Extract the stock symbol from user input
    const stock_symbol = data['stock_symbol'];
    // Build the Yahoo Finance URL
    const link = `https://finance.yahoo.com/quote/${stock_symbol}`;

    // Navigate to the Yahoo Finance page
    await page.goto(link, { waitUntil: 'domcontentloaded' });

    // Extract the stock price using a selector
    const stock_price = parseFloat(
      (await page.textContent('[data-testid="qsp-price"]')) as string,
    );

    // Return the data
    return {
      stock_symbol: stock_symbol,
      stock_price: stock_price,
    };
  },
});

export { stockPriceScraper };
```

Have you noticed we named our scraper `stockPriceScraper`? 

This name is crucial—Botasaurus uses it to connect the scraper with its input controls js file, which we'll create in Step 3.

### Step 2: Add the Scraper

Now let's connect our scraper function to the desktop app. Open `src/scraper/backend/server.ts` and replace all its contents with the code below:

```ts title="src/scraper/backend/server.ts"
import { Server } from 'botasaurus-server/server';
import { stockPriceScraper } from '../src/stockPriceScraper';

// highlight-next-line
Server.addScraper(stockPriceScraper);
```

This single line connects your scraper with the desktop app, making it available in the user interface.

### Step 3: Create the Input Controls

Finally, let's create the user interface. Create a new file named `stockPriceScraper.js` in the `inputs` folder:

```js title="inputs/stockPriceScraper.js"
/**
 * @typedef {import('botasaurus-controls').Controls} Controls // 👈 Enables Code autocomplete
 */

/**
 * Renders the form users see on the Home page.
 * @param {Controls} controls
 */
function getInput(controls) {
  // highlight-start
  controls.text("stock_symbol", {
    label: "Stock Symbol",
    placeholder: "e.g. AAPL",
    defaultValue: "AAPL",
    isRequired: true,
    validate: (value) => {
      const errors = [];
      if (value.length !== 4) {
        errors.push("Length must be 4 characters");
      }
      if (!/^[A-Z]+$/.test(value)) {
        errors.push("Only uppercase letters allowed");
      }
      return errors;
    },
  });
  // highlight-end
}
```

This code creates a text input field where users can enter stock symbols. 

Also, the JSDoc comments enable code completion in VSCode, helping you discover all available input controls.



### 🎉 You're Done!

Congratulations! You've just built a fully functional **Stock Price Scraper** that runs as a desktop application.

**To test it:**

1. Press `Ctrl/⌘ + R` to reload the app  
2. Type any stock symbol (or use the default "AAPL") and click **Run**  
3. Watch as Chrome opens Yahoo Finance and navigates to your stock  
4. Click on the app window to see your results in a table format  




The next section will show you how to build an Amazon Invoice extractor that can save accountants hours of manual work.

---

--- METRICS ---
- Files Extracted: 27
- Estimated Token Budget: ~59820 tokens
- Canonical Reference: https://codewiki.google/github.com/omkarcloud/botasaurus