## 1. Project Overview & Quickstart (browserbase/CONTRIBUTING)
# CONTRIBUTING
Open-source repository browserbase/CONTRIBUTING
### Repository Details
- **Repository:** [browserbase/CONTRIBUTING](https://github.com/browserbase/CONTRIBUTING)
- **Primary Language:** Code
*Note: High-volume repository documentation is actively indexed and synchronized by YakaAI.*
## 2. Official Technical Reference & Guides (browserbase/docs)
## File: README.md
The Browserbase documentation source code has been moved to our monorepo.
Please contact us via [our chat](https://go.crisp.chat/chat/embed/?website_id=c7f89ae9-ae21-4d06-a71b-951e2d2c6887) or [our website](https://www.browserbase.com) if you have any feedback or questions.
---
## File: guides/session-debug-connection/browser-remote-control.mdx
---
title: "Taking a Session's Remote Control"
sidebarTitle: "Remotely control a Session"
"Navigate complex UIs and achieve 10x faster development iterations."
---
This guide covers getting a `debuggerFullscreenUrl` to access a running Session
to debug a local AI Agent iteration or handle complex UI scenarios such as
dealing with iframes or user credentials.
## Unblock complex UI scenarios by giving control to the end-user
While Browserbase helps deal with common scraping problems
([anti-bot mechanism, captchas](/features/stealth-mode),
[reliably download files](/features/sessions#downloading-files)), some scenarios
remain hard to automate for technical or data privacy reasons.
**Dealing with iframe** can be challenging as the loaded content might be
external and could change without notice, adding another fold of scraping
complexity. Interacting with iframes can be delegated to the end user by
forwarding the remote control URL.
**Dealing with user credentials** is another complex scenario as the end-user
might prefer to fill out its credentials directly on the website instead of
storing them elsewhere.
Let's look at the last scenario with an AI Agent for Amazon wishlists.
### Pause our automation while the user enters its credentials
Our AI Agent tries to find the best prices for items on a user's Amazon
wishlist; below is the `retrieveAmazonWishlistItems()` function responsible to
gather the items:
```js
async function retrieveDebugConnectionURL(sessionId) {
const response = await fetch(
`https://www.browserbase.com/v1/sessions/${sessionId}/debug`,
{
method: "GET",
headers: {
'x-bb-api-key': `${process.env.BROWSERBASE_API_KEY}`,
},
}
);
const json = await response.json();
return json.debuggerFullscreenUrl;
}
async function retrieveAmazonWishlistItems(sessionId, page) {
await page.goto(
`https://www.amazon.com/ap/signin?openid.pape.max_auth_age=0&openid.return_to=https%3A%2F%2Fwww.amazon.com%2F%3Fref_%3Dnav_signin&openid.identity=http%3A%2F%2Fspecs.openid.net%2Fauth%2F2.0%2Fidentifier_select&openid.assoc_handle=usflex&openid.mode=checkid_setup&openid.claimed_id=http%3A%2F%2Fspecs.openid.net%2Fauth%2F2.0%2Fidentifier_select&openid.ns=http%3A%2F%2Fspecs.openid.net%2Fauth%2F2.0`,
{
// let's make sure the page is fully loaded before asking for the live debug URL
waitUntil: "domcontentloaded",
}
);
const debugRemoteURL = await retrieveDebugConnectionURL(sessionId)
await sendCredentialsPageUrlToUser(debugRemoteURL)
// now, let's wait for the user to complete the credential flow
await page.waitForSelector('#nav-flyout-wl-items > div > a:nth-child(2) > span', {
timeout: 300000 // give 5 minutes,
visible: false
})
// Once we are here:
// 1. navigate to https://www.amazon.com/hz/wishlist/ls
// 2. collect items and navigates pagination
}
```
## Accelerate your local development with remote debugging
Developing scraping programs requires a lot of slow iterations, involving lots
of `console.log()` and `page.on('pageerror', /* ... */`.
Fortunately, you can develop new automation or AI Agent 10x faster by combining
the live debugging of Sessions and
[Session Debugger](/features/session-debugger):
```mermaid
flowchart LR
run["Run your automation"]
error["An error is raised by the page"]
stuck["A selector mismatches"]
debug["Live inspect and debug the Session"]
failed["An error occurred and the Session closed"]
replay["Inspect on Session Debugger"]
worksout["The automation works as intented"]
run-->error
run-->stuck
run-->failed
run-->worksout
stuck--Request the debug connection URL-->debug
error--Request the debug connection URL-->debug
failed--Look at logs and events played-->replay
worksout--Analyze the Session steps-->replay
replay--fix your code-->run
debug--fix your code-->run
```
Let's see how to implement this local development workflow:
```js
import { chromium } from "playwright-core";
let sessionId;
async function createSession() {
const response = await fetch(`https://www.browserbase.com/v1/sessions`, {
method: "POST",
headers: {
"x-bb-api-key": `${process.env.BROWSERBASE_API_KEY}`,
"Content-Type": "application/json",
},
body: JSON.stringify({
projectId: process.env.BROWSERBASE_PROJECT_ID,
}),
});
const json = await response.json();
return json;
}
async function retrieveDebugConnectionURL(sessionId) {
const response = await fetch(
`https://www.browserbase.com/v1/sessions/${sessionId}/debug`,
{
method: "GET",
headers: {
"x-bb-api-key": `${process.env.BROWSERBASE_API_KEY}`,
},
},
);
const json = await response.json();
return json.debuggerFullscreenUrl;
}
(async () => {
const { id } = await createSession();
sessionId = id;
const browser = await chromium.connectOverCDP(
// we connect to a Session created via the API
`wss://connect.browserbase.com?apiKey=${process.env.BROWSERBASE_API_KEY}&sessionId=${sessionId}`,
);
const defaultContext = browser.contexts()[0];
const page = defaultContext.pages()[0];
await page.goto("https://www.browserbase.com", {
// let's make sure the page is fully loaded before asking for the live debug URL
waitUntil: "domcontentloaded",
});
const debugUrl = await retrieveDebugConnectionURL(sessionId);
console.log(`Session started, live debug accessible here: ${debugUrl}.`);
await page.close();
await browser.close();
})().catch((error) => {
console.log(
`Session failed, replay is accessible here: https://www.browserbase.com/sessions/${sessionId}.`,
);
console.error(error.message);
});
```
Let's review the key snippets used:
Keep these two helpers close by mutualizing them into an `utils.js` file.
Retrieve and print the debug connection URL as soon as the Session starts,
making it accessible in case of a stuck Session or unexpected errors.
Finally, we close the loop of productivity by printing the direct link to the
[Session Debugger](/features/session-debugger) in case of a Session crash.
---
## File: guides/session-debug-connection/integrate-live-session.mdx
---
title: "Integrate a Session live view within your application"
sidebarTitle: "Integrate a Session live view"
"Provide real-time visibility to your end-users on ongoing automation."
mode: "wide"
---
The Session Debug Connection URLs can be used to integrate a live view of ongoing automation within your application.
For example, an application enabling users to configure advanced health checks based on pages' content can benefit
from allowing the users to see the health check recording in case of false positives.
## Integrating a live Session iframe within your application
Store the Session ID of each running Session by creating a Session using the [Session API](/api-reference/create-a-session) to store it and pass it to the connect URL as follows:
```typescript Playwright (TypeScript)
import { chromium } from "playwright-core";
async function createSession() {
const response = await fetch(`https://www.browserbase.com/v1/sessions`, {
method: "POST",
headers: {
'x-bb-api-key': `${process.env.BROWSERBASE_API_KEY}`,
"Content-Type": "application/json",
},
body: JSON.stringify({
projectId: process.env.BROWSERBASE_PROJECT_ID,
}),
});
const json = await response.json();
return json;
}
(async () => {
const { id } = await createSession();
// TODO: store the Session `id` to database
const browser = await chromium.connectOverCDP(
// we connect to the Session created via the API
`wss://connect.browserbase.com?apiKey=${process.env.BROWSERBASE_API_KEY}&sessionId=${sessionId}`,
);
// ...
})()
```
```typescript Puppeteer (TypeScript)
import puppeteer from 'puppeteer';
const fetch = require('node-fetch');
async function createSession() {
const response = await fetch(`https://www.browserbase.com/v1/sessions`, {
method: "POST",
headers: {
'x-bb-api-key': `${process.env.BROWSERBASE_API_KEY}`,
"Content-Type": "application/json",
},
body: JSON.stringify({
projectId: process.env.BROWSERBASE_PROJECT_ID,
}),
});
const json = await response.json();
return json;
}
(async () => {
const { id } = await createSession();
// TODO: store the Session `id` to database
const browser = await puppeteer.connect({
browserWSEndpoint: `wss://connect.browserbase.com?apiKey=${process.env.BROWSERBASE_API_KEY}&sessionId=${id}`,
});
// ...
})();
```
```python Python
from playwright.async_api import async_playwright
import asyncio
import aiohttp
async def create_session():
async with aiohttp.ClientSession() as session:
async with session.post(
'https://www.browserbase.com/v1/sessions',
headers={
'x-bb-api-key': f'{os.environ["BROWSERBASE_API_KEY"]}',
'Content-Type': 'application/json'
},
json={'projectId': f'{os.environ["BROWSERBASE_PROJECT_ID"]}'}
) as response:
json_data = await response.json()
return json_data
async def main():
async with async_playwright() as p:
browser = await p.chromium.connect_over_cdp(
f'wss://connect.browserbase.com?apiKey={os.environ["BROWSERBASE_API_KEY"]}&sessionId={session_id}'
)
# Your code using the browser goes here
if __name__ == '__main__':
loop = asyncio.get_event_loop()
session_id = loop.run_until_complete(create_session())
loop.run_until_complete(main())
```
If a user tries to access the live view of a running automation, retrieve the Session Debug Connection URLs as follows:
```typescript TypeScript
async function retrieveDebugConnectionURL(sessionId: string) {
const response = await fetch(
`https://www.browserbase.com/v1/sessions/${sessionId}/debug`,
{
method: "GET",
headers: {
'x-bb-api-key': `${process.env.BROWSERBASE_API_KEY}`,
},
},
);
const json = await response.json();
return json.debuggerFullscreenUrl;
}
const debugConnectionURL = await retrieveDebugConnectionURL()
// forward `debugConnectionURL` to the front-end
```
```python Python
import aiohttp
import os
async def retrieve_debug_connection_url(session_id):
url = f"https://www.browserbase.com/v1/sessions/{session_id}/debug"
headers = {
'x-bb-api-key': os.environ.get('BROWSERBASE_API_KEY', '')
}
async with aiohttp.ClientSession() as session:
async with session.get(url, headers=headers) as response:
json_data = await response.json()
return json_data.get('debuggerFullscreenUrl', None)
```
Let's forward the `debugConnectionURL` to the front end to create the final live view with an iframe.
The front-end can now display a live running Session by loading the `debugConnectionURL` in a read-only iframe:
```html
```
To enable your end users to interact with the live Session, remove the `style='pointer-events: none;'` line.
---
## File: guides/session-debug-connection/overview.mdx
---
title: "Getting Session Debug Connection URLs"
sidebarTitle: "Introduction"
---
Browserbase offers the [Session Debugger](/features/session-debugger), a
complete tool to inspect completed Sessions. However, many use cases require
direct interaction with the running Session or can benefit from direct debugging
from the local Browser.
## Getting Session Debug Connection URLs
Getting Session Debug Connection URLs is achieved by using the
[Session Debug Connection URL endpoint](/api-reference/retrieve-a-session-debug-connection-urls):
Use the [Session API](/api-reference/create-a-session) to create a new session
([_give it a try from the API Playground_](/api-reference/create-a-session)).
Then, use the [Session API debug
endpoint](/api-reference/retrieve-a-session-debug-connection-urls) to get the
`debuggerFullscreenUrl`.
Finally, open the `debuggerFullscreenUrl` URL in Chrome to start taking action
on the session.
## Using the Session Debug Connection URLs
Navigate complex UIs and achieve 10x faster development iterations.
Provide real-time visibility to your end-users on ongoing automation.
---
## File: guides/authentication.mdx
---
title: "How to deal with authentication flows"
sidebarTitle: "Authentication flows"
---
Authentication flows, along with an anti-bot mechanism, add complexity to
automation. Two-factor authentication and captchas are challenging to overcome,
and regular authentication flows slow down automation.
Fortunately, Browserbase [Stealth Mode](/features/stealth-mode), along with some
tactics, helps to go over authentication flows without interruption and to reuse
sessions for faster automation.
## Accessing an authentication flow with Stealth Mode
Most authentication flows implement mechanisms to prevent automation and
scraping:
- IP addresses restrictions
- User Agent filtering
- Captchas (ex:
[Clerk now features a Captcha on all authentication flows](https://clerk.com/changelog/2024-04-19-visual-captcha))
- Rate limiting
When running your own Browser, dealing with the above requires setup IP
rotations with Proxies along with Captcha solving and fingerprinting code or
libraries.
By connecting your automation to Browserbase, you get opt-in
[Proxies](/features/stealth-mode), automatic fingerprinting, and Captcha solving
without any code change:
```typescript Playwright (TypeScript)
import { chromium } from "playwright-core";
(async () => {
const browser = await chromium.connectOverCDP(
`wss://connect.browserbase.com?apiKey=${process.env.BROWSERBASE_API_KEY}&enableProxy=true`,
);
// Getting the default context to ensure the sessions are recorded.
const defaultContext = browser.contexts()[0];
const page = defaultContext.pages()[0];
await page.goto("https://browserbase.com/sign-in");
await page.close();
await browser.close();
})().catch((error) => console.error(error.message));
```
```typescript Puppeteer (TypeScript)
const puppeteer = require("puppeteer");
(async () => {
const browser = await puppeteer.connect({
browserWSEndpoint: `wss://connect.browserbase.com?apiKey=${process.env.BROWSERBASE_API_KEY}&enableProxy=true`,
});
const pages = await browser.pages();
const page = pages[0];
await page.goto("https://browserbase.com/sign-in");
await page.close();
await browser.close();
})().catch((error) => console.error(error.message));
```
```python Playwright (Python)
from playwright.sync_api import sync_playwright
def main():
with sync_playwright() as p:
browser = p.chromium.connect_over_cdp(
f'wss://connect.browserbase.com?apiKey={os.environ["BROWSERBASE_API_KEY"]}&enableProxy=true'
)
# Getting the default context to ensure the sessions are recorded.
default_context = browser.contexts()[0]
page = default_context.pages()[0]
page.goto("https://browserbase.com/sign-in")
page.close()
browser.close()
if __name__ == "__main__":
main()
```
```python Selenium (Python)
from selenium import webdriver
from selenium.webdriver.remote.remote_connection import RemoteConnection
from selenium.webdriver.common.desired_capabilities import DesiredCapabilities
import requests
import os
def create_session():
url = 'https://www.browserbase.com/v1/sessions'
headers = {'Content-Type': 'application/json', 'x-bb-api-key': os.environ["BROWSERBASE_API_KEY"]}
json = {
"projectId": os.environ["BROWSERBASE_PROJECT_ID"],
}
response = requests.post(url, json=json, headers=headers)
return response.json()['id']
class CustomRemoteConnection(RemoteConnection):
_session_id = None
def __init__(self, remote_server_addr: str, session_id: str):
super().__init__(remote_server_addr)
self._session_id = session_id
def get_remote_connection_headers(self, parsed_url, keep_alive=False):
headers = super().get_remote_connection_headers(parsed_url, keep_alive)
headers.update({'x-bb-api-key': os.environ["BROWSERBASE_API_KEY"]})
headers.update({'session-id': self._session_id})
# enable proxy here
headers.update({'enable-proxy': "true"})
return headers
def run():
session_id = create_session()
custom_conn = CustomRemoteConnection('http://connect.browserbase.com/webdriver', session_id)
options = webdriver.ChromeOptions()
options.debugger_address = "localhost:9223"
driver = webdriver.Remote(custom_conn, options=options)
driver.get("https://www.browserbase.com")
get_title = driver.title
print(get_title)
# Make sure to quit the driver so your session is ended!
driver.quit()
run()
```
## Dealing with OAuth and Session-based authentication flows
Some services rely on OAuth authentication (tokens) or Sessions (cookies).
The cookies used as part of those authentication flows are set by external
services (with `CORS` or `SameSite` attribute restrictions), making them
inaccessible with the Puppeteer or Playwright APIs.
A good practice to handle such advanced authentication flows is to create a
custom Chrome Extension responsible for intercepting and storing cookies or
tokens.
## Dealing with two-factor authentication
Two-step verification (via authenticator apps or SMS) or magic links require
human intervention in the loop. Here are some tactics to cope with 2FA:
### Disable 2FA or create an app password
This approach only applies to authentication flows owned by your team or
company.
For an internal tool, try to turn off the
[two-step verification](https://support.google.com/accounts/answer/1064203?hl=en&co=GENIE.Platform%3DDesktop).
For an authentication flow requiring some level of security,
[try to create an app password](https://support.google.com/accounts/answer/185833?hl=en).
### Enable Remote Control of your Session
If the two-step verification mechanism cannot be bypassed, you should consider
handing back the control to the end user by leveraging the Session Connection
Debug URLs.
Let your end users complete the two-step verification process as part of your
automation.
---
## File: guides/browser-contexts-and-pages.mdx
---
title: "Deep dive: BrowserContexts and Pages"
sidebarTitle: "BrowserContexts & Pages"
"Key concepts not to be confused with and how to use them with Browserbase."
mode: "wide"
---
Most drivers (Playwright, Puppeteer) expose an API through `Browser`,
`BrowserContext`, and `Page` objects.
The boundaries between them can be blurry. This guide teaches you to avoid
common pitfalls and how each concept relates to Browserbase's architecture.
## What are BrowserContexts and Pages?
A typical Playwright or Puppeteer program looks as follows:
```typescript Playwright (TypeScript)
import { chromium } from "playwright-core";
(async () => {
// 1. Get a Browser instance
const browser = await chromium.connectOverCDP(
`wss://connect.browserbase.com?apiKey=${process.env.BROWSERBASE_API_KEY}`
);
// 2. Get a BrowserContext
const defaultContext = browser.contexts()[0];
// 3. Get a Page
const page = defaultContext.pages()[0];
// 4. Act on Page
await page.goto("https://browserbase.com/");
await page.close();
await browser.close();
})().catch((error) => console.error(error.message));
```
```typescript Puppeteer (TypeScript)
import puppeteer from "puppeteer-core";
(async () => {
// 1. Get a Browser instance
const browser = await puppeteer.connect({
browserWSEndpoint:
`wss://connect.browserbase.com?apiKey=${process.env.BROWSERBASE_API_KEY}`
});
// 2. Get a Page
const page = await browser.newPage();
// 3. Act on Page
await page.goto("https://www.browserbase.com");
await page.close();
await browser.close();
})().catch((error) => console.error(error.message));
```
```python Playwright (Python)
from playwright.sync_api import sync_playwright, Playwright
def run(playwright: Playwright):
chromium = playwright.chromium
# 1. Get a Browser instance
browser = chromium.connect_over_cdp('wss://connect.browserbase.com?apiKey=${process.env.BROWSERBASE_API_KEY}')
# 2. Get a BrowserContext
context = browser.contexts[0]
# 3. Get a Page
page = context.pages[0]
# 4. Act on Page
# ...
with sync_playwright() as playwright:
run(playwright)
```
Selenium (Python) do not provide BrowserContext and Pages,
[see Dos and Don't](#dos).
Let's look at the `Browser`, `BrowserContext`, and `Page` definitions by
comparing them to our day-to-day Browser:
| Concept | Drive (Puppeteer/Playwright) | My Browser |
| ---------------- | -------------------------------------------------------------- | ---------------------------------------------------- |
| `Browser` | A Browser instance (_dedicated cache, sessions_) | The running Chrome application |
| `BrowserContext` | A Browser Session (_dedicated cookies, shared cache_) | Browser Window or Chrome Profile or Incognito Window |
| `Page` | A Browser tab (_dedicated history, shared cookies, and cache_) | Tabs |
We can now visualize the relationship between them as follows:
```mermaid
flowchart
subgraph M ["Machine"]
subgraph B1 ["Browser (instance)"]
subgraph BC1 ["BrowserContext 1 (session)"]
subgraph P2 ["Page 2 (tab)"]
dom2["DOM"]-->frames2["Frames"]
events2["Events"]
workers2["Workers (WebWorkers)"]
history2["History"]
end
subgraph P1 ["Page 1 (tab)"]
dom1["DOM"]-->frames1["Frames"]
events1["Events"]
workers1["Workers (WebWorkers)"]
history1["History"]
end
cookies["Cookies"]
ls["LocalStorage"]
routing["Routing"]
P1<-->cookies
P1<-->ls
P2<-->cookies
P2<-->ls
end
P1<-->routing
P2<-->routing
routing<-->cache["Cache"]
end
subgraph MR ["Resources"]
fs["File System"]
cpus["CPUs"]
mem["Memory"]
gpus["GPUS"]
network["Network"]
end
B1<-->cpus
B1<-->mem
B1<-->gpus
routing<-->network
cache<-->fs
ls<-->fs
BC1<-->cache
end
```
`Browser`, `BrowserContext`, and `Page` are encapsulated concepts that enable
developers to control different parts of the Browser.
Browserbase removes the complexity of dealing with multiple `BrowserContext` by
offering on-the-fly browsers with a unique context in which to interact.
## How to use BrowserContexts and Pages with Browserbase
Browserbase assigns each `Browser` to a
[dedicated virtual machine](/under-the-hood) (with isolated resources -- unlike
your local browser) with a unique `BrowserContext` leveraging the
[Persistent Context API](https://playwright.dev/docs/api/class-browsertype#browser-type-launch-persistent-context).
The combination of isolated resources and a unique `BrowserContext`
[enables the support for Chrome Extensions](/features/browsers#loading-a-custom-extension)
and reliable performances.
In short, with Browserbase:
> A Browser Session (`connectOverCDP()`) = Dedicated Browser + Unique
> BrowserContext
Here are some dos and don'ts to help get the best out of Browserbase:
### Dos
```typescript Playwright (TypeScript)
import { chromium } from "playwright-core";
(async () => {
const browser = await chromium.connectOverCDP(
`wss://connect.browserbase.com?apiKey=${process.env.BROWSERBASE_API_KEY}`
);
// Make sure to grab the first BrowserContext and Page
const defaultContext = browser.contexts()[0];
const page = defaultContext.pages()[0];
// Act on Page
await page.goto("https://browserbase.com/");
await page.close();
await browser.close();
})().catch((error) => console.error(error.message));
```
```typescript Puppeteer (TypeScript)
import puppeteer from "puppeteer-core";
(async () => {
const browser = await puppeteer.connect({
browserWSEndpoint:
`wss://connect.browserbase.com?apiKey=${process.env.BROWSERBASE_API_KEY}`
});
// Make sure to grab the first Page
const page = await browser.newPage();
// Act on Page
await page.goto("https://www.browserbase.com");
await page.close();
await browser.close();
})().catch((error) => console.error(error.message));
```
```python Playwright (Python)
from playwright.sync_api import sync_playwright, Playwright
def run(playwright: Playwright):
chromium = playwright.chromium
browser = chromium.connect_over_cdp('wss://connect.browserbase.com?apiKey=${process.env.BROWSERBASE_API_KEY}')
# Make sure to grab the first BrowserContext and Page
context = browser.contexts[0]
page = context.pages[0]
# Act on Page
# ...
with sync_playwright() as playwright:
run(playwright)
```
```python Selenium (Python)
from selenium import webdriver
from selenium.webdriver.remote.remote_connection import RemoteConnection
from selenium.webdriver.common.desired_capabilities import DesiredCapabilities
import requests
import os
def run():
# see create_session and CustomRemoteConnection definition
# at https://docs.browserbase.com/quickstart/selenium
session_id = create_session()
custom_conn = CustomRemoteConnection('http://connect.browserbase.com/webdriver', session_id)
options = webdriver.ChromeOptions()
options.debugger_address = "localhost:9223"
driver = webdriver.Remote(custom_conn, options=options)
driver.get("https://www.browserbase.com")
# Selenium don't have the concept of BrowserContext and Pages
get_title = driver.title
print(get_title)
# Make sure to quit the driver so your session is ended!
driver.quit()
run()
```
See how to setup multiple Browser instances to speed up your Automation.
### Don'ts
Using multiple contexts or pages breaks the Session recording and live debugger URLs. **Instead, [rely on multiple browser instances](/guides/parallelization)**.
```typescript Playwright (TypeScript)
import { chromium } from "playwright-core";
(async () => {
const browser = await chromium.connectOverCDP(
`wss://connect.browserbase.com?apiKey=${process.env.BROWSERBASE_API_KEY}`,
);
const getContent = async (url: string) => {
const context = await browser.newContext();
const page = await context.newPage();
try {
await page.goto(url, {
waitUntil: "domcontentloaded",
});
return await page.content();
} finally {
await page.close();
}
};
await Promise.all(
["https://www.google.com", "https://www.bing.com"].map(async (item) => {
const contents = [];
for (url in item.urls) {
contents.push(await getContent(url));
}
return contents;
}),
);
await browser.close();
})().catch((error) => console.error(error.message));
```
```typescript Puppeteer (TypeScript)
import puppeteer from "puppeteer";
(async () => {
const browser = await puppeteer.connect({
browserWSEndpoint: `wss://connect.browserbase.com?apiKey=${process.env.BROWSERBASE_API_KEY}`,
});
const getContent = async (url) => {
const page = await browser.newPage();
try {
await page.goto(url, {
waitUntil: "domcontentloaded",
});
return await page.content();
} finally {
await page.close();
}
};
await Promise.all(
["https://www.google.com", "https://www.bing.com"].map(async (url) => {
const contents = await getContent(url);
return contents;
}),
);
await browser.disconnect();
})().catch((error) => console.error(error.message));
```
```python Playwright (Python)
from playwright.sync_api import sync_playwright
async def main():
async with sync_playwright() as p:
browser = await p.chromium.connect_over_cdp(
f'wss://connect.browserbase.com?apiKey={os.environ["BROWSERBASE_API_KEY"]}'
)
async def get_content(url):
context = await browser.new_context()
page = await context.new_page()
try:
await page.goto(url, wait_until="domcontentloaded")
return await page.content()
finally:
await page.close()
urls = ["https://www.google.com", "https://www.bing.com"]
contents = await asyncio.gather(*(get_content(url) for url in urls))
print(contents)
await browser.close()
if __name__ == "__main__":
import os
import asyncio
asyncio.run(main())
```
Using multiple pages breaks the Session recording and live debugger URLs.
**Instead, persist the session and reuse the default context and page**.
See how to reuse user sessions across multiple Browser Sessions.
---
## File: guides/parallelization.mdx
---
title: "Speed up your automation with parallelization"
sidebarTitle: "Parallelization"
"Leverage multiple browser instances in parallel with a simple task queue."
---
Some complex automation requires processing multiple pages and actions without
resulting in an exponential duration. For example, a backup automation tool must
access and back up hundreds of pages reliably and efficiently.
As covered in the
[Deep dive: BrowserContexts and Pages](/guides/browser-contexts-and-pages)
guide, using multiple pages or contexts for parallelism will result in issues
with Browserbase's persistent context. Instead, we recommend leveraging multiple
browser instances as a pool to process actions on numerous pages in parallel.
## Enable parallelism with a pool of Browser instances
Clone this GitHub repo to get started with a Playwright parallelism setup
Let's walk through the code of our
[example script](https://github.com/browserbase/quickstart-playwright-js-parallelism/blob/main/index.ts)
processing hundreds of Wikipedia pages efficiently:
The `processBrowserbaseTasks()` utility create 5 Browserbase Sessions and reuse the available pages between tasks:
```ts
import { Page, chromium } from "playwright-core";
export async function processBrowserbaseTasks(
tasks: ((page: Page) => Promise)[]
): Promise {
const tasksQueue = tasks.slice();
const resultsQueue: R[] = [];
const createBrowserSession = async (browserWSEndpoint: string) => {
const browser = await chromium.connectOverCDP(browserWSEndpoint);
const page = await browser.newPage(); // Create a single page for this session
while (true) {
if (tasksQueue.length > 0) {
const task = tasksQueue.shift();
if (task) {
const result = await task(page);
resultsQueue.push(result);
}
} else {
break;
}
}
await page.close();
await browser.close();
};
const browserWSEndpoint = `wss://api.browserbase.com?apiKey=${process.env.BROWSERBASE_API_KEY}&enableProxy=true`;
const sessions = Array.from({ length: 5 }, () =>
createBrowserSession(browserWSEndpoint)
);
await Promise.all(sessions);
return resultsQueue;
}
```
Let's create a function that will be called with a `page` for each Wikipedia URL and return the `url`, `content` tuple:
```ts
const tasks = loadUrlsFromFile("wikipedia_urls.txt").map(
(url) => async (page: Page) => {
console.log(`Processing ${url}...`);
await page.goto(url);
const content = await page.content();
return [url, content];
}
);
```
```ts
const result = await processBrowserbaseTasks(tasks);
result.map(([url, content]) => {
console.log(url, content.substring(0, 200) + "...");
});
```
Let's now run our Wikipedia automation; you will notice that pages are processed by a group of 5:
```bash
$ BROWSERBASE_PROJECT_ID=xxxxxxxxx BROWSERBASE_API_KEY=xxxxxxxxx node dist/index.js
Processing https://en.wikipedia.org/wiki/Patrick_Flynn_(hurler)...
Processing https://en.wikipedia.org/wiki/Environmental_radioactivity...
Processing https://en.wikipedia.org/wiki/Alexi_Ogando...
Processing https://en.wikipedia.org/wiki/Costantino_Maria_Attilio_Barneschi...
Processing https://en.wikipedia.org/wiki/Breaking_bulk...
Processing https://en.wikipedia.org/wiki/New_Hampshire_Route_122...
Processing https://en.wikipedia.org/wiki/David_Hoff...
Processing https://en.wikipedia.org/wiki/Neodesha,_Oklahoma...
Processing https://en.wikipedia.org/wiki/List_of_Bethel_Threshers_head_football_coaches...
Processing https://en.wikipedia.org/wiki/Thysanodonta_boucheti...
Processing https://en.wikipedia.org/wiki/Sturm_und_Drang_(play)...
Processing https://en.wikipedia.org/wiki/Maša_Kolanović...
Processing https://en.wikipedia.org/wiki/Hermitage_of_Sant'Onofrio,_Serramonacesca...
Processing https://en.wikipedia.org/wiki/Bill_Simpson_(racing_driver)...
Processing https://en.wikipedia.org/wiki/Dundee,_Oregon...
Processing https://en.wikipedia.org/wiki/Caragh_McMurtry...
Processing https://en.wikipedia.org/wiki/Palmar_metacarpal_veins...
Processing https://en.wikipedia.org/wiki/2000_Uzbek_presidential_election...
```